Three-dimensional display method, device, electronic equipment, storage medium and program product of sound object

Abstract

Embodiments of the present application provide a three-dimensional display method and device for a sound object, electronic equipment, a storage medium and a program product. The method comprises: in response to a first trigger operation, controlling a movable object operated by the first trigger operation to move according to an operation trajectory of a user, and playing a sound object sound effect corresponding to both the movable object and the operation trajectory in a preset physical space; wherein the first trigger operation represents that the user moves a movable object on a three-dimensional virtual interface; in response to a second trigger operation, controlling the moving speed of the movable object at the current time, performing subsequent movement, and playing a sound object sound effect corresponding to both the movable object and the subsequent movement trajectory corresponding to the subsequent movement in the preset physical space; wherein the second trigger operation represents that the user no longer touches the movable object. The method achieves that the user can not only visually see the movement of the object, but also aurally feel the source and change of the sound.

Description

Technical Field

[0001] This application relates to the field of audio processing, and more particularly to a method, apparatus, electronic device, storage medium, and program product for three-dimensional display of sound objects. Background Technology

[0002] With the rapid development of digital media technology, the production and consumption of audio and video content have permeated all aspects of people's daily lives. Sound object technology, as a crucial means of enhancing the audio experience, is becoming increasingly important. However, currently, most users rely primarily on headphones or speaker systems to perceive the spatial distribution of sound when enjoying sound object content. While this method can provide a certain level of sound object effect, it lacks intuitive visual presentation, and users cannot control the sound objects. Therefore, it fails to fully meet users' needs for an immersive experience.

[0003] Therefore, there is an urgent need for a technological solution that allows users not only to visually see the movement of objects, but also to auditorily perceive the source and changes of sound. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, storage medium, and program product for displaying three-dimensional sound objects, so as to enable users to not only see the movement of objects visually based on three-dimensional display, but also to feel the source and changes of sound auditorily.

[0005] In a first aspect, embodiments of this application provide a method for displaying a three-dimensional sound object, comprising: responding to a first trigger operation, controlling a movable object operated by the first trigger operation to move according to a user's operation trajectory, and playing sound object sound effects corresponding to both the movable object and the operation trajectory within a preset physical space; wherein, the first trigger operation represents the user moving a movable object on a three-dimensional virtual interface; responding to a second trigger operation, controlling the movement speed of the movable object at the current moment to perform subsequent movement, and playing sound object sound effects corresponding to both the movable object and the subsequent movement trajectory corresponding to the subsequent movement within the preset physical space; wherein, the second trigger operation represents the user no longer touching the movable object.

[0006] In one possible implementation, the moving speed of the movable object at multiple times is obtained; based on the moving speed of the movable object at the current time and the moving speed at the multiple times, a speed to be used is determined; if the speed to be used is determined to be greater than a preset speed threshold, the movable object is controlled to move according to the speed to be used.

[0007] In one possible implementation, the movable object is controlled to move in a straight line at a constant speed according to the speed to be used; or, the movable object is controlled to move in a straight line at a deceleration speed according to the speed to be used and the deceleration of the scene corresponding to the at least one movable object; or, the subsequent movement trajectory of a preset shape is generated according to the operation trajectory and the speed to be used, and the movable object is controlled to move according to the subsequent movement trajectory of the preset shape with the speed to be used as the starting speed.

[0008] In one possible implementation, if the movable object is detected to have touched the first edge of the three-dimensional virtual interface during the subsequent movement, the movable object is controlled to bounce off the first edge of the three-dimensional virtual interface and then move.

[0009] In one possible implementation, the angle between the subsequent movement trajectory of the movable object and the normal of the first edge is determined as the angle of incidence; the straight line containing the reflection angle of the angle of incidence is determined as the rebound movement trajectory line of the movable object; the movable object is controlled to move along the rebound movement trajectory line after rebounding from the first edge of the three-dimensional virtual interface.

[0010] In one possible implementation, the volume of the movable object is adjusted in response to a third trigger operation on the movable object.

[0011] In one possible implementation, the display size of the aperture surrounding the movable object is adjusted in real time based on the volume of the sound effect of the movable object at the current moment.

[0012] In one possible implementation, in response to a switching command, the currently displayed first view is switched to a second view; wherein the switching command represents a switch of view; when the first view is a third-party view, the second view is the view of the movable object indicated by the switching command or the user's view; when the first view is the view of the movable object, the second view is a third-party view or the user's view; when the first view is the user's view, the second view is a third-party view or the movable object's view.

[0013] In a possible implementation manner, in response to a selection operation of a user on a second interface, a trajectory indicated by the selection operation is determined as a first preset movement trajectory, and the first preset movement trajectory carries preset time information and preset speed information; wherein, the second interface is a three-dimensional interface or an interface in a virtual reality space; multiple trajectories are displayed in the second interface; the selection operation indicates that a trajectory needs to be selected as the preset movement trajectory; wherein, the first preset movement trajectory is used for a movable object to move on a three-dimensional virtual interface based on the preset movement trajectory.

[0014] In a possible implementation manner, in response to an adjustment instruction issued by the user based on the first preset movement trajectory, the adjustment instruction carries time adjustment information and speed adjustment information; a second preset movement trajectory is generated based on the adjustment instruction.

[0015] In a possible implementation manner, according to the somatosensory information to be sent of each movable object displayed on the three-dimensional virtual interface, a hardware device set in the preset physical space is controlled to send associated somatosensory information.

[0016] In a possible implementation manner, according to the somatosensory information to be sent of each movable object displayed on the three-dimensional virtual interface, conflict information between the somatosensory information to be sent of movable objects of the same category is determined, and according to the conflict information, the associated somatosensory information of the hardware device is determined from the somatosensory information to be sent of movable objects of the same category; the hardware device set in the preset physical space is controlled to send the associated somatosensory information.

[0017] In a second aspect, an embodiment of the present application provides a three-dimensional display device for a sound object, including: a first control unit, configured to, in response to a first trigger operation, control a movable object operated by the first trigger operation to move according to a user's operation trajectory, and play a sound object sound effect corresponding to both the movable object and the operation trajectory in the preset physical space; wherein, the first trigger operation represents that the user moves a movable object on a three-dimensional virtual interface; a second control unit, configured to, in response to a second trigger operation, control the moving speed of the movable object at the current moment to perform subsequent movement, and play a sound object sound effect corresponding to both the movable object and a subsequent movement trajectory corresponding to the subsequent movement in the preset physical space; wherein, the second trigger operation represents that the user no longer touches the movable object.

[0018] In a third aspect, an embodiment of the present application provides a playback device for a sound object, including: a memory, a processor;

[0019] The memory stores computer execution instructions;

[0020] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0022] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0023] In a sixth aspect, embodiments of this application provide a method for displaying a three-dimensional sound object in a sound production system, comprising: responding to a first trigger operation triggered by a user in the sound production system, controlling a movable object operated by the first trigger operation to move on an interface displayed by the sound production system according to the user's operation trajectory, and playing a stereoscopic generated sound effect corresponding to both the movable object and the operation trajectory within a preset physical space; wherein, the first trigger operation represents the user moving a movable object on a three-dimensional virtual interface; responding to a second trigger operation triggered by the user in the sound production system, controlling the movement speed of the movable object at the current moment, performing subsequent movement on the interface displayed by the sound production system, and playing a sound object sound effect corresponding to both the movable object and the subsequent movement trajectory corresponding to the subsequent movement within the preset physical space; wherein, the second trigger operation represents the user no longer touching the movable object.

[0024] In a seventh aspect, embodiments of this application provide a method for displaying a three-dimensional sound object in an interactive audio system, comprising: responding to a first trigger operation triggered by a user in the interactive audio system, controlling a movable object operated by the first trigger operation to move on the interface displayed by the interactive audio system according to the user's operation trajectory, and playing a stereoscopic generated sound effect corresponding to both the movable object and the operation trajectory within a preset physical space; wherein, the first trigger operation represents the user moving the movable object on the three-dimensional virtual interface; responding to a second trigger operation triggered by the user in the interactive audio system, controlling the movement speed of the movable object at the current moment, performing subsequent movement on the interface displayed by the interactive audio system, and playing a sound object sound effect corresponding to both the movable object and the subsequent movement trajectory corresponding to the subsequent movement within the preset physical space; wherein, the second trigger operation represents the user no longer touching the movable object.

[0025] The three-dimensional display method, apparatus, electronic device, storage medium, and program product for sound objects provided in this application calculate and play corresponding sound object sound effects in real time based on the user's operation trajectory when the user touches or moves a movable object. These sound effects change according to the user's actions and the object's position to simulate sound changes in a real environment. When the user stops touching the movable object, the system continues to calculate and play subsequent sound object sound effects based on the object's current movement speed and direction. This ensures that even when the user does not directly touch the object, the sound changes still match the object's movement. Based on the user's operation trajectory and the object's movement state, the system continuously adjusts the parameters of the sound object sound effects to simulate sound changes in a real environment, achieving the effect that the user can not only see the object's movement visually based on a three-dimensional display but also perceive the source and changes of the sound audibly. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0027] Figure 1 A scene diagram illustrating the three-dimensional display of the sound objects provided in this application;

[0028] Figure 2 Flowchart for the 3D display of the sound object provided in this application Figure 1 ;

[0029] Figure 3 Flowchart of the method for displaying three-dimensional sound objects provided in this application Figure 2 ;

[0030] Figure 4 A schematic diagram of the virtual interface for the three-dimensional display method of the sound object provided in this application;

[0031] Figure 5 A schematic diagram illustrating the perspective switching of the three-dimensional display method for sound objects provided in this application;

[0032] Figure 6 A schematic diagram of the trajectory template for the three-dimensional display method of the sound object provided in this application;

[0033] Figure 7 A schematic diagram of a three-dimensional trajectory template for a three-dimensional display method of sound objects provided in this application;

[0034] Figure 8 A schematic diagram of the structure of the three-dimensional display device for the sound object provided in this application;

[0035] Figure 9A schematic diagram of the structure of the three-dimensional display device for the sound object provided in this application.

[0036] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0037] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0038] Figure 1 A scene illustration for the three-dimensional display of the sound objects provided in this application, such as... Figure 1 As shown, the specific application scenarios of this application are as follows:

[0039] With the continuous development of audio technology, sound objects (including mono, stereo, immersive, surround sound, spatial audio, and 3D sound effects) have become a basic configuration of modern audio systems, widely used in music, film, games, and virtual reality. Sound objects provide users with a more realistic and immersive experience through the collaborative work of single or multiple channels. However, despite their already impressive sound performance, sound objects still have some limitations, one of which is that users cannot visually perceive the movement of sound.

[0040] Traditional sound object systems primarily play sound through speakers, allowing users to receive sound information only through their ears, without any corresponding visual feedback. This one-way information transmission method limits the user experience, making it difficult for them to accurately determine the source and path of the sound.

[0041] While sound objects can simulate the spatial location of sounds to some extent, users may still feel confused and find it difficult to accurately track the specific location of each sound in complex environments, such as when multiple people are speaking at the same time or in multi-channel audio scenarios.

[0042] Existing sound object systems typically lack interactivity with users. Users cannot directly participate in the sound scene through visual means, which limits the possibility of immersive experiences.

[0043] To provide a more realistic auditory experience, users want to be able to see the movement trajectory of sound and control and edit sound objects through interaction. This not only enhances spatial awareness but also improves the overall immersion.

[0044] Based on the scenarios described above, it is clear that existing technologies lack intuitive 3D visual presentation and do not allow users to control sound objects. These technical issues prevent the full fulfillment of users' needs for an immersive experience.

[0045] The 3D display method for sound objects provided in this application calculates and plays corresponding sound effects in real time based on the user's touch or movement of the movable object. These sound effects change according to the user's actions and the object's position to simulate sound changes in a real environment. When the user stops touching the movable object, the system continues to calculate and play subsequent sound effects based on the object's current movement speed and direction. This ensures that even when the user does not directly touch the object, the sound changes still match the object's movement. Based on the user's operation trajectory and the object's movement state, the system continuously adjusts the parameters of the sound effects to simulate sound changes in a real environment, achieving the effect that allows the user not only to visually see the object's movement based on a 3D display but also to auditorily perceive the source and changes of the sound.

[0046] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0047] Figure 2 Flowchart for the 3D display of the sound object provided in this application Figure 1 ,like Figure 2 As shown, the method includes:

[0048] S201. In response to the first trigger operation, control the movable object operated by the first trigger operation to move according to the user's operation trajectory, and play sound effects of sound objects corresponding to both the movable object and the operation trajectory in the preset physical space; wherein, the first trigger operation represents the user moving the movable object on the three-dimensional virtual interface;

[0049] For example, the three-dimensional virtual interface includes at least one movable object, which can be in a moving state or a non-moving state. That is, when there are multiple movable objects in the three-dimensional virtual interface, there are three possible scenarios: all movable objects are in a moving state, all movable objects are in a stationary state, or some of the movable objects are in a moving state and some are in a stationary state.

[0050] For example, the movable object can be a geometric shape of a preset size: sphere, cube, cylinder, cone, etc., and the movable object can also have a three-dimensional appearance shape set according to the corresponding sound object effect of the movable object.

[0051] Optionally, the appearance of these geometries can be customized according to user preferences or scene requirements, including attributes such as color, texture, and reflectivity.

[0052] When a sound object is associated with a specific animal or object, the appearance of the movable object can be set accordingly to the shape of that animal or object.

[0053] For example, if the sound object is a frog croaking sound, the appearance of the movable object can be set to the shape of a three-dimensional frog, including skin texture, color, and motion pattern.

[0054] For example, the preset physical space can be any physical space suitable for playing sound effects, such as: a movie theater, opera house, concert hall, museum, car in P gear, private cinema, themed restaurant, amusement park, escape room, a room in a home environment, etc.

[0055] For example, the first triggering operation may be a single click, long press, or swipe gesture on a touch screen device, or on a non-touch device, such as a virtual reality (VR) device or an augmented reality (AR) device, it may be pressing a button, making a gesture, or performing a mouse click or drag operation based on the display screen.

[0056] When a user performs their first trigger action, their action leaves a path or trace in space and time; this is called the action trajectory. For example, when a user wears a VR device and triggers an action using a universal gesture or controller, a three-dimensional action trajectory is drawn within the VR space.

[0057] When swiping with a finger on a touchscreen, the path the finger travels from the starting position to the ending position is the operation trajectory; similarly, in mouse operation, the path the mouse pointer travels is the operation trajectory. Optionally, the movable object can be controlled based on information such as the direction, distance, and speed of the operation corresponding to the operation trajectory.

[0058] S202. In response to the second trigger operation, control the movement speed of the movable object at the current moment, perform subsequent movement, and play the sound object sound effect corresponding to both the movable object and the subsequent movement trajectory within the preset physical space; wherein, the second trigger operation indicates that the user no longer touches the movable object.

[0059] The second trigger indicates that the user has stopped touching the movable object. This means the user has ended direct control of the movable object, and the object will enter a state of autonomous movement. At this point, the system needs to control its subsequent movement based on the movable object's current state and preset rules.

[0060] For example, the second trigger operation can be based on a time interval, such as after the user stops touching, after a short delay, the system starts to control the movable object according to preset rules; or it can be based on specific conditions, such as the user releasing the movable object at a speed greater than a preset value, and then initiating the subsequent movement of the movable object.

[0061] After the user stops touching the object, the system calculates a suitable subsequent movement speed based on information such as the object's current speed, direction, and acceleration. This speed may be related to the speed at which the user stopped touching the object, or it may be adjusted according to the needs of the scenario.

[0062] In an optional embodiment, the movable object may also correspond to a mono sound effect or a stereo sound effect. The sound object may also be: mono, stereo, immersive sound, surround sound, spatial audio, three-dimensional sound effect, etc.

[0063] The three-dimensional display method for sound objects provided in this application calculates and plays corresponding sound object sound effects in real time based on the user's operation trajectory when the user touches or moves the movable object. These sound effects change according to the user's actions and the object's position to simulate sound changes in a real environment. When the user stops touching the movable object, the system continues to calculate and play subsequent sound object sound effects based on the object's current movement speed and direction. This ensures that even when the user does not directly touch the object, the sound changes still match the object's movement. Based on the user's operation trajectory and the object's movement state, the system continuously adjusts the parameters of the sound object sound effects to simulate sound changes in a real environment, achieving the effect that the user can not only visually see the object's movement but also auditorily perceive the source and changes of the sound.

[0064] Figure 3 Flowchart of the method for displaying three-dimensional sound objects provided in this application Figure 2 ,likeFigure 3 As shown, in this embodiment... Figure 2 Based on the embodiments, a method for displaying three-dimensional sound objects is described in detail, which includes:

[0065] S301. In response to the first trigger operation, control the movable object operated by the first trigger operation to move according to the user's operation trajectory, and play the sound object sound effect corresponding to both the movable object and the operation trajectory in the preset physical space; wherein, the first trigger operation represents the user moving the movable object on the three-dimensional virtual interface.

[0066] S302. In response to the second triggering operation, obtain the movement speed of the movable object at multiple times; determine the speed to be used based on the movement speed of the movable object at the current time and the movement speed at the multiple times.

[0067] For example, if it is determined that the speed to be used is greater than a preset speed threshold, the movable object is controlled to move in a straight line at a constant speed according to the speed to be used; or, the movable object is controlled to move in a straight line at a deceleration speed according to the speed to be used and the deceleration of the scene corresponding to the at least one movable object; or, the subsequent movement trajectory of a preset shape is generated according to the operation trajectory and the speed to be used, and the movable object is controlled to move according to the subsequent movement trajectory of the preset shape with the speed to be used as the starting speed.

[0068] For example, the second trigger operation might indicate that the user no longer touches the movable object. That is, the movable object needs to transition from a state directly controlled by the user to a state of autonomous movement according to system rules. The system needs to adjust the movable object's movement state based on certain parameters. By acquiring the movement speed at different times, the system determines the movable object's subsequent trajectory and state changes. Specifically, the system records the movable object's speed information at different points in time. These time points can be evenly distributed, such as recording the speed every 0.1 seconds; or they can be dynamically adjusted according to the movable object's movement state, such as recording speed data more frequently during acceleration or deceleration phases.

[0069] The system collects movement speed data at multiple points in time and then analyzes the motion trend of the movable object. By analyzing this data, the system can understand how the movable object changes from its initial speed to its current speed and predict its possible future speed changes. In other words, it infers its motion pattern by observing its position and speed at different points in time.

[0070] The current movement speed is the speed of a movable object at the instant the user stops touching it. It represents the immediate state of the movable object when the user's operation ends. For example, when the user stops touching a movable object in virtual reality space, the speed of the movable object at that moment is its current movement speed.

[0071] For example, the system compares and analyzes the current movement speed with the movement speeds recorded at multiple previous moments. If the current speed is close to a certain stable speed range from the past, the system may continue to maintain a similar speed strategy; if the current speed changes significantly from the previous speed (such as a sudden acceleration or deceleration), the system will consider this trend and adjust the speed to be used.

[0072] For example: when the system determines that the speed to be used is greater than the preset speed threshold:

[0073] Alternatively, the movable object may move at a constant speed in a straight line. This means that the movable object will maintain its currently determined speed (the speed to be used) and continue moving along a straight line. In this case, the system does not need to make any additional adjustments to the speed of the movable object (such as acceleration or deceleration).

[0074] Optionally, if the speed to be used exceeds a preset speed threshold, in addition to uniform linear movement, decelerated linear movement can also be selected. The system will determine the deceleration based on the scene information of the movable object. For example, preset scene information includes: gravity, zero gravity, damped, undamped; or gravity and undamped, gravity and zero damped, zero gravity and damped, zero gravity and undamped, etc.

[0075] Optionally, the movement of the movable object can be controlled by generating a subsequent movement trajectory of a preset shape based on the operation trajectory and the speed to be used. The system will combine the user's previous operation trajectory (even if the user has stopped touching) and the current speed to be used to determine the future movement path of the movable object.

[0076] For example, when the movable object is a ball, after the player throws the ball (the movable object) (the user stops touching, triggering a second trigger operation), the ball has an initial velocity (the velocity to be used). The system generates a parabolic trajectory (a preset shape) based on the angle at which the ball is launched (the trajectory) and its initial velocity. The player then controls the ball to move along this generated parabolic trajectory, starting from this initial velocity. This makes the motion of the movable object not just a simple uniform or decelerating linear motion, but a more complex curvilinear motion.

[0077] S303. If it is detected that the movable object touches the first edge of the three-dimensional virtual interface during the subsequent movement, the movable object is controlled to bounce off the first edge of the three-dimensional virtual interface and then move.

[0078] For example, the system continuously monitors the position of the movable object in real time during its subsequent movement. This involves constantly acquiring the coordinate information of the movable object in the 3D virtual interface and comparing it with the boundary coordinates of the 3D virtual interface to determine whether the movable object is approaching or touching the edge.

[0079] For example, in a 3D virtual interface with the terminal device screen as the display area, the system periodically (e.g., multiple times per second) collects the position coordinates of movable objects, including their position information on the x, y, and z axes, and then compares them with the coordinate ranges corresponding to the six faces (up, down, left, right, front, and back) of the preset virtual space.

[0080] The edges of a 3D virtual interface have clearly defined coordinate ranges. The method for calculating the coordinates of the edges will differ depending on the shape of the 3D virtual interface (such as a cube, sphere, or irregular shape).

[0081] Taking a common cube-shaped 3D virtual interface as an example, such as Figure 4 As shown, Figure 4 This is a schematic diagram of a virtual interface for the three-dimensional display method of the sound object provided in this application. Assuming that the upper left corner coordinates of the first side (front) of the rectangular three-dimensional virtual interface are (x1, y1, z1) and the lower right corner coordinates of the second side (back) are (x2, y2, z2), then when the coordinates of the movable object satisfy x = x1 and y is between y1 and y2, and z is between z1 and z2, it can be determined that the movable object has touched the left edge.

[0082] The first step of S303 is to determine the angle between the subsequent movement trajectory of the movable object and the normal of the first edge, which is the angle of incidence; to determine the straight line where the reflection angle of the incident angle is located, which is the rebound movement trajectory line of the movable object; and to control the movable object to move along the rebound movement trajectory line after rebounding from the first edge of the three-dimensional virtual interface.

[0083] When a movable object is detected to touch the first edge of the 3D virtual interface during subsequent movement, it is first necessary to determine the angle between the subsequent movement trajectory of the movable object and the normal of the first edge. This angle is the angle of incidence.

[0084] The angle of incidence can be calculated by obtaining the direction vector of the movable object's trajectory and the direction vector of the first edge normal, and then using the formula for the angle between the vectors. For example, suppose the direction vector of the movable object's trajectory is... The direction vector of the first edge normal is The incident angle θ can then be calculated using formula (1):

[0085]

[0086] The system controls movable objects to bounce off the first edge of the 3D virtual interface along a predetermined bounce trajectory. This involves updating the position information of the movable object in each frame or time interval to ensure smooth movement along the bounce trajectory.

[0087] For example, the bounce direction is determined based on the position of the movable object touching the edge and the angle of impact. If the collision is perpendicular to the edge, the bounce direction will be directly perpendicular to the edge plane; if the collision is at an angle, the bounce direction is calculated based on the angle of incidence and the normal direction of the edge. When the movable object is associated with material information, energy loss during the collision process, such as friction and air resistance, can also be calculated. The bounce speed is usually less than the initial velocity. A bounce coefficient can be used to adjust the speed; the value of the bounce coefficient ranges from 0 to 1, and the specific value can be determined based on the properties of the movable object (such as mass and material) and the properties of the impact surface (such as roughness and hardness).

[0088] Based on the calculated rebound direction and velocity, the system updates the trajectory of the movable object. This involves adjusting the direction and magnitude of the movable object's position change in 3D space so that it moves along the new trajectory. For example, if the movable object was originally moving along the positive x-axis, after colliding with the right edge, it might move along the negative x-axis at a new velocity based on the calculated rebound direction and velocity.

[0089] In an alternative embodiment, lighting effects, such as highlight flashes and sparks, are also included at the moment the movable object collides with the edge.

[0090] S304. In response to a third trigger operation on the movable object, adjust the volume of the movable object.

[0091] Optionally, adjusting the volume of the movable object can be achieved by adjusting the adjustment control corresponding to the movable object, or by gestures, such as pinch or magnify gestures.

[0092] Optionally, users can influence the volume of movable objects through various interactive methods. These methods can be diverse, such as using a mouse wheel to scroll, click, and drag a slider in a 3D display simulated on a computer screen, performing swipe gestures on the touchscreen, or using specific keys on a keyboard in a VR environment. In the case of VR-based environments, interaction can be achieved using head-mounted displays, XR controllers, gestures, or headphones with gyroscopes.

[0093] Taking control adjustment as an example, if the system determines that the user has performed a third trigger operation, it will immediately display the first control corresponding to the movable object in the 3D virtual interface. This first control is specifically designed for adjusting the volume of the movable object, and its appearance and display method can be adjusted according to the interface layout and user experience design. For example, the first control may appear near the movable object as a small volume adjustment slider, or it may appear as a floating volume adjustment panel in a fixed position on the interface, and its appearance and style will be adapted to the overall style of the 3D virtual interface.

[0094] The first control is specifically designed to adjust the volume of movable objects. When the user adjusts the volume using the first control, the movable object will adjust its audio output volume in real time according to the user's operation. If the user increases the volume using the first control, the stereo sound of the movable object will increase accordingly; conversely, if the volume is decreased, the audio sound will decrease. This volume adjustment function provides users with a more personalized audio experience, allowing them to control the audio intensity according to their needs and the environment.

[0095] Taking gesture or cursor adjustment as an example, the user selects a movable object and makes a pinching or expanding gesture, or uses the cursor to shrink or enlarge the area of ​​the movable object. The volume level corresponding to the movable object is then determined based on the scaling ratio of the movable object. It should be noted that in AR or VR scenarios, the user's gestures need to be detected by image sensors, infrared sensors, or other devices capable of recognizing user movements. The user's actions are determined based on the recognition results. This application does not impose any limitations on this.

[0096] For example, a user controls volume using gestures on a touch-sensitive AR device. The user uses two fingers to "pinch" or "spread" within the AR's virtual space. Pinching typically refers to bringing two fingers closer together, while spreading means moving them apart. As the user pinches or spreads, the selected object shrinks or expands accordingly. For instance, if it's a volume control slider, it might become longer (increase volume) or shorter (decrease volume) depending on the user's gesture. If the area of ​​the selected movable object represents volume, it might become larger (increase volume) or smaller (decrease volume) depending on the user's gesture. Exemplarily, in response to a fourth trigger action of the displayed first control, the volume of the movable object corresponding to that first control is adjusted.

[0097] Regarding volume adjustment, for example, in a playback scenario with four movable objects (A, B, C, and D), each corresponding to a sound effect, the user can adjust the volume of movable object A and movable object B to 50 dB, and the volume of movable object C and movable object D to 55 dB. Optionally, the user can also adjust the volume of the movable objects based on a percentage of the maximum playable volume of the playback device. For example, if the maximum playable volume of a playback device is 90 dB, the user can choose to adjust the volume of movable object A and movable object B to 50%, which is 45 dB, and the volume of movable object C and movable object D to 60%, which is 54 dB.

[0098] For example, the display size of the aperture surrounding the movable object is adjusted in real time according to the volume of the sound effect of the movable object at the current moment.

[0099] Optionally, the action of adjusting the volume is triggered in the following situations:

[0100] Users might click the grip area on the slider with the mouse pointer and then drag it left or right along the slider's track. This dragging action is the action of adjusting the volume.

[0101] If the user is wearing a head-mounted display, the system tracks the user's eyes using sensors on the display. When the user stares at the first control for an extended period of time, it is considered a trigger for adjusting the volume.

[0102] The system determines whether a volume adjustment action is triggered based on preset rules and logic. These rules may include factors such as the direction of the operation, the magnitude of the operation, and the duration of the operation.

[0103] For example, for mouse wheel scrolling, the system determines whether the scrolling direction is upward (usually indicating an increase in volume) or downward (usually indicating a decrease in volume), as well as the scrolling speed and duration. If the scrolling direction is upward, and the scrolling speed is fast and the duration is short, it may be interpreted as a rapid volume increase command; while if the scrolling direction is downward, and the scrolling is slow and lasts for a period of time, it may be an operation to slowly decrease the volume. Similarly, for touch screen swipe operations, the system also determines the user's intent based on the direction and distance of the swipe to ensure accurate recognition of the fourth trigger operation.

[0104] For example, a head-mounted display (HMD) uses sensors to track the user's eyes. When the user stares at the first control for an extended period, it's considered a trigger for volume adjustment. At this time, the HMD detects whether the user tilts their head up (preset to increase volume) or tilts their head down (preset to decrease volume). Alternatively, the user may make a gesture that conforms to preset rules, such as waving upwards, downwards, leftwards, rightwards, forwards, backwards, pinching, expanding, rotating clockwise, or rotating counterclockwise.

[0105] It should be noted that multiple virtual playback scenes can be preset, and each scene includes various preset sound effects and movable objects, for example:

[0106] Forest scene: bird song sound effects, flowing water sound effects (gurgling water, splashing water), leaf sound effects, insect chirping sound effects.

[0107] Street scene: Car sound effects (engine roar, tire friction with the ground), horn sound effects, pedestrian sound effects (sounds of walking and talking on the street).

[0108] Seaside scene: sound effects of ocean waves (the sound of waves crashing against the shore, including the sound of surging waves and the sound of waves receding), and sound effects of seagulls (the sound of seagulls soaring and calling on the sea).

[0109] Meditation scenarios: voice guidance, meditation sound effects (wooden fish sound effects, xylophone sound effects, singing bowl sound effects, etc.).

[0110] Performance scenarios: guitar sound effects (including string sounds and plucking sounds), piano sound effects, violin sound effects (bow string friction sounds, high and low pitch transitions), etc.

[0111] Film and television sound effects: bullet sound effects (the sound of bullets being fired, flying, and hitting targets in gunfight movies), airplane sound effects (the sound of airplanes taking off, flying, and landing, including engine roars, whistling air, etc.), horse hoof sound effects (the sound of horses running in ancient wars or Western movies), etc.

[0112] Entertainment scenes: horror sound effects, door creaking sound effects, celebration sound effects, etc.

[0113] Users can choose preset playback scenes based on their preferences.

[0114] Optionally, in a meditation setting, the voice guidance can be the voice guidance for different meditation courses, thereby guiding the user's breathing rate or adjusting body posture, etc.

[0115] To better experience the meditation course, users may prefer a louder voice guidance. In this case, they can select the movable object corresponding to the voice guidance and increase its volume, while lowering the volume of the meditation sound effects. Alternatively, users may prefer a more immersive meditation music experience for relaxation. In this case, they can select the movable object corresponding to the meditation sound effects and increase its volume, while lowering the volume of the voice guidance.

[0116] If users are using head-mounted displays or similar devices, they can also experience the audiovisual effects of moving between different movable objects by changing their position, for example:

[0117] Optionally, in a performance setting, users can select their favorite instrument sound effects and position the instruments to simulate the arrangement of instruments in a concert hall, thus creating an immersive performance experience.

[0118] For example, a user might want to simulate a concert in their home environment (preset physical space). The user can choose their favorite songs and have them sung by their favorite preset AI singer. The user can also define the movement trajectory of the AI ​​singer (a movable object) to experience a realistic concert effect at home.

[0119] Optionally, in film and television scenarios, users can add corresponding sound effects based on the selected film and the corresponding plot, and can also plan the trajectory of the sound effects according to their preferences.

[0120] For example, in film clips, if there are scenes of horses running, users can add hoof sound effects and add a movement trajectory from left to right for the hoof sound effects.

[0121] Optionally, in entertainment scenarios, users can add horror sound effects, celebration sound effects, etc., based on their entertainment needs. For example, users can use the above sound effects to guide players to experience and pass through an escape room game.

[0122] Based on the different scenarios mentioned above, users can adjust the volume of movable objects with different sound effects according to the needs of different scenarios, so as to achieve a more immersive music experience.

[0123] Optionally, in the above scenarios, users can specify the playback duration of the scenario via a timer. For example, if a user needs to meditate for half an hour, after selecting the meditation scenario, the user can set a 30-minute countdown, and exit the meditation scenario when the countdown ends.

[0124] It should be noted that each first control is associated with the volume adjustment function of a movable object. This means there is a one-to-one correspondence; only a specific first control can adjust the volume of its corresponding movable object.

[0125] For example, in a complex 3D virtual scene, there are multiple movable objects, each with its own sound characteristics and volume control requirements. When a user wants to adjust the volume of a specific movable object, they need to perform a volume adjustment operation on that object. This mapping relationship makes volume adjustment targeted and does not affect the volume settings of other movable objects.

[0126] This design allows users to easily and precisely adjust the volume of movable objects using the primary control, creating a sound environment that better suits their needs within the 3D virtual interface. Whether in virtual reality music playback, immersive audio experiences, or other application scenarios, the volume can be flexibly controlled according to the actual situation, enhancing the realism and fun of the interaction.

[0127] S305. In response to a switching command, switch the currently displayed first view to a second view; wherein the switching command represents switching views; when the first view is a third-party view, the second view is the view of the movable object indicated by the switching command or the user's view; when the first view is the view of the movable object, the second view is a third-party view or the user's view; when the first view is the user's view, the second view is a third-party view or the movable object's view.

[0128] Switching commands can be initiated by the user. For example, in a virtual reality environment, a user might issue a command to switch perspectives by pressing a specific button, clicking a switch button on the interface, or using voice commands. This type of proactive command stems from the user's desire to change their viewing angle to obtain a better audio playback experience based on their needs in the game and the current context.

[0129] The system can also automatically trigger the switching command. For example, when a user moves the head-mounted display to a specific location, in order to guide the user's attention, display important scene elements, or achieve specific visual effects, the system will automatically send a command to switch the viewpoint according to preset logic.

[0130] Regarding the shift in perspective, Figure 5A schematic diagram illustrating the perspective switching of the three-dimensional display method for the sound object provided in this application, as shown below. Figure 5 As shown,

[0131] Third-party perspective 501: The user observes the entire virtual scene from the perspective of an outsider, just like a camera located at a fixed position in the scene or moving along a specific trajectory, able to see multiple movable objects and the overall picture of the scene.

[0132] Movable object perspective 502: This means that the user will enter the "subjective field of view" of the movable object and can only see the part of the scene that the object sees. For example, if the movable object is the frog mentioned above, then if the user switches to the movable object's perspective, the user's field of view will become the frog's field of view.

[0133] User Perspective 503: In this scenario, users can freely move their viewpoint within the scene, as if they were actually in the virtual environment. For example, in a virtual concert hall, after observing the overall layout of different instruments from a third-party perspective, switching to the user's perspective allows the user to freely turn their head, move around, observe the details of instruments of interest up close, and experience the playback effects while walking between different instruments.

[0134] S306. In response to the user's selection operation on the second interface, determine that the trajectory indicated by the selection operation is a first preset movement trajectory, the first preset movement trajectory carrying preset time information and preset speed information; wherein, the second interface is a three-dimensional interface or an interface in virtual reality space; multiple trajectories are displayed in the second interface; the selection operation indicates that a trajectory needs to be selected as the preset movement trajectory; wherein, the first preset movement trajectory is used for a movable object to move on the three-dimensional virtual interface based on the preset movement trajectory.

[0135] Users interact on a second interface, which is a three-dimensional interface, such as a computer graphics interface with stereoscopic effects, or an interface within a virtual reality space. In these interfaces, users can interact with various elements in an immersive way.

[0136] When users see multiple tracks displayed on the interface, they will select one based on their needs and intentions. This selection can take many forms, such as clicking on a track with a mouse, swiping on a touchscreen, specifying a track via voice command, or using gestures in a virtual reality environment. Regardless of the method used, the user's goal is to choose the track that interests them from among many options.

[0137] Optionally, the second interface is a pre-set interface that includes various trajectory templates. Figure 6A schematic diagram of the trajectory template for the three-dimensional display method of the sound object provided in this application, as shown below. Figure 6 As shown:

[0138] The system will respond to the user's selection and determine the selected trajectory as the first preset movement trajectory. This first preset movement trajectory is not merely a path; it also carries preset time and speed information. The preset time information indicates the duration of the first preset movement trajectory.

[0139] Because the second interface is a 3D interface or an interface in virtual reality space, users can better experience the differences between different preset trajectories. In a 3D interface, trajectories can be displayed in various forms in 3D space, such as... Figure 7 As shown, Figure 7 This is a schematic diagram of a 3D trajectory template for the 3D display method of the sound object provided in this application. Users can observe the characteristics and details of the trajectory from different angles. Furthermore, within the interface of a virtual reality space, users can experience the trajectory effects of different trajectories in an immersive way.

[0140] S307. In response to an adjustment command issued by the user based on the first preset movement trajectory, wherein the adjustment command carries time adjustment information and speed adjustment information; and generate a second preset movement trajectory based on the adjustment command.

[0141] The system modifies and optimizes the first preset movement trajectory based on the time and speed adjustment information in the adjustment command. For time adjustment, if the user requests to arrive at a certain location ahead of schedule, the system may find a more direct path or increase the speed of the movable object; if the arrival is delayed, it may appropriately reduce the speed or add some detours. For speed adjustment, the system adjusts the curvature and length of the corresponding sections of the trajectory according to the speed requirements specified by the user at different stages, ensuring that the movable object can move smoothly according to the new speed requirements.

[0142] Once a user has initially selected a preset movement trajectory for a movable object, in real-world scenarios, further optimization or adjustment of the object's movement may be necessary. For example, in an audio playback scenario, the user initially chooses a trajectory that moves from point A to point C above their head, hovering until reaching point B. However, during the movement, the user finds that the sound effects don't meet their needs. In this case, the user can send signals carrying time and speed adjustment information. Time adjustment information might include extending or shortening the time it takes for the movable object to reach a specific location; for example, the user might want the movable object to hover above point C for a longer period. Speed ​​adjustment information involves changing the speed of the trajectory while hovering above point C. For instance, starting from point A, the speed could be increased to quickly reach the height of point C, and then decreased near point C to extend the hovering time.

[0143] S308. Based on the motion sensing information to be emitted from each movable object displayed on the three-dimensional virtual interface, control the hardware device set in the preset physical space to emit linkage motion sensing information.

[0144] For example, based on the motion sensing information to be emitted by each movable object displayed on the three-dimensional virtual interface, conflict information between the motion sensing information to be emitted by movable objects of the same category is determined. Based on the conflict information, the linkage motion sensing information of the hardware device is determined from the motion sensing information to be emitted by movable objects of the same category. The hardware device set in the preset physical space is controlled to emit the linkage motion sensing information.

[0145] For example, different movable objects correspond to different sound effects. Based on the different sound effect types, there are pre-defined associations with the hardware devices set up in the physical space. For example, the pre-defined hardware devices in the physical space include: audio equipment, aromatherapy devices, massage chair equipment, ambient lighting equipment, etc., as shown in Table 1.

[0146] Table 1

[0147]

[0148] Each movable object corresponds to at least two association modes: the default mode and the user-defined mode.

[0149] For example: a movable pair in the shape of a campfire ball is preset, corresponding to campfire sound effects, corresponding to an air conditioner temperature of 25°C, releasing warm fragrance, adjusting the massage chair to a preset angle, and turning on the firelight jumping ambient light mode.

[0150] If both ice / snow spheres and campfire spheres exist simultaneously, the preset second massage chair angle corresponding to the ice / snow sphere is inconsistent with the preset first massage chair angle corresponding to the campfire sphere; the ice / snow sphere corresponds to an air conditioner temperature of 16℃ and a cold-type fragrance, while the campfire sphere corresponds to an air conditioner temperature of 25℃ and a warm-type fragrance, indicating inconsistent temperatures. These can be identified as conflicting information.

[0151] Regarding temperature perception information, a temperature conflict is identified if the temperature settings of different areas (represented by different movable objects) differ by more than a certain threshold (e.g., ±2℃). For massage chair angle perception information, an angle conflict is considered when the difference in massage chair angle settings associated with different movable objects exceeds a specific angle (e.g., ±10°). Regarding fragrance, a fragrance conflict is considered to exist if different areas expect different types of fragrance to be released, or if the concentration of the same fragrance differs significantly (e.g., more than 30%). Lighting conflict is determined based on the different needs of different areas for light brightness (difference exceeding 20%), color (color difference exceeding a certain standard), or mode (e.g., flicker frequency difference greater than 50%).

[0152] If a temperature conflict is detected, a global temperature balancing strategy is prioritized. The average temperature of all conflicting areas is calculated, and then the temperature control devices (such as air conditioners) in each area are adjusted to near this average temperature. For example, if the temperature in the campfire-shaped ball area is 25℃ and the temperature in the ice and snow-shaped ball area is 20℃, and the calculated average temperature is 22.5℃, then the air conditioning temperatures in both areas are adjusted to approximately 22.5℃.

[0153] When a conflict arises regarding the angle of the massage chair, consider the user's usage frequency and habits. If a user frequently uses a particular angle, then when a conflict occurs, prioritize adjusting the massage chair angle to that user's preferred angle. For example, if a user frequently uses the 45° angle, adjust the massage chair angle to 45° when an angle conflict occurs.

[0154] You can also set default massage chair angles for different functional areas, such as rest areas and entertainment areas (determined based on the attributes of the preset physical space). When a conflict occurs, the angles are adjusted according to the priority of the functional area. For example, the massage chair angle in the rest area will prioritize the specific angle that meets the relaxation needs when there is a conflict.

[0155] To address fragrance clashes, a zoned release strategy can be employed. Based on the intended use of different areas and user needs, fragrances can be divided into different diffusion zones. For example, a warm, woody fragrance can be released in the campfire-shaped sphere area, while a fresh, cool fragrance can be released in the snow-shaped sphere area, reducing interference between fragrances.

[0156] Alternatively, based on fragrance compatibility, choose a complex fragrance that can blend multiple scents to create a relatively harmonious overall atmosphere.

[0157] When lighting conflicts occur, adjustments should be made based on the scene's lighting requirements and atmosphere. In scenes requiring a bright environment (such as activity areas), prioritize higher brightness and more active lighting patterns (such as rapid flashing) to unify the lighting across the entire scene. For example, adjust the brightness of the lights in the campfire-shaped and snow-shaped sphere areas to a higher level, and unify the flashing frequency to a faster pattern.

[0158] In areas where a warm and soft atmosphere is needed (such as rest areas), reduce the overall brightness and use soft light colors and slow flickering or non-flickering modes to make the lighting in different areas harmonious and consistent under this atmosphere.

[0159] Optionally, a user preference database can be established to record each user's preference for different sensory information. For example, some users prefer higher temperatures and stronger fragrances, while others prefer lower temperatures and lighter fragrances.

[0160] When a conflict in sensory information is detected, the central control system queries the user preference database and adjusts the sensory information based on the user's location and historical preferences within the scene. For example, if a user who prefers high temperature and strong fragrance is near the campfire-shaped ball area, while a user who prefers low temperature and light fragrance is near the snow-shaped ball area, when a temperature and fragrance conflict occurs, the temperature and fragrance concentration in the campfire-shaped ball area are appropriately increased, while the relatively low temperature and fragrance concentration in the snow-shaped ball area are maintained.

[0161] After resolving the conflict, continuously monitor changes in haptic feedback. If a new conflict arises, promptly re-execute the conflict resolution strategy. For example, when a user moves within the scene, it may cause a new conflict to arise in the previously resolved haptic feedback, requiring the system to reassess and adjust accordingly. Establish user feedback channels, such as feedback buttons or on-site feedback terminals. Users can provide feedback at any time regarding the comfort level of their current haptic feedback, allowing for further optimization of the haptic feedback adjustment strategy. For example, if a user reports that the fragrance in a certain area is too strong, even if no fragrance conflict is currently detected, the system will appropriately reduce the fragrance concentration in that area.

[0162] The three-dimensional display method for sound objects provided in this application calculates and plays corresponding sound object sound effects in real time based on the user's operation trajectory when the user touches or moves the movable object. These sound effects change according to the user's actions and the object's position to simulate sound changes in a real environment. When the user stops touching the movable object, the system continues to calculate and play subsequent sound object sound effects based on the object's current movement speed and direction. This ensures that even when the user does not directly touch the object, the sound changes still match the object's movement. Based on the user's operation trajectory and the object's movement state, the system continuously adjusts the parameters of the sound object sound effects to simulate sound changes in a real environment, achieving the effect that the user can not only visually see the object's movement but also auditorily perceive the source and changes of the sound.

[0163] Figure 8 A schematic diagram of the structure of the three-dimensional display device for the sound object provided in this application, as shown below. Figure 8 As shown, the three-dimensional display device 80 for sound objects provided in this embodiment includes:

[0164] The first control unit 801 is configured to respond to a first trigger operation, control the movable object operated by the first trigger operation to move according to the user's operation trajectory, and play sound object sound effects corresponding to both the movable object and the operation trajectory within the preset physical space; wherein, the first trigger operation represents the user moving the movable object on the three-dimensional virtual interface;

[0165] The second control unit 802 is used to respond to the second trigger operation, control the movement speed of the movable object at the current moment, perform subsequent movement, and play sound object sound effects corresponding to both the movable object and the subsequent movement trajectory within the preset physical space; wherein, the second trigger operation indicates that the user no longer touches the movable object.

[0166] In one possible implementation, the second control unit 802 is configured to: acquire the moving speed of the movable object at multiple times; determine a speed to be used based on the moving speed of the movable object at the current time and the moving speed at the multiple times; and if the speed to be used is determined to be greater than a preset speed threshold, control the movable object to move according to the speed to be used.

[0167] In one possible implementation, the second control unit 802 is further configured to: control the movable object to move in a uniform linear motion according to the speed to be used; or control the movable object to move in a decelerated linear motion according to the speed to be used and the deceleration of the scene corresponding to the at least one movable object; or generate the subsequent movement trajectory of a preset shape according to the operation trajectory and the speed to be used, and control the movable object to move according to the subsequent movement trajectory of the preset shape with the speed to be used as the starting speed.

[0168] In one possible implementation, the device further includes a detection unit, configured to, if the movable object is detected to have touched a first edge of the three-dimensional virtual interface during the subsequent movement, control the movable object to bounce off the first edge of the three-dimensional virtual interface and then move.

[0169] In one possible implementation, the detection unit is further configured to: determine the angle between the subsequent movement trajectory of the movable object and the normal of the first edge, which is the angle of incidence; determine the straight line where the reflection angle of the angle of incidence lies, which is the rebound movement trajectory line of the movable object; and control the movable object to move along the rebound movement trajectory line after rebounding from the first edge of the three-dimensional virtual interface.

[0170] In one possible implementation, the detection unit is further configured to: if the movable object is detected to touch the second edge of the three-dimensional virtual interface during the subsequent movement, determine a first volume in which the movable object enters the second edge; wherein the first edge is an edge constituting a first virtual space, the second edge is an edge constituting a second virtual space, and the second virtual space is located inside the first virtual space; if the three-dimensional virtual interface has the second edge, then the space inside the second virtual space is the user-visible interface; and determine the display volume of the movable object displayed inside the second edge based on the first volume.

[0171] In one possible implementation, the device further includes a volume unit for adjusting the volume of the movable object in response to a third triggering operation on the movable object.

[0172] In one possible implementation, the device further includes an adjustment unit for adjusting the display size of the aperture surrounding the movable object in real time based on the volume of the sound effect of the movable object at the current moment.

[0173] In one possible implementation, the device further includes: a switching unit, configured to switch the currently displayed first view to a second view in response to a switching command; wherein the switching command indicates a switch view; when the first view is a third-party view, the second view is the view of a movable object indicated by the switching command or the user's view; when the first view is the view of a movable object, the second view is a third-party view or the user's view; when the first view is the user's view, the second view is a third-party view or the view of a movable object.

[0174] In one possible implementation, the device further includes: a template unit, configured to, in response to a user's selection operation on a second interface, determine that the trajectory indicated by the selection operation is a first preset movement trajectory, the first preset movement trajectory carrying preset time information and preset speed information; wherein, the second interface is a three-dimensional interface or an interface in a virtual reality space; the second interface displays multiple trajectories; the selection operation indicates that a trajectory needs to be selected as the preset movement trajectory; wherein, the first preset movement trajectory is used for a movable object to move on the three-dimensional virtual interface based on the preset movement trajectory.

[0175] In one possible implementation, the template unit is further configured to, in response to an adjustment command issued by a user based on the first preset movement trajectory, the adjustment command carrying time adjustment information and speed adjustment information, generate a second preset movement trajectory based on the adjustment command.

[0176] In one possible implementation, the device further includes: a motion sensing unit, which controls the hardware devices set in the preset physical space to emit linkage motion sensing information based on the motion sensing information to be emitted by each movable object displayed on the three-dimensional virtual interface.

[0177] In one possible implementation, the motion sensing unit is further configured to determine conflict information between the motion sensing information to be emitted by movable objects of the same category based on the motion sensing information to be emitted by each movable object displayed on the three-dimensional virtual interface, and to determine the linkage motion sensing information of the hardware device from the motion sensing information to be emitted by movable objects of the same category based on the conflict information; and to control the hardware device set in the preset physical space to emit the linkage motion sensing information.

[0178] The three-dimensional display device for the sound object provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0179] This application provides a method for displaying a three-dimensional sound object in a sound production system, comprising: responding to a first trigger operation triggered by a user in the sound production system, controlling a movable object operated by the first trigger operation to move on the interface displayed by the sound production system according to the user's operation trajectory, and playing a stereoscopic generated sound effect corresponding to both the movable object and the operation trajectory within a preset physical space; wherein, the first trigger operation represents the user moving the movable object on the three-dimensional virtual interface; responding to a second trigger operation triggered by the user in the sound production system, controlling the movement speed of the movable object at the current moment, performing subsequent movement on the interface displayed by the sound production system, and playing a sound object sound effect corresponding to both the movable object and the subsequent movement trajectory within the preset physical space; wherein, the second trigger operation represents the user no longer touching the movable object.

[0180] This application provides a method for displaying a three-dimensional sound object in an interactive audio system, comprising: responding to a first trigger operation triggered by a user in the interactive audio system, controlling a movable object operated by the first trigger operation to move on the interface displayed by the interactive audio system according to the user's operation trajectory, and playing a stereoscopic generated sound effect corresponding to both the movable object and the operation trajectory within a preset physical space; wherein, the first trigger operation represents the user moving the movable object on the three-dimensional virtual interface; responding to a second trigger operation triggered by the user in the interactive audio system, controlling the movement speed of the movable object at the current moment, performing subsequent movement on the interface displayed by the interactive audio system, and playing a sound object sound effect corresponding to both the movable object and the subsequent movement trajectory within the preset physical space; wherein, the second trigger operation represents the user no longer touching the movable object.

[0181] Figure 9 A schematic diagram of the structure of the three-dimensional display device for the sound object provided in this application. Figure 9 As shown, the electronic device 90 provided in this embodiment includes at least one processor 901 and a memory 902. Optionally, the device 90 further includes a communication component 903. The processor 901, memory 902, and communication component 903 are connected via a bus 904.

[0182] In a specific implementation, at least one processor 901 executes computer execution instructions stored in memory 902, causing at least one processor 901 to perform the above-described method.

[0183] The specific implementation process of processor 901 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0184] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0185] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0186] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0187] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0188] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0189] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0190] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0191] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0192] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0193] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0194] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0195] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0196] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for displaying a three-dimensional sound object, characterized in that, The method is applied to a terminal device located within a preset physical space. The terminal device displays a three-dimensional virtual interface, which includes at least one movable object. The movable object is in a moving state or a non-moving state. The method includes: In response to a first trigger operation, the movable object operated by the first trigger operation is controlled to move according to the user's operation trajectory, and sound effects of sound objects corresponding to both the movable object and the operation trajectory are played in the preset physical space; wherein, the first trigger operation represents the user moving the movable object on the three-dimensional virtual interface; In response to the second trigger operation, the movement speed of the movable object at the current moment is controlled to perform subsequent movement, and sound effects of sound objects corresponding to both the movable object and the subsequent movement trajectory are played in the preset physical space; wherein, the second trigger operation indicates that the user no longer touches the movable object.

2. The method according to claim 1, characterized in that, Based on the moving speed of the movable object at the current moment, subsequent movement is performed, including: Obtain the movement speed of the movable object at multiple times; determine the speed to be used based on the movement speed of the movable object at the current time and the movement speed at the multiple times. If it is determined that the speed to be used is greater than a preset speed threshold, the movable object is controlled to move according to the speed to be used.

3. The method according to claim 2, characterized in that, Controlling the movable object to move according to the speed to be used includes: The movable object is controlled to move at a constant linear speed according to the speed to be used; Alternatively, the movable object can be controlled to decelerate and move in a straight line according to the speed to be used and the deceleration of the scene corresponding to the at least one movable object; Alternatively, based on the operation trajectory and the speed to be used, a subsequent movement trajectory of a preset shape is generated, and the movable object is controlled to move according to the subsequent movement trajectory of the preset shape, starting from the speed to be used.

4. The method according to claim 1, characterized in that, The method further includes: If it is detected that the movable object touches the first edge of the three-dimensional virtual interface during the subsequent movement, the movable object is controlled to bounce off the first edge of the three-dimensional virtual interface and then move.

5. The method according to claim 4, characterized in that, Controlling the movable object to move after bouncing off the first edge of the three-dimensional virtual interface includes: The angle between the subsequent movement trajectory of the movable object and the normal of the first edge is determined as the angle of incidence; the straight line containing the reflection angle of the angle of incidence is determined as the rebound movement trajectory line of the movable object. The movable object is controlled to move along the bounce trajectory line after bouncing off the first edge of the three-dimensional virtual interface.

6. The method according to claim 1, characterized in that, The method further includes: In response to a third trigger operation on the movable object, the volume of the movable object is adjusted.

7. The method according to claim 1, characterized in that, The method further includes: The display size of the aperture surrounding the movable object is adjusted in real time based on the volume of the sound effect of the movable object at the current moment.

8. The method according to claim 1, characterized in that, The three-dimensional virtual interface is a three-dimensional interface or a virtual reality interface, and the method further includes: In response to a switching instruction, switching the currently displayed first perspective to a second perspective; wherein, the switching instruction represents switching perspectives; when the first perspective is a third-party perspective, the second perspective is the perspective of the movable object indicated by the switching instruction or the user perspective; when the first perspective is the perspective of the movable object, the second perspective is the third-party perspective or the user perspective; when the first perspective is the user perspective, the second perspective is the third-party perspective or the perspective of the movable object.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: In response to a selection operation by the user on the second interface, determining that the trajectory indicated by the selection operation is a first preset movement trajectory, and the first preset movement trajectory carries preset time information and preset speed information; Wherein, the second interface is a three-dimensional interface or an interface in a virtual reality space; multiple trajectories are displayed in the second interface; the selection operation indicates that a trajectory needs to be selected as the preset movement trajectory; wherein, the first preset movement trajectory is used for the movable object to move on the three-dimensional virtual interface based on the preset movement trajectory.

10. The method according to claim 9, characterized in that, The method further includes: In response to an adjustment instruction issued by the user based on the first preset movement trajectory, the adjustment instruction carries time adjustment information and speed adjustment information; Generating a second preset movement trajectory based on the adjustment instruction.

11. The method according to any one of claims 1-8, characterized in that, The method further includes: According to the somatosensory information to be sent of each movable object displayed in the three-dimensional virtual interface, controlling the hardware devices arranged in the preset physical space to send associated somatosensory information.

12. According to the method described in claim 11, controlling the hardware devices arranged in the preset physical space to send associated somatosensory information according to the somatosensory information to be sent of each movable object displayed in the three-dimensional virtual interface includes: According to the somatosensory information to be sent of each movable object displayed in the three-dimensional virtual interface, determining the conflict information between the somatosensory information to be sent of the movable objects of the same category, and according to the conflict information, determining the associated somatosensory information of the hardware devices from the somatosensory information to be sent of the movable objects of the same category; Controlling the hardware devices arranged in the preset physical space to send the associated somatosensory information.

13. A three-dimensional display device for a sound object, characterized in that, The device is applied to a terminal device, the terminal device is located in a preset physical space, the terminal device displays a three-dimensional virtual interface, the three-dimensional virtual interface includes at least one movable object, the movable object is in a moving state or a non-moving state, and the device includes: A first control unit, configured to, in response to a first trigger operation, control the movable object operated by the first trigger operation to move according to the operation trajectory of the user, and play the sound object sound effect corresponding to both the movable object and the operation trajectory in the preset physical space; wherein, the first trigger operation represents that the user moves the movable object on the three-dimensional virtual interface. The second control unit is used to respond to the second trigger operation, control the movement speed of the movable object at the current moment, perform subsequent movement, and play sound object sound effects corresponding to both the movable object and the subsequent movement trajectory within the preset physical space; wherein, the second trigger operation indicates that the user no longer touches the movable object.

14. A device for playing sound objects, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-12.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-12.

17. A method for displaying three-dimensional sound objects in a sound production system, characterized in that, The method includes: In response to a first trigger operation by the user in the sound production system, the movable object operated by the first trigger operation is controlled to move on the interface displayed by the sound production system according to the user's operation trajectory, and a stereo generated sound effect corresponding to both the movable object and the operation trajectory is played in the preset physical space; wherein, the first trigger operation represents the user moving the movable object on the three-dimensional virtual interface; In response to a second trigger operation initiated by the user in the sound production system, the movement speed of the movable object at the current moment is controlled, and subsequent movement is performed on the interface displayed by the sound production system. The sound object sound effect corresponding to both the movable object and the subsequent movement trajectory is played within the preset physical space. The second trigger operation indicates that the user no longer touches the movable object.

18. A method for displaying three-dimensional sound objects in an interactive audio system, characterized in that, The method includes: In response to a first trigger operation by the user in the interactive audio system, the movable object operated by the first trigger operation is controlled to move on the interface displayed by the interactive audio system according to the user's operation trajectory, and a stereo generated sound effect corresponding to both the movable object and the operation trajectory is played in the preset physical space; wherein, the first trigger operation represents the user moving the movable object on the three-dimensional virtual interface; In response to a second trigger operation initiated by the user in the interactive audio system, the movement speed of the movable object at the current moment is controlled, and subsequent movement is performed on the interface displayed by the interactive audio system. Sound effects of the sound objects corresponding to both the movable object and the subsequent movement trajectory are played within the preset physical space. The second trigger operation indicates that the user no longer touches the movable object.

Images