Sound generation control method, sound generation device, and sound generation control program

By automatically selecting the solver and adjusting parameters in the virtual space, the problems of high workload and difficulty in timbre adjustment of content maker are solved, and a consistent sound signal and realistic virtual space experience are achieved.

CN119072741APending Publication Date: 2024-12-03SONY GROUP CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202380035110.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-05-02
Filing Date
2023-04-21
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In virtual spaces, prior art is difficult to effectively reduce the workload of content producers and generate consistent sound signals, and it is difficult to directly adjust the tone to achieve the sound desired by the producers.

Method used

By automatically selecting multiple solvers in the virtual space and adjusting parameters according to the input of the producer, a second sound signal conforming to predetermined physical rules is generated to reduce workload and generate consistent sound.

Benefits of technology

This achieves reducing the workload of the producer, generating consistent sound signals, enhancing the realistic sound in the virtual space, and avoiding the problem of uncorrelation between parameters.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119072741A_ABST
    Figure CN119072741A_ABST
Patent Text Reader

Abstract

A sound generation control method according to one aspect of the present disclosure comprises a step in which a computer: accepts an input of environmental data indicating conditions set for a virtual space in which a light source and a sound receiving point are arranged, respectively; selecting a plurality of resolvers for calculating sound characteristics at the sound receiving point according to the environment data, and determining a first parameter to be input to each of the plurality of resolvers; accepting a change request for a first sound signal generated based on the first parameter; and adjust the environmental data or the first parameter in response to the change request, and generate a second sound signal using the adjusted environmental data or a second parameter, the second parameter being the adjusted parameter and being re-input to each resolver.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a sound generation control method, a sound generating device, and a sound generation control program. Background Art

[0002] With the development of virtual space technologies such as games, metalanguage, and cross-reality (XR), in order to enhance the user's sense of immersion, the sound reproduced in the virtual space also needs to be based on the real experience of reality. In the study of auditory recognition of sound-emitting bodies in three-dimensional virtual spaces, there are many methods for simulating sound generation. Among them, spatial sound field reproduction technology using physical simulation is designed based on physical phenomena in real space, and when the sound is reproduced in the virtual space in terms of audibility, the reproducibility is high.

[0003] Examples of physical simulation methods in sound fields include wave sound simulation that models and calculates the wave characteristics of sound, and geometric sound simulation that models and calculates the energy propagation of sound, represented by the sound line method and the virtual image method. It is known that the former well shows the apparent microwave phenomenon, and is particularly advantageous for simulating low-frequency sounds that are greatly affected by diffraction, interference phenomena, etc. However, since this method discretizes the space and performs calculations, the calculation load is very large compared to the geometric sound simulation, and it is difficult to perform real-time processing such as sound calculation that follows the player's line of sight in the game. On the other hand, in the latter, although the calculation algorithm is relatively simple and can be realized in real time, because the fluctuation of sound such as diffraction and interference cannot be considered, the application is limited to high-frequency bands with sufficiently short wavelengths in the size of the space.

[0004] Regarding the sound reproduction processing in the sound simulation, a method has been proposed to reproduce a sound close to real hearing with a low amount of calculation by separately calculating the early reflection sound and the high-order reflection sound (late reverberation sound) (for example, the following patent document 1). In addition, a method for adjusting the ratio of the volume between the early reflection sound and the late reverberation sound according to the distance between the sound source and the user (listening point) has been proposed (for example, patent document 2). In addition, a method is known in which the sound space is divided into a plurality of areas in order to calculate the fluctuating sound, and sound processing is performed between the divided boundaries to increase the calculation speed (for example, non-patent document 1). In addition, a deep learning method has been proposed for deriving an expression for performing calculations that satisfy physical rules using machine learning (for example, non-patent document 2).

[0005] Reference List

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2000-267675

[0008] Patent Document 2: JP 2019-165845 A

[0009] Non-patent literature

[0010] Non-patent literature 1: "Efficient and Accurate Sound Propagation Using AdaptiveRectangular Decomposition" N.Raghuvanshi et al., IEEE (2009)

[0011] Non-patent literature 2: "Physics-informed neural networks: Adeep learning framework for solving forward and inverse problems involving nonlinear partial differential equations" M. Raissi, P. Perdikaris, GE Karniadakis, Journal of Computational Physics 378 (2019) Summary of the invention

[0012] Technical issues

[0013] According to the related art, it is possible to improve the reproducibility of sound in a virtual space. The sound of a virtual space is set based on spatial object information and the like. However, in the related art, a content producer needs to set characteristics such as reflection parameters for each object constituting the virtual space, and there is a large workload. In addition, in the related art, since the content producer cannot directly adjust the timbre, there is a lack of intuition, and it may be difficult to generate the sound desired by the producer.

[0014] In addition, for example, in content production, in order to achieve the purpose of amplifying a specific sound characteristic for production or otherwise appropriately expressing the sound characteristic, parameter adjustments that are not necessarily based on physical rules may be performed. When producers perform parameter adjustments for these purposes, there is a problem of collapse of the correlation between parameters that various generators for generating sound signals should physically follow.

[0015] Therefore, the present disclosure proposes a sound generation control method, a sound generating device, and a sound generation control program that can reduce the workload of a content producer and generate a consistent sound signal.

[0016] Solution to the problem

[0017] According to one embodiment of the present disclosure, a sound generation control method includes: causing a computer to perform the following operations: receiving input of environmental data indicating each condition set in a virtual space where sound sources and sound receiving points are arranged; selecting multiple solvers for calculating characteristics of sound at sound receiving points based on the environmental data, and determining a first parameter to be input to each of the multiple solvers; receiving a change request for a first sound signal generated based on a first parameter; and adjusting the environmental data or the first parameter in response to the change request, and generating a second sound signal using the adjusted environmental data or the second parameter, the second parameter being an adjusted parameter and being re-input to each solver. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a diagram showing an overview of a sound generation control method according to an embodiment.

[0019] Figure 2 is a diagram showing details of a sound simulator according to an embodiment.

[0020] Figure 3 is a diagram showing a configuration example of a sound generating device according to an embodiment.

[0021] Figure 4 is a diagram for explaining an overview of sound simulation according to an embodiment.

[0022] Figure 5 : is a flowchart showing the flow of speech generation processing in sound simulation.

[0023] Figure 6 FIG. 1 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0024] Figure 7 FIG. 2 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0025] Figure 8 FIG. 3 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0026] Fig. 9 This is a flowchart (1) showing the flow of solver selection processing in sound simulation.

[0027] Fig.10 is a diagram for explaining the application of the diffraction sound solver.

[0028] Fig.11 Flowchart (2) shows the flow of solver selection processing in sound simulation.

[0029] Fig.12FIG. 4 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0030] Fig.13 FIG. 5 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0031] Fig.14 is a diagram for explaining the parameters of each solver.

[0032] Fig.15 FIG. 6 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0033] Fig.16 is a flowchart (1) showing an example of the flow of a change process according to an embodiment.

[0034] Fig.17 FIG. 7 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0035] Fig.18 is a flowchart (2) showing an example of the flow of a change process according to an embodiment.

[0036] Fig.19 is a flowchart (3) showing an example of the flow of a change process according to an embodiment.

[0037] Fig. 20 FIG. 8 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0038] Fig.21 FIG. 9 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0039] Fig. 22 is a diagram for explaining an example of a parameter control process according to an embodiment.

[0040] Fig.23 : is a diagram showing an overview of a sound generation control method according to a modification.

[0041] Fig.24 is a diagram illustrating an example of output control processing according to a modification.

[0042] Fig.25 is a hardware configuration diagram showing an example of a computer that realizes the function of the sound generating device. DETAILED DESCRIPTION

[0043] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in the following embodiments, the same reference numerals are given to the same parts and repeated descriptions are omitted.

[0044] The present disclosure will be described according to the following item order.

[0045] 1. Implementation Method

[0046] 1-1. Overview of Sound Generation Control Method According to Embodiment

[0047] 1-2. Configuration of Sound Generating Device According to Embodiment

[0048] 1-3. Overview of Sound Simulation According to Embodiment

[0049] 1-4. Adjustment of environmental data (global data)

[0050] 1-5. Learning process through scene setting

[0051] 1-6. Modifications of the Embodiments

[0052] 2. Other Implementation Methods

[0053] 3. Effects of the sound generation control method according to the present disclosure

[0054] 4. Hardware Configuration

[0055] (1. Implementation Method)

[0056] (1-1. Overview of Sound Generation Control Method According to Embodiment)

[0057] First, refer to Figure 1 An overview of a sound generation control method according to an embodiment is described. Figure 1 is a diagram showing an overview of a sound generation control method according to an embodiment.

[0058] The sound generation control method according to this embodiment is as follows: Figure 1 The sound generating device 100 shown is executed. The sound generating device 100 is an example of the sound generating device of the present invention, and is an information processing terminal used by a creator 200 who creates content related to a virtual space such as a game or metatext. For example, the sound generating device 100 is a personal computer (PC), a server device, a tablet terminal, etc.

[0059] The sound generating device 100 includes an output unit such as a display and a speaker, and outputs various types of information to the producer 200. For example, the sound generating device 100 displays a user interface of software related to sound generation (e.g., a sound simulator that generates sound (sound signal) based on input information) on the display. In addition, the sound generating device 100 outputs the generated sound signal from the speaker according to the operation instructed from the producer 200 on the user interface.

[0060] In the embodiment, the sound generating device 100 calculates what kind of sound the sound output from the sound source object (hereinafter referred to as "sound source") reproduces at a listening point in a virtual three-dimensional space such as a game (hereinafter referred to as "virtual space"), and reproduces the calculated sound. That is, the sound generating device 100 performs sound simulation in the virtual space, and performs processing to make the sound emitted in the virtual space close to the real world or the sound desired by the reproduction producer 200.

[0061] For example, a virtual space where the producer 200 wants to set the sound is displayed on a display of the sound generating device 100. The producer 200 sets the position (coordinates) of the sound source (sound generating point) in the virtual space, and sets the sound receiving point (the position where the sound is observed in the virtual space, and is also called the listening point). In the actual space, due to various physical phenomena, a difference occurs between the sound observed near the sound source and the sound observed at the listening point. Therefore, the sound generating device 100 virtually reproduces (simulates) real physical phenomena in the virtual space according to the instructions of the producer 200, and generates a sound signal suitable for the space, so as to enhance the realistic feeling of the sound expression experienced in the virtual space by game players etc. (hereinafter referred to as "users") who use the content.

[0062] The sound signal generated by the sound generating device 100 will be described. Figure 1 The graph 60 shown in schematically shows the loudness of the sound when the sound emitted from the sound source is observed at the sound receiving point. In the case where the sound receiving point can be seen from the sound source, first, the direct sound is observed at the sound receiving point, and then the diffracted sound, the transmitted sound, etc. of the direct sound are observed, and the early reflected sound reflected at the boundary of the virtual space is observed. Whenever the sound is reflected at the boundary, the early reflected sound is observed, and for example, the reflected sound of the first to third orders or the reflected sound reaching 80ms after the direct sound arrives is observed as the early reflected sound. Thereafter, the high-order reflected sound is observed at the sound receiving point as the late reverberation sound. Since the sound emitted from the sound source decays over time, the graph 60 draws an envelope (decay curve) that asymptotically approaches 0 with the direct sound as the vertex.

[0063] The producer 200 designs the sound characteristics of the virtual space so that the sound emitted in the virtual space becomes a realistic sound with a lifelike feeling to the user who listens to the sound at the sound receiving point. Specifically, the producer 200 designs the sound characteristics of the objects arranged in the virtual space and the boundaries of the virtual space (equivalent to the walls, ceiling, etc. of the virtual space).

[0064] In addition, the producer 200 may edit the sound signal illustrated in the graph 60 so that the sound output in the virtual space becomes an ideal sound. However, for example, in the case where the producer 200 changes only the late reverberation sound to the ideal sound, the late reverberation sound is observed after the direct sound, the diffracted sound, and the early reflected sound. Therefore, if the coordination of the direct sound and the early reflected sound is unsuccessful, an unnatural sound may be reproduced as a whole. That is, the entire sound signal of the virtual space (such as the early reflected sound and the late reverberation sound) is required to maintain an appropriate relationship close to the real physical phenomenon.

[0065] Furthermore, in order for the producer 200 to realize ideal sound characteristics in the virtual space, it is necessary to select various generators (referred to as solvers, etc.) required for sound simulation and set parameters to be input to each generator. Such setting work imposes a large workload on the producer 200, hindering the progress of the work.

[0066] Therefore, the sound generating device 100 according to the contents of the present disclosure solves the above-mentioned problems by a sound generation control method according to an embodiment. Specifically, the sound generating device 100 automatically selects a plurality of solvers to generate a sound signal based on a data group in a virtual space, and then determines the parameters. In addition, in the case where the producer 200 changes an optional parameter, the sound generating device 100 automatically adjusts another parameter based on a predetermined physical rule (based on a calculation to be described later), and regenerates a sound signal without unnaturalness as a whole. In other words, in the case of changes made by the producer 200, the sound generating device 100 as a whole makes sense by adjusting other elements used to generate the sound signal. Therefore, the sound generating device 100 can reduce the workload of the producer 200 and generate a consistent sound signal.

[0067] Figure 1 An overview of the flow of the sound generation control method according to the embodiment is shown. Figure 1 The flow of the sound generating device 100 generating the sound signal shown in the diagram 60 is shown.

[0068] The producer 200 uses the user interface provided from the sound generating device 100 to input the input condition 10, which is a condition in a virtual space where a sound source and a sound receiving point are arranged. The input condition 10 includes various types of data in the virtual space, such as objects and space data constituting the virtual space, sound characteristics such as transmittance and reflectivity of the object, the positions of the sound source and the sound receiving point, and the loudness and directivity of the sound emitted from the sound source. Hereinafter, various types of data sets as the input condition 10 may be collectively referred to as environmental data.

[0069] When the input condition 10 is set, the sound generating device 100 generates the sound observed at the sound receiving point based on the input condition 10. For example, the sound generating device 100 inputs the input condition 10 to the generator 30 and the parameter controller 50 for calculating (generating) components that determine the characteristics of the sound (e.g., the early reflection sound and the late reverberation sound), and generates the sound observed at the sound receiving point (corresponding to the sound at the sound receiving point). Figure 1 That is, the generator 30 can be referred to as a calculator for sound simulation, which inputs the input conditions 10 and the output of the parameter controller 50 and outputs the sound characteristics associated with each generator. Figure 1 As shown, the producer 200 generates sound in which the components (direct sound, early reflected sound, etc.) generated by the producer 30 are synthesized, and a sound environment similar to a real space in a virtual space can be reproduced.

[0070] It should be noted that, in addition to Figure 1 In addition to the generator 30 shown, the sound simulator provided by the sound generating device 100 further includes a parameter controller 50. The parameter controller 50 is a functional unit that controls parameters for controlling the generator 30 according to a predetermined physical rule.

[0071] The parameter controller 50 is a calculator that generates parameters according to a predetermined physical rule, and for example, there may be a calculator based on an analytical solution of a theoretical formula or a wave sound simulator that correctly predicts a sound field in a three-dimensional space. That is, the parameter controller 50 is used to control the parameters input to each generator 30 based on the input condition 10, and for example, correlates each generator according to the physical rule and automatically adjusts the parameters to appropriate parameters.

[0072] As described above, in content generation, parameters that conform to physical rules are not necessarily required, and the producer 200 may, for example, exaggerate certain sound characteristics used for generation or may slightly express certain sound characteristics. That is, after generating a sound according to the input condition 10, the producer 200 may desire to change the sound according to the intention of generating the content generated by the producer. As an example, the producer 200 changes the parameters input to the generator 30 for another generator or changes the generator 30 itself. When parameter adjustments for these purposes are optionally performed, there is a concern that the correlation between the parameters of the various generators may collapse. For example, when the characteristics of the early reflection sound change, other characteristics should change accordingly in the case of real sound, but other characteristics are maintained in the sound simulated in the virtual space.

[0073] On the other hand, in the sound generation control method according to the embodiment, the parameter controller 50 generates new parameters or selects an appropriate generator to generate a sound signal that does not cause discomfort (does not contradict the physical rules) according to the intention of the producer 200. For example, when the producer 200 changes the early reflection sound of the sound, the parameter controller 50 reproduces the parameters input to another generator caused by the change according to the physical rules. As described above, in response to the change request from the producer 200 for the sound generated once, the sound generating device 100 performs feedback from the generator 30 to the parameter controller 50, and performs adjustment through the parameter controller 50. Then, the parameter controller 50 uses the reproduction parameters to make the generator 30 calculate the characteristics for constituting the sound signal again, thereby generating a sound that conforms to the predetermined physical rules and does not cause discomfort. Therefore, for example, in the case where the producer 200 changes the early reflection sound, it is possible to avoid generating a sound with discomfort as a whole or manually and automatically adjusting the late reverberation sound to be consistent with the whole. That is, the producer 200 can reduce the adjustment load for realizing the sound environment intended by the producer through the function of the parameter controller 50.

[0074] Note that the above-mentioned automatic adjustment is not necessarily limited to the readjustment of the parameters of the generator 30, and the environmental data input to the physical simulator, i.e., the sound source position, the boundary conditions of the object, etc., may also be automatically corrected based on the changed content. By reading the corrected environmental data, the sound generating device 100 can further increase the correlation between the parameters of the generator 30 and reduce the workload on the generator 200.

[0075] Will refer to Figure 2 The sound generation process performed by the sound generating device 100 is described in more detail. Figure 2 is a diagram showing details of a sound simulator according to an embodiment.

[0076] like Figure 2 As shown, input condition 10 includes sound source information 12, object information 14, sound receiving point information 16, etc. as scene information set in the virtual space. In addition, input condition 10 may include setting information 18 indicating information about the reproduction environment of the content, etc.

[0077] The sound source information 12 includes various types of information such as fluctuation data, the type and size of sound emitted from the sound source, the position and shape of the sound source, and the directionality of the sound.

[0078] The object information 14 includes space data and materials such as walls and ceilings constituting the virtual space, the positions and shapes of objects arranged in the virtual space, the materials of the objects, etc. These are not necessarily the same as the data used to display the video, and data different from the data used for video display can be used for sound representation accordingly, or data of simplified sound representation obtained by reducing the surface, polygon, etc. from the data used for video display can be generated and used. Sound characteristics (sound impedance, etc.) are preset for the material of the object. For example, the producer 200 selects the object to be arranged and the material of the object on the user interface, and specifies an optional position (coordinate) in the virtual space so that the object can be arranged.

[0079] The sound receiving point information 16 indicates the position where the sound emitted from the sound source is to be heard. The sound receiving point corresponds to the position of the head of the character in the game, for example, in the case of game content.

[0080] The setting information 18 is information such as the type of reproduction machine from which the content is reproduced and the platform on which the content is distributed. By setting this information, the sound generating device 100 is able to generate a sound signal in consideration of the characteristics of the reproduction environment. In addition, the setting information 18 may include information associated with the scene for which the producer 200 wants to generate a sound signal (hereinafter referred to as "scene setting"). For example, the scene setting is indicated by the status of the scene set in the current virtual space (such as "daily scene", "tense scene" or "battle scene"). Although the details will be described later, the sound generating device 100 can perform, for example, processing that automatically adjusts the output of each solver in association with each scene setting.

[0081] The generator 30 includes a direct sound solver 32, an early reflection sound solver 34, a diffraction sound solver 36, a transmission sound solver 38, and a late reverberation sound solver 40. When a parameter based on the input condition 10 is input, each solver outputs a value corresponding to the input value.

[0082] Note that there are a variety of solvers depending on the calculation method and the like, and the sound generating device 100 selects a solver according to the input conditions. For example, the sound generating device 100 may select a first late reverberation sound solver that performs calculation based on a geometric method or a second late reverberation sound solver that performs calculation based on a wave analysis method as the late reverberation sound solver 40. In addition, the sound generating device 100 may select not to use a specific solver according to the input conditions. For example, in the case where it is understood that there is no transmission sound between the sound source and the sound receiving point in the virtual space, the sound generating device 100 may select not to use the transmission sound solver 38 to generate a sound signal.

[0083] The parameter controller 50 controls the parameters input to the generator 30. First, in the case where the input condition 10 is input from the generator 200, the parameter controller 50 derives the first parameter (parameter before change) to be input to the generator 30 based on such condition. After the sound signal is generated based on the first parameter, when the first parameter or the sound signal is edited by the producer 200, the parameter controller 50 derives the second parameter (parameter after change) to be input to the generator 30 based on the changed data.

[0084] In an embodiment, the parameter controller 50 has a plurality of models for deriving parameters. For example, the parameter controller 50 includes a simulation model 52 and an analytical solution model 54.

[0085] The simulation model 52 is a calculator modeled by deep learning (PINN: also known as Physics Informed Neural Networks) that satisfies physical rules, for example. The simulation model 52 can calculate the wave component of sound at high speed without solving the wave equation in all spaces.

[0086] The analytical solution model 54 is a calculator that analytically calculates parameters according to the physical rules between solvers. For example, according to known techniques, when the early reflection sound changes, the influence of the data after the change on the late reverberation sound can be analytically calculated. In the case where the producer 200 makes some changes, the analytical solution model 54 derives the second parameter applied after the change by analytically calculating what influence the change has.

[0087] For example, the parameter controller 50 may generate the second parameter having physical consistency by selectively using the simulation model 52 or the analytical solution model 54 according to the changed contents of the producer 200 .

[0088] As described above, after acquiring the input condition 10, the sound generating device 100 transmits the input condition 10 to the parameter controller 50 and selects an appropriate solver. In addition, the sound generating device 100 determines the parameters to be input to each selected solver based on the input condition 10. The sound generating device 100 generates a sound signal as shown in the graph 60 based on the information output from each solver. Thereafter, in the case where the producer 200 changes the parameters to be input to the solver, etc., the sound generating device 100 feeds back to the parameter controller 50 so that an unnatural sound is not generated by the change, thereby automatically adjusting the parameters to be input to another solver. As a result, the sound generating device 100 can generate a sound signal consistent with the sound intended by the producer 200.

[0089] (1-2. Configuration of Sound Generating Device According to Embodiment)

[0090] Next, see Figure 3The configuration of the sound generating device 100 according to the present embodiment will be described. Figure 3 : is a diagram showing a configuration example of the sound generating device 100 of the present embodiment.

[0091] like Figure 3 As shown, the sound generating device 100 includes a communication unit 110, a storage unit 120, a control unit 130, and an output unit 140. Note that the sound generating device 100 may include an input unit (for example, a pointing device such as a touch panel, a keyboard, or a mouse, a voice input microphone, or an image input camera (line of sight, gesture input)) that performs various operation inputs from a producer 200 or the like who operates the sound generating device 100, etc.

[0092] The communication unit 110 is implemented by, for example, a NIC (Network Interface Card), a network interface controller, etc. The communication unit 110 is connected to the network N in a wired or wireless manner, and transmits and receives information to and from an external device or the like via the network N. The network N is implemented by, for example, a wireless communication standard or system such as Bluetooth (registered trademark), the Internet, Wi-Fi (registered trademark), ultra-wideband (UWB), or low-power wide area network (LPWA).

[0093] The storage unit 120 is implemented by, for example, a semiconductor storage element such as a random access memory (RAM) or a flash memory or a storage device such as a hard disk or an optical disk. The storage unit 120 stores various types of data such as voice data output from a sound source, shape data of a virtual space or an object, preset settings of a sound absorption coefficient, and types and preset settings of a solver.

[0094] The control unit 130 is implemented by executing a program (e.g., a sound generation control program according to the present disclosure) stored inside the sound generating device 100 using a random access memory (RAM) or the like as a work area, for example, by a central processing unit (CPU), a micro processing unit (MPU), etc. In addition, the control unit 130 is a controller and can be implemented by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0095] like Figure 3 As shown, the control unit 130 includes an acquisition unit 131, a display control unit 132, an input unit 133, a generation unit 134, and an output control unit 135, and implements or executes the functions and actions of the information processing described below. Figure 1 The processing corresponding to the parameter controller 50 shown in the above etc. It should be noted that the internal configuration of the control unit 130 is not limited to that in Figure 3 The configuration shown in , and may be another configuration as long as information processing described later is performed.

[0096] The acquisition unit 131 acquires various types of data for use by the processing unit in the subsequent stages. For example, the acquisition unit 131 acquires preset data including a virtual space or a sound source as a processing object, settings of a solver for generating sound, etc. In addition, the acquisition unit 131 may appropriately acquire various types of information required by the processing unit in the subsequent stages, such as a library storing the sound absorption coefficient of each material, and a late reverberation sound preset.

[0097] The display control unit 132 performs control to display various types of information about the sound simulator provided by the sound generating device 100 on a display or the like. For example, the display control unit 132 displays Figure 4 and the virtual space shown in the subsequent figures, Figure 6 and the user interfaces shown in subsequent figures, etc.

[0098] Furthermore, in the event that a parameter or environmental data (the shape of an object, etc.) is changed by processing of the input unit 133 or the generation unit 134 in a subsequent stage, the display control unit 132 performs control to change the display on the user interface based on the change.

[0099] The input unit 133 receives input of environment data representing each condition set in a virtual space where sound sources and sound receiving points are arranged. For example, the input unit 133 receives input of environment data from the producer 200 via a user interface.

[0100] After generating the sound signal, the input unit 133 may receive an input of a change in each solver that generated the characteristic of the relevant sound signal. As an example, the input unit 133 receives a change in the setting of the characteristic related to the early reflection sound and the late reverberation sound in the sound signal. For example, the input unit 133 inputs the setting of the early reflection sound and the late reverberation sound desired by the producer 200 according to the operation of the producer 200 performed via the user interface.

[0101] Specifically, the input unit 133 inputs parameters indicating characteristics related to early reflection sounds and late reverberation sounds on the user interface based on data input by the producer 200 using an input device (touch panel, keyboard, pointing device such as mouse, microphone, camera, etc.).

[0102] The generation unit 134 performs each process related to the generation of the sound signal. For example, the generation unit 134 selects a plurality of solvers for calculating the sound characteristics at the sound receiving point according to the environment data input by the input unit 133. In addition, the generation unit 134 determines a first parameter to be input to each of the plurality of selected solvers.

[0103] As described above, each of the plurality of solvers is associated with calculation of characteristics of each of direct sound, early reflected sound, diffracted sound, transmitted sound, and late reverberation sound of the sound at the sound reception point.

[0104] Note that the generation unit 134 selects whether to use a solver for generating the first sound signal or the second sound signal from among a plurality of solvers according to the environmental data. For example, the generation unit 134 selects from the environmental data whether the space to be processed is closed, whether there are geometric obstacles between the sound source and the sound receiving point (for example, whether the sound receiving point can be identified from the sound source), or whether the sound source and the sound receiving point are used as solvers for generating sound signals based on the transmission loss between the sound source and the sound receiving point. This will be referred to later. Fig. 9 etc. describe the details of this processing.

[0105] In addition, the generation unit 134 receives a change request for the first sound signal generated based on the first parameter. In addition, the generation unit 134 automatically adjusts the environment data or the first parameter in response to the change request, and generates a second sound signal using the adjusted environment data or the second parameter that is the adjusted parameter and is re-input to each solver.

[0106] For example, in the case where any one of the first parameters input to the solver corresponding to any one of the direct sound, early reflected sound, diffracted sound, transmitted sound, and late reverberation sound is changed, the generation unit 134 changes the parameter input to the other solver to a second parameter according to the physical rule between the changed solver and the other solver. Here, the physical rule may be based on an analytical solution of a theoretical formula, or may be based on a wave sound simulator that predicts the sound field in a virtual space. The analytical solution according to the theoretical formula is, for example, a relationship between the solvers obtained based on physical calculation analysis. For example, a calculator (simulator) modeled by deep learning that satisfies the physical rules is used to obtain a solution by a wave sound simulator. Reference will be made later. Figure 4 and subsequent figures describe the details of these processes.

[0107] The output control unit 135 performs control to output the sound signal generated by the generation unit 134. For example, the output control unit 135 performs control to output the first sound signal generated by the generation unit 134 or the second sound signal corresponding to the sound after the producer 200 changes the parameters from the speaker 160, an external device, or the like.

[0108] The output unit 140 outputs various types of information. Figure 3As shown, the output unit 140 includes a display 150 and a speaker 160. Under the control of the display control unit 132, the display 150 displays the virtual space as the processing object, or displays a user interface for the producer 200 to input operations. The speaker 160 outputs generated sounds and the like under the control of the output control unit 135.

[0109] (1-3. Overview of Sound Simulation According to Embodiment)

[0110] Will refer to Figure 4 and subsequent figures describe sound simulation according to this embodiment. Figure 4 is a diagram for describing an overview of sound simulation according to an embodiment.

[0111] Figure 4 The virtual space 70 shown in FIG. 2 is an example of environmental data set by the producer 200 for the sound simulation in the game content. The virtual space 70 includes a tunnel-like space 71 with a large volume and a tunnel-like space 72 with a small volume. The space 71 and the space 72 are set as underground spaces, for example. In addition, the virtual space 70 includes a ground space 73 set as a free sound field from the space 71 via the space 72.

[0112] In the case of performing a sound simulation regarding the sound of water discharged from a sewage path existing in the space 71 in the virtual space 70, the producer 200 sets the sound source 75 at a position indicating the sewage path in the space 71. In addition, the producer 200 regards the game character as a sound receiving point, sets a sound receiving point 76 in the underground space, sets a sound receiving point 77 in the ground space 73, and so on. This is a case where the game character moves from the underground space to the ground space 73.

[0113] When receiving input of environmental data, the sound generating device 100 uses a method such as ray tracing based on geometric acoustics to determine the sound reaching the sound receiving point 76 and the sound receiving point 77. In addition, the sound generating device 100 determines whether the transmitted sound reaches the sound receiving point 76 or the sound receiving point 77 based on the material of the object set in the virtual space 70, etc.

[0114] like Figure 4 As shown, at the sound receiving point 76, the sound is mainly configured by the direct sound or the early reflected sound from the sound source 75. In addition, at the sound receiving point 77, the sound is mainly configured by the transmitted sound transmitted through the space 72, the diffracted sound diffracted from the space 71 or the space 72, a combination thereof, etc. In this way, the sound generating device 100 determines the elements constituting the sound at the sound receiving point based on the environmental data, and selects a resolver for generating each element. In addition, the sound generating device 100 determines the parameters to be input to the resolver.

[0115] Note that the sound generating apparatus 100 can select a solver specified by the producer 200. As described above, in the virtual space 70, not only a sound configuration according to physical rules but also an unrealistic sound configuration intended as a work by the producer 200 is preferred. Therefore, in the case where there is a request for change of the solver from the producer 200, the sound generating apparatus 100 changes the solver as requested.

[0116] When determining a solver or an early parameter (first parameter) to be input to the solver based on the environmental data, the sound generating device 100 generates a sound signal observed at the sound receiving point 76 or the sound receiving point 77 .

[0117] Will refer to Figure 5 The generation process of the above-mentioned sound signal is described. Figure 5 : is a flowchart showing the flow of speech generation processing in sound simulation.

[0118] First, the sound generating device 100 acquires voice data of a sound emitted from the sound source 75 (step S101). Furthermore, the sound generating device 100 acquires space data of a space where the sound source 75, the sound receiving point 76, and the sound receiving point 77 exist (step S102).

[0119] Subsequently, the sound generating apparatus 100 calculates a path from the sound source 75 to the sound receiving point 76 or the sound receiving point 77 (step S103). Then, the sound generating apparatus 100 generates a direct sound component using a direct sound solver (step S104).

[0120] Similarly, the sound generating device 100 generates an early reflection sound component using an early reflection sound solver (step S105). Similarly, the sound generating device 100 generates a diffraction sound component using a diffraction sound solver (step S106). At this time, in the case where it is determined in step S103 that the sound is transmitted to the sound receiving point, the sound generating device 100 may generate a transmitted sound component using a transmission sound solver. In addition, the sound generating device 100 generates a late reverberation sound component using a late reverberation sound solver (step S107).

[0121] Finally, the sound generating device 100 synthesizes the corresponding voice signal and outputs the synthesized voice signal (step S108).

[0122] Note that in the case where the sound signal is first generated based on the environmental data, the generation processing according to steps S104 to S107 does not depend on the order between the corresponding steps and can be interchanged. In addition, when considering the case where diffraction occurs after reflection in the propagation path, the signal generated by pre-calculating the reflection attenuation at the boundary from the sound source can be used as an input for calculating the diffracted sound.

[0123] In addition, despite the Figure 4 Although not considered in the example of FIG. 1 , the sound generating device 100 may add a process using a transmission simulation of sound on a boundary surface or a portable simulation technology that simulates the characteristics when a small sound passes through a space such as a window or door of a building. As a result, the sound generating device 100 can achieve a sound performance that is closer to a real space phenomenon.

[0124] In addition, in the case where there are multiple sound sources, the sound generating device 100 may also process all of the above-mentioned processing in parallel for the sounds output from the multiple sound sources. Alternatively, the sound generating device 100 may delay the time until the output until the entire processing is completed and perform the processing sequentially, and further synthesize and output each synthesized signal whose time series of the arriving sounds at the sound receiving point are aligned.

[0125] Next, we will refer to Figure 6 , and subsequent figures describe a user interface used when the producer 200 performs the above-mentioned sound simulation. Figure 6 FIG. 1 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0126] like Figure 6 As shown, user interface 300 includes virtual space display 302 , object selection window 304 , object material parameter display 306 , material selection window 308 , sound source selection window 310 , sound source directivity table 312 , sound receiving point setting display 314 , and setting information 316 .

[0127] The virtual space display 302 displays a virtual space constructed based on the environment data input by the creator 200. Figure 6 In the example, Figure 4 The virtual space 70 shown in FIG. 7 is displayed in the virtual space display 302 .

[0128] The object selection window 304 is a window for the producer 200 to select an object to be added to the virtual space. When the producer 200 adds the object to the virtual space, the sound generating device 100 displays the object on the virtual space based on the shape data preset for the object. In addition, the producer 200 can select the material of the object using the material selection window 308. For each material, the sound characteristics as shown in the object material parameter display 306 are preset for each frequency. When the producer 200 installs the object in the virtual space, the sound generating device 100 acquires such shape and sound characteristics as environmental data, and calculates the sound on the virtual space based on the acquired data.

[0129] In addition, the producer 200 selects a sound source that emits sound in the virtual space from the sound source selection window 310. For example, each icon shown in the sound source selection window 310 is associated with a sound source file that will become each voice data. The producer 200 selects an icon (or a voice file itself) corresponding to the sound to be emitted, thereby determining the sound source on the virtual space. At this time, the producer 200 can select the directionality of the sound emitted from the sound source from a preset using the sound source directivity table 312, or can customize and set the directivity by itself.

[0130] In addition, the producer 200 sets a sound receiving point, which is a coordinate for observing the sound in the virtual space, on the sound receiving point setting display 314. In addition, the producer 200 selects an environment for reproducing the content indicated by the virtual space in the setting information 316. For example, the producer 200 selects whether the environment in which the content is reproduced is a speaker or a headphone, or selects the type of the reproduction console of the content. In addition, the producer 200 can select a scene setting in a situation to be simulated. For example, the producer 200 selects whether such a scene is a "tense scene" or another scene from the preset scene settings.

[0131] In addition, the user interface 300 includes a direct / early reflection sound solver setting 320, a late reverberation sound solver setting 322, a diffraction sound solver setting 324, and a transmission sound solver setting 326. The sound generating apparatus 100 determines whether to use each solver, which type of solver to use, the value of a parameter to be input to the solver, etc. based on the input environment data. Note that the producer 200 can also determine to use a solver desired by himself / herself.

[0132] Furthermore, the user interface 300 includes an execution button 328 as an operation panel. The producer 200 requests execution of each process described later and rejoins through the operation panel.

[0133] Next, we will refer to Figure 7 The use of user interface 300 is described. Figure 7 An example is shown in which the producer 200 inputs environmental data (earlier settings) into the sound simulation. Figure 7 FIG. 2 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0134] When the producer 200 uses the sound of water discharged from a sewage path existing in the virtual space as a sound source, the producer 200 selects an icon corresponding to water from the sound source selection window 310 and drags the icon to a desired coordinate of the virtual space display 302. The sound generating device 100 displays the voice file 340 set to the icon based on the selected icon (in Figure 7In the example of , "water.wav"). In addition, the sound generating apparatus 100 displays the sound source display 344 at the position where the sound source is dragged.

[0135] In addition, the producer 200 operates the sound source directivity table 312 to determine the directivity of the sound source, etc. Figure 7 In the example of , since the sound of water discharged from the sewage path is assumed, the producer 200 selects a line sound source. The sound generating device 100 displays the directionality display 342 (in Figure 7 In the example, “line sound source (corresponding to Φ500)”).

[0136] from Figure 7 Continue, refer to Figure 8 Describes an operation example of sound simulation. Figure 8 FIG. 3 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0137] Figure 8 The state in which the sound generating apparatus 100 selects each solver in response to the input of the environmental data is shown. It is assumed that the producer 200 operates the solver selection display 358 in advance to determine whether to select a solver by himself or automatically.

[0138] In the case of automatically selecting a solver, the sound generating device 100 presents the selected solver to the producer 200. Figure 8 In the example of FIG. 1 , the sound generating device 100 displays the solver selected as the direct / early reflection sound solver on the display 350 (in FIG. 1 ). Figure 8 Similarly, the sound generating device 100 displays the solver selected as the late reverberation sound solver on the display 352, displays the solver selected as the diffraction sound solver on the display 354, and displays the solver selected as the transmission sound solver on the display 356.

[0139] When the producer 200 presses the execution button 328 in this state, the sound generating device 100 generates a sound signal using the selected solver.

[0140] Here, the flow of the solver selection process will be described. Fig. 9 This is a flowchart (1) showing the flow of solver selection processing in sound simulation.

[0141] The sound generating device 100 inputs environmental data including sound source information, space information, object information, sound receiving point information, other settings, etc. according to the instruction of the producer 200 (step S201). Note that the environmental data may include the position and directivity of the sound source, the geometry and material of the entire virtual space, the user environment (such as the platform using the content and the reproduction environment resources), the scene settings, etc.

[0142] Then, the sound generating apparatus 100 recognizes the space shape and the object shape, and performs 3D data processing (step S202 ). The sound generating apparatus 100 determines whether the space as the processing object is closed or open based on the 3D data (step S203 ).

[0143] Note that the sound generating device 100 can selectively determine the space as a processing object. For example, the sound generating device 100 sets the spatial scene as a processing object to an area covering a specified scale multiple (e.g., 1.5 times) of the range in which the sound source and the sound receiving point (game character, etc.) can move. Alternatively, the sound generating device 100 can use ray tracing for image generation to set the area that the sound reaches within a specific time (e.g., 1 second) as a processing object by emitting a light beam from the sound receiving point.

[0144] In addition, whether a space is closed can also be determined by the sound generating device 100 based on an optional standard. For example, the sound generating device 100 may determine that a space in which the ratio of the wall or ceiling in the calculation target space exceeds a specific value (for example, 70%) is a closed space. Figure 4 In the virtual space 70 shown, space 71 and space 72 are determined as closed spaces, and ground space 73 is determined as a non-closed space.

[0145] If the space is closed, the sound generating apparatus 100 determines that there is a late reverberant sound and determines to use a late reverberant sound solver (step S204 ).

[0146] It should be noted that as a late reverberation sound solver, for example, a solver by analytical solution (Analytical Solver) and a solver by geometric method (Numerical Solver) such as ray tracing or sound line method can be used. The sound generating apparatus 100 can automatically determine which late reverberation sound solver to use based on the execution environment (computing resources, etc.) of the content.

[0147] Furthermore, based on the sound absorption coefficient, volume and surface area of ​​the object, the reverberation time important in the late reverberation sound solver is obtained from the following formula (1), which is called Sabine's reverberation formula in the field of architectural sound.

[0148] T=0.161V / αS…(⑴

[0149] In the above formula (1), V represents the target space volume, S represents the total surface area, and α represents the average sound absorption coefficient. It should be noted that the reverberation time is not limited to Savin's equation and can be obtained by another known formula (Eyring's formula, Knudsen's formula, etc.).

[0150] Furthermore, based on the volume of the target space, echo density (diffusion) as another factor of the late reverberant sound can be analytically obtained from the following formula (2).

[0151]

[0152] Similarly, the modal density (Density), which is another factor of the late reverberant sound, can be analytically obtained from the following formula (3) based on the volume of the target space.

[0153]

[0154] Next, the sound generating device 100 determines a transmission path from the sound source to the sound receiving point (step S205 ).

[0155] For example, when it is determined that the transmission amount of sound is greater than a predetermined reference value due to the use of a material with high transmittance for a space or object between a sound source and a sound receiving point, the sound generating device 100 determines that there is transmitted sound and determines to use a transmitted sound solver (step S206).

[0156] On the other hand, when it is determined that the transmission amount of sound is less than a predetermined reference value due to the use of a material with low transmittance for a space or object between the sound source and the sound receiving point, etc., the sound generating device 100 determines that there is no transmitted sound (or the transmitted sound can be ignored) and determines not to use the transmitted sound solver (step S207).

[0157] For example, in order to determine the presence or absence of transmitted sound, the following formula (4) is used. Note that in the following formula (4), m represents the area density of obstacles between the sound source and the sound receiving point.

[0158] TL=2010g(fm)-42[dB]…(4)

[0159] For example, in Figure 4In the virtual space 70 shown, when the game character is located in the space 72, the game character (sound receiving point) cannot be seen from the sound source 75. In addition, it is assumed that the material set in the spaces 71 and 72 is concrete and the transmission loss TL is 40 dB or more. At this time, the sound generating device 100 determines that there is no transmitted sound. In addition, the sound generating device 100 can also calculate the sound based on the assumption that the sound from the sound source 75 to the space 72 is propagated as a diffracted sound described later. On the other hand, since the transmission loss TL between the space 72 and the ground space 73 is assumed to be 40 dB or less, a transmitted sound is generated.

[0160] Next, the sound generating device 100 determines a diffraction path from the sound source to the sound receiving point (step S208 ).

[0161] For example, in the case where the space between the sound source and the sound receiving point or the shape of the object generates diffraction sound, the sound generating apparatus 100 determines that the diffraction sound exists and determines to use the diffraction sound solver (step S209 ).

[0162] Will see Fig.10 Describes the diffraction sound solver. Fig.10 is a diagram for explaining the application of the diffraction sound solver. Fig.10 As shown in FIG. 1 , the sound generating device 100 can obtain a table of frequency characteristic curves for each angle α when the sound generated from the sound source diffracts an obstacle by using a diffraction sound solver. The sound generating device 100 can generate a sound signal by combining signals related to the generated sound signal, taking into account the influence of the diffracted sound. Note that the presence or absence of the diffracted sound can be determined by the sound generating device 100 by, for example, Fig.10 The geometric method described in is automatically determined, or it can be determined by the intention of the producer 200 if the producer 200 believes that the diffracted sound is important.

[0163] return Fig. 9 In step S208, when the shape of the space or object between the sound source and the sound receiving point does not generate diffraction sound (or can be ignored), the sound generating device 100 determines that the diffraction sound does not exist and determines not to use the diffraction sound solver (step S210).

[0164] Thereafter, the sound generating device 100 determines the size of the space (ie, the processing target area of ​​the sound simulation) (step S211 ).

[0165] In the case where the space is larger than a predetermined reference value, the sound generating apparatus 100 limits the area for calculating the early reflection sound (step S212). In addition, the sound generating apparatus 100 determines the complexity of the shape of the space and the shape of the object (step S213). In the case where the complexity is higher than a predetermined reference value, the sound generating apparatus 100 determines parameters to reduce the order calculated by using an early reflection sound solver applied to a limited area and based on geometry (step S214). On the other hand, if the complexity is lower than a predetermined reference value (in a simple case), the sound generating apparatus 100 determines that the early reflection sound solver is based on geometry and has parameters of a low or medium order (step S215).

[0166] The reason for determining the space size as described above is that when the space size increases, the time it takes for the sound to reach the sound receiving point is separated from the direct sound, which does not contribute to the sense of localization or loudness expected in the early reflected sound due to the sound size. Therefore, the sound generating device 100 determines the space size as "large" or "small" based on the case where the early reflected sound is separated from the direct sound by a predetermined time or longer (e.g., 80ms), and changes the calculation method according to the space size. For example, in the case where the sound generating device 100 determines that the space size is "large", the space used to perform calculations related to the early reflected sound is limited to the area where the early reflected sound falls within 80ms. Thereafter, the complexity of the space and the object is calculated in step S213, but in the case of a complex shape, the calculation load increases, and therefore, the sound generating device 100 determines the reflection order as a smaller parameter. Note that in the case where the complexity of the space is low, the sound generating device 100 sets the order up to the early reflected sound area (e.g., within 80ms) in the above (2). Basically, the sound generating device 100 also generates parameters corresponding to the above determination in each branch (steps S213, S217, S237, S241, etc.) described later.

[0167] In the case where it is determined in step S211 that the space is smaller than the predetermined reference value, the sound generating device 100 does not limit the area for calculating the early reflection sound (step S216). As in step S213, the sound generating device 100 determines the complexity of the shape of the space and the shape of the object (step S217). If the complexity is higher than the predetermined reference value, the sound generating device 100 determines an early reflection sound solver based on geometry and having a smaller order of parameters (step S218). In addition, in the case where the complexity is lower than the predetermined reference value (simple case), the sound generating device 100 determines an early reflection sound solver based on geometry and having a smaller order (step S219).

[0168] In addition, reference will be made to Fig.11 The flow in the case where the processing target is an open space in step S203 is described. Fig.11 Flowchart (2) shows the flow of solver selection processing in sound simulation.

[0169] In the case where the processing target is an open space, the sound generating apparatus 100 determines not to use the late reverberation sound solver and the transmission sound solver (steps S230 and S231 ).

[0170] Furthermore, the sound generating device 100 determines the diffraction path from the sound source to the sound receiving point in the same manner as in step S208 (step S232 ).

[0171] For example, in the case where the space between the sound source and the sound receiving point or the shape of the object generates diffraction sound, the sound generating apparatus 100 determines that the diffraction sound exists and determines to use the diffraction sound solver (step S233 ).

[0172] On the other hand, when the shape of the space or object between the sound source and the sound receiving point does not generate diffraction sound (or can be ignored), the sound generating device 100 determines that the diffraction sound does not exist and determines not to use the diffraction sound solver (step S234).

[0173] Thereafter, the sound generating device 100 determines the size of the space in the same manner as in step S211 (step S235 ).

[0174] In the case where the space is larger than a predetermined reference value, the sound generating device 100 limits the area for calculating the early reflection sound (step S236). In addition, the sound generating device 100 determines the complexity of the shape of the space and the shape of the object (step S237). In the case where the complexity is higher than a predetermined reference value, the sound generating device 100 determines that the early reflection sound solver will be applied to a limited area, based on geometry, and used to reduce the order to be calculated (step S238). On the other hand, if the complexity is lower than a predetermined reference value (in a simple case), the sound generating device 100 determines that the early reflection sound solver is based on geometry and has a low or medium order (step S239).

[0175] In addition, in the case where it is determined in step S235 that the space is smaller than the predetermined reference value, the sound generating device 100 does not limit the calculation area of ​​the early reflection sound (step S240). Similar to step S237, the sound generating device 100 determines the complexity of the shape of the space and the shape of the object (step S241). In the case where the complexity is higher than the predetermined reference value, the sound generating device 100 determines that the early reflection sound solver is based on geometry and has a low order or a medium order for calculation (step S242). In addition, in the case where the complexity is lower than the predetermined reference value (in a simple case), the sound generating device 100 determines that the early reflection sound solver is based on geometry and has a medium order or a high order for calculation (step S243).

[0176] Next, we will refer to Fig.12 A display example of the user interface 300 when a solver or a parameter is determined is described. Fig.12 FIG. 4 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0177] Fig.12 The display 360 in FIG. 1 shows an example of a display in the case where the producer 200 sets the sound source and the sound receiving point. As shown in the display 360, the sound generating device 100 performs line of sight determination from the sound source to the sound receiving point, and determines the sound source and the sound receiving point according to the sound source and the sound receiving point. Fig. 9 and Fig.11 The process shown in determines the solver and the parameters of the solver to be applied in the scene.

[0178] Next, we will refer to Fig.13 A display example of the user interface 300 when a solver or a parameter is determined is described. Fig.13 FIG. 5 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0179] exist Fig.12 After the display of , the sound generating device 100 displays the determined solver and parameters on the user interface 300. Fig.13 In the example of FIG. 1 , it is assumed that the producer 200 operates the pull-down of the solver selection display 358 in advance to set the parameter generation mode of the solver in the sound simulation. When the producer 200 presses the execution button 328 after setting the parameter generation mode, the sound generating apparatus 100 generates (calculates) the parameters of each solver.

[0180] For example, the sound generating device 100 displays Fig.13 364, late reflected sound parameter 366, diffracted sound parameter 368, and transmitted sound parameter 370 described in . Note that the sound generating device 100 may synthesize a sound signal based on the generated parameters and output the synthesized sound.

[0181] Here, we will refer to Fig.14 This section shows the solver settings and parameters. Fig.14 is a diagram for explaining the parameters of each solver.

[0182] like Fig.14 As shown, the settings of the early reflection sound solver may include the reflection sound level, reflection order, end time, etc. In this case, the values ​​input to the reflection sound level, reflection order, end time, etc. are parameters. Fig.14 Displays such as “A01” shown in conceptually indicate parameters.

[0183] In addition, as a setting item of the diffraction sound solver, there may be a diffraction sound level. In addition, as a setting item of the transmission sound solver, there may be a transmission sound level. In addition, the setting items of the late reverberation sound solver may include a reverberation level, a delay time of the late reverberation, a decay time, a ratio of a decay time of a high frequency to a decay time of a low frequency, a modal density, an echo density, and the like.

[0184] Next, we will refer to Fig.15 and subsequent figures describe processing in the case where the producer 200 requests a change in generated parameters and the like. Fig.15 FIG. 6 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0185] Fig.13 The parameters shown in are parameters generated by the function of the parameter controller 50, and are calculated based on an analytical solution or a numerical simulation solution based on input conditions, and therefore follow physical rules. That is, the parameters at the current moment are the state of obtaining the correlation of the parameters between the solvers. Therefore, when the producer 200 only manually adjusts some parameters, the physical correlation collapses. There is no problem with this relationship that does not follow physical rules as long as the relationship is intended to be presented as an expression. However, if the relationship is not intended to be so, and the amount of change exceeds the human discrimination limit, there is a possibility that the relationship adversely affects the user's auditory spatial cognition. However, manually adjusting all other relevant parameters of the producer 200 requires complex calculations, which is a very demanding task. Therefore, the sound generating device 100 automatically corrects the parameters used in other solvers so that the parameters can be related to the changed parameters through the function of the parameter controller 50 that conforms to physical rules.

[0186] exist Fig.15 In the example of , the producer 200 selects "Parameter adjustment (local)" from the pull-down of the display 380 when requesting to change a parameter. In addition, the producer 200 selects the parameter requested to be changed and inputs a desired numerical value. Fig.15 In the example of , as shown in the parameter change display 384, the producer 200 changes the "reflected sound level" and "reflection order" among the parameters of the direct / early reflected sound solver. That is, Fig.15 The example shows that the producer 200 has changed Fig.14 Here, the “local” parameter adjustment is used to adjust the parameters associated with one solver controlled by the parameter controller 50 .

[0187] Will refer to Fig.16 Describe the process flow in the event that this change is made. Fig.16 is a flowchart (1) showing an example of the flow of a change process according to an embodiment.

[0188] First, the sound generating device 100 acquires speech data generated based on parameters before the change (step S301 ). Then, the producer 200 changes some parameters related to the early reflection sound (step S302 ).

[0189] The sound generating apparatus 100 determines whether the early reflection sound level has changed in the early reflection sound solver (step S303 ). When the early reflection sound level has not changed (step S303 ; No), the immediately subsequent processing is skipped.

[0190] When the early reflection sound level has changed (step S303; Yes), the sound generating device 100 performs level calculation so that the levels in other solvers are also changed (steps S304, S305, S306, and S307). For example, the sound generating device 100 reflects the same increase or decrease in the level as the early reflection sound level in the parameters of each solver in each solver.

[0191] Thereafter, the sound generating apparatus 100 determines whether the early reflection order has changed in the early reflection sound solver (step S308). When the early reflection order has not changed (step S308; No), the immediately subsequent processing is skipped.

[0192] When the early reflection order changes (step S308; yes), the sound generating apparatus 100 changes the parameters of each solver according to the physical rule based on the change in the order value. As an example, since the start time of the late reverberation sound changes when the early reflection order changes, the sound generating apparatus 100 changes the start time parameter of the late reverberation sound in the late reverberation sound solver (corresponding to Fig.14 Parameter A07 shown in (step S309).

[0193] Similarly, the sound generating apparatus 100 corrects the decay time (corresponding to Fig.14 The sound generating device 100 also corrects the echo density of the late reverberation sound (corresponding to the echo density of the early reflection that has terminated (decreased in sequence) in a manner that is adapted to the echo density of the early reflection that has terminated (decreased in sequence). Fig.14 Parameter A10 shown in (step S311).

[0194] Then, the sound generating apparatus 100 generates a signal in each resolver whose parameter has been changed (step S312). Subsequently, the sound generating apparatus 100 outputs a sound signal generated by combining the signals generated by the corresponding resolvers (step S313).

[0195] Fig.17 Shown in the Fig.162 is a display example of the user interface 300 in the case where the process shown in FIG. 1 changes parameters. Fig.17 FIG. 7 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0196] like Fig.17 As shown in , in the case where the producer 200 increases the early reflection sound level by 3dB and changes the early reflection order from the second order to the first order in the parameters of the early reflection sound solver, the parameters of the late reverberation sound solver and the diffraction sound solver change.

[0197] For example, the late reverberation sound parameter 390 after the change indicates that the late reverberation sound level is increased by 3 dB, the delay time of the late reverberation sound is shortened by 5 ms, etc. In addition, the changed diffraction sound parameter 392 indicates that the diffraction sound level is increased by 3 dB, the setting of the low pass filter (LPF) of the diffraction sound can be changed, etc.

[0198] Note that the producer 200 watches the sound after the parameter change, and in the case where the producer does not like the sound after the parameter change, the producer can return to the process by pressing the return button on the operation panel. Note that the sound generating device 100 can switchably output a first sound signal based on a first parameter and a second sound signal based on a second parameter according to the operator's operation on the user interface 300. As a result, the producer 200 can advance the sound design while easily switching the sound before and after the change.

[0199] also, Fig.16 and Fig.17 An example in which the producer 200 changes the parameters of the early reflection is described, but the producer 200 may change desired parameters such as the late reverberation sound and the diffraction sound.

[0200] Fig.18 The following shows the processing flow when the producer 200 changes the parameters of the late reverberation sound. Fig.18 is a flowchart (2) showing an example of the flow of a change process according to an embodiment.

[0201] First, the sound generating device 100 acquires speech data generated according to parameters before the change (step S401). Thereafter, the producer 200 performs some parameter changes on the late reverberation sound (step S402).

[0202] The sound generating apparatus 100 determines whether the late reverberation sound level changes in the late reverberation sound solver (step S403). When the late reverberation sound level does not change (step S403; No), the immediately subsequent processing is skipped.

[0203] When the late reverberation sound level has been changed (step S403; Yes), the sound generating apparatus 100 performs level calculation so as to change the level in other solvers as well (steps S404, S405, S406, and S407). For example, the sound generating apparatus 100 reflects the same increase or decrease in level as the late reverberation sound level in the parameters of each solver in each solver.

[0204] Thereafter, the sound generating device 100 determines the delay time of the late reverberation (corresponding to Fig.14 Parameter A07 shown in FIG. 1 has been changed in the late reverberation sound solver (step S408). In addition, when the delay time of the late reverberation is not changed (step S408; No), the immediately subsequent processing is skipped.

[0205] In a case where the delay time of the late reverberation has been changed (step S408 ; Yes), the sound generating apparatus 100 adjusts the order of the early reflection sound so that the early reflection sound and the late reverberation sound do not excessively overlap (step S409 ).

[0206] In addition, the sound generating device 100 determines the echo density (corresponding to Fig.14 Whether the parameter A10 shown in FIG. 10 is changed in the late reverberation sound solver (step S410). When the echo density of the late reverberation sound is not changed (step S410; No), the immediately subsequent processing is skipped.

[0207] Since the echo density changes according to the complexity of the object or area to be processed, when the echo density of the reverberant sound at night changes (step S410; yes), the sound generating device 100 performs adjustments such as increasing the target area and the surface that contributes to reflection in a pseudo-range manner (step S411). Note that although Fig.18 Although not shown, even in the case where other parameters are changed, similar to step S409 and step S411, the sound generating apparatus 100 adjusts the parameters between the solver and the environment data by making changes according to the physical rules.

[0208] Then, the sound generating apparatus 100 generates a signal in each resolver whose parameter is changed (step S412). Subsequently, the sound generating apparatus 100 outputs a sound signal generated by combining the signals generated by the respective resolvers (step S413).

[0209] Next, we will refer to Fig.19 The process flow in the case where the producer 200 changes the parameters of the diffracted sound is described. Fig.19 is a flowchart (3) showing an example of the flow of a change process according to an embodiment.

[0210] First, the sound generating device 100 acquires speech data generated according to the parameters before the change (step S501). Thereafter, some parameter changes regarding the diffracted sound are performed by the producer 200 (step S502).

[0211] For example, the sound generating apparatus 100 determines whether the diffraction sound level has changed in the diffraction sound solver (step S503). When the diffraction sound level has not changed (step S503; No), the immediately subsequent processing is skipped.

[0212] When the diffracted sound level has changed (step S503; Yes), the sound generating apparatus 100 performs level calculation so as to change the level in other solvers as well (steps S504, S505, S506, and S507). For example, the sound generating apparatus 100 reflects, in each solver, an increase or decrease in the level corresponding to the diffracted sound level in the parameters of each solver.

[0213] Thereafter, the sound generating device 100 determines whether the setting of the low-pass filter for diffracted sound in the diffracted sound solver has been changed (step S508). For example, the sound generating device 100 determines whether the frequency, order, etc. set in the low-pass filter have been changed. In addition, when the setting of the low-pass filter for diffracted sound is not changed (step S508; No), the immediately subsequent processing is skipped.

[0214] When the setting of the low-pass filter for diffracted sound has been changed (step S508; Yes), the sound generating device 100 adjusts the ratio of the decay time of high frequencies to the decay time of low frequencies in the late reverberation sound (HF ratio) according to the physical rule ( Fig.14 In this case, the sound generating device 100 may recalculate the frequency dependency of the decay time according to the change in the ratio. Fig.19 Although not shown, even in the case where other parameters are changed, similar to step S509, the sound generating apparatus 100 adjusts the parameters between the solver and the environment data by performing the change according to the physical rules.

[0215] Then, the sound generating apparatus 100 generates a signal in each resolver whose parameter is changed (step S510). Next, the sound generating apparatus 100 outputs a sound signal generated by synthesizing the signals generated by the corresponding resolvers (step S511).

[0216] Note that, although not shown, even in the case where the parameters of the transmission sound solver are changed, the sound generating device 100 can be similar to Fig.18 and Fig.19 The processing shown in rescales the parameters.

[0217] (1-4. Adjustment of environmental data (global data))

[0218] In the above description, an example has been described in which when the parameter (local data) given to a certain solver changes, the sound generating apparatus 100 automatically adjusts the parameter of another solver. Here, in the case where the parameter given to a certain solver changes, the sound generating apparatus 100 can change the originally given environmental data so that the changed sound is consistent in the case where the parameter given to the certain solver changes.

[0219] Will see Fig. 20 Such processing is described. Fig. 20 FIG. 8 is a diagram for explaining a user interface of sound simulation according to an example.

[0220] exist Fig. 20 In the example shown, the creator 200 selects "Parameter Adjustment (Global)" from the pull-down on the display 400. This change mode represents a change to the environment data itself that affects the entire virtual space (global), rather than an adjustment between solvers.

[0221] For example, the producer 200 changes the parameter of the early reflection sound to a desired value, as illustrated in the early reflection sound parameter 402 after the change. Then, the producer 200 presses the execution button 328.

[0222] In this case, the sound generating apparatus 100 derives the original environment data so as to realize the sound generated based on the changed early reflection sound parameter 402. For example, the sound generating apparatus 100 sets the object material parameter 404 in which the reflectivity and the like of the object material setting are changed, thereby realizing the sound generated based on the changed early reflection sound parameter 402 according to the physical rules and not unnatural.

[0223] Note that, in the case where the reflectivity of a material or the like changes, the parameters of the corresponding solver also change with the change, and therefore, the sound generating apparatus 100 recalculates the parameters. Fig.21 Describe this. Fig.21 FIG. 9 is a diagram for explaining a user interface of sound simulation according to an embodiment.

[0224] like Fig.21 As shown, when changing the object material parameter 404, the producer 200 selects "parameter generation" from the pull-down of the display 410. In this change mode, when the producer 200 presses the execution button 328, the sound generating device 100 generates the parameters of the corresponding solver.

[0225] Specifically, the sound generating device 100 generates a parameter 412 after the change recalculated based on the object material parameter 404 after the change, and displays the parameter on the user interface 300. If necessary, the producer 200 can further change the value of the parameter 412 after the change. In addition, in "Parameter Adjustment (Global)", another solver parameter using the target environment data can be retroactively recalculated, or it can be reflected from the case of using the same environment data after the next adjustment.

[0226] As described above, in the case where a specific parameter is changed, the sound generating device 100 can perform the following processing: automatically generate another parameter in consideration of the influence on another parameter, or automatically generate environmental data (parameters) required for generating an impulse response according to a predetermined impulse response by inversion calculation using a learning model as described later. In addition, the sound generating device 100 can more accurately cope with wave propagation phenomena such as diffraction, for example, it can change the transmittance, reflectivity, etc. of an object according to the sound intended by the producer 200.

[0227] For example, by using Fig. 22 The machine learning-based wave sound simulation shown in implements this processing. Fig. 22 is a diagram for explaining an example of a parameter control process according to an embodiment.

[0228] Fig. 22 Model 420 is shown as an example of a calculator. Model 420 is an example of a calculator modeled by learning by deep learning that satisfies physical rules (PINN: also known as Physics Informed Neural Networks). By using model 420, the sound generating device 100 can calculate the wave component of the sound at high speed without solving the wave equation in all spaces.

[0229] Note that although the details will be described later, Figure 1 In the case of the sound or combined output desired by the producer 200 (in other words, information derived from the output of the model 420) as shown in the graph 60 in FIG. 1 , the sound generating device 100 can inversely calculate the sound source position and boundary conditions as input information to the simulator itself using the inversion calculator of the model 420. That is, the sound generating device 100 corrects the input information using the inversion calculator and reproduces the parameters of the various generators 30 again through the parameter controller 50, so that the parameters can be updated to parameters that are more suitable for physical rules.

[0230] exist Fig. 22In the example shown, it is assumed that the sound generating device 100 pre-generates the model 420 based on predetermined learning data. For example, the sound generating device 100 performs a learning process related to the model 420 based on the conditions given from the producer 200. The model 420 is a predetermined artificial intelligence and is implemented as a deep neural network (DNN) or an optional machine learning algorithm. For example, the model 420 includes a DNN 424 for implementing the above-mentioned PINN.

[0231] The model 420 is a system generated by a learning process so as to input various types of data (data sets) as training data and output a transfer function 426 as a transfer function at a sound source and a sound receiving point. For example, the transfer function 426 is a Green's function for converting and calculating an impulse response and a sound pressure 430 at a sound receiving point using a predetermined function. That is, the model 420 is an artificial intelligence model generated to receive data of different parameters including environmental data as input and output a transfer function 426 as output. The sound generating device 100 uses, for example, actual measurement values ​​simulated using an FDTD method or the like in a learning environment (in this example, a predetermined virtual space as a processing object) as training data, and performs a learning process of the model 420 so as to output parameters of the transfer function 426 by a method of minimizing an error between a value output from the model 420 and training data for presenting a value to be output. The transfer function 426 defines the shape of a function curve of an impulse response based on a predetermined input. That is, in the model 420, for example, the outputs of n nodes arranged in the layer immediately before the node G as the Green's function can be supplemented with the curve as n sampling points in the time axis direction of the impulse response to generate a Green's function curve. At this time, the sound generating device 100 can perform learning by minimizing the error between the bending shape of the training model and each sampling point.

[0232] The input information 422 to the model 420 is a data set including sound source information, sound receiving point information, environmental data about the structure constituting the space, and data such as boundary conditions (such as the sound impedance of the structure). In the input information 422, the input data of the model 420 includes, for example, coordinate data of the structure constituting the virtual space, coordinate data of the sound receiving point (corresponding to Fig. 22 The "r" shown in Fig. 22 ”), and boundary conditions such as the acoustic impedance of structures or objects constituting the virtual space (corresponding to Figure 1 In addition, since the output of the transfer function 426 formed in the output layer of the model 420 generated by the learning process obtains the impulse response of the sound receiving point at the "selectable time", the input data of the model 420 in the input information 422 may include a parameter representing time (corresponding to Figure 1t shown in Figure 1).

[0233] For example, the sound generating device 100 learns the model 420 so as to generate learning data by variously changing conditions such as data on the structure of the virtual space, the sound impedance, the position data of the sound source and the sound receiving point, and the size of the sound source given from the producer 200, and generates a predetermined impulse response based on the generated learning data. As a result, the sound generating device 100 generates the model 420 capable of outputting an impulse response of a transfer function 426 suitable for a predetermined virtual space through a learning process. Note that since the sound pressure can be derived from the transfer function 426 formed in the output layer of the model 420 using a predetermined function conversion, for example, the sound pressure at the sound receiving point can be obtained using the model 420. Thus, in the case where the sound receiving point can reproduce the sound emitted from the sound generating device 100 with high accuracy.

[0234] As described above, the data set used as the input of the model 420 includes the coordinate data and time data of the sound receiving point and the sound source as parameters. In addition, the model 420 is configured by, for example, a DNN, and is specifically learned so that the output of the transfer function 426 formed in the final output layer of the DNN forms an impulse response curve. In addition, the sound pressure can be calculated by function conversion based on the Green's function output of the model 420. Therefore, the sound generating device 100 is able to represent the sound emitted by a certain sound source as a spatial distribution. In addition, since the input of the model 420 includes time as a parameter, the propagation of the sound emitted from the sound source can also be represented in a time series. In an embodiment, the model 420 learns Fig. 22 It should be noted that it is basic to use "r", "r'", and "t" as parameters as Green's function, but the sound generating device 100 can be improved by setting the sound impedance z (which is Fig. 22 ) and other boundary conditions (e.g., the shape of the object, etc.) as parameters of the input data set to generate Green's functions with various parameters, that is, transfer function 426 as a learning model. Therefore, the following effect is achieved: the sound generating device 100 can automatically generate Green's functions including a large number of parameters that are difficult to design a known algorithm through learning. For example, the Green's function is a function that represents an impulse response. When the impulse response of the sound receiving point is known, the sound pressure at the sound receiving point (in other words, the loudness of the sound) can also be analytically obtained.

[0235] The model 420 generated as described above corresponds to Figure 2 An example simulation model 52 of a parameter controller 50 is shown.

[0236] For example, it is assumed that the producer 200 performs fluctuation editing or the like on the first sound signal generated first. In this case, the sound generating device 100 acquires a sound signal in which the impulse response, the sound pressure, and the like at the sound receiving point are changed. Since this corresponds to the change in the sound signal output of the model 420, the sound generating device 100 inputs the changed impulse response to the output side (output layer) of the model 420, and can calculate the sound source position and boundary conditions by using the inversion calculation of the model 420, wherein the inversion calculation of the model 420 performs calculations to obtain the output from the input side (input layer) of the model 420, and the sound source position and boundary conditions are the input information 422 of the model 420. Specifically, the sound generating device 100 can change the parameters of the sound characteristics such as the transmittance of the spatial structure, or automatically change and set the parameters defining the shape, position, and the like of the objects arranged in the space, so that the sound changed by the producer 200 (the sound signal having a predetermined impulse response, etc.) is output.

[0237] As described above, when the sound generating device 100 receives a request to change the first sound signal, information (impulse response, etc.) corresponding to the changed sound signal is input to a sound simulator modeled by learning of artificial intelligence, and an inversion calculation is performed from the output side of the learned model to the input side, thereby reflecting the data output from the input side of the learned model as the environmental data after adjustment. In this case, the artificial intelligence can be configured by a deep neural network, and the information (data) output based on the inversion calculation of the deep neural network is reflected in the environmental data, that is, the output data is set to the adjusted environmental data. As environmental data, the sound generating device 100, for example, reflects a change in at least one of the material, transmittance, reflectivity, position data, and shape of the structure of the space including the sound source and the sound receiving point or the object arranged in the space.

[0238] As described above, according to the sound generating device 100, the producer 200 can also obtain the setting (environmental data) of the compatible virtual space by directly editing the fluctuation of the desired sound without manually changing the parameters of the solver. That is, the producer 200 can easily construct the virtual space along the desired sound, and can easily configure the objects corresponding to the sound environment of the real space.

[0239] (1-5. Learning process through scene setting)

[0240] Furthermore, the sound generating apparatus 100 may automatically set the parameter by inputting predetermined scene data to a learned model and outputting the parameter to be set using a learning method in which a learning process of a parameter change tendency is performed by the user for each scene setting.

[0241] As described above, the producer 200 can select the scene setting when performing sound design in the virtual space. For example, the producer 200 can set the scene to be "tense scene", "normal scene", "battle scene", etc. for each scene to be edited.

[0242] In the sound design, depending on the scene, there may be a tendency for the producer 200 to change the sound. For example, the sound generating device 100 acquires each change result so that in the case of a "tense scene", the producer 200 tends to reduce the level or order of the early reflection sound, or in the case of a "battle scene", the producer 200 tends to shorten the reverberation time of the late reverberation.

[0243] Then, the sound generating device 100 uses a predetermined learning method (e.g., reinforcement learning, etc.) as described above to learn the trends of these changes. That is, the sound generating device 100 inputs the data of the specified scene into the learning model, outputs parameters such as sound characteristics as its output, and does not perform learning when the user does not change the parameters for output. On the other hand, in the case where the user has changed the parameters, the sound generating device 100 can generate a model that has learned the parameter change trend of the user by correcting the network using the method for learning. As a result, when the scene setting is performed in advance when the parameters (local data) of each solver are first generated, the sound generating device 100 can adjust the parameters based on the local data related to the scene setting. That is, the sound generating device 100 learns the artificial intelligence model by using the data of the correlation as the learning data, wherein the producer 200 manually adjusts the parameters (local data) of the solver according to the scene, so that the parameter controller 50 can automatically generate parameters (local data) close to the intention of the producer 200 according to the change of the scene using the learned model. Thereby, the producer 200 can perform good design work more smoothly. Note that the sound production apparatus 100 may learn the correlation between the scene and the parameters for each producer 200 , or may collectively learn the correlation between the scenes and the parameters for a plurality of producers 200 .

[0244] (1-6. Modifications of the Embodiments)

[0245] The information processing according to the above-described embodiment may be accompanied by various modifications. Hereinafter, modifications of the embodiment will be described.

[0246] Fig.23 FIG. 2 is a diagram showing an overview of a sound generation control method according to a modified example. Fig.23 In the example of FIG. 4 , it is assumed that the parameter controller 50 transmits different parameters using a plurality of transmission paths as shown by the transmission path 440 and the transmission path 442 .

[0247] Fig.24An example of execution based on such transmission is shown in FIG. Fig.24 is a diagram illustrating an example of output control processing according to a modification.

[0248] exist Fig.24 In the example of , when a sound source is selected from the sound source library 500, the sound generating apparatus 100 performs parameter generation based on the selected sound source through the parameter controller 50. In addition, the sound generating apparatus 100 also stores parameters reproduced based on the parameters changed by the producer 200.

[0249] Then, the sound generating device 100 transmits the early parameters 502 generated by the parameter controller 50 and the second parameters 504 adjusted by the producer 200 to the communication unit 506 in the different system.

[0250] The communication unit 506 transmits the early parameters 502 and the second parameters 504 to the producer adjustment data reflection unit 508 in a different system (external device, etc.). At this time, the behavior of the producer adjustment data reflection unit 508 is different between the developer side (including the producer 200) and the general user side that uses the content. For example, the developer side can readjust the parameters based on the received early parameters 502 and the second parameters 504. On the other hand, the general user side is set so that the parameters cannot be adjusted in the producer adjustment data reflection unit 508, and the data output from the generation unit 512 in the subsequent stage is uniquely determined. Note that the sound generating device 100 may have a mechanism for encrypting the early parameters 502 and the producer adjustment data reflection unit 508 and protecting the producer's adjustment technology, etc. from people other than limited developers.

[0251] The generator adjustment data reflecting unit 508 transmits the data 510 obtained via one of the above-mentioned sides to the generating unit 512. The generating unit 512 generates the sound data 514 based on the received data 510 (the parameters adjusted by the developer or the parameters transmitted to the general user). Then, the output control unit 516 receives the sound data 514 generated by the generating unit 512 and outputs the voice corresponding to the sound data 514.

[0252] As described above, the sound generating device 100 may perform control to transmit the first parameter or the first sound signal based on the first parameter and the adjusted second parameter or the second sound signal based on the second parameter, respectively.

[0253] As described above, the sound generating device 100 can select a method of using parameters according to the application of the developer (producer 200, etc.) or the general user by using a plurality of transmission paths. That is, when transmitting the generated parameters, the sound generating device 100 can change its behavior according to the application according to whether it is for the developer or for the general user. As a result, the developer can flexibly edit the parameters, such as returning the parameters to the standard parameters (early parameters) even after the parameter adjustment.

[0254] (2. Other Implementation Methods)

[0255] In addition to each of the above-described embodiments, the processing according to each of the above-described embodiments may be performed in various different modes.

[0256] In the processes described in the above embodiments, all or part of the processes described as being automatically performed may be performed manually, or all or part of the processes described as being manually performed may be performed automatically by a known method. In addition, unless otherwise specified, the processing procedures, specific names, and information including various types of data and parameters shown in the documents and drawings may be optionally changed. For example, the various types of information shown in each drawing are not limited to the information shown.

[0257] In addition, each component of each device shown in the drawings is conceptual in function and is not necessarily physically configured as shown in the drawings. That is, the specific form of distribution and integration of each device is not limited to the form shown, and all or part of it may be configured to be functionally or physically distributed and integrated in optional units according to various loads, usage conditions, etc.

[0258] Furthermore, the above-described embodiments and modifications can be appropriately combined within a range that does not conflict with the processing contents.

[0259] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may be provided.

[0260] (3. Effects of the Sound Generation Control Method According to the Present Disclosure)

[0261] As described above, in the sound generation control method according to the present disclosure, the computer (the sound generating device 100 in the example) receives input of environmental data indicating each condition set in a virtual space where a sound source and a sound receiving point are arranged. In addition, the computer not only selects a plurality of solvers for calculating the characteristics of the sound at the sound receiving point according to the environmental data, but also determines a first parameter to be input to each of the plurality of solvers. In addition, the computer receives a change request for a first sound signal generated based on the first parameter. In addition, the computer automatically adjusts the environmental data or the first parameter in response to the change request, and generates a second sound signal using the adjusted environmental data or the second parameter, wherein the second parameter is an adjusted parameter and is re-input to each solver.

[0262] As described above, the sound generation control method according to the present disclosure automatically selects a plurality of solvers for generating a sound signal, determines its parameters, and automatically adjusts other related parameters to follow predetermined physical rules when there is a request to change the parameters. As a result, the sound generation control method can regenerate the sound signal as a whole without being unnatural. That is, according to the sound generation control method of the present invention, the workload of the producer 200 can be reduced and a consistent sound signal can be generated.

[0263] In addition, in a sound generation control method, when a change request for a first sound signal is received by changing a first parameter input to a predetermined solver, a parameter input to another solver is changed to a second parameter according to the parameter changed in the predetermined solver, and a second sound signal is generated using the second parameter. In addition, each of the plurality of solvers is associated with calculation of characteristics of each of direct sound, early reflected sound, diffracted sound, transmitted sound, and late reverberation sound of the sound at a sound receiving point. In the sound generation control method, in the case where any one of the first parameters input to a solver corresponding to any one of the direct sound, early reflected sound, diffracted sound, transmitted sound, and late reverberation sound is changed, the generation unit changes the parameter input to another solver to a second parameter according to a physical rule between the changed solver and another solver. For example, the physical rule is based on an analytical solution by a theoretical formula or on a wave sound simulator that predicts a sound field in a virtual space.

[0264] As described above, according to the sound generation control method, by changing parameters in consideration of the influence between the parameters, parameter change with physical consistency can be achieved.

[0265] In addition, in the sound generation control method, whether to use as a solver for generating the first sound signal or the second sound signal is selected from among a plurality of solvers according to environmental data. For example, in the sound generation control method, whether a space to be processed is closed, whether there is a geometric obstacle between a sound source and a sound receiving point, or whether to use as a solver for generating the first sound signal or the second sound signal is selected from among a plurality of solvers based on environmental data.

[0266] As described above, according to the sound generation control method, since an appropriate solver is selected based on environmental data input by a producer, the producer can determine a solver for generating an appropriate sound without manually performing settings or the like.

[0267] Furthermore, in the sound generation control method, the environmental data is data specifying a scene set in a virtual space, and the sound generation control method determines the first parameter based on the specified scene.

[0268] As described above, according to the sound generation control method, parameter changes according to the producer's intention can be realized by learning changes performed in a state where a scene is set.

[0269] In addition, in the case of receiving a change request for the first sound signal, the sound generation control method inputs information corresponding to the changed sound signal into a sound simulator modeled by artificial intelligence, and reflects the output information as adjusted environmental data. At this time, the artificial intelligence is a learned deep neural network, and the information (data) output based on the inversion calculation of the deep neural network can be set as the adjusted environmental data. The sound generation control method includes: reflecting the structure of the space including the sound source and the sound receiving point or the change of at least one of the material, transmittance, reflectivity, position data and shape of the object arranged in the space as the adjusted environmental data.

[0270] As described above, according to the sound generation control method, by using the inverse calculation of the simulator generated by machine learning, it is possible to obtain environmental data (the shape of the structure of the space, etc.) for realizing the sound environment in the virtual space. Therefore, the producer can arrange objects without specific awareness and select the materials of the objects consistent with the real sound environment.

[0271] Furthermore, the sound generation control method performs control to transmit the first parameter or the first sound signal based on the first parameter and the adjusted second parameter or the second sound signal based on the second parameter, respectively.

[0272] As described above, according to the sound generation control method, for example, the parameters after adjustment can be flexibly utilized by transmitting the generated parameters to the producer and the general user respectively.

[0273] In addition, the sound generation control method includes: according to the operation of the operator (the producer 200 in the example) on the user interface, the first sound signal based on the first parameter and the second sound signal based on the second parameter are switchably output.

[0274] As described above, according to the sound generation control method, since the producer can easily confirm the sound before and after the change, the sound design work can be smoothly performed.

[0275] (4. Hardware Configuration)

[0276] For example, by having Fig.25 The computer 1000 of the configuration shown in embodies an information device such as the sound generating device 100 according to each of the above-described embodiments. Hereinafter, the sound generating device 100 of this example will be described as an example. Fig.25 1 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the sound generating device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

[0277] The CPU 1100 operates based on the program stored in the ROM 1300 or the HDD 1400, and controls each unit. For example, the CPU 1100 develops the program stored in the ROM 1300 or the HDD 1400 in the RAM 1200, and executes processing corresponding to the various programs.

[0278] The ROM 1300 stores a boot program such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is activated, a program depending on the hardware of the computer 1000 , and the like.

[0279] The HDD 1400 is a computer-readable recording medium that non-transiently records a program executed by the CPU 1100 , data used by the program, and the like. Specifically, the HDD 1400 is a recording medium that records an information processing program according to the present disclosure as an example of program data 1450 .

[0280] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from another device via the communication interface 1500 or transmits data generated by the CPU 1100 to another device.

[0281] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a touch panel, a keyboard, a mouse, a microphone, or a camera via the input / output interface 1600. In addition, the CPU 1100 transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. In addition, the input / output interface 1600 can be used as a medium interface for reading a program recorded in a predetermined recording medium (medium), etc. Examples of the medium include optical recording media such as digital versatile disks (DVDs) or phase-change rewritable disks (PDs), magneto-optical recording media such as magneto-optical disks (MOs), magnetic tape media, magnetic recording media, and semiconductor memories.

[0282] For example, in the case where the computer 1000 functions as the sound generating device 100 of the present example, the CPU 1100 of the computer 1000 implements the functions of the control unit 130 and the like by executing the information processing program loaded on the RAM 1200. In addition, the HDD 1400 stores the sound generation control program and data according to the present disclosure in the storage unit 120. Note that the CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program data, but as another example, these programs may be acquired from another device via the external network 1550.

[0283] It should be noted that the present technology may also have the following configurations.

[0284] (1) A sound generation control method, comprising:

[0285] Make the computer execute

[0286] receiving an input of environmental data indicating each condition set in a virtual space in which a sound source and a sound receiving point are arranged;

[0287] selecting a plurality of solvers for calculating characteristics of sound at a sound receiving point based on the environmental data, and determining a first parameter to be input to each of the plurality of solvers;

[0288] receiving a request to change a first sound signal generated based on a first parameter; and

[0289] The environment data or the first parameter is adjusted in response to the change request, and a second sound signal is generated using the adjusted environment data or the second parameter, which is the adjusted parameter and is re-input to each solver.

[0290] (2) A sound generation control method according to (1), wherein, when a request for changing a first sound signal is received by changing a first parameter input to a predetermined solver, a parameter input to another solver is changed to a second parameter according to the parameter changed in the predetermined solver, and a second sound signal is generated using the second parameter.

[0291] (3) The sound generation control method according to (2), wherein each of the plurality of resolvers is associated with calculation of characteristics of each of direct sound, early reflected sound, diffracted sound, transmitted sound, and late reverberation sound of the sound at the sound receiving point.

[0292] (4) A sound generation control method according to (3), wherein, when any one of first parameters input to a solver corresponding to any one of direct sound, early reflected sound, diffracted sound, transmitted sound, and late reverberation sound is changed, a generation unit changes the parameter input to another solver to a second parameter according to a physical rule between the changed solver and the other solver.

[0293] (5) The sound generation control method according to (4), wherein the physical rule is based on an analytical solution through a theoretical formula.

[0294] (6) The sound generation control method according to (4), wherein the physical rule is based on a wave sound simulator that predicts a sound field of a virtual space.

[0295] (7) The sound generation control method according to any one of (1) to (6), wherein whether to use as a resolver that generates the first sound signal or the second sound signal is selected among the plurality of resolvers according to environmental data.

[0296] (8) A sound generation control method according to (7), wherein it is selected from environmental data whether a space to be processed is closed, whether there are geometric obstacles between a sound source and a sound receiving point, or whether the sound source and the sound receiving point are used as a solver for generating a first sound signal or a second sound signal among multiple solvers based on the transmission loss between the sound source and the sound receiving point.

[0297] (9) The sound generation control method according to any one of (1) to (8), wherein the environmental data is data specifying a scene set in a virtual space, and

[0298] The sound generation control method determines a first parameter based on the specified scenario.

[0299] (10) A sound generation control method according to any one of (1) to (9), wherein, upon receiving a request to change a first sound signal, the sound generation control method inputs information corresponding to the changed sound signal into a sound simulator modeled by artificial intelligence, and reflects the output information as adjusted environmental data.

[0300] (11) A sound generation control method according to (10), wherein the artificial intelligence is a deep neural network, and the environmental data is adjusted based on an inverse calculation of the deep neural network.

[0301] (12) The sound generation control method according to (10) or (11) also includes: reflecting a change in at least one of the material, transmittance, reflectivity, position data and shape of the structure of the space including the sound source and the sound receiving point or the objects arranged in the space as adjusted environmental data.

[0302] (13) The sound generation control method according to any one of (1) to (12) further includes: performing control to transmit the first parameter or the first sound signal based on the first parameter and transmit the adjusted second parameter or the second sound signal based on the second parameter, respectively.

[0303] (14) The sound generation control method according to any one of (1) to (13) further includes: according to an operation of an operator on a user interface, switchably outputting a first sound signal based on the first parameter and a second sound signal based on the second parameter.

[0304] (15) A sound generating device comprising:

[0305] an input unit that receives an input of environmental data indicating each condition set in a virtual space in which a sound source and a sound receiving point are arranged; and

[0306] a generating unit that selects a plurality of solvers for calculating characteristics of sound at a sound receiving point according to the environmental data, and determines a first parameter to be input to each of the plurality of solvers to generate a first sound signal based on the determined first parameter,

[0307] When a change request for the first sound signal is received, the generation unit adjusts the environmental data or the first parameter in response to the change request, and generates a second sound signal using the adjusted environmental data or the second parameter, wherein the second parameter is the adjusted parameter and is re-input into each solver.

[0308] (16) A sound generation control program for causing a computer to:

[0309] an input unit that receives an input of environmental data indicating each condition set in a virtual space in which a sound source and a sound receiving point are arranged; and

[0310] a generating unit that selects a plurality of solvers for calculating characteristics of sound at a sound receiving point according to the environmental data, and determines a first parameter to be input to each of the plurality of solvers to generate a first sound signal based on the determined first parameter,

[0311] When a change request for the first sound signal is received, the generation unit adjusts the environmental data or the first parameter in response to the change request, and generates a second sound signal using the adjusted environmental data or the second parameter, wherein the second parameter is the adjusted parameter and is re-input into each solver.

[0312] Reference Numbers List

[0313] 100 Sound generating device

[0314] 110 Communication unit

[0315] 120 storage units

[0316] 130 Control unit

[0317] 131 Get Unit

[0318] 132 Display control unit

[0319] 133 Input unit

[0320] 134 Generation Unit

[0321] 135 Output control unit

[0322] 140 Output Unit

[0323] 150 Display

[0324] 160 Speakers

[0325] 200 Makers.

Claims

1. A sound generation control method, comprising: Make the computer execute receiving an input of environmental data indicating each condition set in a virtual space in which a sound source and a sound receiving point are arranged; selecting a plurality of solvers for calculating characteristics of the sound at the sound receiving point according to the environmental data, and determining a first parameter to be input to each of the plurality of solvers; receiving a request to change a first sound signal generated based on one of said first parameters; and The environment data or the first parameter is adjusted in response to the change request, and a second sound signal is generated using the adjusted environment data or a second parameter, which is the adjusted parameter and is re-input to each resolver.

2. The sound generation control method according to claim 1, wherein: When a change request for the first sound signal is received by changing the first parameter input to a predetermined solver, a parameter input to another solver is changed to the second parameter according to the parameter changed in the predetermined solver, and the second sound signal is generated using the second parameter.

3. The sound generation control method according to claim 2, wherein: Each of the plurality of solvers is associated with calculation of characteristics of each of direct sound, early reflected sound, diffracted sound, transmitted sound, and late reverberant sound of the sound at the sound receiving point.

4. The sound generation control method according to claim 3, wherein: In the event that any one of the first parameters input to the solver corresponding to any one of the direct sound, the early reflected sound, the diffracted sound, the transmitted sound, and the late reverberation sound is changed, the parameter input to the other solver is changed to the second parameter according to a physical rule between the changed solver and the other solver.

5. The sound generation control method according to claim 4, wherein: The physical laws are based on analytical solutions through theoretical formulas.

6. The sound generation control method according to claim 4, wherein: The physical rules are based on a wave sound simulator that predicts the sound field of the virtual space.

7. The sound generation control method according to claim 1, wherein: Whether to use as a solver for generating the first sound signal or the second sound signal is selected among the plurality of solvers according to the environmental data.

8. The sound generation control method according to claim 7, wherein: From the environmental data, select whether the space to be processed is closed, whether there are geometric obstacles between the sound source and the sound receiving point, or based on the transmission loss between the sound source and the sound receiving point, select whether the sound source and the sound receiving point are used as solvers to generate the first sound signal or the second sound signal among the multiple solvers.

9. The sound generation control method according to claim 1, wherein: The environment data is data that specifies a scene set in the virtual space, and The sound generation control method determines the first parameter based on the specified scenario.

10. The sound generation control method according to claim 1, wherein: In the case of receiving a change request for the first sound signal, the sound generation control method inputs information corresponding to the changed sound signal to a sound simulator modeled by artificial intelligence, and reflects the output information as adjusted environmental data.

11. The sound generation control method according to claim 10, wherein: The artificial intelligence is a deep neural network, and the environmental data is adjusted based on an inverse calculation of the deep neural network.

12. The sound generation control method according to claim 10, further comprising: The reflection constitutes a change in at least one of a structure of a space including a sound source and a sound receiving point or a material, transmittance, reflectance, position data, and shape of an object arranged in the space as the adjusted environmental data.

13. The sound generation control method according to claim 1, further comprising: Control is performed to transmit the first parameter or the first sound signal based on the first parameter and to transmit the adjusted second parameter or the second sound signal based on the second parameter, respectively.

14. The sound generation control method according to claim 1, further comprising: According to an operation of the operator on the user interface, a first sound signal based on the first parameter and a second sound signal based on the second parameter can be output in a switchable manner.

15. A sound generating device, comprising: an input unit that receives an input of environmental data indicating each condition set in a virtual space in which a sound source and a sound receiving point are arranged; as well as a generating unit that selects a plurality of solvers for calculating the characteristics of the sound at the sound receiving point according to the environmental data, and determines a first parameter to be input to each of the plurality of solvers to generate a first sound signal based on the determined first parameter, When a change request for the first sound signal is received, the generation unit adjusts the environmental data or the first parameter in response to the change request, and generates a second sound signal using the adjusted environmental data or second parameter, wherein the second parameter is the adjusted parameter and is re-input into each solver.

16. A sound generation control program for causing a computer to: an input unit that receives an input of environmental data indicating each condition set in a virtual space in which a sound source and a sound receiving point are arranged; and a generating unit that selects a plurality of solvers for calculating the characteristics of the sound at the sound receiving point according to the environmental data, and determines a first parameter to be input to each of the plurality of solvers to generate a first sound signal based on the determined first parameter, in, When a change request for the first sound signal is received, the generation unit adjusts the environmental data or the first parameter in response to the change request, and generates a second sound signal using the adjusted environmental data or second parameter, wherein the second parameter is the adjusted parameter and is re-input to each resolver.

Citation Information

Patent Citations

  • Acoustical signal processor

    JP2000267675A