Acoustic signal control method, learning model generation method, and acoustic signal control program
Generating Green's function through deep learning and physically informed neural networks solves the problems of high calculation load and unnatural sound in virtual space, achieving high-precision and high-speed acoustic signal reproduction, providing a real sound experience.
Patent Information
- Application Number
- CN202380028136.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-25
- Filing Date
- 2023-03-13
- Publication Date
- 2024-11-01
AI Technical Summary
Existing acoustic processing methods are difficult to reproduce real physical phenomena and natural sounds in virtual spaces, especially parameter discontinuities at boundaries lead to high computational loads and unnatural sounds.
Deep learning method is adopted, and Green functions are generated using physically informed neural networks (PINNs), and the wave components of sound are calculated by pre-learning the transfer function to achieve high-speed and natural acoustic signal control in virtual space.
High-precision and high-speed acoustic signal reproduction in virtual space is realized, providing a real sound experience, and reducing calculation load, avoiding the problem of parameter discontinuity at the boundary.
Smart Images

Figure CN118891900A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an acoustic signal control method, a learning model generation method, and an acoustic signal control program. Background Art
[0002] With the development of virtual space technologies such as the metaverse and games, the sound reproduced in the virtual space also needs to be based on actual realism. Currently, embodiments of methods for such acoustic processing include, for example, ray tracing methods for geometric analysis from the sending point to the receiving point of the sound. However, in the ray tracing method, since the wave component of the sound is excluded, it is in principle impossible to achieve diffraction and shielding caused by the wave (taking into account the acoustic effects at the boundaries between spaces, such as doors that exist in the virtual space).
[0003] In contrast, according to acoustic processing using wave equations, phenomena caused by waves can be reproduced. However, compared with the ray tracing method, acoustic processing using wave equations has a significantly large computational load, and it is difficult to perform acoustic operations that follow the player's line of sight in the game. In this regard, a method is known in which an acoustic space is divided into a plurality of regions in order to calculate wave equations, and acoustic processing is performed between the divided boundaries to increase the operation speed (e.g., non-patent document 1 below). A deep learning method has also been proposed for deriving an expression for performing operations that satisfy physical rules using machine learning (e.g., non-patent document 2 below).
[0004] Reference List
[0005] Non-patent literature
[0006] Non-patent document 1: “Efficient and Accurate Sound Propagation Using AdaptiveRectangular Decomposition” N.Raghuvanshi et al., IEEE (2009).
[0007] Non-patent literature 2: "Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations" M. Raissi, P. Perdikaris, GE Karniadakis, Journal of Computational Physics 378 (2019). Summary of the invention
[0008] Technical issues
[0009] According to Non-Patent Document 1, by dividing the space and reducing the calculation load, an arithmetic speed that can be afforded for use in games and the like can be achieved.
[0010] However, in the technology disclosed in non-patent document 1, since changes with discontinuity may occur in parameters related to acoustic processing at the boundary between spaces, real physical phenomena cannot be reproduced, and unnatural sounds may be reproduced in terms of audibility. In the technology disclosed in non-patent document 1, diffraction in the presence of a structure on the boundary surface may not be reproduced naturally.
[0011] Therefore, the present disclosure proposes an acoustic signal control method, a learning model generation method, and an acoustic signal control program that can reproduce an acoustic space with a sense of reality and realize high-speed calculation processing.
[0012] Solution to the problem
[0013] According to an embodiment of the present disclosure, an acoustic signal control method includes: selecting a sound source object to be arranged in a three-dimensional virtual space, and arranging the selected sound source object at a predetermined position in the three-dimensional virtual space in which one or more three-dimensional objects are arranged; inputting position data and acoustic data related to the arranged sound source object, area data in the three-dimensional virtual space, structural data of the three-dimensional object, and position data of each object and sound receiving point at a predetermined time into a first artificial intelligence by a computer, and outputting an acoustic signal at the sound receiving point based on an impulse response at the sound receiving point generated by the output data. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 is a diagram showing an overview of information processing according to the first embodiment.
[0015] Figure 2 : is a view showing a configuration example of the sound generating device according to the first embodiment.
[0016] Figure 3 This is a view (1) for describing the learning process according to the first embodiment.
[0017] Figure 4 This is a view (2) for describing the learning process according to the first embodiment.
[0018] Figure 5 It is a diagram for describing the operation process according to the first embodiment.
[0019] Figure 6 is a flowchart showing the procedure of the learning process according to the first embodiment.
[0020] Figure 7 : is a flowchart showing the procedure of the arithmetic processing according to the first embodiment.
[0021] Figure 8 is a view showing another example of the arithmetic processing according to the first embodiment.
[0022] Fig. 9 is a diagram showing an overview of information processing according to the second embodiment.
[0023] Fig.10 is a flowchart showing an information processing procedure according to the second embodiment.
[0024] Fig.11 is a view (1) showing an embodiment of a user interface provided by a sound generating device.
[0025] Fig.12 1 is a diagram showing an example of operations in a user interface.
[0026] Fig.13 is a view (2) showing an example of operations in the user interface.
[0027] Fig.14 It is a view (3) showing an example of operation in the user interface.
[0028] Fig.15 It is a view (4) showing an example of operation in the user interface.
[0029] Fig.16 is a view (5) showing an example of operations in the user interface.
[0030] Fig.17 is a view (6) showing an operation example in the user interface.
[0031] Fig.18 1 is a diagram showing an example of a learning process according to the third embodiment.
[0032] Fig.19 2 is a diagram showing an example of a learning process according to the third embodiment.
[0033] Fig. 20 is a flowchart showing the procedure of arithmetic processing according to the third embodiment.
[0034] Fig.21 : is a hardware configuration diagram showing an embodiment of a computer that realizes the function of the sound generating device. DETAILED DESCRIPTION
[0035] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in each of the following embodiments, the same reference numerals are given to the same parts and repeated descriptions are omitted.
[0036] The present disclosure is described according to the order of the following items.
[0037] 1. First Implementation
[0038] 1-1. Overview of information processing according to the first embodiment
[0039] 1-2. Configuration of Sound Generating Device According to First Embodiment
[0040] 1-3. Procedure of information processing according to first embodiment
[0041] 1-4. Execution Example of Information Processing According to First Embodiment
[0042] 2. Second Implementation
[0043] 2-1. Overview of information processing according to the second embodiment
[0044] 2-2. Procedure of information processing according to the second embodiment
[0045] 3. Third Implementation
[0046] 4. Fourth Implementation
[0047] 5. Fifth Implementation
[0048] 6. Sixth Implementation
[0049] 7. Seventh Implementation Method
[0050] 8. Eighth Implementation
[0051] 9. Other Implementation Methods
[0052] 10. Effects of the Acoustic Signal Control Method and Learning Model Generation Method According to the Present Disclosure
[0053] 11. Hardware Configuration
[0054] (1. First Embodiment)
[0055] (1-1. Overview of Information Processing According to First Embodiment)
[0056] First, refer to Figure 1 An outline of information processing according to the first embodiment is described. Figure 1 is a diagram showing an overview of information processing according to the first embodiment.
[0057] pass Figure 1The sound generating device 100 shown in performs information processing according to the first embodiment (according to the acoustic signal control method of the present disclosure, etc.). The sound generating device 100 is an embodiment of the acoustic signal control device (information processing terminal) according to the present disclosure, and is an information processing terminal used by a producer 200 that generates content related to a virtual space such as games and metaverse festivals. For example, the sound generating device 100 is a personal computer (PC), a server device, a tablet terminal, etc.
[0058] The sound generating device 100 includes an output unit such as a display and a speaker, and outputs various types of information (or data) to the producer 200. For example, the sound generating device 100 displays a user interface of software related to sound generation on the display. The sound generating device 100 outputs the generated acoustic signal from the speaker according to the operation of the producer 200's instruction on the user interface.
[0059] In the first embodiment, the sound generating device 100 calculates what kind of sound the sound output from the sound source object as the sound emission point of the sound is reproduced at the sound receiving point in a virtual three-dimensional space of a game or the like (hereinafter referred to as the "virtual space"), and reproduces the calculated sound. That is, the sound generating device 100 performs acoustic simulation in the virtual space, and performs processing to make the sound emitted in the virtual space close to the sound of the real world or reproduce the sound desired by the producer 200.
[0060] Figure 1 The virtual space 10 on the game for which the producer 200 wants to design sound is shown. For example, the virtual space 10 is displayed on a display included in the sound generating device 100. The producer 200 sets the position (coordinates) of the sound source object (the sound emission point of the sound) in the virtual space 10, and sets the sound receiving point (the position in the virtual space 10 for listening to the sound). Figure 1 In the example, for example, in the virtual space 10, the producer 200 sets the sound source object as the sound generation point 30, and the position of the game character or the like as the sound receiving point 40. The sound receiving point 40 is a position where the sound output from the sound generation point 30 is observed, and is, for example, the position of the game character operated by the player (more specifically, the position coordinates corresponding to the head of the game character).
[0061] One or more three-dimensional objects (hereinafter referred to as "objects") are arranged in the virtual space 10. Figure 1 Although not shown in the figure, for example, furniture-shaped objects, human-shaped objects, etc. may be arranged in the virtual space 10. In the virtual space 10, walls, ceilings, etc. are also arranged as boundaries constituting the virtual space 10. In addition, in the following description, boundaries such as walls and ceilings may also be handled as one of the objects arranged in the virtual space 10.
[0062] In the actual space, due to various physical phenomena, a difference occurs between the sound observed near the sound source and the sound observed at the sound receiving point. Therefore, the sound generating device 100 virtually reproduces (simulates) the real physical phenomenon in the virtual space 10 according to the instruction of the producer 200, and generates an acoustic signal suitable for the space, so as to enhance the real feeling of the sound expression experienced by the game player etc. (hereinafter referred to as "player") using the content in the virtual space 10.
[0063] Here, an acoustic signal generated by the sound generating device 100 is described. The acoustic signal generated in the virtual space 10 is determined based on factors such as the indoor environment set in the virtual space 10 (parameters such as the volume of the space and the sound absorption coefficient of the wall), the distance between the sound generating point 30 and the sound receiving point 40, and the shape of the space (the presence or absence of obstacles). Embodiments of a method of reproducing such an acoustic signal include geometric simulation such as a sound ray method and wave acoustic simulation in which a wave phenomenon as a sound propagation state is reproduced using a wave equation.
[0064] Among them, in the ray tracing method, since the wave component of the sound is excluded, it is impossible to achieve diffraction and hiding caused by the wave in principle. On the contrary, according to the acoustic processing using the wave equation, the phenomenon caused by the wave can be reproduced. However, compared with the ray tracing method, the acoustic processing using the wave equation has a significantly large calculation load, and it is difficult to perform acoustic operations that follow the player's line of sight in the game. Even if a method of dividing the space to increase the calculation speed is adopted, because a step function change may occur in the parameters related to the acoustic processing at the boundary between the spaces, the real physical phenomenon cannot be reproduced, and unnatural sounds may be reproduced in terms of audibility. This method may cause unnatural reproduction even in diffraction, etc. in the case where there is a structure on the boundary surface. That is, in the virtual space 10, in order to reproduce the sound without making the producer 200 and the player feel strange, while performing high-speed processing, there is a problem of calculating high-speed wave sounds without strange feelings.
[0065] As an embodiment of this physical arithmetic method, a finite difference time domain method (FDTD method) is proposed. In the FDTD method, the sound field to be processed is discretized in space and time, the differential terms of the main equation (wave equation) are expanded into differential equations, and calculations are performed in sequence to obtain solutions. As a method, calculations are performed using a staggered grid that provides sound pressure and particle velocity in a half-integer coordinate system. In order to solve them, it is necessary to perform calculations by discretizing the entire space through coordinates, so from the perspective of real-time operation, very high computer capabilities are required. At this time, it is considered to reduce the amount of operation by reducing the frequency band, but since the frequency of the initial reflected sound depends largely on the sound source, there is a possibility that the reproducibility cannot be improved.
[0066] As a method for solving this problem, the sound generating device 100 performs an acoustic signal control method according to an embodiment. Specifically, the sound generating device 100 uses a method (also called a physically informed neural network (PINN)) that satisfies physical rules in machine learning and performs deep learning. As a result, the sound generating device 100 can calculate the wave component of the sound at high speed without solving the wave equation of the entire space. According to the acoustic signal control method of the present embodiment, since there is no restriction such as the division of the spatial size of the sound field, it is not necessary to divide the virtual space 10 into fine boundaries. In addition, since the acoustic signal control method according to the present embodiment solves the body expression applied to deep learning with the transfer function as the target, it is also possible to calculate the situation of the movement of the sound generating point 30 and the sound receiving point 40 and the influence of the movement of another object in the virtual space 10. That is, in the acoustic signal control method of the present embodiment, by pre-learning the transfer function containing the influence of movement, etc., the waveform sound can be output in real time during operation, so that a real sound experience can be provided to the player.
[0067] In the following, reference will be made to Figure 1 An outline of a process in which the sound generating device 100 performs the acoustic signal control method according to the embodiment is described.
[0068] First, the producer 200 sets the virtual space 10 and the sound generation point 30 and the sound receiving point 40 in the virtual space 10 in the user interface provided by the sound generating device 100. For example, the producer 200 selects any sound source object that sets the intensity, pitch, etc. of the sound on the user interface, and arranges it in the virtual space 10 as the sound generation point 30.
[0069] The sound generating device 100 performs a learning process of the model 60 included in the learning configuration 50 based on the conditions provided by the producer 200. The model 60 is a predetermined artificial intelligence and is implemented as a deep neural network (DNN) or any machine learning algorithm. For example, the model 60 is a DNN for implementing the above-mentioned PINN.
[0070] The model 60 is generated through a learning process so as to receive input of various data (data sets) as training data and output a Green's function 64, which is a transfer function between a sound source and a sound reception. Here, in the first embodiment, the Green's function 64 is used to convert and calculate the impulse response and sound pressure at the sound receiving point 40 using a predetermined function. That is, the model 60 is an artificial intelligence model that is generated to receive various data as input and output the Green's function 64 as output. Although learning will be described in detail later, the model 60 performs a learning process of the model 60 so as to minimize the error between the value output from the model 60 and the training data presenting the output value, using, for example, in a learning environment (in Figure 1 The parameters of the Green's function 64 are outputted using actual measured values simulated using the FDTD method or the like in the virtual space 10 in the embodiment of the present invention as training data. It should be noted that the Green's function 64 itself defines the shape of a function curve of an impulse response based on a predetermined input. That is, in the model 60, for example, the Green's function curve can be obtained by using a Green's function curve arranged immediately before the Green's function curve. Figure 1 The outputs of n nodes in the layer before the node G which is the Green's function in the time axis direction of the impulse response are used as n sampling points to supplement the curve. At this time, the sound generating device 100 can perform learning by minimizing the error between the curved shape of the training model and each sampling point.
[0071] The input information 62 to the model 60 is a data set used as training data of the model 60 (teacher data for input and output of the model 60). In the input information 62, the input data of the model 60 may include, for example, coordinate data of the structure forming the virtual space 10, the sound receiving point 40 (corresponding to Figure 1 ”r” shown in FIG. ), the sound generating point 30 (corresponding to Figure 1 , and boundary conditions such as the acoustic impedance of the structures and objects forming the virtual space 10 (corresponding to Figure 1 Since the output of the Green's function 64 formed in the output layer of the model 60 generated by the learning process obtains the impulse response of the sound receiving point 40 at "any time", the input data of the model 60 in the input information 62 also includes a parameter representing time (corresponding to Figure 1 t in the text).
[0072] For example, the sound generating device 100 generates learning data by variously changing conditions such as acoustic impedance, the positions of the sound generating point 30 and the sound receiving point 40, and the size of the sound source based on the structure of the virtual space 10 given from the producer 200, and performs learning of the model 60 based on the generated learning data. Therefore, the sound generating device 100 generates the model 60 capable of outputting the impulse response of the Green's function 64 suitable for the virtual space 10 through the learning process. It is to be noted that since the Green's function 64 formed in the output layer of the model 60 can obtain the impulse response and the sound pressure using a predetermined function conversion, for example, the sound pressure 66 at the sound receiving point 40 can be obtained using the trained model 60. Therefore, in the case where the sound transmitted from the sound generating point 30 is heard at the sound receiving point 40 with high accuracy, the sound generating device 100 can reproduce the sound.
[0073] Note that, as described above, the data set used as the input of the model 60 includes the coordinate data and time data of the sound receiving point and the sound generating point as parameters. The model 60 includes, for example, a DNN, and is specifically trained in such a manner that the output of the Green's function 64 formed in the final output layer of the DNN forms an impulse response curve. In addition, the sound pressure can be calculated by function conversion based on the Green's function output of the model 60. Therefore, the sound generating device 100 can indicate the sound emitted by a specific sound generating point 30 as a spatial distribution. Figure 1 The distribution 12 shown in FIG. 1 visually represents the sound pressure distribution obtained by the sound generating device 100. For example, in the distribution 12, the dark portion represents high sound pressure, and the light portion represents low sound pressure. Because the input of the model 60 includes time as a parameter, the sound generating device 100 can change the distribution 12 in a time series. That is, the sound generating device 100 can also represent the propagation of the sound emitted from the sound generation point 30 in a time series. In this embodiment, the model 60 learns Figure 1 It should be noted that the Green's function basically includes "r", "r'", and "t" as parameters, but since the sound generating device 100 can set the acoustic impedance z (in Figure 1 ) and other boundary conditions (e.g., the shape of the object, etc.) as input parameters, so it has the effect that Green's function, which includes many parameters that are usually difficult for algorithms to design, can be automatically generated through learning.
[0074] As described above, the sound generating device 100 according to the first embodiment selects a sound source object to be arranged on the virtual space 10 based on the operation of the producer 200, and arranges the sound source object at a predetermined position on the virtual space 10 as a sound generating point 30. Subsequently, the sound generating device 100 inputs the position data and acoustic data related to the sound generating point 30, the area data in the virtual space 10, the structural data of the object, and the position data of each object, sound receiving point, and sound generating point at a predetermined time (they are collectively referred to as coordinate data and boundary conditions) to the model 60. Note that the acoustic data is various setting data as sound, such as the intensity and pitch of the sound set in the sound source object. The area data in the virtual space 10 is, for example, the boundary conditions of the virtual space 10 to be analyzed, the coordinate data forming the boundary of the virtual space, etc. The structural data includes the acoustic impedance set in the object, the coordinate data of the point group forming the object, etc.
[0075] The sound generating device 100 generates an impulse response of the sound receiving point 40 as an output of the Green's function 64 forming a part of the model 60. The sound generating device 100 outputs an acoustic signal at the sound receiving point 40 based on the impulse response of the sound receiving point 40 generated from the data output from the model 60. That is, the sound generating device 100 calculates the acoustic signal at the sound receiving point 40 in the virtual space 10 according to the condition selected on the user interface by the producer 200, and reproduces the acoustic signal based on the arithmetic result from the speaker, etc. At this time, the sound generating device 100 outputs the acoustic signal based on the sound pressure at the sound receiving point calculated from the impulse response at the sound receiving point.
[0076] In this way, the sound generating device 100 can obtain a simulation of the acoustic signal in the virtual space 10 using the trained artificial intelligence (model 60), thereby reproducing the acoustic signal with high accuracy without performing huge computational processing. Although described in detail later, according to the sound generating device 100, the simulation is performed using the model 60 including boundary conditions (e.g., coordinate data and time) as a parameter, for example, by changing the structure in the space and immediately calculating the wave component according to the change, an acoustic simulation can be provided in real time according to the space design desired by the generator 200.
[0077] (1-2. Configuration of Sound Generating Device According to First Embodiment)
[0078] Next, refer to Figure 2 The configuration of the sound generating device 100 according to the first embodiment is described. Figure 2 : is a view showing a configuration example of the sound generating device 100 according to the first embodiment.
[0079] like Figure 2As shown, the sound generating device 100 includes a communication unit 110, a storage unit 120, a control unit 130, and an output unit 140. Note that the sound generating device 100 may include an input device (e.g., a touch panel, a keyboard, a pointing device such as a mouse, a voice input microphone, an image input camera (line of sight, gesture input)) for obtaining various operation inputs from a producer 200 or the like who operates the sound generating device 100.
[0080] The communication unit 110 is implemented by, for example, a network interface card (NIC), etc. The communication unit 110 is connected to a network N (cloud, Internet, local area network, near field communication (NFC), Bluetooth (registered trademark), etc.) in a wired or wireless manner, and transmits information (data) to other information devices, etc., and receives information (data) from other information devices, etc., via the network N.
[0081] The storage unit 120 is implemented by, for example, a semiconductor storage element such as a random access memory (RAM) and a flash memory, or a storage device such as a hard disk and an optical disk. The storage unit 120 stores various data, such as voice data output from the sound generation point 30, shape data of the object, and preset setting values of the sound absorption coefficient of each object. The storage unit 120 stores the "pre-learning model (for learning)" and "post-learning model (for execution)" of the model 60 generated in the learning process described later. In the learning process, the current "learning model for execution" can be copied in the storage area of the "model for learning", the model generated due to the learning process can be copied in the storage area of the "post-learning model (for execution), and the post-learning model can be used in the sound generating device 100.
[0082] The control unit 130 is implemented by executing a program (e.g., an acoustic signal control program according to the present disclosure) stored in the sound generating device 100 using a random access memory (RAM) or the like as a work area, for example, by a central processing unit (CPU), a microprocessing unit (MPU), etc. The control unit 130 is a controller and can be implemented by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) and a field programmable gate array (FPGA).
[0083] like Figure 2 As shown, the control unit 130 includes an input unit 131, a learning unit 132, an operation unit 133, a display control unit 134, and an output control unit 135, and implements or executes the functions and actions of the information processing described below. It should be noted that the internal configuration of the control unit 130 is not limited to Figure 2 The configuration shown in , and may be another configuration as long as information processing described later is performed.
[0084] The input unit 131 obtains input of settings and selections of various types of information (data) from the producer 200. For example, the input unit 131 obtains input of spatial data indicating an area of the virtual space 10, position coordinate data of a sound receiving point and a sound generating point, position coordinate data indicating the structure of an object using an input unit (e.g., a touch panel, a keyboard), conditions related to the virtual space 10, etc., and a pointing device such as a mouse, a voice input microphone, an image input camera (line of sight, gesture input), etc., which obtains various operation inputs on a screen provided to the producer 200 through a user interface.
[0085] For example, the input unit 131 obtains inputs of various settings such as structures such as doors and walls configured in the virtual space 10, boundary conditions of the walls and ceiling of the virtual space 10, parameters such as sound absorption coefficients of objects configured in the virtual space 10, and positions of the sound emission point 30 and the sound receiving point 40 from the producer 200. The input unit 131 may obtain inputs of these settings from the generator 200 at the learning stage of the model 60, or the generator 200 may obtain inputs of these settings at the acoustic design stage of the content to be generated after learning the model 60.
[0086] The input unit 131 acquires input of a reproduction request or the like of a sound reproduced at the sound receiving point 40 using the model 60 from the producer 200. For example, the input unit 131 acquires a selection input of a sound source object arranged in the virtual space 10 from the producer 200.
[0087] The input unit 131 acquires an input of a request for arrangement of an object from the producer 200. At this time, the input unit 131 may provide the producer 200 with materials to be set in the object, etc., via a user interface for selection. Fig.12 For example, the input unit 131 determines structural data (parameters set for each object) in an object based on input from a user specifying at least any one of material, transmittance, reflectance, or position data of the object.
[0088] It should be noted that the input unit 131 can acquire various data for processing by the processing unit in a subsequent stage from the storage unit 120 or the communication unit 110 as needed, even in the case where no explicit instruction is input by the producer 200 or the like. For example, the input unit 131 acquires spatial data of the virtual space 10 to be processed from the storage unit 120 or the communication unit 110. Specifically, the input unit 131 acquires spatial information indicating an area of the virtual space 10, coordinate position data indicating the configuration of an object, conditions regarding the virtual space 10, and the like from input from the user, the storage unit 120, or the communication unit 110.
[0089] The input unit 131 may acquire voice data as the source of the sound output from the sound generation point 30 as the sound source from the input from the generator 200, the storage unit 120, or the communication unit 110. The input unit 131 may access a database such as a library stored in the storage unit 120 (in which the sound absorption coefficient of each material, etc. is recorded), and may appropriately acquire various types of information required by the processing unit at a subsequent stage. For example, the database is stored in the storage unit 120, a server on the network N accessible via the communication unit 110, or a storage device accessible via a network device, an information device, etc. Note that the storage unit 120 may be formed using a so-called virtualization technology in which the control unit 130 is managed in a unified storage space including a storage device accessible on the network N. In this case, the control unit 130 is stored in the storage unit 120, and processing may be performed through the storage unit 120 via an operating system that performs processing through the operation of the control unit. Therefore, in the following description, unless explicitly described, the storage unit 120 also includes a storage device accessible on the network N.
[0090] The learning unit 132 performs learning of the model 60 for reproducing an acoustic signal in the virtual space 10. The learning of the model 60 corresponds to learning of a transfer function for obtaining an impulse response in an acoustic system. Specifically, the learning unit 132 generates artificial intelligence based on predetermined teacher data, the artificial intelligence having as inputs acoustic data related to a sound source object configured in the virtual space 10, area data in the virtual space, structural data of the object, and position data of each object and a sound receiving point at a predetermined time, and a transfer function indicating a relationship when a sound emitted by the sound source object is observed at the sound receiving point 40 as an output through learning.
[0091] As described above, the artificial intelligence according to this embodiment is a machine learning algorithm, and more specifically, a learning model (also referred to as PINN) that forms the Green's function 64 in the output layer of the DNN.
[0092] Note that the teacher data according to the first embodiment may be data acquired based on acoustic simulation in the virtual space 10. For example, the teacher data is simulated data obtained in an acoustic simulator inputting settings about the virtual space 10 by the FDTD method or the like. Note that the teacher data is not limited to this embodiment, and may be, for example, acoustic data simulated by various existing methods (impulse response in the first embodiment). The teacher data may be actual measurement data measured in a shape similar to that of the virtual space 10.
[0093] The learning unit 132 performs learning so as to minimize the error with reference to the error between the impulse response obtained by the Green's function 64 output from the model 60 and the teacher data. Figure 3and 4 Such a learning process is described.
[0094] Figure 3 is a view (1) for describing the learning process according to the first embodiment. Figure 3 As shown, the learning unit 132 inputs the input information 62 as learning data into the model 60 in the learning configuration 50, and outputs an unknown function (Green's function 64 in the first embodiment) based on the main equation in the output layer.
[0095] The learning unit 132 performs a known acoustic simulation based on the input information 62 in the simulation environment 71 and obtains actual measurement data 72 of the impulse response through the simulation. The learning unit 132 obtains a Green's function 74 based on the actual measurement data 72.
[0096] The learning unit 132 calculates the error as the difference between the output of the Green's function 64 and the output of the Green's function 74, and continues to learn the model 60 of the parameters of the output Green's function 64 until the error becomes sufficiently smaller than the threshold value ε which is optionally defined. It should be noted that the Green's function 64 itself defines the shape of the function curve of the impulse response by output based on the predetermined input of the model 60. That is, in the model 60, for example, the Green's function curve can be obtained by using the Green's function curve arranged immediately before Figure 1 The outputs of n nodes in the layer before the node G in which the Green's function is a Green's function are used as n sampling points in the time axis direction of the impulse response to supplement the curve to generate. At this time, the learning unit 132 can perform learning by minimizing the error between the curved shape of the training model and each sampling point. In this case, in the case where the above-mentioned Green's function 74 is based on n sampling points of the actual measurement data 72, the learning unit 132 can calculate the error by comparing each sampling point.
[0097] Note that in the first embodiment, not only the difference from the teacher data based on acoustic simulation but also other errors can be used as the error function. Thus, the accuracy of the model 60 can be improved. Figure 4 This process is described.
[0098] Figure 4 is a view (2) for describing the learning process according to the first embodiment. Figure 4 As shown, the learning unit 132 derives three types of values as parameters forming the error function.
[0099] First, the learning unit 132 obtains the error between the output of the Green's function 64 formed in the output layer of the model 60 (i.e., the impulse response according to the predetermined data input) and the master equation corresponding to the input of the model 60 (step S11). That is, the learning unit 132 calculates the feasibility of the master equation (difference of differential equations) in the PINN as the error MSEG , error MSE G is one of the parameters of the error function.
[0100] It should be noted that, when the sound receiving point position is represented by r, the sound source position is represented by r', time is represented by t, a certain differential operator is represented by L, a field quantity (e.g., potential) is represented by φ, and an output force or a wave source (e.g., a sound source) is represented by ζ, the transfer function in the sound field is represented by the following expression (1). In the following embodiment, an embodiment based on an orthogonal coordinate system is described, but a polar coordinate system may be used. In the case where a polar coordinate system is used for coordinate data, the following formulas may also be appropriately converted into formulas corresponding to polar coordinates.
[0101] L φ (r, t) = -ζ(r, t)…(1)
[0102] In this case, the Green's function G in the sound field is expressed by the following expression (2).
[0103] LG(r,r',t)=-δ((r,t)-(r',t))...(2)
[0104] In the above expression (2), considering reflection and sound absorption as boundary conditions, the Green function G satisfies any boundary condition among the Dirichlet boundary condition, the Neumann boundary condition, and the Robin boundary condition. For example, the Dirichlet boundary condition satisfies the condition of the following expression (3).
[0105] G(r,r',t)=0···(3)
[0106] The Neumann boundary condition satisfies the following expression (4): In the expression (4), n represents a normal vector pointing outward from the boundary surface.
[0107]
[0108] The Robin boundary condition satisfies the condition of the following expression (5). Note that A and B in expression (5) represent constants other than 0.
[0109]
[0110] When the Green's function G is expressed by the above-mentioned Expression (2), an expression showing the establishment of a main equation is expressed by the following Expression (6).
[0111] LG(r,r',t)=-δ((r,t)-(r',t))···(6)
[0112] According to the above expression (6), the error MSE, which is one of the parameters of the error function, is GIt is represented by the following expression (7).
[0113]
[0114] like Figure 4 As shown in, the learning unit 132 can use the difference between the output (i.e., impulse response) from the Green's function 64 (i.e., the output layer of the model 60) and the output from the Green's function obtained when the data obtained by inverting the sound generation point and the sound reception point in the input information is input to the model 60 as an error MSE of one parameter of the error function. r (Step S12) This is based on the idea that even if the positions of the sound generation point and the sound reception point are reversed, under the premise of establishing a transfer function, the difference between the original value and the reversed value is used as an evaluation function. Error MSE r It is represented by, for example, the following expression (8).
[0115]
[0116] like Figure 4 As shown, the learning unit 132 calculates the error MSE0 which is the difference between the output of the Green's function 64 (ie, the output layer of the model 60) and the teacher data based on the following expression (9), for example (step S13).
[0117]
[0118] The learning unit 132 derives the error function MSE including these three parameters (step S14). That is, the error function MSE is represented by, for example, the following expression (10).
[0119] MSE = MSE G +MSE r +MSE0···(10)
[0120] The learning unit 132 determines whether the solution of the expression (10) becomes a value less than the threshold value ε (step S15). In the case where the solution of the expression (10) is less than the threshold value ε (step S15; yes), the learning unit 132 determines that the learning is sufficiently performed and completes the learning (step S16). On the contrary, in the case where the solution of the expression (10) is greater than the threshold value ε (step S15; no), the learning unit 132 determines that the learning is insufficient and continues the learning (step S17).
[0121] As described above, the learning unit 132 performs learning of the model 60 so that the error MSE G The sum of minimized, error MSE Gis the error between the output of the Green's function 64 (i.e., the output layer of the model 60) and the output of the main equation corresponding to the input, and the error MSE between the output of the Green's function 64 (i.e., the output layer of the model 60) and the output of the Green's function 64 when the position data of the sound generation point and the sound reception point are reversed r , and the error MSE0 between the teacher data and the output of the Green's function 64. Therefore, the learning unit 132 can perform learning of the model 60 capable of reproducing the space with high accuracy. Note that the learning method of the model 60 is not limited to the above-mentioned embodiment. For example, the learning unit 132 can perform learning of the model 60 using an adversary learning method called a generative adversary network (GAN) or the like.
[0122] Note that, as described above, since boundary conditions are included in the input when learning the model 60, this has nothing to do with the error function represented by expression (10). Therefore, the error function is not affected by the boundary conditions and does not change according to the boundary conditions.
[0123] return Figure 2 , the description continues. The operation unit 133 performs various operations using the model 60. The model 60 is a model trained by the learning process, and is stored in the storage area of the "learned model (for execution)" of the storage unit 120, for example, as described above, and can be accessed by the operation unit 133. For example, the operation unit 133 inputs various types of data including the sound receiving point 40 acquired by the input unit 131 to the model 60, and derives the Green's function 64 corresponding to the impulse response at the sound receiving point 40 using the parameters output from the model 60.
[0124] The operation unit 133 can also calculate the sound pressure 66 at the sound receiving point 40 by performing a predetermined function conversion operation based on the Green function 64 formed in the output layer of the model 60. Figure 5 Describe this. Figure 5 It is a diagram for describing the operation process according to the first embodiment.
[0125] Figure 5 is a plan view showing the virtual space 14 calculated by the operation unit 133. Figure 5 In the embodiment of FIG. 1 , the operation unit 133 calculates the sound pressure of the sound transmitted from the sound generation point 30 at the sound reception point 40 .
[0126] exist Figure 5In the embodiment shown in , in a case where the sound pressure obtained by the operation unit 133 is represented by a function Φ of the observation position (coordinate) r and time t, when the boundary of the area to be analyzed (virtual space 14) is represented by Σ, when the coordinates of the sound source position are represented by r', the intensity of the sound source is represented by S, and the volume of the area to be analyzed is represented by V, the function Φ is represented by the following expression (11).
[0127]
[0128] The operation unit 133 calculates the sound pressure at the sound receiving point 40 based on the above expression (11). As a result, even in the virtual space 14 including the structure of the object 16 and the like, the operation unit 133 can obtain the sound pressure at the sound receiving point 40 in consideration of the influence.
[0129] return Figure 2 , the description is continued. The display control unit 134 controls the output unit 140 to output various types of information. Specifically, the display control unit 134 controls the display 150 to display a user interface used by the producer 200, etc.
[0130] Similar to the display control unit 134, the output control unit 135 controls the output unit 140 to output various types of information. Specifically, when the input unit 131 obtains the input of the voice reproduction operation of the producer 200 via the user interface, the output control unit 135 controls the speaker 160 to output an acoustic signal in response to the request.
[0131] For example, the output control unit 135 outputs the reproduced sound at the sound receiving point 40 of the sound transmitted from the sound generating point 30 based on the operation result of the operation unit 133. Specifically, the output control unit 135 outputs the acoustic signal from the speaker 160 based on the sound pressure at the sound receiving point 40 calculated from the impulse response at the sound receiving point 40.
[0132] The output unit 140 outputs various types of information. Figure 2 As shown, the output unit 140 includes a display 150 and a speaker 160. Under the control of the display control unit 134, the display 150 displays the virtual space 10 to be processed, and displays a user interface for the input unit 131 to obtain operation input from the producer 200. The speaker 160 outputs an acoustic signal generated from the operation result, etc. under the control of the output control unit 135.
[0133] (1-3. Procedure of Information Processing According to First Embodiment)
[0134] Next, refer to Figure 6 and Figure 7 The procedure of information processing according to the first embodiment is described.
[0135] First, refer to Figure 6 The procedure of the learning process according to the first embodiment is described. Figure 6 is a flowchart showing the procedure of the learning process according to the first embodiment.
[0136] The sound generating device 100 inputs the training data to the model 60 for learning (specifically, the input layer of the DNN) (step S21). The training data is data that may be required for the learning process of forming the Green's function in the output layer of the DNN, and includes, for example, position coordinate data of the sound generating point and the sound receiving point, position coordinate data indicating the spatial structure, boundary conditions such as the acoustic impedance of the structure, time parameters, etc.
[0137] The sound generating device 100 forms a Green's function in the output layer of the DNN (step S22 ). Subsequently, the sound generating device 100 performs an operation for performing an impulse response output of a predetermined time according to the formed Green's function (step S23 ).
[0138] Thereafter, the sound generating device 100 calculates an error based on the error function represented by the above expression (10), such as an error between the training data acquired from the storage unit 120 and an arithmetic result for performing an impulse response output corresponding to a predetermined time from the above Green's function, and determines whether the error evaluation is a predetermined value or more (step S24). It should be noted that the error evaluation is, that is, an error amount, and the predetermined value corresponds to, for example, Figure 4 for any value of the threshold ε shown in .
[0139] In the case where the error estimate is lower than the predetermined value (step S24; No), the sound generating apparatus 100 adjusts the parameters of the DNN (step S25), and returns to S21 again to repeat the learning process.
[0140] In contrast, in the case where the error evaluation is greater than the predetermined value (step S24; Yes), the sound generating device 100 completes the learning and stores the derived Green's function in the storage unit 120 (step S26).
[0141] Next, refer to Figure 7 The procedure of the arithmetic processing according to the first embodiment is described. Figure 7 : is a flowchart showing the procedure of the arithmetic processing according to the first embodiment.
[0142] The sound generating device 100 obtains the training model 60 stored in the storage unit 120 and inputs data to the obtained training model 60 (step S31). The data includes the position coordinate data of the sound generating point and the sound receiving point in the virtual space to be operated, the position coordinate data representing the spatial structure, the boundary conditions such as the acoustic impedance of the structure, time, etc.
[0143] The sound generating device 100 applies expression (11) based on the impulse response, which is the output of the Green's function formed in the output layer of the model 60, and performs operation to derive the sound pressure (step S32). The sound generating device 100 further calculates the sound pressure at the sound receiving point by applying data such as the position coordinate data of the sound receiving point to expression (11) (step S33).
[0144] (1-4. Execution Example of Information Processing According to First Embodiment)
[0145] The sound generating device 100 can obtain an acoustic signal at a sound receiving point in real time even in a virtual space whose structure changes according to time by using the training model 60. Figure 8 Such an embodiment is described.
[0146] Figure 8 is a view showing another example of the arithmetic processing according to the first embodiment. Figure 8 An example is shown in which a structure (object) as a wall 20 exists between the sound generation point 30 and the sound receiving point 40 in the virtual space 10 .
[0147] In this embodiment, the generator 200 performs a process of deleting half of the wall 20 by operating the cursor 24 displayed on the user interface while reproducing the acoustic signal at the sound receiving point 40. Through this operation, the wall 20 interrupted between the sound generating point 30 and the sound receiving point 40 in the virtual space 10 is changed to the wall 21 having half the volume of the wall 20.
[0148] The sound generating device 100 substitutes data reflecting this change into the model 60 having the Green's function in the output layer. Specifically, when the wall 20 is changed to the wall 21, the sound generating device 100 inputs the position coordinate data (structural data in the virtual space 10) into the model 60 having the Green's function in the output layer, and recalculates the output value at the sound receiving point 40. That is, the sound generating device 100 obtains the sound pressure distribution after the structure in the virtual space 10 is changed, as shown in the distribution 18.
[0149] As a result, in the case where the wall 20 changes in the virtual space 10, the producer 200 can confirm in real time what type of sound change occurs at the sound receiving point 40. That is, in the virtual space 10, the sound at the sound receiving point 40 continuously changes due to the change in the structure of the wall 20 arranged at the center. The generator 200 can be changed via the user interface operation, and the sound that continuously changes following the change can be listened to at the sound receiving point 40. In this way, according to the sound generating device 100, the operability of the acoustic design of the generator 200 can be improved.
[0150] (2. Second Embodiment)
[0151] (2-1. Overview of Information Processing According to Second Embodiment)
[0152] Next, the second embodiment is described. In the first embodiment, the sound generating device 100 reproduces an acoustic signal at a sound receiving point by generating a model 60 having a Green's function, and the model 60 outputs data corresponding to the impulse response in the output layer through a learning process. Here, on the basis of the training model 60 in the first embodiment, the producer 200 can perform acoustic design by inputting a desired impulse response in a virtual space, that is, by inputting an ideal sound in a virtual space as a condition. For example, the producer 200 specifies the shape of the function of the impulse response using a user interface. The sound generating device 100 extracts n sample values as many as the input nodes of the trained model 60 from the specified shape of the function, and inputs the n sample values to each node. In the second embodiment, it is assumed that the training model of the first embodiment has a function of performing an inverse operation at the input / output of each node. That is, for example, the sound generating device 100 according to the second embodiment receives an input at the output of each node of the first embodiment, and performs an inverse operation at each node to generate an output. By providing the trained model 60 with an inverse operation function, the sound generating device 100 can estimate the parameters that the external shape of a specific structure in the environment should have when the output of a predetermined impulse response is conditioned. For example, the sound generating device 100 sets a specific structure as a processing target, thereby generating sound parameters that satisfy the specific structure. In addition, the second embodiment includes the generation of a model for obtaining a derivation of an ideal shape obtained by changing the shape of the structure to change the external shape.
[0153] The sound generating device 100 according to the second embodiment is provided with an inverse operation different from the operation according to the first embodiment in the training model 60, and can output the condition data of the acoustic environment having the impulse response as an input by performing the inverse operation. That is, in the case where the trained model 60 in the first embodiment is made into a DNN, the sound generating device 100 can newly set the output layer forming the Green's function in the first embodiment as the input layer, regard the input layer including the acoustic impedance z in the first embodiment as the output layer, continue the data calculation flow in the direction starting from the output layer (that is, the input layer of the second embodiment) to the intermediate layer and from the intermediate layer to the input layer (the output layer of the second embodiment), and finally output the acoustic impedance z, which is one of the data of the input layer of the first embodiment. That is, the sound generating device 100 according to the second embodiment can generate the acoustic impedance of the system as the output in the input layer of the model 60 learned in the first embodiment by acquiring the data of the impulse response in the input unit 131, inputting the acquired data to the output layer of the training model, and performing the above-mentioned inverse operation at each node in the model 60 of the first embodiment based on the model 60 trained in the first embodiment. As a result, the sound generating device 100 can establish a virtual space in which the producer 200 implements an ideal impulse response. According to the sound generating device 100, in order to realize the ideal impulse response, data such as the arrangement (position coordinates) of a new object for making the acoustic impedance of the entire system close to the ideal value can be output and presented to the producer 200.
[0154] refer to Fig. 9 Such processing is described. Fig. 9 1 is a diagram showing an outline of information processing according to the second embodiment. Similar to the first embodiment, the sound production device 100 provides the producer 200 with a user interface in which the virtual space 10 is displayed.
[0155] The virtual space 10 includes a sound generating point 30, a sound receiving point 40, and a wall 20. In the virtual space 10, the generator 200 inputs a desired impulse response 70 as a sound of a sound reproduced at the sound receiving point 40. It should be noted that the producer 200 may input an impulse response created by himself / herself, or may select an existing impulse response in which reverberation information according to various environments and the like are preset from a library.
[0156] The sound generating device 100 inputs the data of the impulse response 70 to the input layer of the model 80 as input information. Note that the model 80 is a DNN having a Green's function in a part of the input layer, and has an inverse operation function different from the operation of the first embodiment at each node of the trained model 60 in the first embodiment. The sound generating device 100 may convert the sound pressure 82 into an impulse response and then input it to the Green's function 84 formed in the input layer of the model 80.
[0157] In this embodiment, the trained model 80 outputs the acoustic impedance 86, which is a boundary condition. That is, the model 80 outputs the acoustic impedance in the virtual space 10 for realizing the ideal impulse response 70 at the sound receiving point 40. Note that, although the acoustic impedance is output in the present embodiment, in the output of the model 80, any input at the time of learning the model 60 (that is, spatial information indicating the area of the virtual space 10, position coordinate data indicating the configuration of the object, and input regarding the condition of the virtual space 10) may be output.
[0158] Thereafter, the sound generating device 100 may arrange the new object 90 in the virtual space 10 based on the output acoustic impedance 86. That is, the sound generating device 100 may change the virtual space 10 by adding the object 90 to the virtual space 10 so that the boundary condition satisfies the acoustic impedance 86 as the entire system. As described above, specific information (position coordinates, structural data (e.g., material), etc.) of a predetermined sound source object in the virtual space 10 may be output instead of the acoustic impedance 86 in the model 80.
[0159] As described above, the sound generating device 100 according to the second embodiment obtains the input of the request related to the output of the acoustic signal desired by the generator 200 (for example, the designation of the desired impulse response). In the case of obtaining the input of the request, the sound generating device 100 inputs the output of the acoustic signal desired by the generator 200 to the model 80, and outputs at least any one of the information about the virtual space 10 or the object required to realize the output of the acoustic signal desired by the generator 200 (for example, the acoustic impedance) or the information about the sound source object from the output data. As the information related to the virtual space 10 or the object, the sound generating device 100 outputs at least any one of the change of the boundary condition of the space to be analyzed, the change of the structural data of the object that has been arranged, or the structural data of the object that should be newly arranged in the virtual space 10.
[0160] Therefore, the sound generating device 100 can implement an acoustic design that satisfies the ideal of the producer 200. The producer 200 can realize the ideal sound he or she imagines without manually changing the configuration of objects in the virtual space 10, which is time-consuming and laborious.
[0161] (2-2. Procedure of Information Processing According to Second Embodiment)
[0162] Next, refer to Fig.10 The procedure of information processing according to the second embodiment is described. Fig.10 is a flowchart showing an information processing procedure according to the second embodiment.
[0163] The sound generating device 100 obtains data input of an ideal impulse response for sound output in the sound generating device via the user interface according to the instruction of the producer 200 (step S41). For example, the data of the ideal impulse response may be obtained from a server on the network via the communication unit 110, or may be obtained from a predetermined storage area of the storage unit 120 to be input. The producer 200 may input separately via the input unit 131. The sound generating device 100 inputs the input impulse response data to the model 80 (i.e., the Green's function formed in the input layer of the model 80) (step S42).
[0164] Subsequently, the sound generating device 100 performs an inverse operation toward the input layer of the DNN, including obtaining the parameters of the Green's function formed in the input layer of the model 80 through the inverse operation (step S43). Through such processing, the sound generating device 100 can calculate the acoustic impedance in the virtual space 10 based on the trained model 80.
[0165] Thereafter, the sound generating device 100 calculates the coordinates of the additionally required structure based on the calculated acoustic impedance (step S44). The sound generating device 100 adds the added structure to the initial condition (step S45). For example, the sound generating device 100 adds the position coordinate data, acoustic impedance, etc. of the added structure to the initial condition.
[0166] After adding the structure, the sound generating device 100 verifies whether there is a contradiction in the virtual space 10 (step S46). For example, the sound generating device 100 verifies whether the impulse response at the sound receiving point 40 can be maintained or whether there is a change in the acoustic impedance of the entire virtual space 10 by arranging the additional structure.
[0167] In the case where there is a contradiction in the verification result (step S46; Yes), the sound generating apparatus 100 changes the shape of the added structure (step S47), and adds the information of the structure to the initial condition again.
[0168] On the contrary, in the case where there is no contradiction in the verification result (step S46; No), the sound generating apparatus 100 displays the structure on the screen including the user interface (step S48).
[0169] It should be noted that although the above describes an embodiment in which the physical quantity obtained by the sound generating device 100 is the acoustic impedance, the sound generating device 100 can output other physical quantities through an inverse operation based on the trained model. For example, the sound generating device 100 is configured to output the acoustic impedance based on the ideal impulse response. Conversely, the position coordinate data of the sound generation point 30 (sound source position) for obtaining the sound of the ideal impulse response can be calculated through an inverse operation based on the trained model. As a result, the sound generating device 100 can output the position data of the sound source object that realizes the output of the acoustic signal desired by the generator 200 at the sound receiving point 40 as information about the object after the virtual space is changed.
[0170] Alternatively, the sound generating device 100 may satisfy the ideal boundary condition by changing the position data of an existing object. For example, when the acoustic impedance is implemented in the virtual space 10, the sound generating device 100 may set a specific structure as a processing target and generate a model for obtaining an ideal shape by changing only the shape of the target structure.
[0171] (3. Third Embodiment)
[0172] The information processing described in the first and second embodiments can be performed via a user interface provided by the sound generating apparatus 100. Information processing via such a user interface will be described as a third embodiment.
[0173] Fig.11 is a view ( 1 ) showing an embodiment of a user interface 300 provided by the sound generating device 100 . Fig.11 A display example of the user interface 300 displayed on a display or the like connected to the sound generating device 100 is shown.
[0174] The user interface 300 is used when the producer 200 causes the sound generating device 100 to perform a learning process, and when the acoustic design of a virtual space in content is performed using a trained model or the like.
[0175] The user interface 300 includes, for example, an impulse response display operation area 302. In the impulse response display operation area 302, for example, a waveform graph or the like indicating an impulse response indicated as a result of operation according to the first embodiment is displayed.
[0176] Alternatively, in the case where the sound producing device 100 performs the arithmetic processing according to the second embodiment, a waveform of an impulse response that is ideal for the producer 200 is displayed in the impulse response display operation area 302. For example, the producer 200 causes the sound producing device 100 to read a file holding an ideal impulse response, thereby displaying a waveform indicating the impulse response in the impulse response display operation area 302. The producer 200 can edit the impulse response displayed in the impulse response display operation area 302.
[0177] The user interface 300 includes a sound pressure map display operation area 304. In the sound pressure map display operation area 304, for example, a sound pressure distribution or the like represented as a calculation result according to the first embodiment is displayed.
[0178] The user interface 300 includes a condition display area 306. Information on each condition set in the virtual space being edited is displayed in the condition display area 306. When the producer 200 selects any object, information on each condition set for the object is displayed in the condition display area 306.
[0179] The producer 200 may input various conditions set in the virtual space of the content in the condition display area 306. For example, the producer 200 may input the acoustic impedance, transmittance, reflectance, position coordinates, etc. of the virtual space and each object arranged in the virtual space.
[0180] The user interface 300 includes a virtual space display area 310. In the virtual space display area 310, the state of the virtual space designed by the producer 200 is displayed. For example, the virtual space display area 310 displays the texture of the wall or ceiling of the virtual space and the object 312 arranged in the virtual space. It should be noted that although Fig.11 The virtual space displayed in the virtual space display area 310 is shown in three dimensions, but the virtual space may be displayed in two dimensions such as a plan view.
[0181] The user interface 300 includes a learning button 314, an execution button 316, and an inverse operation button 318. The learning button 314 is a button for causing the sound generating apparatus 100 to perform the learning process described in the first and second embodiments in the virtual space designed by the producer 200. The execution button 316 is a button for causing the sound generating apparatus 100 to perform the arithmetic operation process described in the first embodiment in the virtual space designed by the producer 200. The inverse operation button 318 is a button for causing the sound generating apparatus 100 to perform the arithmetic operation process described in the second embodiment in the virtual space designed by the producer 200.
[0182] The user interface 300 includes an object selection window 320. In the object selection window 320, objects that can be arranged in the virtual space by the producer 200 are displayed. The producer 200 can arrange the object in the virtual space by selecting a desired object and moving it to the virtual space display area 310. When the object is arranged, the sound generating device 100 obtains the position coordinate data and shape information of the object and stores them in the virtual space.
[0183] The user interface 300 includes a sound source selection window 322. In the sound source selection window 322, sound source objects that can be arranged in the virtual space by the producer 200 are displayed. The sound source object may include various types of information about the acoustic signal to be emitted in advance, such as sound pressure, pitch, and directivity. The producer 200 can arrange the sound source object in the virtual space by selecting a desired sound source object and moving it to the virtual space display area 310. When the sound source objects are configured, the sound generating device 100 obtains the position coordinate data, sound pressure, and timbre information of these sound source objects and stores them as sound emission points in the virtual space.
[0184] The user interface 300 includes a material selection window 324. In the material selection window 324, materials that can be set in an object arranged in the virtual space by the producer 200 are displayed. The producer 200 can select a desired material and apply the material to an object arranged in the virtual space display area 310, thereby providing conditions (acoustic impedance, sound absorption coefficient, etc.) for the object.
[0185] In this way, the sound generating device 100 obtains the input of the request for the arrangement of the object from the producer 200 through the input unit 131 via the user interface 300. The sound generating device 100 determines the structural data of the object in the virtual space based on the designation of at least any one of the material, transmittance, reflectance, or position data of the object contained in the input of the request from the producer 200. That is, the producer 200 can give the object condition by selecting a familiar material without specifying a numerical value such as acoustic impedance, so that the work of acoustic design can be easily performed. It should be noted that in the case where the input unit 131 obtains the input of the request for the arrangement of the object from the generator 200, the sound generating device 100 can output an acoustic signal at a sound receiving point recalculated based on the structural data of the object after the change is requested. As a result, the producer 200 can confirm the change caused by the operation such as the arrangement of the object in real time, so that the operability of the acoustic design is improved.
[0186] The user interface 300 includes a time display area 326. In the time display area 326, time information in the case of performing various types of processing is displayed. For example, in the case of causing the sound generating device 100 to calculate an impulse response and a sound pressure at any sound receiving point, and reproducing an acoustic signal based on the arithmetic result, the passage of reproduction time is displayed in the time display area 326.
[0187] See also Figures 12 to 17 Examples of operations that may be performed by producer 200 in user interface 300 are described below. Fig.12 is a view ( 1 ) showing an example of operations in the user interface 300 .
[0188] like Fig.12As described in FIG. 3 , in the user interface 300, the producer 200 operates the cursor 340 and selects an object desired to be arranged in the virtual space from the object selection window 320. The producer 200 moves the selected object 342 to the virtual space display area 310 by a drag operation or the like.
[0189] The sound generation device 100 obtains shape information related to the configuration object 342 and position data (position coordinate data) of the object 342 configured in the virtual space. The sound generation device 100 updates the area data in the virtual space based on the acquired information. In this way, the producer 200 can quickly form a desired virtual space by using the user interface 300.
[0190] Then, refer to Fig.13 Another operating embodiment is described. Fig.13 is a view ( 2 ) showing an example of operations in the user interface 300 .
[0191] like Fig.13 As shown, in the user interface 300, the producer 200 operates the cursor 340 and selects a sound source object desired to be arranged in the virtual space from the sound source selection window 322. The producer 200 moves the selected sound source object 344 to the virtual space display area 310 by a drag operation or the like.
[0192] The sound generating device 100 obtains the sound information set in the configured sound source object 344 and the position data (position coordinate data) of the sound source object 344 configured in the virtual space. The sound generating device 100 updates the sound generation point in the virtual space based on the obtained information. In this way, the producer 200 can visually set where in the virtual space the sound is transmitted by using the user interface 300.
[0193] Then, refer to Fig.14 Another operating embodiment is described. Fig.14 is a view (3) showing an example of operations in the user interface 300.
[0194] like Fig.14 As shown, in the user interface 300, the producer 200 operates a cursor 340 and selects a material set as an object from the material selection window 324. The producer 200 moves the selected material toward the object 312 arranged in the virtual space display area 310 by a drag operation or the like.
[0195] When the cursor 340 is superimposed on the object 312, the sound generating device 100 applies the selected material to the object 312. Specifically, the sound generating device 100 applies boundary conditions such as acoustic impedance and sound absorption coefficient set for each material to the object 312. The sound generating device 100 updates the boundary conditions of the entire virtual space based on the acquired information. In this way, the producer 200 can perform numerical setting of acoustic impedance, etc., which is generally difficult, through an intuitive operation of selecting a material using the user interface 300.
[0196] Then, refer to Fig.15 Another operating embodiment is described. Fig.15 is a view (4) showing an example of operations in the user interface 300.
[0197] like Fig.15 As shown, in the user interface 300, after the formation of the virtual space and the setting of the sound receiving point and the sound generating point are completed, the producer 200 operates the cursor 340 and presses the learning button 314. When the learning button 314 is pressed, the sound generating device 100 performs a learning process under the set environment. Note that in the learning process, the sound generating device 100 can perform a process of automatically performing a plurality of trials while gradually changing the positions of the sound receiving point and the sound generating point.
[0198] In this way, by performing learning in a state where specific boundary conditions are set, the sound generating device 100 can perform learning of a model corresponding to the specific boundary conditions. For example, the sound generating device 100 can maintain a trained model suitable for each environment, such as a trained model trained in an environment assuming a narrow room or a model trained in an environment close to a free sound field. As a result, the sound generating device 100 can calculate an acoustic signal with higher accuracy by using a trained model suitable for arithmetic processing described later.
[0199] Then, refer to Fig.16 Another operating embodiment is described. Fig.16 is a view (5) showing an example of operations in the user interface 300.
[0200] like Fig.16As shown, in the user interface 300, after the formation of the virtual space and the setting of the sound receiving point and the sound generating point are completed, the producer 200 operates the cursor 340 and presses the execution button 316. When the execution button 316 is pressed, the sound generating device 100 performs arithmetic processing under the set environment. Specifically, the sound generating device 100 inputs the position coordinate data of the set sound receiving point or the sound generating point, the boundary conditions of the virtual space, etc. into the training model 60, and derives the impulse response at the sound receiving point. In addition, the sound generating device 100 calculates the sound pressure at the sound receiving point based on the derived impulse response. The sound generating device 100 generates an acoustic signal based on the arithmetic result and outputs the generated acoustic signal from a speaker or the like.
[0201] Then, refer to Fig.17 Another operating embodiment is described. Fig.17 is a view (6) showing an example of operations in the user interface 300.
[0202] like Fig.17 As shown in FIG. 3 , in the user interface 300, the producer 200 completes the formation of the virtual space and the setting of the sound receiving point and the sound generating point, and further sets the ideal impulse response at the sound receiving point, and then operates the cursor 340 and presses the inverse operation button 318. When the inverse operation button 318 is pressed, the sound generating device 100 performs an inverse operation of the boundary conditions for realizing the ideal impulse response of the producer 200. Specifically, the sound generating device 100 inputs the position coordinate data of the set sound receiving point or the sound generating point, the impulse response of the DNN for which pre-learning is performed set by the producer 200, and obtains the boundary conditions output from the DNN.
[0203] The sound generating device 100 calculates the shape, boundary conditions, shape, etc. of the object 346 required to implement the acquired boundary conditions. In the case of configuring the object 346, the sound generating device 100 verifies that there is no contradiction in the boundary conditions of the virtual space acquired from the DNN, and then displays the new object 346 in the virtual space display area 310. Note that in the case where the producer 200 does not accept the object 346 proposed to be arranged by the sound generating device 100, he / she may request the sound generating device 100 to change the shape, position coordinate data, etc. In this case, the sound generating device 100 changes the shape and configuration of the object 346 so that there is no contradiction in the boundary conditions of the virtual space.
[0204] Next, an embodiment of a learning process using the user interface 300 is described. Fig.18 is a diagram (1) showing an example of a learning process according to the third embodiment;
[0205] As described above, the sound generating device 100 may perform learning of a plurality of models 60 according to the boundary conditions. In this regard, the sound generating device 100 may perform learning of the model 60 separately for a case where the direct sound emitted from the sound source reaches the sound receiving point and a case where the direct sound emitted from the sound source does not reach the sound receiving point (i.e., a case where the sound observed at the sound receiving point is only the diffracted sound or the indirect sound). Note that the direct sound refers to the sound observed at the sound receiving point, there is no obstruction between the sound source and the sound receiving point, and even if there is once, there is no reflection or diffraction of the sound emitted from the sound source by the wall or the ceiling.
[0206] exist Fig.18 In the illustrated embodiment, a sound source object 360, a sound receiving point 362, and an object 364 are arranged in the virtual space display area 310. In this example, since the direct sound emitted from the sound source object 360 is interrupted by the object 364, it does not reach the sound receiving point 362.
[0207] The sound generating device 100 can be set in this way in a situation where direct sound cannot be observed at the sound receiving point 362, and then perform the learning process under such a condition. Specifically, in which one or more objects ( Fig.18 In the case where the object 364 in the embodiment of the present invention is arranged in a position and range in which the direct sound generated by the sound source object 360 in the virtual space is blocked from reaching the sound receiving point 362 and the sound receiving point 362 between the sound source object 360, the sound generating device 100 can generate the model 60 trained using only the indirect sound from the sound source object 360 as input as acoustic data. As a result, the sound generating device 100 can obtain the model 60 trained in an environment that does not include direct sound.
[0208] Thereafter, the sound generating device 100 moves the sound source object 360 and sets a situation in which direct sound can be observed at the sound receiving point 362 . Fig.19 Such an embodiment is shown in . Fig.19 is a view (2) showing an example of a learning process according to the third embodiment;
[0209] exist Fig.19 In the illustrated example, when the sound source object 360 moves in the virtual space display area 310 , the direct sound emitted from the sound source object 360 is not interrupted by the object 364 and reaches the sound receiving point 362 .
[0210] Even under the condition that direct sound can be observed at the sound receiving point in this way, the sound generating device 100 performs the same Fig.18 As a result, the sound generating device 100 can obtain the model 60 trained in an environment containing direct sound.
[0211] For example, the sound generating apparatus 100 may selectively use the trained model 60 in the case of performing the arithmetic processing according to the first embodiment in a game or the like. Fig.18 In the case where an object 364 in the embodiment of the present invention exists in a position and range that blocks the direct sound generated by the sound source object from reaching the sound receiving point between the sound receiving point in the virtual space to be analyzed and the sound source object, the sound generating device 100 can output an acoustic signal at the sound receiving point using artificial intelligence (training model 60) that only considers the indirect sound generation from the sound source object. Fig. 20 Such a process is described. Fig. 20 is a flowchart showing the procedure of arithmetic processing according to the third embodiment.
[0212] like Fig. 20 As shown, the sound generating device 100 determines the sound source position according to the operation of the producer 200, the setting of the sound source object in the game, etc. (step S51).
[0213] Next, the sound generating device 100 determines whether there is an object that blocks the direct sound between the sound source position and the listening point (e.g., the position of the game character) (step S52). For example, the sound generating device 100 determines whether the vision of the game character at the listening point (e.g., within the visual field that the player can visually recognize on the game screen) includes the sound source (the character, object, etc. that emits the sound).
[0214] In the case where there is an object blocking the direct sound between the sound source position and the listening point (step S52); yes), the sound generating device 100 refers to the case where there is no object blocking the direct sound. Fig.18 The sound generating device 100 calculates the acoustic effect at the listening point using the model 60 trained under the direct sound environment from the storage unit 120 shown in the embodiment of FIG.
[0215] On the contrary, if there is no object blocking the direct sound between the sound source position and the listening point (step S52; No), the sound generating device 100 refers to the sound source position including the following: Fig.19 The sound generating device 100 uses the trained model 60 trained in the environment of the direct sound from the storage unit 120 described in the embodiment of FIG. Then, the sound generating device 100 calculates the acoustic effect at the listening point using the model 60 trained by the direct sound and the indirect sound (step S54).
[0216] In this way, depending on the environment in which arithmetic processing is performed, the sound generating device 100 can selectively use the model 60 trained under conditions close to the environment to perform operations on sound. Thus, the sound generating device 100 can reproduce the sound of the virtual space more realistically.
[0217] (4. Fourth Embodiment)
[0218] In the information processing described in each of the above-mentioned embodiments, an embodiment in which there is one sound generation point or one sound receiving point is described, but there may be a plurality of sound generation points or sound receiving points. For example, in the case where a plurality of sound generation points are arranged, the sound generating device 100 calculates the impulse response of the sound reaching the sound receiving point from each sound generation point, and generates an acoustic signal based on the calculation result. Then, the sound generating device 100 reproduces the acoustic signal at the sound receiving point by combining the acoustic signals generated at the respective sound generation points.
[0219] In this embodiment, for example, in the case where the position of the sound generation point or the sound receiving point is a physical quantity to be obtained in the second embodiment, the condition that there are multiple sound sources can be prepared as a check box on the user interface, and the producer 200 can check the check box if he wants to generate multiple sound sources. As a result, the sound generating device 100 can output multiple sound sources and sound receiving points to the virtual space according to the conditions input by the producer 200.
[0220] (5. Fifth Embodiment)
[0221] In the above-mentioned first embodiment, an embodiment in which the sound generating device 100 performs a learning process using teacher data based on acoustic simulation is described. Here, the sound generating device 100 may perform a learning process using not only teacher data based on acoustic simulation but also actual measurement data, a solution obtained by numerical calculation, etc. as teacher data. For example, the sound generating device 100 uses actual measurement data in a space reproduced in a similar shape or data obtained by numerical calculation as teacher data. As a result, the sound generating device 100 can improve the accuracy of learning and operation.
[0222] (6. Sixth Implementation Method)
[0223] In each of the above-mentioned embodiments, the boundary conditions set for the virtual space and the objects in the virtual space may change over time. For example, the sound generating device 100 may perform learning processing or arithmetic processing in the virtual space, wherein the virtual space includes data in which at least one of the shape, position coordinate data, acoustic impedance, transmittance, and reflectivity of the structure in the virtual space changes over time. Therefore, the sound generating device 100 can provide a sound field with a more realistic sense of change in acoustic characteristics relative to a state in which the physical properties of the structure itself change (for example, a depiction in which cloth burns or metal melts).
[0224] (7. Seventh Implementation Method)
[0225] In the data processing described in each of the above-mentioned embodiments, the sound generating device 100 may process a frequency band corresponding to a duration (generally referred to as reverberation time) until the energy density of the sound decays by 60 decibels as a divided analysis target. Specifically, the sound generating device 100 divides the sound to be analyzed into a reverberation sound region and a direct sound region, which is a region that contributes to diffraction. Among them, the reverberation sound region is a region where the contribution of the wave component is high compared to the direct sound region. That is, the sound generating device 100 may divide the region where the contribution of the wave component is high and other regions, and set the region where the contribution of the wave component is high as an arithmetic target, thereby providing a sound field that follows the visual information while reducing the total operation amount.
[0226] The sound generating device 100 may use a sound ray tracing method, which is a geometric method that does not include wave sounds, for the above-mentioned reverberant sound region. Note that in order to provide a sound without a strange feeling in a frequency band that is a boundary between the reverberant sound region and the direct sound region, the sound generating device 100 may calculate a buffer of about 30% before and after the reverberation time by two methods of the reverberant sound region and the direct sound region, and calculate a continuous value by adding a physical quantity such as sound pressure. For example, the sound generating device 100 may provide a natural sound at high speed by calculating the direct sound as a wave and the reverberant sound as a sound ray, and combining them as a sound simulator, for example, in a region within the reverberation time.
[0227] (8. Eighth Implementation Method)
[0228] In the second embodiment, the sound generating device 100 may further acquire shape data of an object and position coordinate data of a sound source and position coordinate data of a sound receiving point in addition to the conditions (impulse response) input by the generator 200. That is, the sound generating device 100 may output not only acoustic impedance but also shape data of an additional object and the like through inverse operation.
[0229] In the above process, the sound generating device 100 can perform a process of reducing the error by adding the influence of the numerical error in the inverse operation to the component. For example, by defining the error value as an unknown object, the sound generating device 100 can output a new object that satisfies the inverse operation within the range of the numerical error.
[0230] (9. Other Implementation Methods)
[0231] In addition to each of the above-described embodiments, the processing according to each of the above-described embodiments may be performed in various different modes.
[0232] In each process described in the above embodiments, all or part of the process described as being automatically performed may be performed manually, or all or part of the process described as being manually performed may be automatically performed by a known method. Unless otherwise specified, the processes, specific names, and data including various data and parameters described in this document and shown in the accompanying drawings may be optionally changed. For example, the various data shown in the various figures are not limited to the data shown in the figures.
[0233] Each component of each device shown in the drawings is conceptual in function and is not necessarily physically configured as shown in the drawings. That is, the specific form of distribution and integration of each device is not limited to the form shown, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.
[0234] The above-described embodiment and modified examples can be appropriately combined within a range in which the processing contents do not conflict with each other.
[0235] The effects described in this specification are merely embodiments and are not limited, and there may be other effects.
[0236] (10. Effects of the Acoustic Signal Control Method and the Learning Model Generation Method According to the Present Disclosure)
[0237] As described above, the acoustic signal control method according to the present invention includes: selecting a sound source object to be arranged in a three-dimensional virtual space; arranging the selected sound source object at a predetermined position where one or more three-dimensional objects are arranged in the three-dimensional virtual space; inputting position data and acoustic data related to the arranged sound source object, area data within the three-dimensional virtual space, structural data of the three-dimensional object, and position data of each object and sound receiving point at a predetermined time into a first artificial intelligence (in this embodiment, model 60), and outputting an acoustic signal at the sound receiving point based on an impulse response at the sound receiving point generated from the output data.
[0238] In this way, the acoustic signal control method according to the present disclosure generates an acoustic signal at a sound receiving point by using artificial intelligence having as input the structure of a virtual space, the coordinates of an object arranged in the virtual space, structural data including acoustic impedance, etc. As a result, the acoustic signal control method does not require huge calculations for solving a wave equation in the entire area, and can reproduce an acoustic space with a sense of reality and achieve high-speed arithmetic processing.
[0239] At this time, the acoustic signal control method outputs an acoustic signal based on the sound pressure at the sound receiving point calculated from the impulse response at the sound receiving point.
[0240] In this way, since the acoustic signal control method can obtain the sound pressure from the impulse response, the output of the acoustic signal including the sound pressure at the sound receiving point with high reproducibility can be achieved.
[0241] The acoustic signal control method includes: obtaining input of a request regarding the arrangement of a three-dimensional object from a user (in an embodiment, a producer 200) through an input unit 131; and determining structural data based on a specification from the user of at least any one of material, transmittance, reflectivity, and position data regarding the three-dimensional object included in the requested input.
[0242] In this way, the acoustic signal control method obtains input of a request such as the arrangement of objects in a virtual space from a user via a user interface, etc., and reflects the obtained data in the control of the generated acoustic signal. As a result, the user can quickly and easily perform acoustic design of content.
[0243] The acoustic signal control method outputs an acoustic signal at a sound receiving point recalculated based on structural data of the three-dimensional object after the request is changed, when an input of a request related to arrangement of a three-dimensional object is obtained from a user.
[0244] In this way, the acoustic signal control method can output the sound before and after the change of the sound environment of the virtual space that changes with the user's operation. As a result, the user can immediately confirm the changed sound, thereby improving operability.
[0245] The acoustic signal control method, in a three-dimensional virtual space, uses the first artificial intelligence generated by only considering the indirect sound from the sound source object to output the acoustic signal at the sound receiving point while blocking the direct sound generated by the sound source object from reaching the position and range between the sound receiving point and the sound source object.
[0246] In this way, the acoustic signal control method can output an acoustic signal using artificial intelligence trained according to a situation assumed in a virtual space. According to this acoustic signal control method, since the acoustic signal can be output using artificial intelligence further trained according to the situation, the reproducibility of the acoustic signal can be improved, and the realism of the sound can be improved.
[0247] The acoustic signal control method also includes: obtaining input of a request for output of an acoustic signal desired by a user; inputting the output of the acoustic signal desired by the user into a first artificial intelligence (in an embodiment, model 80), and upon obtaining the input of the request, outputting at least one of the following items from the data output by the inverse operation: data related to the three-dimensional virtual space or the three-dimensional object required to achieve the output of the acoustic signal desired by the user, or data related to the sound source object.
[0248] In this way, the acoustic signal control method can perform an operation opposite to that of the first artificial intelligence, which derives data related to the virtual space (acoustic impedance, etc.) based on a request such as an impulse response desired by the user. According to this acoustic signal control method, the user can realize a sound environment that is ideal for the user himself without manually setting an object newly arranged in the virtual space.
[0249] The acoustic signal control method outputs at least any one of changes in boundary conditions of the space to be analyzed, changes in structural data of already arranged three-dimensional objects, or structural data of three-dimensional objects that should be newly arranged in the three-dimensional virtual space as data related to the three-dimensional virtual space or the three-dimensional objects.
[0250] In this way, the acoustic signal control method can make suggestions to the user, such as outputting objects for achieving a sound environment that is ideal for the user.
[0251] The acoustic signal control method outputs position data of a sound source object where output of a user-desired acoustic signal is achieved at a sound receiving point as data on the sound source object.
[0252] In this way, the acoustic signal control method can automatically arrange objects for achieving an ideal sound environment for the user without annoying the user.
[0253] The learning model generation method of the present invention comprises: selecting a sound source object arranged in a three-dimensional virtual space, arranging the selected sound source object at a predetermined position in the three-dimensional virtual space where one or more three-dimensional objects are arranged, and generating a first artificial intelligence, wherein the first artificial intelligence has as input acoustic data about the arranged sound source object, area data in the three-dimensional virtual space, structural data of the three-dimensional object, and position data of each object and sound receiving point at a predetermined time, and has a transfer function indicating the relationship when the sound emitted from the sound source object is observed as output at the sound receiving point when learning is performed based on predetermined teacher data.
[0254] In this way, the learning model generation method is able to achieve high-speed operation without unnaturally outputting acoustic signals even in a virtual space by generating artificial intelligence (training model) that learns the relationship between various data sets in the virtual space.
[0255] The learning model generating method generates the first artificial intelligence by performing learning of the first artificial intelligence to minimize the sum of the error between the transfer function output from the first artificial intelligence and the main equation corresponding to the input, the error between the transfer function output from the first artificial intelligence and the transfer function output by inverting the position data of the arrangement of the sound source object and the position data of the sound receiving point, and the error between predetermined teacher data and the transfer function output from the first artificial intelligence.
[0256] In this way, the learning model generation method is able to generate high-performance artificial intelligence capable of deriving arithmetic results with small differences from acoustic simulation results that require a large amount of calculation by advancing learning based on multiple error evaluations.
[0257] The learning model generation method is the following first artificial intelligence: in a three-dimensional virtual space, one or more three-dimensional objects are configured as acoustic data within the position and range of the sound receiving point between the sound receiving point and the sound source object where the direct sound generated by the occluded sound source object reaches the sound receiving point, and only the indirect sound from the sound source object is used as input to generate the first artificial intelligence.
[0258] In this way, the learning model generation method can generate artificial intelligence (training models) that are different between the case where direct sound is included and the case where direct sound is not included, thereby reproducing an acoustic signal that is more suitable for the environment assumed in the virtual space at the time of calculation.
[0259] (11. Hardware Configuration)
[0260] For example, by having Fig.21 The computer 1000 of the configuration shown in embodies an information device such as the sound generating device 100 according to each embodiment described above. Hereinafter, the sound generating device 100 according to the first embodiment will be described as an example. Fig.21 1 is a hardware configuration diagram showing an embodiment of a computer 1000 that realizes the functions of the sound generating device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected through a bus 1050.
[0261] The CPU 1100 operates based on the program stored in the ROM 1300 or the HDD 1400, and controls each unit. For example, the CPU 1100 develops the program stored in the ROM 1300 or the HDD 1400 in the RAM 1200, and executes processing corresponding to various programs.
[0262] The ROM 1300 stores a boot program such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is activated, a program depending on the hardware of the computer 1000 , and the like.
[0263] The HDD 1400 is a computer-readable recording medium that non-transiently records a program executed by the CPU 1100 , data used by the program, and the like. Specifically, the HDD 1400 is a recording medium that records an information processing program according to the present disclosure as an embodiment of the program data 1450 .
[0264] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from another device via the communication interface 1500 or transmits data generated by the CPU 1100 to another device.
[0265] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard and a mouse via the input / output interface 1600. The CPU 1100 transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 can be used as a medium interface for reading a program recorded in a predetermined recording medium (medium), etc. For example, the medium is an optical recording medium such as a digital versatile disk (DVD) or a phase change rewritable disk (PD), a magneto-optical recording medium such as a magneto-optical disk (MO), a magnetic tape medium, a magnetic recording medium, a semiconductor memory, etc.
[0266] For example, in the case where the computer 1000 is used as the sound generating apparatus 100 according to the first embodiment, the CPU 1100 of the computer 1000 implements the functions of the control unit 130 and the like by executing the information processing program loaded on the RAM 1200. The HDD 1400 stores the acoustic signal control program and data according to the present disclosure in the storage unit 120. Note that the CPU 1100 reads the program data 1450 from the HDD 1400 to execute, but as another embodiment, these programs may be acquired from another device via the external network 1550.
[0267] It should be noted that the present technology can also have the following configurations.
[0268] (1) An acoustic signal control method comprising:
[0269] Select a sound source object to be arranged in the three-dimensional virtual space;
[0270] Arranging the selected sound source object at a predetermined position in the three-dimensional virtual space where one or more three-dimensional objects are arranged;
[0271] Position data and acoustic data related to the arranged sound source object, area data in the three-dimensional virtual space, structural data of the three-dimensional object, and position data of each object and sound receiving point at a predetermined time are input into the first artificial intelligence, and based on the impulse response at the sound receiving point generated according to the output data, the acoustic signal of the sound receiving point is output.
[0272] (2) The acoustic signal control method according to (1), comprising:
[0273] The acoustic signal is output based on the sound pressure at the sound receiving point calculated from the impulse response at the sound receiving point.
[0274] (3) The acoustic signal control method according to (1) or (2), wherein
[0275] The area data in the three-dimensional virtual space is the boundary conditions of the space to be analyzed and the coordinate information forming the boundary of the three-dimensional virtual space.
[0276] (4) The acoustic signal control method according to any one of (1) to (3), wherein
[0277] The structural data of the three-dimensional object is the acoustic impedance set in the three-dimensional object and the coordinate information of the point group forming the three-dimensional object.
[0278] (5) The acoustic signal control method according to (4), further comprising:
[0279] obtaining input from a user requesting arrangement of the three-dimensional objects; and
[0280] The structural data is determined based on a designation from the user regarding at least any one of material, transmittance, reflectance, and position data of the three-dimensional object included in the input of the request.
[0281] (6) The acoustic signal control method according to (5), comprising:
[0282] When an input of a request related to the arrangement of the three-dimensional object is obtained from a user, the acoustic signal at the sound receiving point is output, the acoustic signal being recalculated based on the structural data of the three-dimensional object after being changed by the request.
[0283] (7) The acoustic signal control method according to any one of (1) to (6), comprising:
[0284] In the three-dimensional virtual space, when one or more three-dimensional objects are located between the sound receiving point and the sound source object, blocking the direct sound generated by the sound source object from reaching the position and range of the sound receiving point, the first artificial intelligence output generated by only considering the indirect sound from the sound source object is used to output the acoustic signal at the sound receiving point.
[0285] (8) The acoustic signal control method according to any one of (1) to (7), wherein
[0286] The first artificial intelligence is the machine learning algorithm.
[0287] (9) The acoustic signal control method according to any one of (1) to (8), wherein
[0288] The first artificial intelligence is deep neural network.
[0289] (10) The acoustic signal control method according to (9), wherein:
[0290] The output of a deep neural network is the output of a Green's function.
[0291] (11) The acoustic signal control method according to any one of (1) to (10), further comprising:
[0292] Obtaining a requested input related to output of an acoustic signal desired by a user; and
[0293] When the input of the request is obtained, the output of the acoustic signal desired by the user is input to the first artificial intelligence, and at least any one of the information related to the three-dimensional virtual space or the three-dimensional object and the information related to the sound source object required to achieve the output of the acoustic signal desired by the user is output based on the data output through the inverse operation.
[0294] (12) The acoustic signal control method according to (11), comprising:
[0295] Output at least any one of changes in boundary conditions of the space to be analyzed, changes in structural data of three-dimensional objects that have already been arranged, and structural data of three-dimensional objects that should be newly arranged in the three-dimensional virtual space as information related to the three-dimensional virtual space or the three-dimensional objects.
[0296] (13) The acoustic signal control method according to (11) or (12), comprising:
[0297] Position data of the sound source object is output as information related to the sound source object, in which the output of the acoustic signal desired by the user at the sound receiving point is achieved.
[0298] (14) A learning model generation method, characterized in that it includes:
[0299] Select a sound source object to be arranged in the three-dimensional virtual space;
[0300] placing the selected sound source object at a predetermined position on the three-dimensional virtual space where one or more three-dimensional objects are placed; and
[0301] A first artificial intelligence is generated by learning based on predetermined teacher data, the first artificial intelligence taking as input acoustic data related to an arranged sound source object, area data in the three-dimensional virtual space, structural data of the three-dimensional object, and position data of each object and a sound receiving point at a predetermined time, and taking as output a transfer function indicating the relationship when the sound emitted from the sound source object is observed at the sound receiving point.
[0302] (15) A learning model generation method according to (14), wherein:
[0303] The predetermined teacher data is data acquired based on acoustic simulation in a three-dimensional virtual space or data observed in a real space.
[0304] (16) The learning model generation method according to (15), comprising:
[0305] The first artificial intelligence is generated by performing learning of the first artificial intelligence to minimize the sum of the error between the transfer function output from the first artificial intelligence and the main equation corresponding to the input, the error between the transfer function output from the first artificial intelligence and the transfer function output by inverting the position data of the arrangement of the sound source object and the position data of the sound receiving point, and the error between predetermined teacher data and the transfer function output from the first artificial intelligence.
[0306] (17) A learning model generation method according to any one of (14) to (16), wherein:
[0307] The first artificial intelligence is the machine learning algorithm.
[0308] (18) A learning model generation method according to any one of (14) to (17), wherein:
[0309] The first artificial intelligence is deep neural network, and
[0310] The output of a deep neural network is the output of a Green's function.
[0311] (19) The learning model generation method according to any one of (14) to (18), comprising:
[0312] In the three-dimensional virtual space, when the one or more three-dimensional objects are arranged between the sound receiving point and the sound source object so as to block the direct sound generated by the sound source object from reaching the position and range of the sound receiving point, the first artificial intelligence is generated which uses only the indirect sound from the sound source object as input for training as the acoustic data.
[0313] (20) Acoustic signal control program, which causes
[0314] computer
[0315] An acoustic signal control device for executing an acoustic signal control method, the acoustic signal control method comprising:
[0316] Selecting a sound source object to be arranged in the three-dimensional virtual space;
[0317] placing the selected sound source object at a predetermined position in the three-dimensional virtual space where one or more three-dimensional objects are placed; and
[0318] Position data and acoustic data related to the arranged sound source object, area data in the three-dimensional virtual space, structural data of the three-dimensional object, and position data of each object and sound receiving point at a predetermined time are input to the first artificial intelligence, and based on the impulse response at the sound receiving point generated according to the output data, the acoustic signal at the sound receiving point is output.
[0319] Reference Numbers List
[0320] 10. Virtual Space
[0321] 100 Sound generating device
[0322] 110 Communication unit
[0323] 120 storage units
[0324] 130 Control unit
[0325] 131 Input unit
[0326] 132 study units
[0327] 133 Arithmetic Unit
[0328] 134 Display control unit
[0329] 135 Output control unit
[0330] 140 Output Units
[0331] 150 Display
[0332] 160 Speakers
[0333] 200 Producers
[0334] 300 User interface.
Claims
1. An acoustic signal control method, comprising: By computer Select a sound source object to be arranged in the three-dimensional virtual space; Arranging the selected sound source object at a predetermined position in the three-dimensional virtual space where one or more three-dimensional objects are arranged; Position data and acoustic data related to the arranged sound source object, area data in the three-dimensional virtual space, structural data of the three-dimensional object, and position data of each object and sound receiving point at a predetermined time are input into the first artificial intelligence, and based on the impulse response at the sound receiving point generated according to the output data, the acoustic signal at the sound receiving point is output.
2. The acoustic signal control method according to claim 1, comprising: The acoustic signal is output based on the sound pressure at the sound receiving point calculated from the impulse response at the sound receiving point.
3. The acoustic signal control method according to claim 1, wherein: The area data in the three-dimensional virtual space is the boundary conditions of the space to be analyzed and the coordinate information forming the boundary of the three-dimensional virtual space.
4. The acoustic signal control method according to claim 1, wherein: The structural data of the three-dimensional object is the acoustic impedance set in the three-dimensional object and the coordinate information of the point group forming the three-dimensional object.
5. The acoustic signal control method according to claim 4, further comprising: obtaining input from a user requesting arrangement of the three-dimensional objects; and The structural data is determined based on a designation from the user regarding at least any one of material, transmittance, reflectance, and position data of the three-dimensional object included in an input of the request.
6. The acoustic signal control method according to claim 5, comprising: When an input of a request related to the arrangement of the three-dimensional object is obtained from a user, the acoustic signal at the sound receiving point is output, the acoustic signal being recalculated based on the structural data of the three-dimensional object after being changed by the request.
7. The acoustic signal control method according to claim 1, comprising: In the three-dimensional virtual space, when one or more three-dimensional objects are located between the sound receiving point and the sound source object, blocking the direct sound generated by the sound source object from reaching the position and range of the sound receiving point, the first artificial intelligence output generated by only considering the indirect sound from the sound source object is used to output the acoustic signal at the sound receiving point.
8. The acoustic signal control method according to claim 1, wherein: The first artificial intelligence is a machine learning algorithm.
9. The acoustic signal control method according to claim 1, wherein The first artificial intelligence is a deep neural network.
10. The acoustic signal control method according to claim 9, wherein The output of the deep neural network is the output of the Green's function.
11. The acoustic signal control method according to claim 1, further comprising: Obtaining a requested input related to output of an acoustic signal desired by a user; and When the input of the request is obtained, the output of the acoustic signal desired by the user is input to the first artificial intelligence, and at least any one of the information related to the three-dimensional virtual space or the three-dimensional object and the information related to the sound source object required to achieve the output of the acoustic signal desired by the user is output based on the data output through the inverse operation.
12. The acoustic signal control method according to claim 11, comprising: Output at least any one of changes in boundary conditions of the space to be analyzed, changes in structural data of three-dimensional objects that have already been arranged, and structural data of three-dimensional objects that should be newly arranged in the three-dimensional virtual space as information related to the three-dimensional virtual space or the three-dimensional objects.
13. The acoustic signal control method according to claim 11, comprising: Position data of the sound source object is output as information related to the sound source object, in which the output of the acoustic signal desired by the user at the sound receiving point is achieved.
14. A learning model generation method, comprising: By computer Select a sound source object to be arranged in the three-dimensional virtual space; placing the selected sound source object at a predetermined position on the three-dimensional virtual space where one or more three-dimensional objects are placed; and A first artificial intelligence is generated by learning based on predetermined teacher data, the first artificial intelligence taking as input acoustic data related to an arranged sound source object, area data in the three-dimensional virtual space, structural data of the three-dimensional object, and position data of each object and a sound receiving point at a predetermined time, and taking as output a transfer function indicating the relationship when the sound emitted from the sound source object is observed at the sound receiving point.
15. The learning model generation method according to claim 14, wherein: The predetermined teacher data is data acquired based on acoustic simulation in the three-dimensional virtual space or data observed in a real space.
16. The learning model generation method according to claim 15, comprising: The first artificial intelligence is generated by performing learning of the first artificial intelligence to minimize the sum of the error between the transfer function output from the first artificial intelligence and the main equation corresponding to the input, the error between the transfer function output from the first artificial intelligence and the transfer function output by inverting the position data of the arrangement of the sound source object and the position data of the sound receiving point, and the error between predetermined teacher data and the transfer function output from the first artificial intelligence.
17. The learning model generation method according to claim 16, wherein: The first artificial intelligence is a machine learning algorithm.
18. The learning model generation method according to claim 17, wherein: The first artificial intelligence is a deep neural network, and The output of the deep neural network is the output of the Green's function.
19. The learning model generation method according to claim 14, comprising: In the three-dimensional virtual space, when the one or more three-dimensional objects are arranged between the sound receiving point and the sound source object so as to block the direct sound generated by the sound source object from reaching the position and range of the sound receiving point, the first artificial intelligence is generated which uses only the indirect sound from the sound source object as input for training as the acoustic data.
20. An acoustic signal control program, computer An acoustic signal control device for executing an acoustic signal control method, the acoustic signal control method comprising: Selecting a sound source object to be arranged in the three-dimensional virtual space; placing the selected sound source object at a predetermined position in the three-dimensional virtual space where one or more three-dimensional objects are placed; and Position data and acoustic data related to the arranged sound source object, area data in the three-dimensional virtual space, structural data of the three-dimensional object, and position data of each object and sound receiving point at a predetermined time are input to the first artificial intelligence, and based on the impulse response at the sound receiving point generated according to the output data, the acoustic signal at the sound receiving point is output.
