Acoustic signal processing method, program, acoustic signal processing device, and acoustic signal reproduction system

Abstract

An acoustic signal processing method according to one embodiment of the present disclosure comprises: an acquisition step (S10) for acquiring an acoustic signal; a determination step (S20) for determining, on the basis of priority information indicating a priority associated with the acoustic signal, whether or not to execute a plurality of acoustic processes executed in accordance with a pipeline on the acoustic signal; and an acoustic processing step (S30) for executing the acoustic processing determined to be executed in the determination step on the acoustic signal.

Description

Technical Field

[0001] The present disclosure relates to an acoustic signal processing method, a program, an acoustic signal processing device, and an acoustic signal reproducing system. Background Art

[0002] Conventionally, there is a device that performs a predetermined sound processing on a sound signal, generates a sound (sound wave) based on the sound signal on which the sound processing has been performed, and provides the sound to a user. Patent document 1 discloses a device that performs a plurality of sound processing on a sound signal in a pipeline (pipeline processing).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2021 / 180938 Summary of the invention

[0006] Problem that the invention aims to solve

[0007] The present disclosure provides an audio signal processing method and the like that can appropriately perform multiple audio processings executed in a pipeline on an audio signal. In particular, the present disclosure provides an audio signal processing method that can appropriately perform pipeline processing even on devices with limited computing power, such as smartphones and head-mounted displays, without causing noticeable deterioration in sound quality and acoustic effects.

[0008] Means used to solve problems

[0009] A sound signal processing method according to one embodiment of the present disclosure is a sound signal processing method performed by a sound signal processing device, comprising: an acquisition step of acquiring a sound signal; a determination step of determining whether to perform a plurality of sound processings performed in a pipeline on the sound signal, based on priority information indicating a priority associated with the sound signal; and a sound processing step of performing the sound processing determined to be performed in the determination step on the sound signal.

[0010] A program according to one aspect of the present disclosure is a program for causing a computer to execute the above-mentioned acoustic signal processing method.

[0011] A sound signal processing device according to one embodiment of the present disclosure includes: an acquisition unit that acquires a sound signal; a determination unit that determines whether to execute a plurality of sound processings executed in accordance with a pipeline on the sound signal, based on priority information indicating a priority associated with the sound signal; and a sound processing unit that performs the sound processing determined by the determination unit on the sound signal.

[0012] An acoustic signal reproduction system according to one aspect of the present disclosure includes: the acoustic signal processing device described above; and a driver that generates a sound based on the acoustic signal on which the acoustic processing determined to be executed by the determination unit has been executed.

[0013] Effects of the Invention

[0014] According to the present disclosure, it is possible to provide an acoustic signal processing method and the like that can appropriately perform a plurality of acoustic processes executed in accordance with pipelines on an acoustic signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a block diagram showing the functional structure of the sound signal reproduction system according to the embodiment.

[0016] Figure 2 This is a diagram for explaining a plurality of sound processes executed in accordance with pipelines by the sound signal processing device according to the embodiment.

[0017] Figure 3 It is a diagram showing priority information of an implementation method.

[0018] Figure 4 This is a diagram showing a first example of a plurality of sound processes executed in accordance with pipelines by the sound signal processing device according to the embodiment.

[0019] Figure 5 This is a diagram showing a second example of a plurality of sound processes executed in accordance with pipelines by the sound signal processing device according to the embodiment.

[0020] Figure 6 This is a flowchart showing the processing procedure of the priority updating process of the acoustic signal processing device according to the embodiment.

[0021] Figure 7 This is a flowchart showing the processing procedure of the sound signal processing device according to the embodiment. DETAILED DESCRIPTION

[0022] (Insights that form the basis of this disclosure)

[0023] In recent years, the development of technologies related to virtual experience from the user's perspective, such as virtual reality (VR) or augmented reality (AR), has continued to advance. In VR or AR, users can personally feel (i.e., immerse themselves in) as if they were in a virtual space. In particular, in order to enhance the sense of immersion by combining a stereoscopic auditory experience with a stereoscopic visual experience, technologies related to a stereoscopic auditory experience are also valued in VR or AR. In order to achieve a stereoscopic auditory experience, in other words, in order to reproduce stereoscopic sound, audio processing (signal processing for an audio signal) is performed. At this time, sometimes multiple audio processing is performed on the audio signal according to the pipeline. In other words, sometimes multiple audio processing is performed on the audio signal one by one in sequence.

[0024] Here, conventionally, in an apparatus that executes a plurality of sound processing in a pipeline, there is a problem that the sound processing executed on the sound signal does not change depending on, for example, the type of device used by the user or the load state of the device, etc. Therefore, conventionally, there is a problem that, for example, in some cases, sound processing that is not desirable to be executed is unnecessarily executed, and the plurality of sound processing executed in a pipeline cannot be appropriately executed on the sound signal according to the situation.

[0025] In view of the above-mentioned problems, the inventors of the present application created the present disclosure.

[0026] Hereinafter, the embodiments will be described in detail with reference to the drawings.

[0027] In addition, the embodiments described below are all general or specific examples. The numerical values, shapes, materials, constituent elements, configuration positions and connection forms of constituent elements, steps, step sequences, etc. shown in the following embodiments are all examples, and their purport is not to limit the present disclosure. In addition, among the constituent elements in the following embodiments, constituent elements not recorded in the independent technical solutions of the present disclosure will be described as arbitrary constituent elements. In addition, the figures are not necessarily strictly shown. In each figure, the same figure numerals are marked for substantially the same structure, and repeated descriptions are sometimes omitted or simplified.

[0028] (Implementation Method)

[0029] [structure]

[0030] First, the configuration of the sound signal reproduction system according to the embodiment will be described.

[0031] Figure 1 2 is a block diagram showing a functional configuration of an acoustic signal reproduction system 200 according to an embodiment.

[0032] The sound signal reproduction system 200 is a system that applies sound processing to a sound signal and generates sound based on the sound signal to which the sound processing is applied. Specifically, the sound signal reproduction system 200 is a system that reproduces stereo sound by performing a plurality of sound processing on the sound signal in accordance with a pipeline. For example, the sound signal reproduction system 200 is used together with a display device that displays an image for visual VR or AR experience and a device for stereoscopic image reproduction (both not shown).

[0033] The sound signal reproduction system 200 is implemented, for example, by a computer such as a personal computer or a smart phone and a head mounted display (HMD) not shown worn by a user. In addition, in the sound signal reproduction system 200 having a function as a renderer that generates a sound signal with sound effects added, all or part of the functions of the renderer may be assumed by a server. That is, all or part of the acquisition unit 110, the determination unit 120, the sound processing unit 130, and the storage unit 140 may also exist in a server not shown. In this case, the sound signal reproduction system 200 is implemented, for example, by combining a computer such as a personal computer or a smart phone, an HMD not shown worn by a user, and a server not shown. In addition, the computer, the HMD, and the server may be connected to each other in a communicative manner using the same network, or may be connected to each other in different networks. In the case of connecting to different networks, since the possibility of communication delay increases, it is also possible to allow the server to perform processing only when the computer, the HMD, and the server are connected to each other in a communicative manner using the same network. In addition, according to the amount of data of the bit stream received by the sound signal reproduction system 200, it can be determined whether the server assumes all or part of the functions of the renderer.

[0034] The sound signal reproduction system 200 changes the presentation sound according to the user's head movement, so that the user perceives that the user is moving his head in the three-dimensional sound field. Therefore, the sound signal reproduction system 200 moves the three-dimensional sound field in the opposite direction to the user's movement.

[0035] like Figure 1 As shown, the sound signal reproduction system 200 includes the sound signal processing device 100 , a communication module 210 , a sensor 220 , and a driver 230 .

[0036] The sound signal processing device 100 is a computing device for performing various signal processing in the sound signal reproduction system 200. The sound signal processing device 100 includes a processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor) and a memory, and performs various functions by executing a program stored in the memory by the processor. The functional structure of the sound signal processing device 100 will be described later.

[0037] The acoustic signal processing device 100 is communicably connected to the communication module 210 , the sensor 220 , and the driver 230 .

[0038] The communication module 210 is an interface device for receiving a bit stream to the sound signal reproduction system 200. The communication module 210 includes, for example, an antenna and a signal converter, and receives a bit stream from an external device such as a server by wireless communication. Specifically, the communication module 210 uses the antenna to receive a wireless signal such as a bit stream containing various information such as a sound signal converted into a format for wireless communication, and the signal converter performs reconversion from the wireless signal to various information such as the sound signal. In this way, the sound signal processing device 100 (more specifically, the acquisition unit 110) acquires various information such as the sound signal from the external device by wireless communication.

[0039] In addition, the communication between the sound signal reproduction system 200 and the external device may be performed by wired communication. In addition, the input data may be stored in a storage device such as the storage unit 140 .

[0040] The sensor 220 is a device for detecting the position of the user. Specifically, the sensor 220 detects 6DoF (position (x, y, z) and angle (yaw, pitch, roll) as the position of the user. The sensor 220 is composed of a combination of various sensors for detecting motion, such as a gyro sensor and an acceleration sensor. For example, when the sound signal reproduction system 200 is implemented by a smartphone, the sensor 220 is built into the smartphone.

[0041] In addition, the sensor 220 may be built into an external device such as an HMD that operates according to the movement of the user's head. In addition, as the sensor 220, an external camera or the like may be used to capture the movement of the user's head, and the movement of the user's head may be detected by processing the captured image.

[0042] The sensor 220 moves with the user and detects the speed and amount of the user's motion. For example, when built in an HMD, the sensor 220 detects the speed and amount of the user's head motion.

[0043] For example, as the speed of movement of the user's head, the sensor 220 can detect the angular velocity of rotation with at least one of the three axes orthogonal to each other in the sound space as the rotation axis, and can also detect the acceleration of displacement with at least one of the above three axes as the displacement direction.

[0044] As the amount of movement of the user's head, the sensor 220 can, for example, detect the amount of rotation with at least one of the three axes orthogonal to each other in the sound space as the rotation axis, or detect the amount of displacement with at least one of the above three axes as the displacement direction.

[0045] In addition, the sensor 220 may be able to detect the user's location, which may be achieved through a camera or a GPS (Global Positioning System) receiver, etc.

[0046] In addition, the sensor 220 may include a thermometer such as a thermocouple for detecting the temperature of the sound signal processing device 100, and a sensor for detecting the remaining amount of a battery connected to the sound signal processing device 100. The acquisition unit 110 may acquire information indicating the temperature of the sound signal processing device 100 and information indicating the remaining amount of the battery from the sensor 220.

[0047] The driver 230 is a device that generates sound (sound waves) based on an acoustic signal. Specifically, the driver 230 generates sound based on an acoustic signal that has been subjected to an acoustic process determined to be executed by the determination unit 120 described below. The driver 230 is implemented, for example, by earphones or headphones.

[0048] Next, the configuration of the sound signal processing device 100 will be described.

[0049] The sound signal processing device 100 includes an acquisition unit 110 , a determination unit 120 , and a sound processing unit 130 .

[0050] The acquisition unit 110 is a processing unit that acquires various information such as an acoustic signal and priority information. The acquisition unit 110 is configured to be able to communicate with the communication module 210 , for example, and acquires a bit stream including an acoustic signal and the like from an external device via the communication module 210 .

[0051] The bitstream includes, for example, an audio signal, priority information, and audio object information.

[0052] The driver 230 generates sound based on the acoustic signal on which the acoustic processing has been performed by the acoustic signal processing device 100 .

[0053] The priority information is information indicating the priority of each of the plurality of sound processings executed by the sound processing unit 130 in accordance with the pipeline in association with the sound signal. For example, when there are a plurality of sound signals acquired by the acquisition unit 110, the priority information includes information indicating a plurality of priorities corresponding to the plurality of sound signals. The priority is expressed, for example, by a numerical value. For example, the priority information indicates the priority individually for each of the plurality of sound signals.

[0054] In addition, the audio signal may be monophonic or multi-channel such as stereophonic. That is, the bitstream may include two or more audio signals each having monophonic audio information, or may include one or more multi-channel audio signals having multi-channel audio information.

[0055] The audio object information is information related to an object (audio object) that generates sound based on an acoustic signal, i.e., reproduces the acoustic signal. The audio object information includes, for example, information indicating the position of the audio object arranged in the sound space, information related to the orientation of the audio object, information related to the directionality of the sound emitted by the audio object, information indicating whether the audio object is biological information, and information indicating whether the audio object is a moving object. For example, the acoustic signal corresponds to one or more audio objects indicated by the audio object information.

[0056] The acquisition unit 110 also acquires position information indicating the position of the user from the sensor 220. By acquiring the audio object information included in the bitstream and the position information acquired from the sensor 220, the acquisition unit 110 acquires spatial information including the audio object information and the position information.

[0057] For example, in the acoustic signal processing apparatus 100 , conversion of an acoustic signal is performed so that a sound emitted by an audio object is perceived as a sound reaching a listening point (specifically, a user's position in an acoustic space) from the audio object.

[0058] Additionally, the position and orientation of the listening point may vary, for example in coordination with the movement of the user's head.

[0059] In addition, although the audio object information is described as information related to an object that generates an acoustic signal, the bitstream includes not only sound source objects such as audio objects, but also information related to non-sound source objects that do not generate an acoustic signal. In addition, the priority information, the audio object information, and the non-sound source object information can be referred to as meta-information as information for controlling acoustic processing for an acoustic signal in the acoustic signal reproduction system 200.

[0060] In addition, the meta information may be information for describing a scene expressed in the sound space. Here, the scene is a term for a collection of all elements representing a three-dimensional image and an audio event in the sound space, which is modeled by the sound signal reproduction system 200 using the meta information. That is, the meta information here may include not only information for controlling the sound processing, but also information for controlling the image processing. Of course, the meta information may include information for controlling only one of the sound processing and the image processing, or may include information for controlling both.

[0061] The sound signal reproduction system 200 uses the meta-information included in the bit stream and the position information of the interacting user obtained by appending, etc. to perform sound processing on the sound information (sound signal), thereby generating a virtual sound effect. In this embodiment, the case of performing initial reflection processing, obstacle processing, entrance processing, blocking processing and reverberation processing in the sound effect is described, but other sound processing can also be performed using the meta-information. For example, consider adding sound effects such as distance attenuation effect, positioning and Doppler effect. In addition, information on switching on and off all or part of the sound effects and priority information of these sound effects can be added to the bit stream as meta-information.

[0062] In addition, all or part of the meta-information may be obtained from outside the bit stream of the sound information. For example, one of the meta-information for controlling the sound and the meta-information for controlling the image may be obtained from outside the bit stream, or both of the meta-information may be obtained from outside the bit stream. In addition, when the meta-information for controlling the image is included in the bit stream obtained by the sound signal reproduction system 200, the sound signal reproduction system 200 may have a function of outputting the meta-information that can be used to control the image to a display device that displays an image, or a stereoscopic image reproduction device that reproduces a stereoscopic image.

[0063] In addition, as an example, the encoded meta-information includes information related to the sound space including the audio object and the obstacle object, and information related to the positioning position when the sound image of the sound is positioned at a specified position in the sound space (that is, perceived as sound arriving from a specified direction). Here, an obstacle object is an object that can affect the sound perceived by the user, such as by blocking the sound or reflecting the sound, before the sound emitted by the audio object reaches the user. In addition to stationary objects, obstacle objects may include animals such as humans or moving objects such as machines. In addition, when there are multiple audio objects in the sound space, for any audio object, other audio objects may become obstacle objects. Both non-sound sources such as building materials and non-living things and sound source objects that emit sound may be obstacle objects.

[0064] The meta-information includes all or part of information respectively indicating the shape of the sound space, the shape and position of obstacle objects existing in the sound space, the shape and position of audio objects existing in the sound space, and the position and orientation of the user in the sound.

[0065] The sound space can be either a closed space or an open space. In addition, the meta-information includes information indicating the reflectivity of structures such as floors, walls or ceilings that may reflect sound in the sound space, and the reflectivity of obstacle objects present in the sound space. Here, the reflectivity is the energy ratio between the reflected sound and the incident sound, and is set for each frequency band of the sound. Of course, the reflectivity can be uniformly set regardless of the frequency band of the sound. In the case where the sound space is an open space, parameters such as uniformly set attenuation rate, diffracted sound, initial reflected sound, etc. can be used as meta-information.

[0066] In the above description, reflectivity is cited as a parameter related to the obstacle object and the audio object included in the meta information, but the meta information may also include information other than reflectivity. For example, the meta information may include information related to the material of the object as information related to both the sound source object and the non-sound source object. Specifically, the meta information may include parameters such as diffusivity, transmittance, and sound absorption coefficient.

[0067] In addition, as information related to the sound source object, meta-information may include volume, radiation characteristics (directivity), reproduction conditions, the number and type of sound sources emitted from an object, and information on the sound source area in the specified object. In the reproduction conditions, for example, it can be determined whether it is a continuous sound or a sound triggered by an event. The sound source area in the object can be determined by the relative relationship between the user's position and the position of the object, or it can be determined based on the object. In the case of determining by the relative relationship between the user's position and the position of the object, based on the side where the user sees the object, the user can perceive that if viewed from the user's perspective, sound A is emitted from the right side of the object and sound B is emitted from the left side. In the case of determining based on the object, it can be made fixed what kind of sound is emitted from which area of ​​the object regardless of the direction in which the user is viewing. For example, the user can perceive that the high sound comes from the right side when viewing the object from the front, and the low sound comes from the left side. In this case, when the user goes around to the back of the object, the user can perceive that the low sound comes from the right side and the high sound comes from the left side when viewed from the back.

[0068] Meta-information related to the space may include time until initial reflected sound, reverberation time, ratio of direct sound to diffuse sound, etc. When the ratio of direct sound to diffuse sound is zero, the user can perceive only the direct sound.

[0069] In addition, the sound information and the meta information can be stored in one bit stream or in multiple bit streams respectively. Similarly, the sound information and the meta information can be stored in one file or in multiple files separately.

[0070] In the case where the sound information and meta-information are stored separately in a plurality of bitstreams or a plurality of files, information indicating other bitstreams or files associated with one or a part of the bitstreams or one or a part of the files may be included, and information indicating other bitstreams or files respectively associated with all the bitstreams or files may be included in the meta-information. Here, the associated bitstream or file refers to, for example, a bitstream or file that may be used simultaneously during sound processing. In addition, the information received by the sound signal processing device 100 may include a bitstream or file in which information indicating other associated bitstreams or files is summarized and recorded. Here, the information indicating other associated bitstreams or files refers to, for example, an identifier indicating other bitstreams, a file name indicating other files, or a URL (Uniform Resource Locator), a URI (Uniform Resource Identifier), etc. In this case, the acquisition unit 110 specifies or acquires a bitstream or file based on the information indicating other associated bitstreams or files. In addition, the bitstream may also include information indicating other associated bitstreams and information indicating a bitstream or file associated with another bitstream or file. Here, the file including information indicating the associated bitstream or file may be a control file such as a manifest file used for content distribution.

[0071] In addition, for example, the acquisition unit 110 acquires device information related to the state of the sound signal processing device 100. The device information includes, for example, at least one of information indicating the remaining amount of a battery for driving the sound signal processing device 100, information indicating the amount of heat generated by the sound signal processing device 100, and information indicating the processing capacity of the sound signal processing device 100. The acquisition unit 110 acquires the device information from, for example, an external device such as the sensor 220, the storage unit 140, or a server.

[0072] The decision unit 120 is a processing unit that determines whether to execute a plurality of sound processings executed in accordance with the pipeline on each sound signal based on priority information indicating a priority associated with the sound signal. For example, the priority information indicates the priority of each of the plurality of sound processings. The decision unit 120 determines whether to execute each of the plurality of sound processings on the sound signal based on whether the priority of each of the plurality of sound processings is greater than or equal to a threshold value. The threshold value is, for example, represented by a numerical value. The decision unit 120 determines whether the priority of each of the plurality of sound processings is greater than or equal to the threshold value, for example, by comparing the priority value with the numerical value of the threshold value.

[0073] The threshold value can be determined arbitrarily without particular limitation. For example, information indicating the threshold value is stored in the storage unit 140 .

[0074] In addition, the information indicating the threshold value may be included in the bit stream together with the sound signal, etc., or may be acquired from an external device such as a server.

[0075] Furthermore, the determination unit 120 may calculate the threshold value based on a predetermined condition. For example, the determination unit 120 may determine the threshold value based on device information related to the state of the acoustic signal processing apparatus 100 .

[0076] The sound processing unit 130 is a processing unit that performs a predetermined sound processing on the sound signal. Specifically, the sound processing unit 130 performs the sound processing determined by the determination unit 120 to be performed on the sound signal. More specifically, based on the sound signal and the spatial information acquired by the acquisition unit 110, the sound processing unit 130 performs processing so that a sound based on the sound signal is generated in the sound space from a predetermined position to a predetermined direction with a predetermined directivity, and the sound is generated by the audio object indicated by the spatial information. For example, the sound processing unit 130 performs processing to convert the acquired sound signal as a digital signal into an analog signal, and outputs the converted sound signal. For example, the sound signal is output from the sound processing unit 130 to the driver 230, and the driver 230 emits a sound based on the sound signal.

[0077] Each processing unit such as the acquisition unit 110 , the determination unit 120 , and the sound processing unit 130 is realized by, for example, a processor and a memory storing a control program executed by the processor.

[0078] The storage unit 140 is a storage device that stores various information such as thresholds, etc. The storage unit 140 also stores a HRIR DB (HRIR database) representing HRIR (Head-Related Impulse Response).

[0079] HRIR is the response characteristic when one impulse is generated. Specifically, HRIR is the response characteristic converted from the frequency domain to the time domain by Fourier transforming the head transfer function, which expresses the change of the sound generated by the surrounding objects including the auricle, the head and the shoulders as a transfer function. HRIR DB is a database including such information.

[0080] The storage unit 140 is implemented by, for example, a HDD (Hard Disk Drive) or a flash memory.

[0081] [Processing order]

[0082] Next, the processing procedure of the acoustic signal processing device 100 will be described.

[0083] Figure 2 This is a diagram for explaining a plurality of sound processes executed in accordance with pipelines by the sound signal processing device 100 according to the embodiment.

[0084] First, the acquisition unit 110 acquires a bit stream including a plurality of sound signals, priority information and audio object information associated one-to-one with the sound signals from an external device such as a server via the communication module 210. In addition, the acquisition unit 110 acquires position information indicating the position of the user, for example, from the sensor 220. In addition, for example, the acquisition unit 110 acquires device information from the storage unit 140 or the like.

[0085] Based on the priority information acquired by the acquisition unit 110 , the determination unit 120 determines whether to execute each of the plurality of audio processes executed by the audio processing unit 130 in accordance with the pipeline.

[0086] For example, the sound processing unit 130 performs an initial reflection process (S110), an obstacle process (S120), an entrance process (S130), an interruption process (S140), and a reverberation process (S150) on the sound signal according to the pipeline. Specifically, the sound processing unit 130 performs an initial reflection process, an obstacle process, an entrance process, an interruption process, and a reverberation process on the sound signal in this order. For the sound signal, the initial reflection process, the obstacle process, the entrance process, the interruption process, and the reverberation process are examples of sound processing.

[0087] The propagation of sound includes, for example, initial reflection, obstruction, portaling, occlusion, and reverberation. In sound processing, processing corresponding to the propagation of these sounds is performed on the sound signal.

[0088] For example, in the initial reflection processing, processing of the sound reflected by a wall or the like is performed.

[0089] In addition, for example, in obstacle processing, processing of sound diffracted by the surroundings of the obstacle is performed.

[0090] Furthermore, for example, in entrance processing, processing of the sound that passes through the wall accompanied by diffraction from a door or the like is performed.

[0091] In addition, for example, in the blocking process, processing of the sound propagating through the wall is performed.

[0092] In addition, for example, in reverberation processing, processing of sound reverberating in a room is performed.

[0093] Figure 3is a diagram showing priority information of an implementation method. Figure 3 In the example, (A) to (D) represent different sound signals, (1) to (5) represent different sound processing, and the numerical value in the rectangular box represents the priority. For example, (1) represents initial reflection processing, (2) represents obstacle processing, (3) represents entrance processing, (4) represents blocking processing, and (5) represents reverberation processing. In this example, the priority is set, for example, 0 is low and 3 is high (for example, required processing), and the larger the numerical value, the higher the priority of the sound processing to be executed.

[0094] For example, it is assumed that the acquisition unit 110 acquires the acoustic signals (A) to (D). Figure 3 In the example shown, the priority information indicates that, for example, for the sound signal (A), the priority of the initial reflection processing is 2, the priority of the obstacle processing is 2, the priority of the entrance processing is 3, the priority of the blockage processing is 3, and the priority of the reverberation processing is 3. In addition, the priority information indicates that, for example, for the sound signal (B), the priority of the initial reflection processing is 1, the priority of the obstacle processing is 1, the priority of the entrance processing is 0, the priority of the blockage processing is 1, and the priority of the reverberation processing is 2.

[0095] In addition, the above-mentioned priority information is determined based on the following insights. For example, the initial reflection processing represented by the sound processing (1) sets the priority of the sound signal including the attack sound whose time from the beginning of the sound to the maximum volume is short to high. In addition, in the obstacle processing of the sound processing (2) and the blocking processing of (4), the sound signal whose main part of the energy is in the low frequency band is set to high priority. In addition, the priority of the sound processing also depends on the image information presented at the same time as the sound signal. For example, when the image information is a stone block with a church, the priority of the reverberation processing for the church bell is set to high, etc.

[0096] Priority information may be given to the acoustic signals (A) to (D) in advance, or the acoustic signals (A) to (D) may be analyzed and the priority information may be calculated when the method, device or system using the inventive technique is started.

[0097] Thus, the priority information indicates the priority of the plurality of sound processing (1) to (5) for each sound signal (A) to (D). For example, the decision unit 120 decides whether to execute the plurality of sound processing (1) to (5) for each sound signal (A) to (D) based on the priority information.

[0098] Figure 4 1 is a diagram showing a first example of a plurality of sound processings executed in a pipeline by the sound signal processing device 100 according to the embodiment. Figure 4The example shown is a diagram showing an example of a case where the acquisition unit 110 repeatedly acquires the acoustic signals (A) to (D). In addition, it is assumed that the priorities associated with the acoustic signals (A) to (D) are Figure 3 The priority information shown in the table shows the priority. Figure 4 In the figure, among (1) to (5) indicating the sound processing included in the sound signals (A) to (D), the processing without a shaded line indicates that the corresponding sound processing is performed on the sound signal, and the processing with a shaded line indicates that the corresponding sound processing is not performed on the sound signal.

[0099] The sound processing unit 130 determines whether to perform sound processing on the sound signal based on the sound signal and the spatial information, for example. For example, depending on the type of the sound signal or the positional relationship between the user and the driver 230, there is sound processing that is not originally necessary. Therefore, the sound processing unit 130 determines whether to perform sound processing (1) to (5) on the sound signals (A) to (D), for example, based on the sound signal and the spatial information. In some cases, depending on the determination result, some or all of the sound processing (1) to (5) are not performed on the sound signals (A) to (D) (processing is turned off based on processing control).

[0100] exist Figure 4 In the example shown, for the sound signals (A) to (D) acquired by the acquisition unit 110 at time t0, the sound processing (3) is performed on each of the sound signals (A) to (C), and the other sound processing is performed on the sound signals (A) to (D). In addition, for example, for the sound signals (A) to (D) acquired by the acquisition unit 110 at time t1, the sound processing (3) is not performed on each of the sound signals (A) and (C), and the other sound processing is performed on the sound signals (A) to (D).

[0101] Furthermore, the sound processing unit 130 performs or does not perform a plurality of sound processing on the sound signal based on the result of the determination by the determination unit 120 based on the priority information.

[0102] For example, if the priority is greater than the threshold, the decision unit 120 instructs the audio processing unit 130 to perform the audio processing corresponding to the priority, and if the priority is less than the threshold, the decision unit 120 instructs the audio processing unit 130 not to perform the audio processing corresponding to the priority ( Figure 2 That is, as the threshold increases, the amount of audio processing that is not performed increases. In other words, as the threshold increases, the amount of processing (amount of calculation) decreases. Figure 4 In the example shown, the priority is 0 at any time point from time t0 to time point t7, and therefore there is no audio processing that is not executed based on the priority level.

[0103] Figure 5 2 is a diagram showing a second example of a plurality of sound processings executed in a pipeline by the sound signal processing device 100 according to the embodiment. Figure 5 The example shown is a diagram showing an example of a case where the acquisition unit 110 repeatedly acquires the acoustic signals (A) to (D). In addition, it is assumed that the priorities associated with the acoustic signals (A) to (D) are Figure 3 The priority information shown in the table shows the priority. Figure 5 In the figure, among (1) to (5) indicating the sound processing included in the sound signals (A) to (D), respectively, the processing without a hatched line indicates that the corresponding sound processing is performed on the sound signal, and the processing with a hatched line (slashed hatching and dotted hatching) indicates that the corresponding sound processing is not performed on the sound signal.

[0104] exist Figure 5 In the example shown, Figure 4 Similarly to the example shown in the figure, the sound processing unit 130 determines whether to perform the sound processing (1) to (5) on the sound signals (A) to (D) based on the sound signal and the spatial information, and in some cases, based on the determination result, does not perform part or all of the sound processing (1) to (5) on the sound signals (A) to (D). Figure 5 The process control shown by the diagonal hatching is turned off).

[0105] In addition, Figure 5 In the example shown, the threshold value gradually changes as the time t0 to t7 approaches. For example, the decision unit 120 determines the threshold value based on the device information. For example, the device information may also include information indicating the amount of heat generated by the sound signal processing device 100. When the sound signal processing device 100 continues to be used or a high load is applied by processing, the amount of heat generated by the sound signal processing device 100 increases. In addition, for example, when the sound signal processing device 100 continues to be used, the remaining amount of the battery connected to the sound signal processing device 100 and used to drive the sound signal processing device 100 decreases. For example, the decision unit 120 changes the threshold value so that the threshold value becomes higher as the amount of heat generated increases or as the remaining amount of the battery decreases. That is, the decision unit 120 changes the threshold value so that the number of unexecuted sound processings among the multiple sound processings increases as the amount of heat generated increases or the remaining amount of the battery decreases.

[0106] exist Figure 5 In the example shown, the sound signals (A) to (D) acquired by the acquisition unit 110 at time t0 are Figure 4 In the example shown, the sound processing (3) is not performed on each of the sound signals (A) to (C), and other sound processing is performed on the sound signals (A) to (D).

[0107] For example, regarding the acoustic signals (A) to (D) acquired by the acquisition unit 110 at time t1, Figure 4 In the example shown, processing (3) is not performed on the sound signals (A) and (C). Figure 5 In the example shown, Figure 4 Unlike the example shown, the acoustic processing (3) is not performed on the acoustic signal (B) based on the priority information.

[0108] In addition, for example, the threshold value initially set may be arbitrary. The threshold value may be predetermined, or may be determined based on, for example, the performance of the sound signal processing device 100 (for example, the performance of a processor included in the sound signal processing device 100). For example, the sound signal processing device 100 sets the threshold value so that the threshold value becomes lower as the performance of the sound signal processing device 100 becomes higher.

[0109] In addition, the priority information (that is, the priority indicated by the priority information) can be changed.

[0110] Figure 6 Detailed Description of the Invention This is a flowchart showing the processing procedure of the priority updating process performed by the acoustic signal processing device 100 according to the embodiment.

[0111] First, the acquisition unit 110 acquires priority information ( S210 ).

[0112] In addition, the acquisition unit 110 acquires spatial information (S220). Steps S210 and S220 are performed, for example, by acquiring a bit stream and position information. Steps S210 and S220 may be performed in any order.

[0113] Next, the decision unit 120 calculates the positional relationship between the user and the audio object based on the spatial information (S230). For example, the decision unit 120 calculates the position of the user in the sound space corresponding to the position of the user in the real space, and calculates the relative positional relationship between the position of the user in the sound space and the audio object (S230).

[0114] Next, the decision unit 120 updates the priority information stored in the storage unit 140 based on the information ( S240 ).

[0115] In addition, when the priority information is not stored in the storage unit 140 , the determination unit 120 may determine the priority based on the information. The determined priority information indicating the priority may be stored in the storage unit 140 .

[0116] Refer again Figure 2The sound processing unit 130 performs the sound processing determined to be performed as described above on the sound signal, and further performs binauralization on the sound signal (S160). Specifically, the sound processing unit 130 performs the following signal processing on the sound signal, applies the HRIR DB based on the coordinates and orientation of the user in the sound space (i.e., the position and orientation of the listening point) so that the sound reaches the user from the position of the audio object, thereby making the sound reaching the user from the position of the audio object configured at a predetermined position in the sound space a reproducible sound signal.

[0117] Then, the sound processing unit 130 outputs the sound signal subjected to the signal processing to the driver 230 (S170). Thus, the sound processing unit 130 causes the driver 230 to generate a sound wave based on the waveform signal indicated by the sound signal, and presents the sound wave to the user.

[0118] The sound signal processing device 100 described above performs the following processing.

[0119] Figure 7 This is a flowchart showing the processing procedure of the sound signal processing device 100 according to the embodiment.

[0120] First, the acquisition unit 110 acquires an acoustic signal ( S10 ).

[0121] Next, the decision unit 120 decides whether to execute a plurality of acoustic processes executed in accordance with pipelines on each acoustic signal based on priority information indicating a priority associated with the acoustic signal ( S20 ).

[0122] Next, the sound processing unit 130 performs the sound processing determined to be performed by the determination unit 120 on the sound signal ( S30 ).

[0123] The sound signal on which the sound processing determined to be executed by the determination unit 120 is executed is output to the driver 230 , for example.

[0124] In addition, in step S20, when the decision unit 120 determines not to execute all of the plurality of audio processing, the audio processing unit 130 may output the audio signal to the driver 230 without executing the audio processing. In this case, for example, the audio processing unit 130 executes steps S160 and S170 instead of executing Figure 2 Steps S110 to S150 are shown.

[0125] [Effects, etc.]

[0126] Hereinafter, the techniques obtained from the disclosure of this specification are exemplified, and the effects and the like obtained by the exemplified techniques are described.

[0127] Technique 1 is a sound signal processing method performed by a sound signal processing device 100, which includes: an acquisition step (S10) of acquiring a sound signal; a determination step (S20) of determining whether to perform a plurality of sound processings performed in a pipeline on each sound signal based on priority information indicating a priority associated with the sound signal; and a sound processing step (S30) of performing the sound processing determined to be performed in the determination step on the sound signal.

[0128] In addition, the plurality of sound processing include, for example, an initial reflection processing ( S110 ), an obstacle processing ( S120 ), an entrance processing ( S130 ), an interruption processing ( S140 ), and a reverberation processing ( S150 ).

[0129] Thus, based on the priority information, one or more of the multiple sound processings executed according to the pipeline are executed on the sound signal, or all the sound processing is not executed. Therefore, for example, by setting the priority indicated by the priority information based on the user's preference, the intention of the producer of the music based on the sound signal, or the state of the device that generates the sound based on the sound signal, appropriate sound processing corresponding to them is executed on the sound signal. Therefore, according to the sound signal processing method of one embodiment of the present invention, it is possible to appropriately execute multiple sound processings executed according to the pipeline on the sound signal. For example, according to the sound signal processing method of one embodiment of the present invention, in order not to execute unnecessary sound processing, the processing amount related to the sound processing can be reduced.

[0130] Technology 2 is the sound signal processing method described in Technology 1, wherein the priority information indicates the priority of each of the plurality of sound processings, and in the determination step, based on whether the priority of each of the plurality of sound processings is above a threshold, it is determined whether to execute each of the plurality of sound processings on the sound signal.

[0131] For example, the priority and the threshold are respectively represented by numerical values.

[0132] In this way, by comparing the priority and the threshold, it is possible to easily determine whether to execute each of the plurality of acoustic processes on the acoustic signal.

[0133] Technique 3 is the sound signal processing method according to Technique 2, wherein in the determination step, the threshold value is determined based on device information related to a state of the sound signal processing device.

[0134] Thus, a threshold value is determined according to the state of sound signal processing device 100 . Therefore, whether or not to execute each of the plurality of sound processes on the sound signal can be easily determined according to the state of sound signal processing device 100 .

[0135] Technique 4 is the sound signal processing method described in Technique 3, wherein the device information includes at least one of information indicating the remaining amount of a battery for driving the sound signal processing device 100, information indicating the amount of heat generated by the sound signal processing device 100, and information indicating the processing capability of the sound signal processing device 100.

[0136] Thus, whether or not to execute each of the plurality of sound processings on the sound signal can be easily determined based on at least one of the remaining amount of a battery for driving the sound signal processing device 100, the amount of heat generated by the sound signal processing device 100, and the processing capacity of the sound signal processing device 100, which are particularly related to the sound processing, among the states of the sound signal processing device 100.

[0137] Technique 5 is the acoustic signal processing method according to Techniques 1 to 4, wherein in the determination step, the priority indicated by the priority information is determined based on position information indicating the position of the user of acoustic signal processing apparatus 100 .

[0138] In order to provide better sound to the user, the important sound processing among the multiple sound processing is changed according to the positional relationship between the user and the audio object. Therefore, by changing the priority based on the position of the user, the particularly important sound processing can be appropriately executed.

[0139] Technique 6 is a program for causing a computer to execute the sound signal processing method described in any one of Techniques 1 to 5.

[0140] This achieves the same effects as the acoustic signal processing method according to one embodiment of the present disclosure.

[0141] Technology 7 is an audio signal processing device 100, which includes: an acquisition unit 110 for acquiring an audio signal; a determination unit 120 for determining whether to perform multiple audio processings executed in a pipeline on each audio signal based on priority information indicating a priority associated with the audio signal; and an audio processing unit 130 for performing the audio processing determined by the determination unit 120 on the audio signal.

[0142] This achieves the same effects as the acoustic signal processing method according to one embodiment of the present disclosure.

[0143] Technique 8 is an acoustic signal reproduction system 200 including: the acoustic signal processing device 100 described in Technique 7; and a driver 230 that generates sound based on an acoustic signal that has been subjected to the acoustic processing determined to be executed by the determination unit 120 .

[0144] Thus, the user is provided with sound based on the sound signal on which the sound processing has been appropriately performed by the sound signal processing device 100 .

[0145] (Other embodiments)

[0146] As mentioned above, although embodiment was described, this disclosure is not limited to the said embodiment.

[0147] For example, the sound signal reproduction system 200 described in the above embodiment may be implemented as a single device having all the components, or each function may be assigned to a plurality of devices and implemented by the cooperation of these plurality of devices. In the latter case, for example, as a device corresponding to the processing module, in the sound signal processing device 100, an information processing device such as a smartphone, a tablet terminal, or a personal computer may be used.

[0148] Furthermore, for example, the sound signal processing device 100 can be realized as a sound signal processing device that is connected to a playback device that includes only the driver 230 and performs only sound processing on the acquired sound signal and outputs it to the playback device. In this case, the sound signal processing device 100 can be realized as hardware including a dedicated circuit or as software for causing a general-purpose processor to execute specific processing.

[0149] Furthermore, in the above-described embodiment, a process executed by a specific processing unit may be executed by another processing unit. Furthermore, the order of a plurality of processes may be changed, or a plurality of processes may be executed in parallel.

[0150] In the above embodiments, each component can be implemented by executing a software program suitable for each component. Each component can be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0151] In addition, each component element can be implemented by hardware. For example, each component element can be a circuit (or integrated circuit). These circuits can constitute a circuit as a whole, or they can be separate circuits. In addition, these circuits can be general circuits or dedicated circuits.

[0152] In addition, the general or specific aspects of the present disclosure may be implemented by a device, a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM. In addition, the general or specific aspects of the present disclosure may be implemented by any combination of a device, a system, a method, an integrated circuit, a computer program, and a recording medium.

[0153] For example, the present disclosure may be implemented as sound signal processing information executed by a computer, or may be implemented as a program for causing a computer to execute the sound signal processing method. The present disclosure may be implemented as a computer-readable non-transitory recording medium having such a program recorded thereon.

[0154] In addition, the present invention also includes methods obtained by implementing various conceivable modifications to the embodiments by those skilled in the art, or methods achieved by arbitrarily combining constituent elements and functions in the embodiments without departing from the gist of the present disclosure.

[0155] Industrial Applicability

[0156] The present disclosure is useful for audio devices.

[0157] Description of Reference Numerals

[0158] 100 Audio signal processing device

[0159] 110 Acquisition Department

[0160] 120 Decision Department

[0161] 130 Sound Processing Department

[0162] 140 Storage

[0163] 200 sound signal reproduction system

[0164] 210 communication module

[0165] 220 Sensors

[0166] 230 Drive

Claims

1. A method for processing an audio signal, performed by an audio signal processing device, wherein: include: Obtaining steps, obtaining an audio signal; A determination step of determining whether to perform a plurality of sound processings executed in accordance with pipelines on the sound signals, respectively, based on priority information indicating a priority associated with the sound signals; as well as The sound processing step performs the sound processing determined to be performed in the determination step on the sound signal.

2. The sound signal processing method according to claim 1, wherein: The priority information indicates the priority of each of the plurality of audio processings. In the determination step, whether or not to execute each of the plurality of acoustic processes on the acoustic signal is determined based on whether or not the priority of each of the plurality of acoustic processes is equal to or greater than a threshold value.

3. The sound signal processing method according to claim 2, wherein: In the determination step, the threshold value is determined based on device information related to a state of the acoustic signal processing device.

4. The sound signal processing method according to claim 3, wherein: The device information includes at least one of information indicating a remaining amount of a battery for driving the sound signal processing device, information indicating a heat generation amount of the sound signal processing device, and information indicating a processing capability of the sound signal processing device.

5. The sound signal processing method according to claim 1, wherein: In the determination step, the priority level indicated by the priority level information is determined based on position information indicating the position of a user of the acoustic signal processing device.

6. A program, wherein: The program is for causing a computer to execute the sound signal processing method according to any one of claims 1 to 5.

7. An audio signal processing device, wherein: have: an acquisition unit, for acquiring an audio signal; a decision unit that determines whether to execute a plurality of sound processings executed in accordance with pipelines on each of the sound signals based on priority information indicating a priority associated with the sound signal; as well as The sound processing unit performs the sound processing determined to be performed by the determination unit on the sound signal.

8. A sound signal reproduction system, wherein: have: The sound signal processing device according to claim 7; and The driver generates a sound based on the sound signal after executing the sound processing determined to be executed by the determination unit.

Images