Stereoscopic sound processing method, stereoscopic sound processing device, and program

Abstract

A stereoscopic sound processing method for reproducing a stereoscopic sound using an AR device (1a), the stereoscopic sound processing method comprising: acquiring change information indicating a change in a space in which the AR device (1a) is located in a process in which the AR device (1a) outputs sound-containing content (S110); determining one or more sound processes based on the change information from among a plurality of sound processes for rendering sound information representing the sound (S150); executing sound processing only for the determined one or more sound processes among the plurality of sound processes (S160); the sound information is rendered on the basis of a first processing result of each of the one or more executed sound processes (S170).

Description

Technical Field

[0001] The present disclosure relates to a stereo sound processing method, a stereo sound processing device, and a program. Background Art

[0002] Patent Document 1 discloses a technique for acquiring acoustic feature quantities (acoustic characteristics) of a room space using devices such as a measurement microphone array and a measurement speaker array.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2012-242597 Summary of the invention

[0006] Problems to be solved by the invention

[0007] The acoustic feature quantity of the real space obtained by the technology of the above-mentioned patent document 1 is sometimes used when rendering the sound information representing the sound output from the AR (Augmented Reality) device. At this time, it can be expected that during the use of the AR device in the space, changes in the space will occur, such as people entering and leaving the space, objects in the space moving or increasing or decreasing, etc. That is, it can be expected that the acoustic feature quantity of the space will change during the use of the AR device.

[0008] It is desirable that the sound outputted from the AR device easily reflects such changes in the space during use. However, Patent Document 1 does not disclose a technique for easily reflecting changes in the space during use.

[0009] Therefore, an object of the present disclosure is to provide a stereophonic sound processing method, a stereophonic sound processing device, and a program that can easily reflect changes in acoustic feature quantities due to changes in space in rendering of sound information.

[0010] Means used to solve problems

[0011] A stereo sound processing method related to a technical solution of the present disclosure is a stereo sound processing method for reproducing stereo sound using an AR (Augmented Reality) device. In the process of outputting content containing sound in the above-mentioned AR device, change information representing changes in the space where the above-mentioned AR device is located is obtained; among multiple sound processings used to render sound information representing the above-mentioned sound, one or more sound processings based on the above-mentioned change information are determined; sound processing is performed only for the above-mentioned one or more sound processings determined among the above-mentioned multiple sound processings; and the above-mentioned sound information is rendered based on the first processing result of each of the above-mentioned one or more sound processings executed.

[0012] A stereo sound processing device according to a technical solution of the present disclosure is a stereo sound processing device for reproducing stereo sound using an AR device, comprising: an acquisition unit, which acquires change information indicating changes in a space in which the AR device is located during a process of outputting content including sound in the AR device; a determination unit, which determines one or more sound processings based on the change information among a plurality of sound processings used to render sound information indicating the sound; a sound processing unit, which performs sound processing only for the one or more sound processings determined among the plurality of sound processings; and a rendering unit, which renders the sound information based on a first processing result of each of the one or more sound processings executed.

[0013] A program according to one technical solution of the present disclosure is a program for causing a computer to execute the above-mentioned stereo processing method.

[0014] Effects of the Invention

[0015] According to a technical aspect of the present disclosure, it is possible to realize a stereophonic processing method or the like that can easily reflect changes in acoustic feature quantities due to changes in space in rendering of sound information. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a block diagram showing the functional structure of a stereo processing device according to an embodiment.

[0017] Figure 2 This is a flowchart showing the operation of the stereo processing device according to the embodiment before the AR device is used.

[0018] Figure 3 This is a flowchart showing the operation of the stereo processing device according to the embodiment when using the AR device.

[0019] Figure 4 This is a diagram for explaining the insertion of a shape model into a space represented by spatial information.

[0020] Figure 5 This is a diagram for explaining the first example of changes occurring in space and sound processing.

[0021] Figure 6 This is a diagram for explaining the second example of changes occurring in space and sound processing. DETAILED DESCRIPTION

[0022] The stereo sound processing method of the first technical solution of the present disclosure is a stereo sound processing method for reproducing stereo sound using an AR (Augmented Reality) device. In the process of outputting content including sound in the above-mentioned AR device, change information representing changes in the space in which the above-mentioned AR device is located is obtained; among multiple sound processings used to render sound information representing the above-mentioned sound, one or more sound processings based on the above-mentioned change information are determined; sound processing is performed only for the above-mentioned one or more sound processings determined among the above-mentioned multiple sound processings; and the above-mentioned sound information is rendered based on the first processing result of each of the above-mentioned one or more sound processings executed.

[0023] Thus, when a change occurs in the space, only one or more determined sound processings among the multiple sound processings are executed, so the amount of calculation for reflecting the change in the space in the sound information can be reduced compared to the case where all of the multiple sound processings are executed. Therefore, according to the stereo sound processing method, the increase in the amount of calculation when a change occurs in the space is suppressed, so the change in the sound feature quantity caused by the change in the space can be easily reflected in the rendering of the sound information.

[0024] In addition, for example, the stereo sound processing method related to the second technical solution of the present disclosure may be, in the stereo sound processing method related to the first technical solution, rendering the above-mentioned sound information based on the above-mentioned first processing results of each of the above-mentioned one or more sound processings and the pre-acquired second processing results of each of the other one or more sound processings among the above-mentioned multiple sound processings except the above-mentioned one or more sound processings, in the rendering of the above-mentioned sound information.

[0025] Thus, since the second processing result acquired in advance is used as the processing result of one or more other sound processing, the amount of calculation can be reduced compared with the case where some calculation is performed for one or more other sound processing.

[0026] In addition, for example, the stereo sound processing method of the third technical solution of the present disclosure may be, in the stereo sound processing method of the first technical solution or the second technical solution, wherein the change information includes information indicating an object that has changed in the space; and in determining the one or more sound processings, the one or more sound processings are determined based on at least one of the sound characteristics of the object and the position of the object.

[0027] Thus, by determining one or more sound processing according to at least one of the acoustic characteristics of the object and the position of the object, it is possible to generate sound information that more appropriately includes the influence of the object. Therefore, it is possible to generate sound information that can output a more appropriate sound according to the spatial situation at that point in time.

[0028] In addition, for example, the stereo sound processing method of the fourth technical solution of the present disclosure may be such that, in the stereo sound processing method of the third technical solution, the sound characteristics of the object and the position of the object are used in determining the one or more sound processings; based on the position of the object, it is determined whether to perform the one or more sound processings corresponding to the object; and when it is determined that the one or more sound processings are to be performed, the one or more sound processings are determined based on the sound characteristics of the object.

[0029] In this way, it is determined whether or not one or more audio processes are to be executed, so that execution of unnecessary audio processes can be suppressed.

[0030] In addition, for example, the stereo sound processing method related to the fifth technical solution of the present disclosure may be, in the stereo sound processing method related to any one of the first to fourth technical solutions, wherein the above-mentioned change information includes information representing objects that have changed in the above-mentioned space; and one or more of the above-mentioned sound processing is performed using a shape model that simplifies the above-mentioned objects.

[0031] Thus, since a shape model that simplifies the object is used, the amount of computation in the sound processing can be reduced compared to the case where the shape of the object itself is used. In particular, by using a shape model for an object whose motion is difficult to predict (such as a person), the amount of computation can be effectively reduced. Therefore, according to the stereo sound processing method, changes in the sound feature quantity caused by changes in the space can be more easily reflected in the rendering of the sound information.

[0032] In addition, for example, the stereo sound processing method of the sixth technical solution of the present disclosure may, in the stereo sound processing method of the fifth technical solution, read out a shape model corresponding to the object from a storage unit in which a plurality of shape models are pre-stored, based on the type of the above-mentioned object, thereby obtaining the above-mentioned shape model.

[0033] Thus, since the shape model only needs to be read, the amount of calculation for acquiring the shape model can be reduced compared to the case where the shape model is generated by calculation or the like.

[0034] In addition, for example, the stereo sound processing method of the 7th technical solution of the present disclosure may insert the above-mentioned shape model into the spatial information representing the above-mentioned space in the stereo sound processing method of the 5th technical solution or the 6th technical solution; and determine the above-mentioned one or more sound processing based on the above-mentioned spatial information after the above-mentioned shape model is inserted.

[0035] This makes it possible to reproduce the situation in the space at that point in time using the shape model. By using such spatial information, it is possible to determine one or more acoustic processing suitable for the situation in the space at that point in time.

[0036] The stereo sound processing device of the eighth technical solution of the present disclosure is a stereo sound processing device for reproducing stereo sound using an AR device, and comprises: an acquisition unit, which acquires change information indicating changes in the space where the AR device is located during the process of outputting content including sound in the AR device; a determination unit, which determines one or more sound processings based on the change information among a plurality of sound processings used to render sound information indicating the sound; a sound processing unit, which performs sound processing only for the one or more sound processings determined among the plurality of sound processings; and a rendering unit, which renders the sound information based on the first processing result of each of the one or more sound processings executed. In addition, the program of the ninth technical solution of the present disclosure is a program for causing a computer to execute the stereo sound processing method of any one of the first to seventh technical solutions.

[0037] This can produce the same effects as the above-mentioned stereo processing method.

[0038] In addition, these inclusive or specific technical solutions may also be implemented by a system, method, integrated circuit, computer program or non-temporary recording medium such as a computer-readable CD-ROM, or by any combination of a system, method, integrated circuit, computer program and recording medium. The program may be pre-stored in a recording medium or provided to the recording medium via a wide area communication network including the Internet.

[0039] Hereinafter, the embodiments will be described in detail with reference to the drawings.

[0040] In addition, the embodiments described below are all inclusive or specific examples. The numerical values, constituent elements, configuration positions and connection forms of constituent elements, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit the present disclosure. In addition, the constituent elements of the following embodiments that are not recorded in the independent claims are described as arbitrary constituent elements.

[0041] In addition, each figure is a schematic diagram and does not necessarily illustrate the figure strictly. Therefore, for example, the scales in each figure are not necessarily the same. In addition, in each figure, the same reference numerals are given to substantially the same structure, and repeated descriptions are omitted or simplified.

[0042] In the present specification, numerical values ​​and numerical ranges are not strictly defined but are substantially equivalent, and include differences of, for example, several percentage points (or 10%).

[0043] (Implementation Method)

[0044] Below, refer to Figure 1 to Figure 6 A stereophonic sound processing method according to the present embodiment and a stereophonic sound processing device for executing the stereophonic sound processing method will be described.

[0045] [1. Structure of stereo sound processing device]

[0046] First, refer to Figure 1 The structure of the stereo processing device according to the present embodiment will be described. Figure 1 1 is a block diagram showing the functional structure of the stereo processing device 10 according to the present embodiment.

[0047] like Figure 1 As shown, the stereo sound processing device 10 is included in the stereo sound reproduction system 1, and the stereo sound reproduction system 1 includes a sensor 20 and a sound output device 30 in addition to the stereo sound processing device 10. The stereo sound reproduction system 1 is built into the AR device, for example, but at least one of the stereo sound processing device 10 and the sensor 20 may be implemented by a device external to the AR device.

[0048] The stereo sound reproduction system 1 is a system for rendering sound information (sound signal) and outputting (reproducing) sound based on the rendered sound information so that sound corresponding to the indoor space (hereinafter simply referred to as space) where a user wearing an AR device is located is output from the sound output device 30 of the AR device.

[0049] As long as the indoor space is enclosed to some extent, it can include living rooms, lobbies, meeting rooms, corridors, stairs, bedrooms, etc.

[0050] AR devices are glasses-type AR wearable terminals (so-called smart glasses) or head-mounted displays for AR that users can wear, but they can also be portable terminals such as smartphones or tablet-type information terminals. In addition, augmented reality refers to the technology of using information processing devices to further add information to the real environment such as scenery, terrain, objects, etc. in real space.

[0051] The AR device includes a display unit, a camera (an example of the sensor 20), a speaker (an example of the sound output device 30), a microphone, a processor, a memory, etc. In addition, the AR device may also include a depth sensor, a GPS (Global Positioning System) sensor, a LiDAR (Laser Imaging Detection and Ranging), etc.

[0052] When rendering sound information, the acoustic features of the space are required as spatial information. Therefore, a technology has been studied for obtaining spatial information of the real space in which the AR device is used before the AR device is used, and inputting the previously obtained spatial information into a processing device for rendering when the AR device is started (or before it is started). The spatial information including the acoustic features can be obtained, for example, by measuring the space in advance, or by calculating with a computer. In addition, the spatial information includes, for example, the size and shape of the space, the acoustic features of building materials such as the walls that constitute the space, the acoustic features of objects in the space, the position and shape of objects in the space, etc.

[0053] In addition, it is conceivable that changes in the space may occur during the use of the AR device, such as people entering and exiting the space, objects in the room moving or increasing or decreasing, etc. If such changes in the space occur, the sound feature quantity (sound characteristics) of the space changes. Therefore, in order to output the sound corresponding to the space from the AR device, it is necessary to render again (additional rendering), but the computational load of the stereo sound processing device 10 as a processing device may increase. In particular, in the case of objects corresponding to the movements of people and the like that are difficult to predict, high-speed sensing is required, which may increase the computational load of the stereo sound processing device 10.

[0054] Therefore, the following describes a stereo sound processing device 10 that can reduce the amount of calculation when performing additional rendering as a device that easily reflects changes in the sound feature quantity of the space caused by changes in the space where the AR device is located in the rendering of the sound. In addition, the use of the AR device means that the user is using the activated AR device, specifically, it means that the content including sound is being output in the AR device worn by the user.

[0055] The stereo sound processing device 10 is an information processing device for reproducing stereo sound using an AR device, and includes an acquisition unit 11 , an update unit 12 , a storage unit 13 , a control unit 14 , a sound processing unit 15 , and a rendering unit 16 .

[0056] During the use of the AR device, the acquisition unit 11 acquires change information indicating changes in the space where the user wearing the AR device is located from the sensor 20. The change in the space is a change in an object located in the space, such as a change in the acoustic feature quantity of the space, and may include, for example, a change in at least one of the shape and size, such as movement of an object in the space (position change), increase or decrease of objects in the space, deformation of an object in the space, and the like.

[0057] The change information includes information indicating the object that has changed in the space. The change information may also include, for example, information indicating the type of the object that has changed in the space and the position of the object in the space. The types of objects include moving objects (mobile bodies) such as people, pets, and robots (for example, autonomous mobile robots), and fixed objects such as tables and partitions, but are not limited to these.

[0058] In addition, the change information may include, for example, an image showing an object in the space (for example, an object that has changed in the space). In this case, the acquisition unit 11 may also have a function of detecting that a change has occurred in the space. The acquisition unit 11 may also have a function of detecting the type of the object and the position of the object in the space from the image by image processing, etc. The acquisition unit 11 may also function as a detection unit that detects that a change has occurred in the space.

[0059] The acquisition unit 11 is configured to include, for example, a communication module (communication circuit).

[0060] The update unit 12 performs processing for reproducing the current status of the real space in the space represented by the spatial information obtained in advance. The update unit 12 can also be said to perform processing for updating the spatial information obtained in advance according to the current status of the real space. When an object is added, the update unit 12 inserts (configures) a shape model (object) corresponding to the type of object (hereinafter also referred to as the target object) contained in the change information to the position of the space represented by the spatial information corresponding to the position of the target object in the space represented by the spatial information. The update unit 12 determines the shape model based on the type of the target object and the establishment of a corresponding table between the type of the target object and the shape model. Based on the type of the object, the update unit 12 reads the shape model corresponding to the object from the storage unit 13 in which multiple shape models are pre-stored, thereby obtaining the shape model. In advance, for example, refers to before outputting content containing sound in the AR device, but is not limited to this.

[0061] A shape model is a model that simplifies an object (imitates an object), for example, represented by a three-dimensional shape. A three-dimensional shape is a shape corresponding to an object, for example, a corresponding shape model is pre-set for each object type. The three-dimensional shape is, for example, a prism, a cylinder, a cone, a sphere, a plate, etc., but is not limited to these. For example, if the object is a person, a quadrangular prism can also be set as the shape model.

[0062] In addition, the shape model may be formed by a combination of two or more three-dimensional shapes, as long as the shape can reduce the amount of computation required for sound processing according to the shape of the actual object. Figure 4 The space 200a) shown in (b) is also recorded as updated space information.

[0063] Furthermore, when the number of objects decreases, the updating unit 12 removes the target object from the previously acquired spatial information. Furthermore, when the object moves, the updating unit 12 moves the target object in the previously acquired spatial information to the position of the target object included in the change information. Furthermore, when the object is deformed, the updating unit 12 deforms the target object in the previously acquired spatial information into the shape of the target object included in the change information.

[0064] The storage unit 13 is a storage device that stores various tables used by the update unit 12 and the control unit 14. In addition, the storage unit 13 may store space information acquired in advance. In advance means before the user uses the AR device in the target space.

[0065] The control unit 14 determines one or more sound processing based on the change information among a plurality of sound processings used to render the sound information (original sound information) representing the sound output from the AR device. The control unit 14 may also determine one or more sound processings based on the type of object, for example. The control unit 14 may also determine one or more sound processings based on at least one of the sound feature quantity (sound characteristics) of the object and the position of the object, for example. In addition, the control unit 14 may also determine one or more sound processings based on the spatial information after the shape model is inserted. In addition, when there are multiple objects, the control unit 14 may also determine one or more sound processings for each of the multiple objects. In this way, the control unit 14 functions as a determination unit that determines one or more sound processings.

[0066] The plurality of acoustic processing includes at least two of processing on reflection of sound in space, processing on reverberation of sound, processing on occlusion of sound, processing on distance attenuation of sound, processing on diffraction of sound, and the like.

[0067] Reflection refers to the phenomenon that sound incident on an object at a certain angle is bounced back by the object. Reverberation is the phenomenon that sound generated in a space can be heard through reverberation such as reflection. The time it takes for the sound pressure level to decay to a certain degree (for example, 60 dB) after the sound source stops is defined as the reverberation time. Masking refers to the effect of sound attenuation when there is some object (shield) between the sound source and the listening point. Distance attenuation refers to the phenomenon that sound attenuates according to the distance between the sound source and the listening point. Diffraction refers to the phenomenon that when there is an object between the sound source and the listening point, the sound is detoured by reflection, so that the sound can be heard from a direction different from the actual direction of the sound source.

[0068] The sound processing unit 15 performs one or more sound processings determined by the control unit 14. The sound processing unit 15 performs the sound processing only for the one or more sound processings among the multiple sound processings. The sound processing unit 15 performs one or more sound processings respectively based on the updated spatial information and the properties of the object, and calculates the processing results of each of the one or more sound processings. The processing results include coefficients (such as filter coefficients) used in rendering. The processing results of each of the one or more sound processings are an example of the first processing result. In addition, the multiple sound processings are preset.

[0069] The rendering unit 16 renders (additionally renders) the originally stored sound information using the processing results of the one or more sound processing. The rendering unit 16 outputs the result of convolution operation of the sound information using the coefficients obtained by the one or more sound processing as sound control information. For details of the processing of the rendering unit 16, use Figure 6 Rendering is a process of adjusting sound information according to the indoor environment of a space so that the sound is output at a predetermined volume and from a predetermined sound output position.

[0070] The sensor 20 is installed to be able to sense the position and posture in the space, and sense the changes in the space. In addition, the sensor 20 is arranged in the space and is connected to the stereo processing device 10 so as to be communicable. The sensor 20 can sense the shape, position, etc. of an object in the space. In addition, the sensor 20 can also be able to determine the type of the object in the space. The sensor 20 is composed of, for example, an imaging device such as a camera.

[0071] The sensor 20 may also determine whether the AR device is located in the space where the sensor 20 is installed and whether the AR device is activated by acquiring the position information and the information indicating that the AR device is in use from the AR device.

[0072] The sound output device 30 outputs sound based on the sound control information acquired from the stereo sound processing device 10. The sound output device 30 includes a speaker, a processing unit such as a CPU, and the like.

[0073] [2. Operation of the stereo sound processing device]

[0074] Next, refer to Figure 2 to Figure 6 The operation of the stereo processing device 10 configured as described above will be described.

[0075] First, refer to Figure 2 The following describes the procedures before using the AR device. Figure 2 1 is a flowchart showing the operation (stereo sound processing method) of the stereo sound processing device 10 according to the present embodiment before the use of the AR device. Figure 2 The processing shown may be executed by a device other than the stereo processing device 10 .

[0076] like Figure 2 As shown in FIG. 1 , the acquisition unit 11 acquires space information including acoustic feature quantities of space ( S10 ). The acquisition unit 11 acquires the space information from the sensor 20 , for example.

[0077] Next, the sound processing unit 15 executes each of the plurality of sound processes using the spatial information ( S20 ).

[0078] Next, the rendering unit 16 performs rendering processing on the sound information using the processing results of each of the plurality of sound processings (an example of the second processing result) (S30). In the rendering unit 16, as a rendering process, the processing results (e.g., coefficients) of each of the plurality of sound processings are integrated, and a convolution operation is performed on the sound information using the integrated processing result. In the rendering unit 16, for example, as a sound processing, a BRIR (Binaural Room Impulse Response) that reflects the characteristics of a person's head or the characteristics of a space (sound processing such as reflection or reverberation) is calculated, and the calculated BRIR is convoluted with the sound information. In addition, the sound processing is not limited to this, and HRIR (Head Related Impulse Response) or the like may be calculated, or other sound processing may be used. In this way, sound information that can reproduce a sound corresponding to the spatial information obtained in advance is generated.

[0079] Next, refer to Figure 3 to Figure 6 The following describes the operations of using the AR device. Figure 3 1 is a flowchart showing the operation (stereo sound processing method) of the stereo sound processing device 10 according to the present embodiment when using an AR device. Figure 3 In the following, the operation when the acquisition unit 11 functions as a detection unit is described.

[0080] The acquisition unit 11 acquires sensing data obtained by the sensor 20 sensing the space where the AR device is located during the use of the AR device (S110). The sensing data includes information indicating the shape and size of the space, the size and position of objects in the space, etc. The acquisition unit 11 acquires the sensing data, for example, periodically or in real time. The sensing data is an example of change information.

[0081] Next, the acquisition unit 11 determines whether there is a change in space (change in space) based on the sensing data (S120). The acquisition unit 11 determines whether there is a change in space based on the spatial information acquired in step S10 or the most recently acquired sensing data, and the sensing data acquired in step S110. The acquisition unit 11 determines "yes" in step S110 when there is movement, increase, decrease, deformation, etc. of objects in the space. In addition, the following describes an example in which the comparison object of the sensing data acquired in step S110 is the spatial information acquired in step S110. In addition, the following describes an action in which an object increases in the real space as an example.

[0082] Next, when the acquisition unit 11 determines that there is a change in the space (Yes in S120), the update unit 12 inserts a simple object (shape model) into the space (spatial information) (S130). Inserting a shape model into the space is an example of updating the spatial information.

[0083] Figure 4 This is a diagram for explaining the insertion of a shape model 210 into a space 200 represented by space information. Here, an example in which an object included in the change information is a person will be described.

[0084] Figure 4 (a) shows a space 200 represented by spatial information acquired in advance and a shape model 210 which is a simple object corresponding to a person.

[0085] Figure 4 (b) shows the space 200a represented by the space information after a simple object (shape model 210) is inserted into the space 200. Figure 4 In (b), the shape model 210 is inserted into the space 200a. The shape model 210 is inserted at a position in the space 200a corresponding to the position of the object in the real space. The position of the object in the real space is included in the sensing data obtained from the sensor 20.

[0086] When the acquisition unit 11 determines that there is no spatial change (No in S120 ), the updating unit 12 returns to step S110 and continues the process.

[0087] Next, the control unit 14 determines whether there is an influence on the acoustic feature quantity of the space 200a represented by the spatial information inserted into the shape model 210 (S140). The control unit 14 makes the determination of step S140 based on at least one of the properties of the scene of the space, the properties of the sound source, and the position of the object. This determination is equivalent to determining whether to perform acoustic processing corresponding to the object (for example, whether additional rendering is required). In addition, when multiple types of objects are added, the control unit 14 may also perform the determination of step S140 on each of the multiple types of objects.

[0088] The properties of the scene include the acoustic features of the object (virtual object) reproduced by the AR device. The properties of the sound source are the properties of the sound represented by the sound information, for example, whether it is a reverberating sound such as a car engine sound or a muffled sound.

[0089] The control unit 14 may determine whether there is an influence on the acoustic feature quantity of the space based on information related to the objects added in the space, for example. The control unit 14 may determine whether there is an influence on the acoustic feature quantity of the space based on, for example, the number of objects added, the size or shape of the objects added, etc. The control unit 14 may determine that there is an influence on the acoustic feature quantity of the space when, for example, the number of objects added is greater than a predetermined number, or when the size of the objects added is greater than a predetermined size.

[0090] In addition, the control unit 14 may also determine whether there is an influence on the sound feature quantity of the space, for example, based on the position of the object (real object) contained in the spatial information obtained in advance and the distance between one of the objects (virtual objects) reproduced by the AR device and the added object (real object). When the distance is below the specified distance, the control unit 14 determines that there is an influence on the sound feature quantity of the space because it can be imagined that the sound feature quantity of the space changes due to the interaction between objects. This is equivalent to determining that sound processing is performed, that is, additional rendering is performed. In addition, when the distance is larger than the specified distance, the control unit 14 determines that there is no influence on the sound feature quantity because it can be imagined that the influence on the sound feature quantity of the space caused by the interaction between objects is small. This is equivalent to determining not to perform sound processing, that is, not to perform additional rendering.

[0091] In addition, the distance used in determining whether or not the acoustic feature quantity of the space is affected may be set according to the acoustic feature quantity of the object (virtual object) and the properties of the sound source, and stored in the storage unit 13. In addition, the control unit 14 may also use the properties (hardness, softness, etc.) of each object in the space in the determination of step S140.

[0092] Alternatively, the control unit 14 may perform the determination in step S140 using a table in which the properties of the object (for example, hardness, size, etc.) and whether or not to execute the audio processing are associated with each other.

[0093] Figure 5 This is a diagram for explaining the first example of changes occurring in space and sound processing. Figure 5 (a) shows the situation in the real space 300 when a user U wearing the AR device 1a is in the real space 300 and one person 50 is added during the use of the AR device 1a. In addition, the sound output device 40 is a virtual object reproduced by the AR device 1a, and is an object that does not actually exist in the real space 300. In this case, the stereo sound processing device 10 reproduces the sound output from the sound output device 40 and reaches the user U.

[0094] It is considered that the effect of the addition of one person 50 on the acoustic feature quantity of the real space 300 is small, so in this case, no additional rendering processing is performed. The control unit 14 may also determine not to perform additional rendering processing when the number of the added people 50 is less than the specified number. In addition, the control unit 14 may also determine that there is no effect, such as not performing additional rendering processing, when the distance between the added person 50 and the user U is greater than the specified distance.

[0095] The additional rendering process is a process of executing the audio process in parallel with the use of the AR device and executing the rendering using the processing result of the executed audio process.

[0096] Figure 6 This is a diagram for explaining the second example of changes occurring in space and sound processing. Figure 6 (a) shows a situation in the real space 300 when a user U wearing the AR device 1 a is in the real space 300 and a plurality of people 50 are added to the use of the AR device 1 a.

[0097] The addition of multiple persons 50 is considered to have a greater impact on the acoustic feature quantity of the real space 300, so additional rendering processing is performed in this case. For example, the control unit 14 may determine that there is an impact when the number of additional persons 50 is greater than a predetermined number, for example, and perform additional rendering processing on the sound information.

[0098] Refer again Figure 3 If the control unit 14 determines that there is an influence ("Yes" in S140), the control unit 14 proceeds to step S150, and if it determines that there is no influence ("No" in S140), the control unit 14 proceeds to step S110 and continues the process. In this way, the control unit 14 functions as a determination unit.

[0099] Next, if the control unit 14 determines that there is an influence ("Yes" in S140), it determines one or more sound processing based on the change information (S150). The control unit 14 may also determine one or more sound processing based on the type of object, for example. The control unit 14 may also use a table that establishes a correspondence between the type of object and one or more sound processing to determine one or more sound processing that needs to be performed on the object determined to have an influence. The table is made according to the properties of the object. For example, when the object is hard, it affects the reflection characteristics as the sound feature quantity, so the correspondence is established with one or more sound processing including processing related to the reflection of sound. In this way, the control unit 14 may also determine one or more sound processing based on the sound characteristics of the object.

[0100] In addition, the control unit 14 may determine one or more sound processing based on the positional relationship between the sound emitting device 40, the user U, and the object and the size of the object. For example, if an object larger than a specified size is added between the sound emitting device 40 and the user U, it may affect the shielding, so the control unit 14 may determine one or more sound processing including processing related to the shielding of the sound. In addition, if an object smaller than a specified size is added between the sound emitting device 40 and the user U, the effect on the sound feature quantity of the space is small, so it may be determined as "No" in step S140.

[0101] Furthermore, the table may be a table in which the acoustic feature quantity (acoustic characteristics) of the object is associated with one or more acoustic processes.

[0102] Next, the sound processing unit 15 executes one or more sound processings determined by the control unit 14 (S160). In other words, the sound processing unit 15 does not execute any sound processing other than the determined one or more sound processings among the plurality of sound processings in step S160.

[0103] exist Figure 6 In the audio processing (initial) shown in (b) of FIG. Figure 2 In step S20 shown in FIG. 1 , five different audio processes, A, B (B1), C, D (D1), and E (E1), are executed. Figure 6 The sound processing (additional part) shown in (b) is Figure 3The sound processing executed in step S150 shown in the figure is only the three sound processings B (B2), D (D2), and E (E2) determined as one or more sound processings. In addition, B1 and B2, D1 and D2, and E1 and E2 are sound processings related to the same sound feature quantity, and the spatial information used in the processing is different. The processing results of each of the sound processings B (B2), D (D2), and E (E2) are examples of the first processing results, and the processing results of each of the sound processings A and C are examples of the second processing results.

[0104] In this way, in step S150, only a part of the sound processing executed in step S20 is executed. In other words, in step S150, all of the multiple sound processing executed in step S20 are not executed. As a result, the amount of calculation of the stereo sound processing device 10 can be reduced compared to the case where all five sound processing are executed.

[0105] Next, the rendering unit 16 uses the processing results of the one or more audio processes to perform rendering processing (additional rendering processing) on ​​the sound information (S170). Figure 6 The sound processing (initial) and (additional part) shown in (b) are respectively processed and rendered ( Figure 6 The rendering unit 16 performs rendering using the processing results of each of the five audio processings A, B (B2), C, D (D2), and E (E2). The rendering unit 16 uses the processing result of the audio processing B (B2) in priority over B (B1). The same is true for the audio processings D (D2) and E (E2). It can also be said that the rendering unit 16 uses the processing result of the audio processing using the latest spatial information in one audio processing in priority over the past processing result of the audio processing.

[0106] In this way, the stereo sound processing device 10 renders the sound information in the rendering (additional rendering) of the sound information in use of the AR device based on the processing results of each of the one or more sound processings (an example of the first processing result), and the processing results of each of the one or more sound processings other than the one or more sound processings among the multiple sound processings (an example of the second processing result), and the second processing result obtained in advance. In addition, it can also be said that the stereo sound processing device 10 suppresses the recalculation of each of the other one or more sound processings, and only recalculates the necessary sound processing corresponding to the added object.

[0107] Refer again Figure 3 The rendering unit 16 outputs the sound information (sound control information) after the rendering process (additional rendering process) to the sound output device 30 (S180). As a result, the sound output device 30 can output the sound corresponding to the situation in the space at that time.

[0108] In addition, the processing of steps S110 to S180 is performed while the AR device is in use.

[0109] in addition, Figure 5 The sound processing shown in (b) corresponds to Figure 6 The sound processing (initial) shown in (b).

[0110] (Other embodiments)

[0111] The above is a description of the stereo sound processing method and the like related to one or more technical solutions based on the implementation method, but the present disclosure is not limited to the implementation method. As long as it does not deviate from the main purpose of the present disclosure, various forms after the present embodiment is applied to the present embodiment that can be thought of by those skilled in the art, or forms constructed by combining the constituent elements of different implementation methods may also be included in the present disclosure.

[0112] For example, in the above-mentioned embodiment, an example in which the stereo sound processing device includes both an updating unit and a control unit is described, but it is sufficient to include at least one of the updating unit and the control unit. For example, the stereo sound processing device may also include only the updating unit of the updating unit and the control unit. Such a stereo sound processing device is a stereo sound processing device for reproducing stereo sound using an AR device, and includes: an updating unit (insertion unit), which obtains change information representing changes in the space where the AR device is located during the process of outputting content including sound in the AR device, and inserts a shape model of an object that has been changed and is simply represented by the change information into the space represented by the spatial information of the space obtained in advance; an audio processing unit, which uses a simplified shape model of the object to perform audio processing for multiple audio processings used to render sound information representing sound; and a rendering unit, which renders the sound information based on the processing results of each of the multiple audio processings performed. In addition, the present disclosure can also be implemented as a stereo sound processing method performed by the stereo sound processing device and a program for causing a computer to perform the stereo sound processing method.

[0113] In addition, in the above-mentioned embodiment, the example in which the change of the object in use of the AR device is the change of the real object is described, but it is not limited to this, and it can also be the change of the virtual object. That is, the change of the object in use of the AR device can also be the movement, increase, decrease, deformation, etc. of the virtual object. In this case, the acquisition unit of the stereo sound processing device obtains the change information from the display control device that controls the display of the AR device.

[0114] In addition, in the above embodiment, an example in which the stereo sound processing device is mounted in an AR device is described, but it can also be mounted in a server. In this case, the AR device and the server are connected in a communicative manner (for example, wirelessly). In addition, the stereo sound processing device can also be used indoors and be mounted or connected to any device that outputs sound. The device can also be a fixed-mounted audio device or a game console (for example, a portable game console).

[0115] In addition, in the above-mentioned embodiment, an example is described in which the updating unit inserts the shape model as it is into the space, but the present invention is not limited thereto. For example, the size (e.g., height) of the shape model may be changed according to the sensing data and then inserted into the space. In addition, the updating unit may combine a plurality of shape models based on the shape of the object included in the sensing data to generate a new shape model corresponding to the shape of the object, and insert the generated new shape model into the space.

[0116] Furthermore, the changes in the space of the above-mentioned embodiments may include changes in the space itself. The changes in the space itself may be changes in at least one of the size and shape of the space itself, for example, by opening or closing a door or partition disposed between two spaces.

[0117] In addition, in the above-mentioned embodiment, the case of using a shape model is described, but it is not limited to this. For a part of the objects, the shape of the object itself can also be used to perform the processing after step S140. The control unit can also determine whether to replace the shape of the object with the shape model based on the type of the object or the shape of the object contained in the change information between steps S120 and S130. In addition, the control unit can also execute step S130 only when it is determined to be replaced, and insert the shape of the object itself into the space when it is determined not to be replaced. The control unit can also determine not to replace based on the type of the object or the shape of the object, for example, when it is assumed that the amount of calculation in the sound processing is less than a specified amount. The control unit can also perform the judgment based on a table that establishes a correspondence between the type of the object or the shape of the object and whether to replace. In addition, the table is set in advance and stored in the storage unit.

[0118] In addition, in the above-mentioned embodiments, each component may be formed by dedicated hardware, or implemented by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory.

[0119] In addition, the order in which each step in the flowchart is executed is used for illustrative purposes to specifically describe the present disclosure, and may be an order other than the above. In addition, a part of the above steps may be executed simultaneously (in parallel) with other steps, or a part of the above steps may not be executed.

[0120] In addition, the division of the functional blocks in the block diagram is an example, and multiple functional blocks can also be implemented as one functional block, or one functional block can be divided into multiple blocks, or a part of the function can be transferred to other functional blocks. In addition, it is also possible that a single hardware or software processes the functions of multiple functional blocks with similar functions in parallel or in a time-sharing manner.

[0121] In addition, the stereo sound processing device of the above-mentioned embodiment can be implemented as a single device or by multiple devices. For example, at least a part of the components of the stereo sound processing device can also be implemented by a device such as a server that can communicate with the AR device. In the case where the stereo sound processing device is implemented by multiple devices, the components of the stereo sound processing device can be distributed to multiple devices in any way. In the case where the stereo sound processing device is implemented by multiple devices, the communication method between the multiple devices is not particularly limited, and it can be wireless communication or wired communication. In addition, wireless communication and wired communication can be combined between devices.

[0122] In addition, each component element described in the above-mentioned embodiment can also be implemented as software, and can typically be implemented as an integrated circuit, i.e., LSI. They can be formed into one chip individually, or they can be formed into one chip in a way that includes part or all of them. Here, it is set as LSI, but according to the difference in integration, there are also cases called IC, system LSI, super LSI, and ultra-large-scale LSI. In addition, the method of integrated circuitization is not limited to LSI, and can also be implemented by a general-purpose circuit (general-purpose circuit that executes a dedicated program by a dedicated circuit) or a general-purpose processor. It is also possible to utilize a FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconstruct the connection or setting of the circuit unit inside the LSI. Furthermore, if a technology for integrated circuitization that replaces LSI appears through the progress of semiconductor technology or other derived technologies, then of course, the technology can also be used to integrate the components.

[0123] System LSI is a multifunctional LSI manufactured by integrating multiple components into one chip. Specifically, it is a computer system composed of a microprocessor, ROM (Read Only Memory), RAM (Random Access Memory), etc. The computer program is stored in the ROM. The system LSI achieves its functions by the microprocessor operating according to the computer program.

[0124] In addition, a technical solution of the present disclosure may also be to enable a computer to execute Figure 2 and Figure 3 A computer program comprising the characteristic steps included in any one of the stereo processing methods shown.

[0125] In addition, for example, the program may also be a program for causing a computer to execute. In addition, a technical solution of the present disclosure may also be a non-temporary recording medium that is readable by a computer and records such a program. For example, such a program may also be recorded in a recording medium and distributed or circulated. For example, the distributed program may also be installed in a device having another processor, and by causing the processor to execute the program, the device may be enabled to perform the above-mentioned processing.

[0126] In addition, the sound information (sound signal) rendered in the present disclosure may be obtained from a storage device (not shown) or the storage unit 13 outside the stereo processing device 10 as an encoded bit stream including the sound information (sound signal) and metadata. For example, the sound information may be obtained from the stereo processing device 10 as a bit stream encoded in a prescribed format such as MPEG-H 3D Audio (ISO / IEC 23008-3). In this case, an extraction unit (not shown) may be included in the stereo processing device 10, and the extraction unit performs decoding processing on the bit stream encoded based on the above-mentioned MPEG-H 3D Audio, etc. For example, the extraction unit functions as a decoder. The extraction unit decodes the encoded bit stream and provides the decoded sound signal and metadata to the control unit 14. In addition, the extraction unit may exist outside the stereo processing device 10, and the control unit 14 may obtain the decoded sound signal and metadata.

[0127] As an example, the encoded sound signal includes information about the target sound reproduced by the stereo sound processing device 10. The target sound here refers to the sound emitted by the sound source object (virtual object) existing in the sound reproduction space or the natural environment sound, and may include, for example, mechanical sound or the sound of animals including humans, etc. In addition, when there are multiple sound source objects in the sound reproduction space, the stereo sound processing device 10 may also obtain multiple sound signals corresponding to the multiple sound source objects.

[0128] Metadata is, for example, information used in the stereo sound processing device 10 to control the sound processing of sound information. Metadata can also be information used to describe the properties of a scene represented by a virtual space (sound reproduction space). The scene mentioned here is a term that refers to a collection of all elements of a three-dimensional image and sound event modeled by the stereo sound processing device 10 using metadata. That is, the metadata mentioned here includes not only information for controlling the sound processing such as sound feature quantities, but also information for controlling the image processing. Of course, the metadata can include information for controlling only one of the sound processing and the image processing, or information used for controlling both.

[0129] The stereo sound processing device 10 performs sound processing on the sound information by using metadata included in the bitstream and the position information of the interactive user U obtained from the sensor 20, etc., to generate a virtual sound effect. For example, it is possible to consider adding sound effects such as generation of reflected sound, processing related to shielding, processing related to diffraction sound, distance attenuation effect, positioning, sound image localization processing or Doppler effect. In addition, information for switching on / off all or part of the sound effects can also be added as metadata. The control unit 14 can also determine one or more sound processing for the object based on the spatial information or metadata after the shape model is inserted.

[0130] In addition, all or part of the metadata may be obtained from outside the bitstream of the audio information. For example, metadata for controlling the audio or metadata for controlling the video may be obtained from outside the bitstream, or both metadata may be obtained from outside the bitstream.

[0131] Furthermore, when the bit stream obtained by the stereo processing device 10 includes metadata for controlling the image, the stereo processing device 10 may also have a function of outputting the metadata that can be used for controlling the image to a display device that displays the image or a stereo image reproduction device that reproduces the stereo image.

[0132] In addition, as an example, the encoded metadata includes: information about the sound reproduction space including the sound source object and the obstacle object that emits the sound, and information about the positioning position when the sound image of the sound is positioned at a specified position in the sound reproduction space (that is, so that it is perceived as a sound arriving from a specified direction), that is, information about the specified direction. Here, the obstacle object is an object that may affect the sound perceived by the user U, such as blocking or reflecting the sound, during the period from the sound emitted by the sound source object to the sound reaching the user U. In addition to stationary objects, obstacle objects may also include moving bodies such as animals such as humans or machinery. In addition, when there are multiple sound source objects in the sound reproduction space, for any sound source object, other sound source objects may become obstacle objects. In addition, non-sound source objects such as building materials or inanimate objects and sound source objects that emit sound may also become obstacle objects. In addition, the sound source objects and obstacle objects described here can be virtual objects or real objects contained in the spatial information of the real space obtained in advance.

[0133] The spatial information constituting the metadata may include not only the shape of the sound reproduction space, but also information indicating the shape and position of obstacle objects existing in the sound reproduction space and the shape and position of sound source objects existing in the sound reproduction space. The sound reproduction space may be either a closed space or an open space, and the metadata includes information on the reflectivity of structures such as floors, walls, or ceilings that can reflect sound in the sound reproduction space and the reflectivity of obstacle objects existing in the sound reproduction space. Here, the reflectivity is the energy ratio of reflected sound to incident sound, and is set for each frequency band of sound. Of course, the reflectivity may also be set uniformly regardless of the frequency band of the sound. In the case where the sound reproduction space is an open space, for example, uniformly set parameters such as attenuation rate, diffracted sound, or initial reflected sound may also be used.

[0134] In the above description, reflectivity is listed as a parameter related to an obstacle object or a sound source object included in metadata, but metadata may include information other than reflectivity. For example, metadata related to both a sound source object and a non-sound source object may include information related to the material of the object. Specifically, metadata may include parameters such as diffusivity, transmittance, or sound absorption.

[0135] As information related to the sound source object, it may also include volume, radiation characteristics (directivity), reproduction conditions, the number and type of sound sources emitted from an object, or information on the sound source area in the specified object. In the reproduction conditions, for example, it may also be set whether it is a sound that flows continuously or a sound triggered by an event. The sound source area in the object can be set by the relative relationship between the position of the user U and the position of the object, or it can be set based on the object. In the case of setting by the relative relationship between the position of the user U and the position of the object, based on the side of the object viewed by the user U, the user U can perceive that sound X is emitted from the right side of the object and sound Y is emitted from the left side when viewed by the user U. In the case of setting based on the object, which sound is emitted from which area of ​​the object can be fixed regardless of the direction in which the user U looks. For example, the user U can perceive that when looking at the object from the front, high sound flows from the right side and low sound flows from the left side. In this case, if the user U goes around to the back of the object, the user U can perceive that the low sound flows from the right side and the high sound flows from the left side when viewed from the back.

[0136] Metadata related to the space may include time until initial reflected sound, reverberation time, ratio of direct sound to diffuse sound, etc. When the ratio of direct sound to diffuse sound is zero, the user U can perceive only direct sound.

[0137] Information indicating the position and orientation of the user U is obtained from information other than the bitstream. For example, position information obtained by performing self-position estimation using sensing information obtained from the sensor 20 may be used as information indicating the position and orientation of the user U. In addition, the sound information and metadata may be stored in one bitstream or in multiple bitstreams. Similarly, the sound information and metadata may be stored in one file or in multiple files.

[0138] In the case where the sound information and metadata are stored in multiple bitstreams, respectively, information indicating other associated bitstreams may be included in one or a part of the bitstreams storing the sound information and metadata. In addition, information indicating other associated bitstreams may be included in metadata or control information of each bitstream of the multiple bitstreams storing the sound information and metadata. In the case where the sound information and metadata are stored in multiple files, information indicating other associated bitstreams or files may be included in one or a part of the files storing the sound information and metadata. In addition, information indicating other associated bitstreams or files may be included in metadata or control information of each bitstream of the multiple bitstreams storing the sound information and metadata.

[0139] Here, the associated bitstreams or files are, for example, bitstreams or files that may be used simultaneously during audio processing. In addition, information indicating other associated bitstreams may be recorded together in the metadata or control information of one bitstream among multiple bitstreams storing sound information and metadata, or may be recorded separately in the metadata or control information of two or more bitstreams among multiple bitstreams storing sound information and metadata. Similarly, information indicating other associated bitstreams or files may be recorded together in the metadata or control information of one file among multiple files storing sound information and metadata, or may be recorded separately in the metadata or control information of two or more files among multiple files storing sound information and metadata. In addition, a control file that records information indicating other associated bitstreams or files may be generated separately from multiple files storing sound information and metadata. In this case, the control file may not store sound information and metadata.

[0140] Here, the information indicating the other associated bitstreams or files is, for example, an identifier indicating the other bitstream, a file name indicating the other file, a URL (Uniform Resource Locator) or a URI (Uniform Resource Identifier), etc. In this case, the acquisition unit 11 determines or acquires the bitstream or file based on the information indicating the other associated bitstream or file. In addition, the information indicating the other associated bitstream may be included in the metadata or control information of at least a part of the bitstreams among the multiple bitstreams storing the sound information and metadata, and the information indicating the other associated files may be included in the metadata or control information of at least a part of the files among the multiple files storing the sound information and metadata. Here, the file containing the information indicating the associated bitstream or file is, for example, a control file such as a manifest file for distributing content.

[0141] The extraction unit (not shown) decodes the encoded metadata and provides the decoded metadata to the control unit 14. The control unit 14 provides the acquired metadata to the sound processing unit 15 and the rendering unit 16. Here, the control unit 14 may provide metadata required by each processing unit instead of providing the same metadata to each of the multiple processing units such as the sound processing unit 15 and the rendering unit 16.

[0142] In addition, the acquisition unit 11 acquires detection information, which includes the rotation amount or displacement amount detected by the sensor 20, and the position and orientation of the user U. The acquisition unit 11 determines the position and orientation of the user U in the sound reproduction space based on the acquired detection information. More specifically, the acquisition unit 11 determines that the position and orientation of the user U represented by the acquired detection information are the position and orientation of the user U in the sound reproduction space. Next, the update unit 12 updates the position information included in the metadata based on the determined position and orientation of the user U. Therefore, the metadata provided by the control unit 14 to the audio processing unit 15 and the rendering unit 16 is metadata including the updated position information.

[0143] In the present embodiment, the stereo sound processing device 10 has a function as a renderer that generates a sound signal with sound effects added, but the server may assume all or part of the functions of the renderer. That is, all or part of the extraction unit (not shown), the acquisition unit 11, the update unit 12, the storage unit 13, the control unit 14, the sound processing unit 15, and the rendering unit 16 may also exist in the server (not shown). In this case, the sound signal generated in the server or the synthesized sound signal is received by the stereo sound processing device 10 via a communication module (not shown) and reproduced by the sound output device 30.

[0144] Industrial Applicability

[0145] The present disclosure is useful in an apparatus or the like that processes sound information representing sound output by an AR device.

[0146] Description of symbols

[0147] 1Stereo sound reproduction system

[0148] 1a AR device

[0149] 10Stereo sound processing device

[0150] 11 Acquisition Department

[0151] 12 Update Department

[0152] 13 Storage

[0153] 14Control Unit (Decision Unit)

[0154] 15Sound Processing Department

[0155] 16 Rendering Department

[0156] 20 sensors

[0157] 30, 40 sound device

[0158] 50 people

[0159] 200, 200a space

[0160] 210 shape model

[0161] 300 Real Space

[0162] UUser

Claims

1. A stereo sound processing method for reproducing stereo sound using an augmented reality device, i.e., an AR device, wherein: During the process of outputting the content including the sound in the AR device, obtaining change information indicating the change in the space where the AR device is located, determining one or more sound processings based on the change information among a plurality of sound processings for rendering sound information representing the sound, executing the audio processing only for the determined one or more audio processings among the plurality of audio processings, The sound information is rendered based on the first processing results of each of the one or more audio processes that have been executed.

2. The stereo sound processing method according to claim 1, wherein: In rendering the sound information, the sound information is rendered based on the first processing results of each of the one or more sound processings and the pre-acquired second processing results of each of the other one or more sound processings among the plurality of sound processings other than the one or more sound processings.

3. The stereo sound processing method according to claim 1 or 2, wherein: The change information includes information indicating an object that has changed in the space. In determining the one or more acoustic processing, the one or more acoustic processing is determined based on at least one of an acoustic characteristic of the object and a position of the object.

4. The stereo sound processing method according to claim 3, wherein: In determining the one or more acoustic processing, the acoustic characteristics of the object and the position of the object are used, Based on the position of the object, determining whether to execute the one or more audio processing corresponding to the object, If it is determined that the one or more acoustic processing is to be executed, the one or more acoustic processing is determined based on the acoustic characteristics of the object.

5. The stereo sound processing method according to claim 1 or 2, wherein: The change information includes information indicating an object that has changed in the space, and the one or more acoustic processes are performed using a simplified shape model of the object.

6. The stereo sound processing method according to claim 5, wherein: Based on the type of the object, a shape model corresponding to the object is read from a storage unit that stores a plurality of shape models in advance, thereby acquiring the shape model.

7. The stereo sound processing method according to claim 5, wherein: inserting the above-mentioned shape model into the spatial information representing the above-mentioned space, The one or more acoustic processes are determined based on the spatial information after the shape model is inserted.

8. A stereo sound processing device for reproducing stereo sound using an augmented reality device (AR device), wherein: have: an acquisition unit that acquires change information indicating a change in a space where the AR device is located during a process in which the content including the sound is outputted by the AR device; a determination unit that determines one or more sound processings based on the change information among a plurality of sound processings for rendering sound information representing the sound; an audio processing unit that performs audio processing only on the one or more audio processings determined among the plurality of audio processings; and The rendering unit renders the sound information based on a first processing result of each of the one or more audio processings executed.

9. A program for causing a computer to execute the stereophonic sound processing method according to claim 1 or 2.

Images