Information processing system, information processing method, and information processing program

Abstract

To enable execution of content disposed in a space even in a situation where positioning by a positioning satellite is difficult, and reduce the cost of providing a sound source in accordance with the space.SOLUTION: A unit held by a user scans a space to acquire a distance to an object present in a space, coordinate information of the object in the space is calculated on the basis of the acquired distance, an anchor virtually disposed in the space in advance is detected on the basis of the calculated coordinate information and position information of the anchor, reverberation characteristics in the space are calculated, a sound source is adjusted on the basis of the calculated reverberation characteristics, and provision of content related to the adjusted sound source to the user is controlled when the anchor is detected.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there is a technology that sets an area within a predetermined distance from a geographical coordinate position as a geofence, and starts an event such as providing content to a mobile terminal when the mobile terminal, which can acquire position information by a positioning satellite such as a GPS (Global Positioning System), enters the set geofence. For example, Patent Document 1 discloses a technology in which a mobile client device has a GPS receiver, and provides print data such as an advertisement when the mobile client device enters within a threshold distance of a geographical position defined by the geofence.

[0003] There is also a technology for executing an event arranged in a space in VR (Virtual Reality) by superimposing an image on a video captured by a camera, etc. For example, Patent Document 2 discloses a technology related to a device for arranging a pre-registered object on an image captured by a camera and playing it back. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-208077 A [Patent Document 2] JP 2019-186921 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology of Patent Document 1, the accuracy of positioning varies depending on the reception state of radio waves from the positioning satellite, and for example, the accuracy of positioning by the positioning satellite may decrease or positioning may be impossible in places with poor visibility in the sky or indoors. Therefore, in a system in which an event is executed according to the positioning result by the positioning satellite, the accuracy of detection of whether or not the terminal has entered the geofence may decrease, causing the accuracy of the start of the event to decrease, such as a delay in the timing of the start of the event.

[0006] Furthermore, in Patent Document 2, in order to accurately detect the user's position, multiple sensors need to be installed, and the cost of building the system may increase in order to improve the accuracy of the start conditions for starting an event virtually placed in space.

[0007] Furthermore, when playing sound sources as part of an event, the reverberation characteristics differ depending on factors such as the shape of the space in which the sound source is played, and so content providers are required to adjust each sound source according to the reverberation characteristics of each individual space, which can increase the costs of providing content.

[0008] The present invention has been made in consideration of the above-mentioned circumstances, and has as its object to provide an information processing system, an information processing method, and an information processing program that can execute content placed in a space even in situations where positioning using positioning satellites is difficult, and that can reduce the cost of providing sound sources according to the space. [Means for solving the problem]

[0009] In order to solve the above problems, the information processing system is an information processing system for providing content to a user, and includes an acquisition unit that is held by the user and scans a space to acquire a distance to an object present in the space, a coordinate information calculation unit that calculates coordinate information of the object in space based on the distance acquired by the acquisition unit, an anchor detection unit that detects an anchor based on the coordinate information calculated by the coordinate information calculation unit and position information of an anchor that has been virtually placed in advance in the space, a reverberation characteristic calculation unit that calculates reverberation characteristics in the space, a sound source adjustment unit that adjusts the sound source based on the reverberation characteristics calculated in the reverberation characteristic calculation unit, and a content control unit that controls the provision of content related to the sound source adjusted in the sound source adjustment unit to the user when an anchor is detected by the anchor detection unit. Effect of the Invention

[0010] According to one embodiment of the present invention, the device is held by a user, scans a space to obtain the distance to an object present in the space, calculates coordinate information of the object in the space based on the obtained distance, detects an anchor based on the calculated coordinate information and position information of an anchor that has been virtually placed in the space in advance, calculates reverberation characteristics in the space based on the calculated coordinate information, adjusts a sound source based on the calculated reverberation characteristics, and when an anchor is detected, controls the provision of content related to the adjusted sound source to the user, thereby making it possible to execute content placed in the space even in situations where positioning using a positioning satellite is difficult, and reducing the cost of providing a sound source according to the space. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram illustrating an example of a system configuration of an information processing system according to an embodiment. [Diagram 2] 1A to 1C are diagrams illustrating a method for installing an anchor in an embodiment. [Diagram 3] 1A to 1C are diagrams illustrating a method for installing an anchor in an embodiment. [Figure 4]1A to 1C are diagrams illustrating a method for installing an anchor in an embodiment. [Diagram 5] 1A to 1C are diagrams illustrating a method for installing an anchor in an embodiment. [Figure 6] 1A and 1B are diagrams illustrating an example of sound reflection in a space in an embodiment. [Figure 7] 3A is a diagram illustrating an example of an impulse sound source, (B) attenuation of sound by the impulse sound source, and (C) an impulse response in an embodiment. FIG. [Figure 8] 2 is a block diagram showing an example of a hardware configuration of an information processing terminal according to an embodiment. FIG. [Figure 9] 4 is a flowchart showing a first example of the operation of the information processing system in the embodiment. [Figure 10] 10 is a flowchart showing a second example of the operation of the information processing system in the embodiment. [Figure 11] FIG. 4 is a diagram illustrating an example of point cloud data of a measurement space in the embodiment. [Figure 12] FIG. 11 is a diagram illustrating an example of a histogram of the x-axis of point cloud data in the embodiment. [Figure 13] FIG. 13 is a diagram showing an example of calculation when curve fitting is performed on positive values ​​of a histogram in an embodiment. [Figure 14] FIG. 13 is a diagram showing an example in which the position of a user and the position of an anchor are plotted in a measurement space in an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] An information processing system, an information processing method, and an information processing program according to an embodiment of the present invention will be described in detail below with reference to the drawings.

[0013] First, the configuration of an information processing system will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the system configuration of an information processing system according to an embodiment.

[0014] In FIG. 1, an information processing system 1 includes an information processing terminal 10, an information processing terminal 20, an information processing device 30, and a content execution device 40.

[0015] The information processing terminal 10 is a terminal for setting content (sometimes called an event) that is virtually placed in a space and executed. The information processing terminal 20 is a terminal held by a user and allows the user to receive the content that is virtually placed in a space. The information processing device 30 is a device that stores the settings of the content placed in the information processing terminal 10 and provides the content to the user holding the information processing terminal 20 based on the stored content settings. Furthermore, the content execution device 40 is a device that provides the content to the user.

[0016] The content in this embodiment is information that is executed by the movement of the information processing terminal 20 and that operates a device in the information processing system 1. The device in the information processing system 1 is, for example, the information processing terminal 20 (or the information processing terminal 10) or the content execution device 40.

[0017] The operation of the device in the information processing system 1 is, for example, the reproduction of content, data communication, or the operation of an actuator. The reproduction of content is, for example, the reproduction of a sound source, the display of an image (moving image or still image), or the display of a 3D object. The data communication is, for example, the communication between the information processing terminal 20 and the information processing device 30, or the communication between the information processing device 30 and the content execution device 40. In the data communication, for example, the terminal information of the information processing terminal 20 (for example, a device ID, location information, acceleration sensor measurement value, or camera captured image) is provided to the information processing device 30. In the data communication, for example, the content execution device 40 (for example, an audio device or a video device) may be controlled by data communication from the information processing terminal 20. In addition, the operation of the actuator is the operation of an actuator such as a motor, an air cylinder, or an air valve, for example, an animal-shaped object or a door driven by a motor or the like, or an operation to inflate a balloon that is inflated by air. In the information processing system 1, the content is provided to the user by the operation of these devices.

[0018] In the following embodiment, the execution of content is exemplified by the reproduction of a sound source in the information processing terminal 20 and the content execution device 40. However, the execution of content in the information processing system 1 is not limited to this.

[0019] The information processing terminal 10 and the information processing terminal 20 are devices that are held and carried (carried) by users, and are, for example, a smartphone, a tablet PC, or a head-mounted, glasses-type, or wristwatch-type wearable device. The information processing device 30 is a device that is communicably connected to the information processing terminal 10 and the information processing terminal 20 via the network 9, and is, for example, a server device. Note that the user in this embodiment refers to the user of the information processing terminal 10, the information processing terminal 20, or the information processing device 30, and may be different users or the same user. The user may be a dedicated operator, or a general user other than a dedicated operator.

[0020] The information processing terminal 10 has an acquisition unit 11, a display unit 12, an anchor placement unit 13, an anchor information provision unit 14, a content setting unit 15, and a content information provision unit 16. The information processing terminal 20 has an acquisition unit 21, a coordinate information calculation unit 22, a coordinate information provision unit 23, and a content execution unit 24. The information processing device 30 has an anchor information storage unit 31, an anchor detection unit 32, an acoustic characteristic storage unit 33, a reverberation characteristic calculation unit 34, a sound source adjustment unit 35, and a content control unit 36. The content execution device 40 has a content execution unit 41. The above-mentioned functional units of the information processing system 1 in this embodiment will be described as functional modules realized by the information processing program (software) in this embodiment.

[0021] The acquisition unit 11 and the acquisition unit 21 acquire scan data obtained by scanning a sensor that scans a space. The sensor that scans a space is a sensor that scans the shape of an object that exists in the space, and is, for example, a distance measurement sensor that measures the distance to an object that exists in the space. For example, a sensor using LiDAR (Light Detection And Ranging) technology can be used as the distance measurement sensor. For example, laser light is used in LiDAR. LiDAR may use ultraviolet light, infrared light, or near infrared light. LiDAR can measure the shape of an object that exists in the space in three dimensions by measuring the distance from the sensor to the object (distance measurement). Measuring the shape of an object in three dimensions means expressing the shape of an object that exists in the space as coordinate information in the space. The coordinate information in the space is a two-dimensional or three-dimensional coordinate value relative to a reference point (coordinate) in the space. For example, the surface shape of an object that exists in the space can be expressed as a set of coordinate values ​​(point cloud data). The set of coordinate values ​​may be expressed as, for example, a line, a surface, or a solid. The coordinate information is information that indicates the surface shape of the space, and may be called space information.

[0022] The scan data obtained by scanning the space may be, for example, images captured by a stereo camera (not shown) included in the information processing terminal 10 or the information processing terminal 20. The scan data obtained by scanning the space may also include information acquired by a position sensor or an acceleration sensor (not shown) included in the information processing terminal 10 or the information processing terminal 20.

[0023] The display unit 12 displays the image captured by the camera on a display (e.g., a display such as a liquid crystal display). For example, the display unit 12 displays the image captured by the camera of a smartphone on the display of the smartphone, thereby allowing the user to check the image capture results. The display unit 12 may also display the shape of an object based on coordinate information calculated based on the scan data acquired by the acquisition unit 11.

[0024] The anchor placement unit 13 provides the user with an operation (UI: User Interface) for placing a virtual anchor on the captured image displayed on the display unit 12 based on the scan data acquired by the acquisition unit 11. A virtual anchor is a mark (marker) virtually installed (placed) in space, and has position information (coordinate values) indicating a specific position in the space. The anchor in this embodiment is, for example, a mark indicating a position where a content such as a sound source is placed. The shape of the mark is, for example, a sphere, a cube, or an icon image, and may be displayed on the display unit 12 and visually recognized by the user. The user can place the anchor by operating the UI provided by the anchor placement unit 13. Note that, in this embodiment, the case where the anchor placement unit 13 of the information processing terminal 10 provides a UI for placing the anchor has been exemplified, but the UI for placing the anchor may be provided by the information processing terminal 20. In other words, the user of the information processing terminal 20 may be able to place the anchor.

[0025] An object existing in space is photographed by a camera, and its shape is acquired based on the scan data acquired by acquisition unit 11. Therefore, coordinate information indicating the shape of an object included in the photographed image displayed on display unit 12 is specified. The anchor placement unit 13 displays the photographed image on display unit 12 and provides a UI that enables the user to place an anchor in the photographed image, thereby enabling the user to acquire the coordinate value at which the anchor is placed without being aware of the spatial coordinate value. The user can set the position at which the anchor is placed while checking the photographed image displayed on display unit 12.

[0026] The anchor placement unit 13 further provides the user with an operation to correct the position at which the anchor is placed by using the captured image displayed on the display unit 12. Since the position at which the anchor is placed is determined while checking the two-dimensional captured image displayed on the display unit 12, the placement position may be unclear in the depth direction of the display screen. For example, the anchor placement unit 13 can correct the position at which the anchor is placed by moving the position of an anchor once placed a predetermined distance in the depth direction (e.g., 1 m or 50 cm, etc.). By correcting the position at which the anchor is placed, it is possible to improve the efficiency of anchor placement compared to the case where the anchor is re-placed.

[0027] The anchor information providing unit 14 provides the information processing device 30 with anchor information related to the anchors placed by the anchor placement unit 13. The anchor information related to the anchor includes, for example, coordinate information in space. The coordinate information in space is coordinate information based on scan data obtained by scanning the space, and is, for example, coordinate information indicating a relative position from an object with a large feature value existing in the space. The feature value of the space is, for example, a distribution (histogram) of color data existing in the space, the shape of the object, or vector data of color data. When recognizing the space, a place with similar feature values ​​can be recognized as the same place. Since the position of an object with a large feature value has a large difference in feature value from the surroundings, it is possible to reduce the error in the position of the object in the coordinate information. On the other hand, since the position of an object with a small feature value has a small difference in feature value from the surroundings, the error in the position of the object in the coordinate information is large. By using the relative position from an object with a large feature value existing in the space as coordinate information, it is possible to reduce the error in the position of the object in the coordinate information, for example, even when placing an anchor at the position of an object with a small feature value. The above-mentioned coordinate information is based on scan data obtained by scanning a space, and is therefore relative position information from an object existing in the space. However, the coordinate information may include, for example, coordinate information calculated from a relative distance from a reference point (beacon) that emits radio waves, absolute coordinate information consisting of longitude, latitude, and altitude calculated from radio waves obtained from a positioning satellite, etc. For example, by using coordinate information based on two calculation methods, the coordinate information can be double-checked, thereby improving the reliability of the coordinate information.

[0028] In this embodiment, the term "provision" may refer to either push-type transmission or pull-type transmission. In this embodiment, the term "acquisition" may refer to either pull-type reception or push-type reception.

[0029] The content setting unit 15 provides the user with an operation screen (UI) for a content placement operation for placing content in relation to anchor information provided to the information processing device 30. The content setting unit 15 also provides the user with an operation screen (UI) for a start condition setting operation for setting a start condition for starting the content to be placed in the content placement operation. The user can perform the content placement operation by operating the UI provided by the content setting unit 15. Note that, in the present embodiment, the case has been exemplified in which the content setting unit 15 of the information processing terminal 10 provides a UI for performing the content placement operation, but the UI for performing the content placement operation may be provided by the information processing terminal 20. In other words, the user of the information processing terminal 20 may be able to place the content.

[0030] The content placement operation is an operation of placing a content (event) to be executed for anchor information, for example, an operation of assigning a content to be played to a placed anchor. The content allocation includes, for example, an operation of selecting and assigning (associating) a playlist of a sound source that has been created in advance or that has been created newly. A playlist is a process of determining a sound source to be played using one or more sound source files, and by assigning a playlist name, it is possible to easily identify the sound source to be played. By providing the content placement operation to a user, for example, it becomes possible to change the content to be executed simply by changing the allocation of content to an anchor, and it becomes possible to improve the operability of content placement. Note that the content to be placed may be content playback such as image playback, data communication, or actuator operation, as described above. In the content placement operation, for example, a content that operates an actuator that drives an animal-shaped object may be placed on an anchor placed near the object.

[0031] The start condition for content execution is a condition for content provision to be executed in the information processing terminal 20 or the like, and in this embodiment, is a condition for determining whether the information processing terminal 20 has entered the geofence of the anchor at which the information processing terminal 20 is installed. The geofence can be set, for example, by the distance between the anchor and the information processing terminal 20, the shooting direction of the camera of the information processing terminal 20, the altitude of the information processing terminal 20 (such as the relative altitude difference with respect to the anchor or the absolute altitude based on sea level), the display time of the anchor, or the stop time of the information processing terminal 20. The geofence may also include a condition such as whether the distance between the anchor and the information processing terminal 20 is approaching or receding. The geofence may also include the fact that the anchor is photographed by multiple information processing terminals 20. The geofence may also include the behavior of the user acquired from an acceleration sensor (not shown) or the like.

[0032] The content setting unit 15 provides the user with a UI for adjusting the sound source when arranging the sound source as the content. Adjusting the sound source means adjusting (setting) the reproduction mode of the sound reproduced from the sound source according to the space. The reproduction mode of the sound source is, for example, the volume, frequency characteristics, or reverberation characteristics.

[0033] <Loudness adjustment> The loudness of a sound emitted from a sound source in a space and heard by a user is affected by the distance from the sound source to the user, or the reflection of the sound by objects in the space. Therefore, when playing back a sound source, it is necessary to adjust the sound source so that the sound is at an appropriate volume. The content setting unit 15 provides a UI for setting an appropriate sound volume.

[0034] <Frequency characteristic adjustment> The frequency characteristics of a sound emitted from a sound source in a space and heard by a user change due to the reflection of the sound by objects in the space. Therefore, for better expression, it is desirable to adjust the sound source to have appropriate frequency characteristics. Frequency characteristic adjustment means adjusting the frequency distribution of the sound emitted from the sound source. Frequency specific adjustment includes, for example, shelving type adjustment that collectively adjusts a range of sounds higher (or lower) than a specific frequency, or peak-dip type adjustment that adjusts a range of sounds around a specific frequency. The content setting unit 15 provides a UI for setting appropriate frequency characteristics.

[0035] <Reverberation characteristics adjustment> The reverberation characteristics of a sound emitted from a sound source in a space and heard by a user vary depending on the reflection of the sound by objects in the space. Therefore, it is necessary to adjust the sound source so that the reverberation characteristics are appropriate when playing the sound source. The content setting unit 15 provides a UI for setting appropriate reverberation characteristics. A specific method for adjusting the reverberation characteristics will be described later.

[0036] The content setting unit 15 may be able to set a playback mode of the sound source other than the above. For example, the content setting unit 15 may be able to set the playback time, playback speed, etc. of the sound source. Furthermore, when the content is the operation of an actuator, the content setting unit 15 may be able to set the motion range, motion speed, etc. of the actuator.

[0037] The content execution mode may also include a mode in which the content execution mode changes depending on the distance between the anchor and the information processing terminal 20, the shooting direction of the camera of the information processing terminal 20, the altitude of the information processing terminal 20, the display time of the anchor, or the stop time of the information processing terminal 20, etc. By setting the playback conditions, it becomes possible to perform various effects using the sound source.

[0038] The content information providing unit 16 provides the information processing device 30 with content placement information for which the content placement operation has been performed in the content setting unit 15, and the start condition set in the start condition setting operation. By providing the placement information and the start condition to the information processing device 30, it becomes possible to store this information in the information processing device 30. By storing this information in the information processing device 30, it becomes possible to read and use the stored information. Note that the placement of anchors or the setting of content may be performed in multiple information processing terminals 10. When the placement of anchors or the like is performed in multiple information processing terminals 10, the execution result of the placement of anchors or the like in one information processing terminal 10 is stored in the information processing device 30 and is reflected in the other information processing terminals 10.

[0039] The coordinate information calculation unit 22 calculates coordinate information of the object in space based on the distance to the object acquired by the acquisition unit 21. The acquisition unit 21 scans the space as described above to acquire the distance to the object existing in the space. The coordinate information calculation unit 22 can measure the shape of the object by calculating coordinate information of the surface of the object from multiple distances to the surface of the object.

[0040] Here, a specific calculation example of the coordinate information in the coordinate information calculation unit 22 will be described with reference to Figs. 11 to 14. Fig. 11 shows point cloud data in a measurement space acquired by the acquisition unit 11 or the acquisition unit 21. Fig. 11 illustrates a case where the measurement space is box-shaped (cubic). The box-shaped measurement space is composed of six faces. The faces in the measurement space can be calculated by calculating dense points on each coordinate axis in three dimensions of the point cloud data. The dense points on each coordinate axis can be calculated by obtaining the distribution on each coordinate axis of the point cloud data.

[0041] <Histogram calculation> Figure 12 shows the histogram of the x-axis of the point cloud data in Figure 11. The histogram is the distribution of the point cloud data on the x-axis. In Figure 12, the point cloud data on the x-axis coordinate has peaks at approximately -4.5 and 3.3. This indicates that there are two planes on the x-axis in the measurement space.

[0042] <Curve fitting calculations> FIG. 13 is a calculation example in which curve fitting is performed on the positive values ​​of the histogram in FIG. 12. By performing curve fitting on the histogram, the peak can be calculated as a wall on the coordinate axis. FIG. 13 has a peak at approximately 3.3, and it is calculated that this point is one surface (wall) on the x-coordinate axis. In FIG. 13, a smoothing spline method is used for curve fitting. The smoothing spline is a method of estimating a function using a spline curve while balancing the smoothness based on the second derivative from an observed value that includes noise. FIG. 13 is calculated using a three-dimensional spline curve. Note that FIG. 13 uses a smoothing spline method for curve fitting, but for example, a method using a Gaussian function for curve fitting may be used.

[0043] The coordinate information calculation unit 22 can calculate two planes on each coordinate axis by the above-mentioned method, and calculate the coordinate information of a box shape on six planes. The coordinate information may include the position of the user who receives the content, and the position information of the anchor detected by the anchor detection unit 32, which will be described later, relative to the coordinate information in the measurement space calculated by the coordinate information calculation unit 22. Fig. 14 illustrates an example in which the user's position (Listener) and the anchor's position (Sound) are plotted in the measurement space.

[0044] 11 to 14, the case where coordinate information is calculated in a measurement space composed of six surfaces has been described, but the measurement space may have a more complicated shape. For example, the measurement space may have a shape in which pillars and beams are included in a room. The measurement space may also have a curved surface, such as a dome-shaped ceiling.

[0045] The coordinate information providing unit 23 provides the coordinate information calculated by the coordinate information calculating unit 22 to the information processing device 30. The information processing device 30, which will be described later, can detect an anchor from the coordinate information of the object provided by the coordinate information providing unit 23.

[0046] The content execution unit 24 executes the provision of content to the user when the information processing terminal 20 including the acquisition unit 21 enters the geofence. The content execution unit 24 plays back a sound source in a speaker (not shown) of the information processing terminal 20, for example.

[0047] In this embodiment, the OSC (Open Sound Control) protocol may be used to provide the sound source. By using the OSC protocol in the content execution unit 24, it is possible to provide the sound source in real time. For example, if the sound source is a live sound source, by using the OSC protocol, it is possible to share the live sound source in real time among a plurality of information processing terminals 20. In addition, in this embodiment, a sound source format (e.g., Ambisonics) for realizing stereophonic sound may be used to provide the sound source. By using a sound source format for realizing stereophonic sound in the content execution unit 24, it is possible to easily provide a stereophonic sound source. In addition, in this embodiment, a platform for game development (e.g., Wwise, etc.) may be used to provide the sound source. By using a platform for game development in the content execution unit 24, it is possible to easily provide events in a game. In addition, in this embodiment, the sound source may be reproduced using a head related transfer function (HRTF) to provide the sound source. The head related transfer function is a transfer function that expresses changes in sound caused by surrounding objects including the auricle, the human head, and shoulders. In the content execution unit 24, the characteristics of the change in sound from the sound source to both ears are quantified using a head-related transfer function, making it possible to recognize the sense of direction of the sound source and the sense of distance to the sound source. When adjusting the sound source, the adjustment result can be expressed as the above-mentioned OSC, etc.

[0048] The anchor information storage unit 31 readably stores anchor information related to the anchors arranged by the anchor arrangement unit 13. As described above, the anchor information can include position information of the anchor. The anchor information storage unit 31 can store multiple pieces of anchor information. The multiple pieces of anchor information stored in the anchor information storage unit 31 are read by the anchor detection unit 32 and used to detect the multiple anchors arranged in space. The anchor information stored in the anchor information storage unit 31 may be provided to the information processing terminal 20. In this embodiment, the anchor detection is performed by the anchor detection unit 32, but the anchor detection may be performed by the information processing terminal 20.

[0049] The anchor detection unit 32 detects an anchor virtually placed in space based on the coordinate information provided by the coordinate information providing unit 23 and the position information of the anchor stored in the anchor information storage unit 31 and virtually placed in space in advance. Here, the coordinate information and the position information are coordinates of a point expressed in three dimensions or two dimensions in space. The coordinate information and the position information may be the above-mentioned feature amount of space. The feature amount of space is a distribution (histogram) of color data existing in the space, a shape of an object, or vector data of color data, etc. The anchor detection unit 32 can detect an anchor virtually placed in space based on the scan data of the space, and therefore can detect an anchor placed in space even in a situation where positioning by a positioning satellite is difficult. This improves the accuracy of the provision conditions of the content placed in the space and enables the provision of content using a geofence set in three dimensions.

[0050] <Detecting anchors by coordinate distance> For example, the anchor detection unit 32 may compare the position information of the anchor stored in the anchor information storage unit 31 with the coordinate information provided by the coordinate information providing unit 23, and detect the anchor based on whether a point included in the coordinate information is within a predetermined distance from the position information.

[0051] <Anchor detection using features> In addition, the anchor detection unit 32 may calculate features from the coordinate information provided by the coordinate information provision unit 23, and detect the anchor based on whether or not the features are similar to the features stored in the anchor information storage unit 31 at the time the anchor was placed.

[0052] The anchor detection unit 32 may detect anchors by sequentially referring to a plurality of pieces of anchor information stored in the anchor information storage unit 31. By the anchor detection unit 32 referring to a plurality of pieces of anchor information, it is possible to easily add or delete anchors to be placed.

[0053] Furthermore, the anchor detection unit 32 may detect anchors based on a detection order of anchors expected in advance. For example, if anchor B is expected to be detected after anchor A, the anchor detection unit 32 detects anchor A and then preferentially detects anchor B, thereby reducing the load of anchor detection and shortening the detection time.

[0054] Furthermore, when an anchor has already been detected for one user and content corresponding to that anchor has already been provided, the anchor detection unit 32 may exclude the already detected anchor from the detection target. By excluding the already detected anchor from the detection target, it becomes possible to prevent duplicate provision of content associated with the anchor.

[0055] Furthermore, the anchor detection unit 32 may count the number of detections when one anchor is detected multiple times by one user or by different users. By counting the number of detections, it becomes possible to change the content provided depending on the number of detections. The provision of content can be controlled by the content control unit 36, which will be described later.

[0056] The anchor detection unit 32 may detect the anchor based on a measurement value acquired from a GNSS (Global Navigation Satellite System) sensor or an acceleration sensor (not shown). By detecting the anchor in combination with the measurement values ​​of these sensors, the detection accuracy of the anchor can be improved. For example, outdoors, the GNSS sensor may be able to measure the current position accurately. Furthermore, by using an acceleration sensor, the movement of the information processing terminal 20 may be recorded and the current position may be detected accurately by comparing it with the position before the movement.

[0057] The acoustic characteristic storage unit 33 stores in advance acoustic characteristics related to the surface of an object existing in a space. The acoustic characteristics related to the surface of an object are acoustic characteristics (sometimes called reflection characteristics) when a sound emitted from a sound source is reflected on the surface of the object. The acoustic characteristics stored in the acoustic characteristic storage unit 33 are, for example, the reflectance of sound (sometimes called sound absorption rate or attenuation rate). For example, the reflectance is high on the surface of a hard material such as metal because the sound transmitted inside the material is small, and the reflectance is low on the surface of a porous material because the sound transmitted inside is large. The acoustic characteristics stored in the acoustic characteristic storage unit 33 may also include the surface shape of the object. In a surface shape with irregularities, reflected sounds may interfere with each other, causing the reflection characteristics of a specific frequency to change. The acoustic characteristics stored in the acoustic characteristic storage unit 33 may also include acoustic characteristics for each frequency. The reflectance of sound on the surface of an object differs for each frequency of sound. For example, a porous material has a low reflectance for a high-frequency sound, and a high reflectance for a low-frequency sound. By storing in advance in the acoustic characteristics memory unit 33 the acoustic characteristics related to the surface of an object existing in space, it is possible to calculate the changes in sound that occur when sound is emitted from a sound source, reflected by the surface of the object, and reaches the user's position.

[0058] The reverberation characteristics calculation unit 34 calculates the reverberation characteristics in the space based on the coordinate information calculated by the coordinate information calculation unit 22. The reverberation characteristics are characteristics of a sound that is emitted from a sound source, reflected on the surface of an object in the space, and reaches the user with a time difference. The reverberation characteristics calculated by the reverberation characteristics calculation unit 34 are exemplified below. Note that the reverberation characteristics shown below are examples of changes in sound that can be heard at the user's position in response to an impulse sound source whose volume changes from a constant volume to 0 at a predetermined time (impulse response), but the reverberation characteristics calculated by the reverberation characteristics calculation unit 34 are not limited to this.

[0059] <Reverberation time> Reverberation time is a characteristic that indicates the length of time that sound reverberates in a space. Reverberation time is the time (impulse response) until the intensity of the sound at the observation point (user's position) attenuates by 60 dB after a sound source is played and emitted into the space and the volume reaches a steady state and then the sound source is stopped (impulse sound source). Reverberation time can be calculated using the following formula:

[0060]

number

[0061] <Initial reverberation time> The initial reverberation time is the reverberation time calculated from the decay slope of the first 10 dB of the reverberation decay. The initial reverberation time is a characteristic that represents the sense of reverberation felt by humans. The decay waveform used for the initial reverberation time is the same as the reverberation time.

[0062] <Time center of gravity> This characteristic is expressed by the following formula.

[0063]

number

[0064] <clarity> Here, a method for calculating clarity will be described with reference to Fig. 7. Fig. 7 is a diagram for explaining an example of (A) an impulse sound source, (B) attenuation of sound by the impulse sound source, and (C) an impulse response in an embodiment.

[0065] In Fig. 7(A), the x-axis represents the passage of time (t), and the y-axis represents the sound pressure (p) at the sound source. The impulse sound source changes from a steady-state sound pressure p1 to a sound pressure of 0 (playback stopped) at time t0. In Fig. 7(B), the y-axis represents the sound pressure (E) at the user's position (measurement position). Measurement of sound pressure E begins at time t0. Sound pressure E decays over time.

[0066] In FIG. 7(C), the y-axis is the ratio (logarithm) of the direct sound that arrives at the measurement position directly from the sound source to the reverberation sound that arrives at the measurement position after reflecting off the target object, calculated using the following formula, and is a characteristic called C80.

[0067]

number

[0068] <Direct sound total energy ratio> The direct sound total energy ratio is the energy ratio of the direct sound to the whole sound (direct sound + reverberation sound) calculated by the following formula. The direct sound total energy ratio is mainly used as an indicator of the intelligibility of the human voice.

[0069]

number

[0070] The above-mentioned reverberation characteristics may be measured in the space in which the sound source is reproduced, or may be calculated by storing reverberation characteristics in other spaces with similar coordinate information in the space and inferring them from the shape of the object in the space and the acoustic characteristics stored in the acoustic characteristics storage unit 33. For example, reverberation characteristics for different types of spaces (e.g., concert halls, churches, rooms surrounded by concrete, corridors, etc.) may be stored in advance, and the reverberation characteristics in the space may be calculated by modifying the stored reverberation characteristics based on the coordinate information calculated by the coordinate information calculation unit.

[0071] Next, an example of the calculation of reverberation characteristics in the reverberation characteristics calculation unit 34 will be described. The calculation of reverberation characteristics can be performed using, for example, the Mirror Image Method by Allen and Berkley (hereinafter, sometimes abbreviated as the "MI method"). In the MI method, the following parameters are used. Speed ​​of sound [m / s] Sampling rate [Hz] Anchor position coordinates (x,y,z) [m] User position coordinates (x,y,z) [m] Room dimensions (x,y,z) [m] Reverberation time of the room [m] or reflection coefficient at each wall

[0072] Here, the speed of sound can be calculated based on the temperature of the room in which the measurement is performed. Speed ​​of sound = 331.5+0.6t[m / s] (t: Celsius, 1 atm) The sampling rate used is the one actually used in the device. Additionally, the anchor location, user location, and room dimensions are calculated in the manner described above.

[0073] The reverberation time of a room [m] or the reflection coefficient at each wall can be calculated from the reverberation time calculated from a single anchor position and user position. The reverberation time can be calculated by measuring the impulse response and calculating the reverberation time based on the measured impulse response.

[0074] <Impulse response measurement> The impulse response is measured in a room (measurement space) whose shape has been estimated by calculating coordinate information. The impulse response is measured by measuring the impulse response described above, and specifically, it can be measured using a TSP (Time Stretched Pulse) signal, pink noise, a balloon, or clapping.

[0075] <Calculating reverberation time> The reverberation time is calculated from the impulse response obtained in the measurement of the impulse response. For example, the calculation method specified in ISO3382-2:2008 can be used to calculate the reverberation time.

[0076] The reverberation characteristics calculation unit 34 calculates the reverberation characteristics by the MI method based on the parameters obtained by the above-mentioned method. The reverberation characteristics calculation unit 34 may calculate the reverberation characteristics by a method other than the MI method. For example, the reverberation characteristics calculation unit 34 may also use the sensor information obtained by the acquisition unit 21 to estimate the material of each wall using an image recognition technology based on deep learning, and calculate the reflection coefficient using the information on the material, etc. There are cases where the material, etc. of a building material can be identified by its appearance, such as its shape. The reverberation characteristics calculation unit 34 can estimate the reverberation characteristics by learning the correspondence between the sensor information and the reflection coefficient by deep learning.

[0077] The sound source adjustment unit 35 adjusts the sound source based on the reverberation characteristics calculated by the reverberation characteristics calculation unit 34. As described above, the adjustment of the sound source is volume adjustment, frequency characteristic adjustment, reverberation characteristic adjustment, or the like. By adjusting the sound source, the sound source adjustment unit 35 can easily adjust the sound source to be provided according to reverberation characteristics that differ depending on the shape of the space in which the sound source is reproduced. Note that the sound source adjustment by the sound source adjustment unit 35 may be set to be performed or not (enabled / disabled). For example, when the sound source adjustment unit 35 is set to be enabled, the sound source adjustment unit 35 may automatically adjust the sound source when the reverberation characteristics are calculated, while when the sound source adjustment unit 35 is set to be disabled, the sound source adjustment may be manually performed.

[0078] By adjusting the sound source, for example, it is possible to reproduce the sound source with the reverberation or volume intended by the content provider. Also, by adjusting the sound source, for example, when reproducing the sound source in an event hall, it is possible to give the user a reverberation feeling as if they were in a church.

[0079] The content control unit 36 ​​controls the provision of the content when the start condition is satisfied according to the arrangement information or the start condition. The content control unit 36 ​​may control the provision of the content in a predetermined content execution mode according to a preset start condition. For example, the content control unit 36 ​​may provide the content by changing the content execution mode, such as the playback mode of the sound source, according to the identification information of the information processing terminal 20 (pre-registered attribute information of the user such as gender, age, language used, nationality, or religion, or device information acquired from the information processing terminal 20). The identification information may be acquired, for example, in association with the user's account information. The content execution unit 24 executes the content changed according to the identification information, thereby enabling the execution of various contents.

[0080] The content execution device 40 is a device different from the information processing terminal 20 that provides content to a user, and is, for example, a device that plays a sound source or operates an actuator. The content execution unit 41 controls the execution of content according to an instruction from the content control unit 36. The content execution unit 41 may, for example, cooperate with the content execution unit 24 to control the provision of content.

[0081] The above-mentioned functional units of the information processing system 1 are merely examples of functions, and do not limit the functions of the information processing system 1. For example, the information processing terminal 10, the information processing terminal 20, or the information processing device 30 does not need to have all the functional units described above, and may have only some of the functional units. The information processing system 1 may also have functions other than those described above.

[0082] For example, the functional units of the information processing terminal 10 may be realized in the information processing device 30. Similarly, the functional units of the information processing terminal 20 may be realized in the information processing device 30. Furthermore, the functional units of the information processing device 30 may be realized in the information processing terminal 10 or the information processing terminal 20. For example, the function of the anchor detection unit 32 may be realized in the information processing terminal 20.

[0083] As described above, each of the above functional units has been described as being realized by software, but at least one of the above functional units may be realized by hardware.

[0084] Also, any of the above functional units may be implemented by dividing one functional unit into a plurality of functional units. Also, any two or more of the above functional units may be implemented by integrating them into one functional unit. Fig. 1 shows the functions of the information processing system 1 in the form of functional blocks, and does not indicate, for example, that each functional unit is configured by a separate program file or the like.

[0085] Furthermore, the information processing terminal 10, the information processing terminal 20, or the information processing device 30 may be a device realized by one housing, or may be a system realized by a plurality of devices connected via a network or the like. For example, the information processing terminal 10, the information processing terminal 20, or the information processing device 30 may have some or all of its functions realized by another virtual device, such as a cloud service provided by a cloud computing system. That is, the information processing terminal 10, the information processing terminal 20, or the information processing device 30 may have at least one or more of the above-mentioned functional units realized by another device.

[0086] Next, a method for installing an anchor will be described with reference to Figures 2 to 5. Figures 2 to 5 are diagrams for explaining a method for installing an anchor in an embodiment.

[0087] 2, a display screen 1000 is displayed on the display unit 12. The display screen 1000 has an icon 1001 indicating an anchor setting screen, an anchor installation button 1002, an exit button 1003, a save button 1004, and a naming button 1005 for naming the playlist.

[0088] Display screen 1000 is a display screen displayed on information processing terminal 10, and displays an image captured by the camera of information processing terminal 10. Anchor installation button 1002 is a button for installing an anchor anywhere in the shooting range displayed on display screen 1000. For example, by operating anchor installation button 1002, an anchor can be installed at the center position of display screen 1000 or at a cursor position (not shown) or the like within display screen 1000. By operating anchor installation button 1002, a user can transition to a screen for placing an anchor, which will be described in FIG. 3.

[0089] An exit button 1003 is a button for ending the settings shown on the display screen 1000, and a save button 1004 is a button for saving the current settings. A name button 1005 is a button for naming the playlist. By operating the name button 1005, the user can input a name in a text box or the like.

[0090] In FIG. 3, a display screen 2000 is displayed on the display unit 12. The display screen 2000 has a trial placement button 2001. The display screen 2000 is a captured image that is enlarged by moving the shooting range displayed on the display screen 1000. The trial placement button 2001 is a button for trial placement of an anchor. A user can place an anchor substantially in the center of the display screen 2000 by operating the trial placement button 2001. Note that trial placement means placement of an anchor before the placement position is confirmed, and the user can trial experience the anchor that he or she placed and the sound source set to the anchor.

[0091] FIG. 4 is a diagram showing a third display example of the display device used in the information processing terminal in the embodiment.

[0092] In FIG. 4, the display screen 1000 is displayed on the display unit 12. The display screen 3000 has an anchor position correction button 3001, an anchor position determination button 3002, and an anchor 3003. The anchor position correction button 3001 is a button for correcting the position of the anchor 3003 set by the trial placement button 2001. The user can correct the position of the set anchor 3003 to the rear side or the front side by operating the anchor position correction button 3001. When the position of the anchor 3003 is corrected to the rear side, the size of the anchor 3003 is displayed smaller, whereas when the position of the anchor 3003 is corrected to the front side, the size of the anchor 3003 is displayed larger. The display size of the anchor 3003 may be changed by a setting operation not shown.

[0093] The anchor position determination button 3002 is a button for determining the installation position of the anchor 3003. By operating the anchor position determination button 3002, the user can transition to a screen for completing the installation of the anchor, which will be described with reference to FIG.

[0094] 5, a display screen 1000 is displayed on the display unit 12. A display screen 4000 has an end button 4001 and an anchor 4002. The user operates the end button 4001 to end the setting of the anchor.

[0095] Next, reflection of sound in a space will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining an example of reflection of sound in a space in the embodiment.

[0096] In FIG. 6, the sound source is assumed to be placed at an anchor position O inside a room R. The anchor position O is the position in space of the anchor installed in the above-mentioned procedure. The acquisition unit 21 of the information processing terminal 20 scans the space to acquire the distance to an object existing in the space. The user position S is the position where the user hears the sound source, and is inside the geofence at the anchor position O.

[0097] The sound emitted from the sound source at the anchor position O is divided into direct sound that reaches the user position S directly and reflected sound (reverberant sound) that reaches the user position S after reflecting off the walls of the room R. While Fig. 6 shows an example of reverberant sound that reaches the user position S after reflecting once at reflection point F, there are multiple paths that the reverberant sound takes depending on the direction in which the sound diffuses from the sound source.

[0098] The direct sound reaches the user position S at a distance of L1. On the other hand, the reverberant sound reaches the user position S at a distance of L2+L3. Therefore, the time difference until the direct sound reaches the user position S changes according to the distance. Also, the reverberant sound that reaches the user position S changes according to the acoustic characteristics inside the room R. The reverberant characteristics in the room R can be measured by playing a pulse sound source at the anchor position O and measuring the change in the sound (sound pressure, etc.) at the user position S with an acoustic analyzer or the like. Note that a dummy head with microphones placed at the ear positions of a human dummy can be used to measure the reverberant characteristics. By using the dummy head, for example, the acoustic characteristics perceived by humans in the measurement space can be measured.

[0099] Next, the hardware configuration of the information processing terminal 10 will be described with reference to Fig. 8. Fig. 8 is a block diagram showing an example of the hardware configuration of the information processing terminal 10 in the embodiment. Note that the hardware configurations of the information processing terminal 20 and the information processing device 30 are similar to that of the information processing terminal 10, and therefore descriptions thereof will be omitted.

[0100] The information processing terminal 10 includes a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, an I / O device 104, and a communication I / F (Interface) 105. The information processing terminal 10 is a device that executes the information processing program described in FIG.

[0101] The CPU 101 controls the user terminal by executing an information processing program stored in the RAM 102 or the ROM 103. The information processing program is obtained, for example, from a recording medium on which the program is recorded or a program distribution server via a network, and installed in the ROM 103, and is read out and executed by the CPU 101.

[0102] The I / O device 104 has an operation input function and a display function (operation display function). The I / O device 104 is, for example, a touch panel. The touch panel enables the user of the information processing terminal 10 to input operations using a fingertip, a touch pen, or the like. In this embodiment, the I / O device 104 is described as using a touch panel having an operation display function, but the I / O device 104 may have a display device having a display function and an operation input device having an operation input function separately. In that case, the display screen of the touch panel can be implemented as the display screen of the display device, and the operation of the touch panel can be implemented as the operation of the operation input device. The I / O device 104 may be realized in various forms such as a head-mounted type, a glasses type, or a wristwatch type display.

[0103] The communication I / F 105 is an I / F for communication. The communication I / F 105 executes short-distance wireless communication such as wireless LAN, wired LAN, infrared, etc. Although the figure illustrates only the communication I / F 105 as an I / F for communication, the information processing terminal 10 may have an I / F for each of a plurality of communication methods.

[0104] Next, the operation of the information processing system 1 will be described with reference to FIGS.

[0105] 9 is a flowchart showing a first example of the operation of the information processing system in the embodiment. The flowchart shown in the first example explains the operation related to the arrangement of anchors and sound sources.

[0106] 9, the information processing system 1 provides a UI for setting a sound source on the display screen of the information processing terminal 10 (step S11). After executing the process of step S11, the information processing system 1 acquires scan data from a sensor that scans a space (step S12).

[0107] After executing the process of step S12, the information processing system 1 judges whether or not to place an anchor (step S13). The judgment of whether or not to place an anchor can be made, for example, based on whether or not the user has completed the operation of setting the anchor. If it is judged that the anchor is not to be placed (step S13: NO), the information processing system 1 repeats the processes of steps S12 to S13 and waits for the anchor to be placed.

[0108] When it is determined that the anchor has been placed (step S13: YES), the information processing system 1 determines whether or not to modify the position of the anchor (step S14). When it is determined that the position of the anchor should be modified (step S14: YES), the information processing system 1 modifies the position of the anchor (step S15).

[0109] After executing the process of step S15, or when it is determined that the anchor position is not to be corrected (step S14: NO), the information processing system 1 determines whether or not a content list has been selected (step S16). The selection of a content list includes saving the selected content list. If there is no content list for the desired content, the information processing system 1 may provide a UI for acquiring new content and creating a content list. If it is determined that a content list has not been selected (step S16: NO), the information processing system 1 repeats the process of step S16 and waits for a content list to be selected.

[0110] On the other hand, if it is determined that a content list has been selected (step S16: YES), the information processing system 1 provides a UI for inputting execution conditions for executing the content on the display screen of the information processing terminal 10 (step S17).

[0111] After executing the process of step S17, the information processing system 1 judges whether the input of the playback conditions is completed (step S18). If it is judged that the input is not completed (step S18: NO), the information processing system 1 repeats the process of step S18 and waits for the input to be completed.

[0112] On the other hand, when it is determined that the input of the playback conditions is completed (step S18: YES), the information processing system 1 provides the anchor information input in steps S13 to S15 from the information processing terminal 10 to the information processing device 30 (step S19).

[0113] After executing the process of step S19, the information processing system 1 provides the information relating to the sound source input in steps S16 to S18 from the information processing terminal 10 to the information processing device 30 (step S20).

[0114] When the process of step S20 is completed, the information processing system 1 stores the anchor information provided in step S19 in the information processing device 30 (step S21). After the process of step S21 is completed, the information processing system 1 stores the sound source information provided in step S20 in the information processing device 30 (step S22), and ends the process shown in the flowchart.

[0115] 10 is a flowchart showing a second example of the operation of the information processing system 1 in the embodiment. The flowchart shown in the second example explains the operation related to the reproduction of a sound source as the provision of content.

[0116] 10, the information processing system 1 provides a UI for playing a sound source on the display screen of the information processing terminal 20 (step S31). The UI for playing a sound source is, for example, an AR (Augmented Reality) application program for enjoying the playback of a sound source placed in a space.

[0117] After executing the process of step S32, the information processing system 1 acquires scan data from a sensor that scans a space (step S32).

[0118] After executing the process of step S32, the information processing system 1 determines whether or not an anchor is detected (step S33). If it is determined that an anchor is not detected (step S33: NO), the information processing system 1 repeats the processes of steps S32 to S33 and waits for an anchor to be detected.

[0119] On the other hand, if it is determined that an anchor has been detected (step S33: YES), the information processing system 1 causes the detected anchor to be displayed on the display screen of the information processing terminal 20 (step S34).

[0120] After executing the process of step S34, the information processing system 1 determines whether or not the playback condition is satisfied (step S35). Whether or not the playback condition is satisfied can be determined, for example, by whether or not the information processing terminal 20 acquires playback condition information from the information processing device 30. The playback condition can include whether or not the information processing terminal 20 is inside a geofence. The information processing system 1 can determine that the playback condition is satisfied when the information processing terminal 20 enters the geofence.

[0121] If it is determined that the playback condition is not satisfied (step S35: NO), the information processing system 1 repeats the processes of steps S32 to S35 and waits until the playback condition is satisfied.

[0122] On the other hand, if it is determined that the reproduction condition is satisfied (step S35: YES), the information processing system 1 starts reproduction of the sound source (step S36). The reproduction of the sound source is, for example, reproduction of an impulse sound source.

[0123] After executing the process of step S36, the information processing system 1 calculates the reverberation characteristics (step S37). As described above, the calculation of the reverberation characteristics can be performed based on the measurement of the impulse response in the space of the object. After executing the process of step S36, the information processing system 1 adjusts the sound source and stores the adjustment results (step S38). By calculating the reverberation characteristics and adjusting the sound source, it is possible to easily adjust the sound source according to the space and reduce the cost of providing the sound source.

[0124] After executing the process of step S38, the information processing system 1 judges whether or not to end the playback (step S39). The decision to end the playback may be made, for example, when the terminal state no longer satisfies the playback conditions, when playback of a previously prepared sound source has ended, or when the user explicitly stops playback. If it is determined not to end the playback (step S39: NO), the information processing system 1 repeats the processes of steps S37 to S39 and waits for the playback to end. On the other hand, if it is determined to end the playback (step S39: YES), the information processing system 1 ends the operation shown in the flowchart.

[0125] It should be noted that the illustrated flowchart shows only one example of the operation and does not limit the operation.

[0126] The above-mentioned various processes of the present embodiment may be performed by recording a program for realizing the functions constituting the device described in the present embodiment in a computer-readable recording medium, and reading and executing the program recorded in the recording medium into a computer system. The "computer system" referred to here may include hardware such as an OS and peripheral devices. In addition, if a WWW system is used, the "computer system" also includes a homepage providing environment (or display environment). In addition, the "computer-readable recording medium" refers to a storage device such as a flexible disk, a magneto-optical disk, a ROM, a writable non-volatile memory such as a flash memory, a portable medium such as a CD-ROM, or a hard disk built into a computer system.

[0127] Furthermore, the term "computer-readable recording medium" includes a memory that holds a program for a certain period of time, such as a volatile memory (e.g., DRAM (Dynamic Random Access Memory)) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. The above-mentioned program may be transmitted from a computer system that stores the program in a storage device or the like to another computer system via a transmission medium, or by a transmission wave in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above-mentioned program may also be a program for realizing part of the above-mentioned functions. Furthermore, the above-mentioned function may be realized in combination with a program already recorded in the computer system, that is, a so-called difference file (difference program).

[0128] Although an embodiment of the present invention has been described above with reference to the drawings, the specific configuration is not limited to this embodiment, and various modifications are also included within the scope that does not deviate from the spirit of the present invention. [Explanation of symbols]

[0129] 1. Information Processing Systems 10. Information processing terminal 11 Acquisition Department 12 Display section 13 Anchor placement section 14 Anchor Information Department 15 Content Settings 16 Content Information Department 20 Information processing terminal 21 Acquisition Department 22 Coordinate information calculation unit 23 Coordinate information provision section 24 Content Execution Department 30 Information processing device 31 Anchor information storage unit 32 Anchor detection section 33 Acoustic characteristics memory section 34 Reverberation characteristic calculation section 35 Sound source adjustment section 40 Content execution device 41 Content Execution Department 101 CPU 102 RAM 103 ROM 104 I / O equipment 105 Communication I / F 1000 display screen 1001 Icon 1002 Anchor installation button 1003 Exit button 1004 Save button 1005 Naming button 1005 2000 display screen 2001 Trial Placement Button 3000 display screen 3001 Anchor position correction button 3002 Anchor position determination button 3003 Anchor 4000 display screen 4001 Exit button< / clarity>

Claims

1. An information processing system for providing a user with content related to a sound source, comprising: an acquisition unit that is held by a user and scans a space to acquire a distance to an object present in the space; a coordinate information calculation unit that calculates coordinate information of the object in space based on the distance acquired by the acquisition unit; an anchor detection unit that detects the anchor based on the coordinate information calculated by the coordinate information calculation unit and position information of an anchor that is virtually placed in advance in space; a reverberation characteristic calculation unit for calculating a reverberation characteristic in a space; a sound source adjustment unit that adjusts a sound source based on the reverberation characteristics calculated by the reverberation characteristics calculation unit; a content control unit that controls provision of content related to the sound source adjusted by the sound source adjustment unit to a user when an anchor is detected by the anchor detection unit; An information processing system comprising:

2. The reverberation characteristic calculation unit calculates, as the reverberation characteristic, a characteristic until the intensity of a sound perceived by a user from an impulse sound source attenuates to a predetermined amount; The information processing system according to claim 1 , wherein the sound source adjustment unit adjusts reverberation characteristics of the sound source itself based on the reverberation characteristics.

3. 3. The information processing system according to claim 1, wherein the reverberation characteristics calculation unit calculates the reverberation characteristics by reading out the reverberation characteristics that are pre-recorded in correspondence with the user's position calculated based on the coordinate information.

4. 4. The information processing system according to claim 1, wherein the reverberation characteristics calculation unit calculates the reverberation characteristics based on an actual measurement value of an impulse response to an impulse sound source that is actually measured in advance at a position of the user calculated based on the coordinate information.

5. An acoustic characteristic storage unit that stores acoustic characteristics related to the surface of the object in advance, The information processing system according to claim 1 , wherein the reverberation characteristic calculation unit calculates the reverberation characteristic in the space further based on the acoustic characteristic stored in the acoustic characteristic storage unit.

6. The information processing system according to claim 1 , wherein the reverberation characteristics calculation unit calculates a time center of gravity as the reverberation characteristics.

7. The information processing system according to claim 1 , wherein the reverberation characteristics calculation unit calculates the reverberation characteristics based on an energy ratio between a direct sound heard directly from a sound source and a reverberation sound heard from the sound source due to reflection from an object.

8. 8. The information processing system according to claim 1, wherein the reverberation characteristic calculation unit calculates the reverberation characteristic based on an energy ratio between a direct sound heard directly from a sound source, a reverberation sound heard from the sound source by reflection at an object, and a total sound which is a sum of the direct sound.

9. The information processing system according to claim 1 , wherein the reverberation characteristics calculation unit calculates the reverberation characteristics based on a cross-correlation function between a sound pressure heard through the left ear and a sound pressure heard through the right ear.

10. An information processing method for providing a user with content related to a sound source, comprising: an acquisition step of holding the device by a user, scanning a space, and acquiring a distance to an object present in the space; a coordinate information calculation step of calculating coordinate information of the object in space based on the distance acquired in the acquisition step; an anchor detection step of detecting the anchor based on the coordinate information calculated in the coordinate information calculation step and position information of an anchor virtually placed in advance in space; A reverberation characteristic calculation step of calculating a reverberation characteristic in a space; a sound source adjustment step of adjusting a sound source based on the reverberation characteristics calculated in the reverberation characteristics calculation step; a content control step of controlling provision of content related to the sound source adjusted in the sound source adjustment step to a user when an anchor is detected in the anchor detection step; An information processing method comprising:

11. On the computer, An acquisition process in which the device is held by a user and scans a space to acquire a distance to an object present in the space; a coordinate information calculation process for calculating coordinate information of the object in space based on the distance acquired in the acquisition process; an anchor detection process for detecting the anchor based on the coordinate information calculated in the coordinate information calculation process and position information of an anchor virtually placed in advance in space; A reverberation characteristic calculation process for calculating reverberation characteristics in a space; a sound source adjustment process for adjusting a sound source based on the reverberation characteristics calculated in the reverberation characteristics calculation process; a content control process for controlling provision of content related to the sound source adjusted in the sound source adjustment process to a user when an anchor is detected in the anchor detection process; An information processing program for executing the above.

Images