Information processing system, information processing method, and information processing program

Abstract

To enable execution of content disposed in a space even in a situation where positioning by a positioning satellite is difficult, and reduce the cost of providing a sound source in accordance with the space.SOLUTION: A unit held by a user scans a space to acquire a distance to an object present in a space, coordinate information of the object in the space is calculated on the basis of the acquired distance, an anchor virtually disposed in the space in advance is detected on the basis of the calculated coordinate information and position information of the anchor, reverberation characteristics in the space are calculated, a sound source is adjusted on the basis of the calculated reverberation characteristics, and provision of content related to the adjusted sound source to the user is controlled when the anchor is detected.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004] , , , , , , ,

[0005] , , , , , ,

[0001] The present invention relates to an information processing system, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, there is a technique of setting a range within a predetermined distance from a geographical coordinate position as a geofence, and starting an event such as providing content to a mobile terminal when the mobile terminal enters the geofence where position information can be acquired by a positioning satellite such as GPS (Global Positioning System). For example, Patent Document 1 discloses a technique of providing print data such as an advertisement when a mobile client device having a GPS receiver enters within a threshold distance of a geographical position defined by a geofence.

[0003] Also, there has been a technique of executing an event arranged in space in VR (Virtual Reality) where an image is superimposed on a video captured by a camera or the like. For example, Patent Document 2 discloses a technique related to a device that arranges and reproduces an object registered in advance on a captured image captured by a camera.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technology described in Patent Document 1, the accuracy of positioning varied depending on the reception status of radio waves from positioning satellites. For example, in areas with poor visibility of the sky or indoors, the accuracy of positioning by positioning satellites decreased, or positioning became impossible. As a result, in systems where events are executed according to the positioning results from positioning satellites, the accuracy of detecting whether a terminal has entered a geofence decreased, which could lead to delays in the timing of event initiation and other decreases in the accuracy of event initiation.

[0006] Furthermore, in Patent Document 2, it was necessary to install multiple sensors in order to accurately detect the user's position, and improving the accuracy of the start conditions for events virtually placed in space sometimes increased the cost of building the system.

[0007] Furthermore, when playing sound sources as part of an event, the reverberation characteristics differ depending on the shape of the space in which the sound source is played. Therefore, content providers need to adjust each sound source according to the reverberation characteristics of each individual space, which can increase the cost of providing the content.

[0008] This invention has been made in view of the above circumstances, and one of its objectives is to provide an information processing system, an information processing method, and an information processing program that can execute content placed in space even in situations where positioning by positioning satellites is difficult, and that can reduce the cost of providing sound sources appropriate to the space. [Means for solving the problem]

[0009] To solve the above problems, the information processing system is an information processing system for providing content to a user, and comprises: an acquisition unit held by the user that scans space and acquires the distance to an object present in space; a coordinate information calculation unit that calculates the coordinate information of the object in space based on the distance acquired by the acquisition unit; an anchor detection unit that detects an anchor based on the coordinate information calculated by the coordinate information calculation unit and the position information of an anchor that has been virtually placed in space in advance; a reverberation characteristic calculation unit that calculates the reverberation characteristics in space; a sound source adjustment unit that adjusts the sound source based on the reverberation characteristics calculated by the reverberation characteristic calculation unit; and a content control unit that controls the provision of content related to the sound source adjusted by the sound source adjustment unit to the user when an anchor is detected by the anchor detection unit. [Effects of the Invention]

[0010] According to one embodiment of the present invention, the system is held by the user, scans the space to obtain the distance to an object present in the space, calculates the coordinate information of the object in space based on the obtained distance, detects an anchor based on the calculated coordinate information and the position information of an anchor virtually placed in the space in advance, calculates the reverberation characteristics in the space based on the calculated coordinate information, adjusts the sound source based on the calculated reverberation characteristics, and controls the provision of content related to the adjusted sound source to the user when an anchor is detected. This makes it possible to execute content placed in space even in situations where positioning by positioning satellites is difficult, and to reduce the cost of providing sound sources appropriate to the space. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing an example of the system configuration of an information processing system in an embodiment. [Figure 2] This is a diagram illustrating the method of installing anchors in an embodiment. [Figure 3] This is a diagram illustrating the method of installing anchors in an embodiment. [Figure 4]This is a diagram for explaining the method of installing an anchor in an embodiment. [Figure 5] This is a diagram for explaining the method of installing an anchor in an embodiment. [Figure 6] This is a diagram for explaining an example of sound reflection in a space in an embodiment. [Figure 7] This is a diagram for explaining an example of (A) an impulse sound source, (B) attenuation of sound by the impulse sound source, and (C) an impulse response in an embodiment. [Figure 8] This is a block diagram showing an example of the hardware configuration of an information processing terminal in an embodiment. [Figure 9] This is a flowchart showing a first example of the operation of an information processing system in an embodiment. [Figure 10] This is a flowchart showing a second example of the operation of an information processing system in an embodiment. [Figure 11] This is a diagram showing an example of point cloud data of a measurement space in an embodiment. [Figure 12] This is a diagram showing an example of a histogram of the x-axis of point cloud data in an embodiment. [Figure 13] This is a diagram showing a calculation example when a curve fitting is performed for positive values of a histogram in an embodiment. [Figure 14] This is a diagram showing an example in which the positions of a user and an anchor are plotted with respect to a measurement space in an embodiment.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, an information processing system, an information processing method, and an information processing program in an embodiment of the present invention will be described in detail with reference to the drawings.

[0013] First, the configuration of the information processing system will be described using FIG. 1. FIG. 1 is a block diagram showing an example of the system configuration of the information processing system in the embodiment.

[0014] In FIG. 1, the information processing system 1 includes an information processing terminal 10, an information processing terminal 20, an information processing apparatus 30, and a content execution apparatus 40.

[0015] The information processing terminal 10 is a terminal for setting content (which may be referred to as an event) that is virtually arranged and executed in space. The information processing terminal 20 is a terminal held by a user for the user to receive the provision of content virtually arranged in space. The information processing apparatus 30 is an apparatus that stores the setting of the content arranged in the information processing terminal 10 and causes the user who holds the information processing terminal 20 to be provided with the content based on the stored content setting. Also, the content execution apparatus 40 is an apparatus that provides content to the user.

[0016] The content in this actual embodiment is information for operating a device in the information processing system 1 that is executed by the movement of the information processing terminal 20. The devices in the information processing system 1 are, for example, the information processing terminal 20 (or the information processing terminal 10), or the content execution apparatus 40, etc.

[0017] In the information processing system 1, the operation of the devices includes, for example, content playback, data communication, or actuator operation. Content playback includes, for example, the playback of sound sources, the display of images (videos or still images), or the display of 3D objects. Data communication includes, for example, communication between the information processing terminal 20 and the information processing device 30, or communication between the information processing device 30 and the content execution device 40. In data communication, for example, terminal information from the information processing terminal 20 (e.g., device ID, location information, acceleration sensor measurement values, or camera captured images) is provided to the information processing device 30. In addition, in data communication, for example, the content execution device 40 (e.g., audio equipment or video equipment) may be controlled by data communication from the information processing terminal 20. Actuator operation refers to the operation of actuators such as motors, air cylinders, and air valves, and includes, for example, moving animal-shaped objects or doors driven by motors, or inflating balloons that inflate with air. In the information processing system 1, content is provided to the user through the operation of these devices.

[0018] In the following embodiments, the execution of content is exemplified by the playback of sound sources on the information processing terminal 20 and the content execution device 40. However, the execution of content in the information processing system 1 is not limited to this.

[0019] Information processing terminals 10 and 20 are devices held and carried by the user, such as a smartphone, tablet PC, or wearable device such as a head-mounted, glasses-type, or wristwatch-type device. Information processing device 30 is a device that is communicatively connected to information processing terminals 10 and 20 via the network 9, such as a server device. In this embodiment, the user refers to the user of information processing terminal 10, information processing terminal 20, or information processing device 30, and these may be different users or the same user. Furthermore, the user may be a dedicated operator or a general user other than a dedicated operator.

[0020] The information processing terminal 10 includes an acquisition unit 11, a display unit 12, an anchor placement unit 13, an anchor information provision unit 14, a content setting unit 15, and a content information provision unit 16. The information processing terminal 20 includes an acquisition unit 21, a coordinate information calculation unit 22, a coordinate information provision unit 23, and a content execution unit 24. The information processing device 30 includes an anchor information storage unit 31, an anchor detection unit 32, an acoustic characteristics storage unit 33, a reverberation characteristics calculation unit 34, a sound source adjustment unit 35, and a content control unit 36. The content execution device 40 includes a content execution unit 41. In this embodiment, each of the above-mentioned functional units of the information processing system 1 will be described as a functional module realized by the information processing program (software) in this embodiment.

[0021] Acquisition units 11 and 21 acquire scan data scanned by a sensor that scans space. A sensor that scans space is a sensor that scans the shape of an object in space, for example, a distance measuring sensor that measures the distance to an object in space. As a distance measuring sensor, for example, a sensor using LiDAR (Light Detection And Ranging) technology can be used. In LiDAR, for example, laser light is used. LiDAR may use ultraviolet, infrared, or near-infrared light. LiDAR can measure the shape of an object in space in three dimensions by measuring the distance from the sensor to the object (distance measuring). Measuring the shape of an object in three dimensions means representing the shape of an object in space as coordinate information in space. Coordinate information in space is a relative two-dimensional or three-dimensional coordinate value from a reference point (coordinate) in space. For example, the surface shape of an object in space can be represented as a set of coordinate values ​​(point cloud data). The set of coordinate values ​​may be represented as a line, a plane, or a solid, for example. Coordinate information is information that indicates the surface shape in space, and is sometimes called spatial information.

[0022] The scan data obtained by scanning the space may, for example, be images captured by a stereo camera (not shown) on the information processing terminal 10 or the information processing terminal 20. Furthermore, the scan data obtained by scanning the space may include information acquired by a position sensor or acceleration sensor (not shown) on the information processing terminal 10 or the information processing terminal 20.

[0023] The display unit 12 displays the captured image taken by the camera on a display device (for example, a liquid crystal display). The display unit 12 allows the user to check the shooting results by displaying the captured image taken by the smartphone's camera on the smartphone's display, for example. The display unit 12 may also display the shape of an object based on coordinate information calculated based on the scan data acquired by the acquisition unit 11.

[0024] The anchor placement unit 13 provides the user with an operation (UI: User Interface) to place virtual anchors on the captured image displayed on the display unit 12, based on the scan data acquired by the acquisition unit 11. A virtual anchor is a mark (marker) virtually placed in space and has positional information (coordinate values) that represents a specific location in space. In this embodiment, an anchor is, for example, a mark indicating the position where content such as a sound source is placed. The shape of the mark can be, for example, a sphere, a cube, or an icon image, as long as it is displayed on the display unit 12 and visible to the user. The user can place anchors by operating the UI provided by the anchor placement unit 13. In this embodiment, the example given is that the anchor placement unit 13 of the information processing terminal 10 provides the UI for placing anchors, but the UI for placing anchors may also be provided on the information processing terminal 20. That is, the user of the information processing terminal 20 may be allowed to place anchors.

[0025] Objects in space are captured by the camera, and their shapes are acquired based on the scan data obtained by the acquisition unit 11. Therefore, the coordinate information indicating the shape of an object included in the captured image displayed on the display unit 12 is identified. The anchor placement unit 13 displays the captured image on the display unit 12 and provides a UI that allows the user to place anchors in the captured image, so that the user can obtain the coordinate values ​​where the anchors are placed without being aware of the coordinate values ​​in space. The user can set the position where to place the anchors while checking the captured image displayed on the display unit 12.

[0026] The anchor placement unit 13 further provides the user with the ability to correct the placement position of the anchors using the captured image displayed on the display unit 12. Since the placement position of the anchors is determined while checking the two-dimensional captured image displayed on the display unit 12, the placement position may be unclear in the depth direction of the display screen. For example, the anchor placement unit 13 makes it possible to correct the placement position of an anchor by moving the position of an anchor that has already been placed by a predetermined distance (for example, 1m or 50cm) in the depth direction. By correcting the placement position of the anchors, it is possible to improve the efficiency of anchor placement compared to repositioning the anchors.

[0027] The anchor information provision unit 14 provides the information processing device 30 with anchor information relating to the anchors placed in the anchor placement unit 13. The anchor information relating to the anchors includes, for example, coordinate information in space. Coordinate information in space is coordinate information based on scan data obtained by scanning space, and is, for example, coordinate information that indicates the relative position from an object with large feature quantities that exists in space. Spatial feature quantities are, for example, the distribution (histogram) of color data that exists in space, the shape of an object, or vector data of color data. When recognizing space, locations with similar feature quantities can be recognized as the same location. The location of an object with large feature quantities has a large difference in feature quantities from its surroundings, so the error in the object's position in the coordinate information can be reduced. On the other hand, the location of an object with few feature quantities has a small difference in feature quantities from its surroundings, so the error in the object's position in the coordinate information is large. By using the relative position from an object with large feature quantities that exists in space as coordinate information, it becomes possible to reduce the error in the object's position in the coordinate information even when, for example, an anchor is placed at the location of an object with small feature quantities. The coordinate information described above is based on scan data obtained by scanning space, and therefore represents relative position information from objects in space. However, the coordinate information may also include, for example, coordinate information calculated from the relative distance from a reference point (beacon) that emits radio waves, or absolute coordinate information consisting of longitude, latitude, and altitude calculated from radio waves acquired from positioning satellites. For example, by using coordinate information based on two calculation methods, the coordinate information can be checked twice, thereby improving the reliability of the coordinate information.

[0028] In this embodiment, "providing" may refer to either push-type or pull-type transmission. Similarly, in this embodiment, "receiving" may refer to either pull-type or push-type reception.

[0029] The content setting unit 15 provides the user with an operation screen (UI) for content placement operations, which allows the user to place content in relation to anchor information provided to the information processing device 30. The content setting unit 15 also provides the user with an operation screen (UI) for start condition setting operations, which allows the user to set start conditions for the content to be placed in the content placement operation. The user can perform content placement operations by operating the UI provided by the content setting unit 15. In this embodiment, the example given is that the content setting unit 15 of the information processing terminal 10 provides the UI for content placement operations, but the UI for content placement operations may also be provided on the information processing terminal 20. In other words, the user of the information processing terminal 20 may be able to place the content.

[0030] Content placement operations are operations that place content (events) to be executed in relation to anchor information, such as assigning content to be played to a placed anchor. Content assignment includes operations such as selecting and assigning (associating) a pre-created or newly created audio playlist. A playlist is a set of one or more audio files that determines which audio to be played, and assigning a playlist name makes it easier to identify which audio to play. By providing content placement operations to users, it becomes possible to change the content to be executed simply by changing the content assignment to an anchor, for example, thereby improving the usability of content placement. The content to be placed may be, as described above, playback of content such as image playback, data communication, or actuator operation. In content placement operations, for example, content that operates an actuator that drives an object may be placed in relation to an anchor placed near an animal-shaped object.

[0031] The initiation conditions for content execution are the conditions for content provision to be executed on the information processing terminal 20, etc., and in this embodiment, these are the conditions for determining whether or not the information processing terminal 20 has entered the geofence of the anchor on which it is installed. The geofence can be set by, for example, the distance between the anchor and the information processing terminal 20, the shooting direction of the camera of the information processing terminal 20, the altitude of the information processing terminal 20 (relative altitude difference with the anchor, or absolute altitude with respect to sea level, etc.), the display time of the anchor, or the stop time of the information processing terminal 20. The geofence may also include conditions such as whether the distance between the anchor and the information processing terminal 20 is approaching or moving away. The geofence may also include the fact that the anchor is being photographed by multiple information processing terminals 20. Furthermore, the geofence may include user behavior acquired from an acceleration sensor, etc. (not shown).

[0032] The content setting unit 15 provides the user with a UI for adjusting sound sources when arranging sound sources as content. Adjusting sound sources means adjusting (setting) the playback characteristics of the sound played from the sound source according to the space. Examples of sound source playback characteristics include volume, frequency characteristics, or reverberation characteristics.

[0033] <Loudness adjustment> The loudness of a sound emitted from a sound source in a space and heard by the user is affected by the distance from the sound source to the user, as well as sound reflections from objects in the space. Therefore, when playing a sound source, it is necessary to adjust the sound source to achieve an appropriate loudness. The content setting unit 15 provides a UI for setting an appropriate loudness.

[0034] <Frequency response adjustment> The frequency characteristics of sound emitted from a sound source in a space and heard by the user change due to sound reflection from objects in the space. Therefore, it is desirable to adjust the sound source to achieve appropriate frequency characteristics for better expression. Frequency characteristic adjustment is the adjustment of the frequency distribution of sound emitted from a sound source. Frequency specific adjustments include, for example, shelving type adjustments that adjust the frequency range above (or below) a specific frequency, or peak-dip type adjustments that adjust the frequency range around a specific frequency. The content setting unit 15 provides a UI for setting appropriate frequency characteristics.

[0035] <Adjusting reverberation characteristics> The reverberation characteristics of sound emitted from a sound source and heard by the user in a space vary depending on factors such as sound reflection from objects present in the space. Therefore, when playing back a sound source, it is necessary to adjust the sound source to achieve appropriate reverberation characteristics. The content setting unit 15 provides a UI for setting appropriate reverberation characteristics. The specific method for adjusting reverberation characteristics will be described later.

[0036] The content setting unit 15 may also be configured to allow setting of playback modes for sound sources other than those described above. For example, the content setting unit 15 may be configured to allow setting of the playback time, playback speed, etc., of the sound source. Furthermore, if the content is the operation of an actuator, the content setting unit 15 may be configured to allow setting of the operating range or operating speed, etc., of the actuator.

[0037] Furthermore, the execution mode of the content may include those in which the execution mode of the content changes depending on the distance between the anchor and the information processing terminal 20, the shooting direction of the camera of the information processing terminal 20, the altitude of the information processing terminal 20, the display time of the anchor, or the stop time of the information processing terminal 20, as described above. By setting playback conditions, it becomes possible to create various effects using sound sources.

[0038] The content information provision unit 16 provides the information processing device 30 with the content placement information of the content placed in the content setting unit 15 and the start conditions set in the start condition setting operation. By providing the placement information and start conditions to the information processing device 30, the information processing device 30 can store this information. By storing this information in the information processing device 30, the stored information can be read and used. Note that anchor placement or content setting may be performed on multiple information processing terminals 10. If anchor placement, etc. is performed on multiple information processing terminals 10, the result of anchor placement, etc., in one information processing terminal 10 is saved in the information processing device 30 and reflected in the other information processing terminals 10.

[0039] The coordinate information calculation unit 22 calculates the spatial coordinate information of an object based on the distance to the object acquired by the acquisition unit 21. The acquisition unit 21 scans space as described above to acquire the distance to objects present in space. The coordinate information calculation unit 22 can measure the shape of an object by calculating the coordinate information of the object's surface from multiple distances to the object's surface.

[0040] Here, a specific example of coordinate information calculation in the coordinate information calculation unit 22 will be explained using Figures 11 to 14. Figure 11 shows point cloud data in the measurement space acquired by the acquisition unit 11 or acquisition unit 21. Figure 11 illustrates the case where the measurement space is box-shaped (cube). The box-shaped measurement space is composed of six faces. The faces in the measurement space can be calculated by calculating the cluster points on each coordinate axis in the three dimensions of the point cloud data. The cluster points on each coordinate axis can be calculated by obtaining the distribution of the point cloud data on each coordinate axis.

[0041] <Calculation of histogram> Figure 12 shows the histogram of the point cloud data in Figure 11 along the x-axis. The histogram is the distribution of the point cloud data along the x-axis. In Figure 12, the point cloud data along the x-axis has peaks at approximately -4.5 and 3.3. This indicates that there are two planes along the x-axis in the measurement space.

[0042] <Calculation of curve fitting> Figure 13 shows an example of a curve fitting calculation for the positive values ​​of the histogram in Figure 12. By performing a curve fitting on the histogram, its peaks can be calculated as walls on that coordinate axis. In Figure 13, there is a peak at approximately 3.3, and it is calculated that this point is one surface (wall) on the x-coordinate axis. In Figure 13, the smoothing spline method is used as the curve fitting. The smoothing spline method is a method that estimates a function using a spline curve while balancing the degree of smoothness based on the second derivative with the observed values ​​observed with noise. Figure 13 is calculated using a 3D spline curve. Note that although Figure 13 uses the smoothing spline method as the curve fitting, a method using a Gaussian function as the curve fitting could also be used, for example.

[0043] The coordinate information calculation unit 22 can calculate two planes for each coordinate axis using the method described above, and calculate the coordinate information of a box shape on six planes. The coordinate information may include the position of the user receiving the content relative to the coordinate information in the measurement space calculated by the coordinate information calculation unit 22, and the position information of the anchor detected by the anchor detection unit 32, which will be described later. Figure 14 shows an example of plotting the user's position (Listener) and the anchor's position (Sound) relative to the measurement space.

[0044] Although Figures 11 to 14 illustrate the calculation of coordinate information in a measurement space composed of six surfaces, the measurement space may have a complex shape. For example, the measurement space may have columns and beams within a room. Furthermore, the measurement space may have curved surfaces, such as a dome-shaped ceiling.

[0045] The coordinate information providing unit 23 provides the coordinate information calculated by the coordinate information calculation unit 22 to the information processing device 30. The information processing device 30, which will be described later, can detect anchors from the coordinate information of the object provided by the coordinate information providing unit 23.

[0046] The content execution unit 24 provides content to the user when the information processing terminal 20, which includes the acquisition unit 21, enters the geofence. For example, the content execution unit 24 plays a sound source in a speaker (not shown) of the information processing terminal 20.

[0047] In this embodiment, the OSC (Open Sound Control) protocol may be used for providing sound sources. By using the OSC protocol in the content execution unit 24, it becomes possible to provide sound sources in real time. For example, if the sound source is a live sound source, by using the OSC protocol, it becomes possible to share the live sound source in real time among multiple information processing terminals 20. In addition, in this embodiment, a sound source format for realizing spatial sound (e.g., Ambisonics) may be used for providing sound sources. By using a sound source format for realizing spatial sound in the content execution unit 24, it becomes possible to easily provide spatial sound sources. In addition, in this embodiment, a game development platform (e.g., Wwise) may be used for providing sound sources. By using a game development platform in the content execution unit 24, it becomes possible to easily provide events in games. In addition, in this embodiment, sound source playback may be performed using a head-related transfer function (HRTF). A head-related transfer function is a transfer function that expresses the changes in sound caused by peripheral objects including the earlobe, head, and shoulders. In the content execution unit 24, the characteristics of sound changes from the sound source to both ears are quantified using head-related transfer functions, making it possible to recognize the direction of the sound source and the distance to the sound source. When adjusting the sound source, the adjustment results can be expressed as the OSC, etc., as described above.

[0048] The anchor information storage unit 31 stores anchor information relating to anchors placed in the anchor placement unit 13 in a readable format. As described above, the anchor information may include anchor position information, etc. The anchor information storage unit 31 can also store multiple anchor information. Multiple anchor information stored in the anchor information storage unit 31 is read out by the anchor detection unit 32 and used to detect multiple anchors placed in space. The anchor information stored in the anchor information storage unit 31 may also be provided to the information processing terminal 20. In this embodiment, the case in which anchor detection is performed by the anchor detection unit 32 is illustrated, but anchor detection may also be performed by the information processing terminal 20.

[0049] The anchor detection unit 32 detects anchors virtually placed in space based on coordinate information provided by the coordinate information provision unit 23 and position information of anchors virtually placed in space in advance and stored in the anchor information storage unit 31. Here, the coordinate information and position information are the coordinates of points represented in three or two dimensions in space. The coordinate information and position information may also be the spatial feature quantities described above. Spatial feature quantities include the distribution of color data present in space (histogram), the shape of an object, or vector data of color data. Since the anchor detection unit 32 can detect anchors virtually placed in space based on spatial scan data, it can detect anchors placed in space even in situations where positioning by positioning satellites is difficult. This improves the accuracy of the conditions for providing content placed in space and enables the provision of content using geofencing set in three dimensions.

[0050] <Detection of anchors using coordinate distance> For example, the anchor detection unit 32 may detect an anchor by comparing the anchor's position information stored in the anchor information storage unit 31 with the coordinate information provided by the coordinate information provision unit 23, and determining whether a point included in the coordinate information and its position information are within a predetermined distance.

[0051] <Detection of anchors using features> Alternatively, the anchor detection unit 32 may calculate feature quantities from the coordinate information provided by the coordinate information provision unit 23 and detect anchors based on whether or not they approximate the feature quantities stored in the anchor information storage unit 31 at the time of anchor placement.

[0052] The anchor detection unit 32 may detect anchors by sequentially referring to multiple anchor information stored in the anchor information storage unit 31. By referring to multiple anchor information, the anchor detection unit 32 can easily add or delete anchors to be placed.

[0053] Furthermore, the anchor detection unit 32 may detect anchors based on a predetermined anchor detection order. For example, if the detection of anchor B is expected after anchor A, the anchor detection unit 32 can reduce the load on anchor detection and shorten the detection time by prioritizing the detection of anchor B after detecting anchor A.

[0054] Furthermore, the anchor detection unit 32 may exclude already detected anchors from detection if an anchor has already been detected for a single user and content corresponding to that anchor has already been provided. By excluding already detected anchors from detection, it becomes possible to prevent duplicate provision of content associated with anchors.

[0055] Furthermore, the anchor detection unit 32 may be configured to count the number of detections if a single anchor is detected multiple times by one user or by different users. By counting the number of detections, it becomes possible to change the content provided according to the number of detections. The provision of content can be controlled by the content control unit 36, which will be described later.

[0056] Furthermore, the anchor detection unit 32 may also detect anchors based on measurements obtained from a GNSS (Global Navigation Satellite System) sensor or an acceleration sensor (not shown). By detecting anchors in combination with the measurements from these sensors, the accuracy of anchor detection can be improved. For example, outdoors, it may be possible to determine the current position accurately using a GNSS sensor. Also, by using an acceleration sensor, it may be possible to record the movement of the information processing terminal 20 and determine the current position accurately by comparing it with the position before the movement.

[0057] The acoustic properties memory unit 33 pre-stores the acoustic properties relating to the surface of an object in space. The acoustic properties relating to the surface of an object refer to the acoustic properties (sometimes called reflection properties) when sound emitted from a sound source is reflected off the surface of the object. The acoustic properties stored in the acoustic properties memory unit 33 include, for example, the reflectivity of sound (sometimes called sound absorption coefficient or attenuation rate). For example, the reflectivity is high on the surface of a hard material such as metal because less sound penetrates into the material, while the reflectivity is low on the surface of a porous material because more sound penetrates into the material. The acoustic properties stored in the acoustic properties memory unit 33 may also include the surface shape of the object. On an uneven surface, reflected sound may interfere with each other, causing the reflection properties at a specific frequency to change. The acoustic properties stored in the acoustic properties memory unit 33 may also include acoustic properties for each frequency. The reflectivity of sound on the surface of an object differs depending on the sound frequency. For example, high-frequency sounds have low reflectivity on porous materials, while low-frequency sounds have high reflectivity. In the acoustic characteristics memory unit 33, by pre-storing the acoustic characteristics of the surface of an object present in space, it becomes possible to calculate the changes in sound from when it is emitted from a sound source until it is reflected off the surface of the object and reaches the user's position.

[0058] The reverberation characteristics calculation unit 34 calculates the reverberation characteristics in space based on the coordinate information calculated by the coordinate information calculation unit 22. Reverberation characteristics are the characteristics of sound emitted from a sound source, reflected off the surfaces of objects in space, and reaching the user with a time delay. Examples of reverberation characteristics calculated by the reverberation characteristics calculation unit 34 are shown below. Note that the reverberation characteristics shown below are examples of the change in sound (impulse response) that can be heard at the user's position for an impulse sound source in which the volume changes from a certain volume to 0 at a predetermined time, but the reverberation characteristics calculated by the reverberation characteristics calculation unit 34 are not limited to this.

[0059] <Reverberation time> Reverberation time is a characteristic that indicates the length of time a sound resonates in a space. Reverberation time is the time it takes for the sound intensity to decay by 60 dB at the observation point (the user's position) after a sound source is played and radiated into space, and the sound source is stopped after the volume reaches a steady state (impulse source). Reverberation time can be calculated using the following formula.

[0060]

number

[0061] <Initial reverberation time> Initial reverberation time is the reverberation time calculated from the decay slope of the first 10 dB portion of the reverberation decay. Initial reverberation time represents the reverberation sensation perceived by humans. The decay waveform used for initial reverberation time is the same as that used for reverberation time.

[0062] <Time Centroid> This characteristic is expressed by the following equation.

[0063]

number

[0064] <clarity> Here, we will explain the method for calculating Clarity using Figure 7. Figure 7 is a diagram illustrating an example of (A) an impulse sound source, (B) sound attenuation by the impulse sound source, and (C) an impulse response in an embodiment.

[0065] In Figure 7(A), the x-axis represents the passage of time (t), and the y-axis represents the sound pressure at the sound source (p). The impulse sound source changes from a steady state sound pressure p1 to a sound pressure of 0 (playback stopped) at time t0. In Figure 7(B), the y-axis represents the sound pressure at the user's position (measurement position) (E). Measurement of sound pressure E begins at time t0. Sound pressure E decays over time.

[0066] In Figure 7(C), the y-axis represents the ratio (logarithmic scale) of the direct sound that reaches the measurement position directly from the sound source and the reverberation sound that reaches the measurement position after being reflected by the object, calculated using the following formula. This characteristic is called C80.

[0067]

number

[0068] <Total energy ratio of direct sound> The direct sound total energy ratio is the ratio of direct sound energy to total energy (direct sound + reverberation) calculated using the following formula. The direct sound total energy ratio is mainly used as a measure of intelligibility in human voices.

[0069]

number

[0070] The reverberation characteristics described above may be measured in the space where the sound source is reproduced, or they may be calculated by storing reverberation characteristics in other spaces with similar spatial coordinate information and inferring them from the spatial shape of the object and the acoustic characteristics stored in the acoustic characteristics storage unit 33. For example, reverberation characteristics in different types of spaces (e.g., concert hall, church, concrete-enclosed room, corridor, etc.) may be stored in advance, and the reverberation characteristics in a space may be calculated by modifying the stored reverberation characteristics based on the coordinate information calculated by the coordinate information calculation unit.

[0071] Next, an example of reverberation characteristic calculation in the reverberation characteristic calculation unit 34 will be described. For calculating reverberation characteristics, for example, the Mirror image method by Allen and Berkley (hereinafter sometimes abbreviated as "MI method") can be used. In the MI method, the following parameters are used. Speed ​​of sound [m / s] Sampling rate [Hz] Anchor position coordinates (x, y, z) [m] User's position coordinates (x, y, z) [m] Room dimensions (x, y, z) [m] Room reverberation time [m], or reflection coefficient of each wall.

[0072] Here, the speed of sound can be calculated from the temperature of the room where it was measured. Speed ​​of sound = 331.5+0.6t [m / s] (t: Celsius, 1 atm) Furthermore, the sampling rate used will be the sampling rate actually used within the device. Furthermore, the anchor positions, user positions, and room dimensions are calculated using the method described above.

[0073] The reverberation time [m] of a room, or the reflection coefficient of each wall, can be calculated from the reverberation time calculated from an anchor point or the user's position. The reverberation time can be calculated by measuring the impulse response and then calculating the reverberation time based on the measured impulse response.

[0074] <Measurement of impulse response> The impulse response is measured within a room (measurement space) whose shape has been estimated by calculating coordinate information. The impulse response is measured as described above, and can be specifically performed using measurements with TSP (Time Stretched Pulse) signals, pink noise, balloons, or handclaps.

[0075] <Calculation of reverberation time> The reverberation time is calculated from the impulse response obtained during the measurement of the impulse response. For example, the calculation method specified in ISO 3382-2:2008 can be used to calculate the reverberation time.

[0076] The reverberation characteristics calculation unit 34 calculates the reverberation characteristics using the MI method based on the parameters obtained by the method described above. The reverberation characteristics calculation unit 34 may also calculate the reverberation characteristics using a method other than the MI method. For example, the reverberation characteristics calculation unit 34 may also use the sensor information obtained by the acquisition unit 21 to estimate the material of each wall using deep learning-based image recognition technology, and then calculate the reflection coefficient using that material information. In some cases, the material of a building material can be identified by its appearance, such as its shape. The reverberation characteristics calculation unit 34 can estimate the reverberation characteristics by learning the correspondence between sensor information and the reflection coefficient using deep learning.

[0077] The sound source adjustment unit 35 adjusts the sound source based on the reverberation characteristics calculated by the reverberation characteristics calculation unit 34. Sound source adjustment includes volume adjustment, frequency characteristic adjustment, or reverberation characteristic adjustment, as described above. By adjusting the sound source, the sound source adjustment unit 35 makes it possible to easily adjust the sound source provided according to the reverberation characteristics, etc., which differ depending on the shape of the space in which the sound source is reproduced. The sound source adjustment by the sound source adjustment unit 35 may be set to be enabled or disabled. For example, if the sound source adjustment unit 35 is enabled, it may automatically adjust the sound source when the reverberation characteristics are calculated, while if it is disabled, the sound source adjustment may be performed manually.

[0078] By adjusting the sound source, it becomes possible to reproduce the sound source with the reverberation or volume intended by the content provider. Furthermore, by adjusting the sound source, it becomes possible to give users, for example, a sense of reverberation in an event hall that makes them feel as if they are in a church.

[0079] The content control unit 36 ​​controls the provision of content when the start condition is met, according to the placement information or start condition. The content control unit 36 ​​may also control the provision of content in a predetermined content execution manner according to the pre-set start condition. For example, the content control unit 36 ​​may change the content execution manner, such as the playback manner of the sound source, according to the identification information of the information processing terminal 20 (attribute information such as the gender, age, language used, nationality or religion of the user that has been registered in advance, or device information obtained from the information processing terminal 20, etc.) and provide the content. The identification information may be obtained, for example, linked to the user's account information. The content execution unit 24 can execute a variety of content by executing the content that has been changed according to the identification information.

[0080] The content execution device 40 is a device different from the information processing terminal 20 that provides content to the user, and is, for example, a device that plays sound sources or operates actuators. The content execution unit 41 controls the execution of content according to instructions from the content control unit 36. The content execution unit 41 may, for example, cooperate with the content execution unit 24 to control the provision of content.

[0081] The above-mentioned functional units of the information processing system 1 are merely examples of functions and do not limit the functions of the information processing system 1. For example, the information processing terminal 10, information processing terminal 20, or information processing device 30 do not need to have all of the above-mentioned functional units, and may have only some of them. Furthermore, the information processing system 1 may have other functions not mentioned above.

[0082] For example, the functional units of the information processing terminal 10 may be implemented in the information processing device 30. Similarly, the functional units of the information processing terminal 20 may be implemented in the information processing device 30. Furthermore, the functional units of the information processing device 30 may be implemented in either the information processing terminal 10 or the information processing terminal 20. For example, the function of the anchor detection unit 32 may be implemented in the information processing terminal 20.

[0083] Furthermore, as described above, each of the above functional units has been explained as being implemented by software. However, at least one of the above functional units may be implemented by hardware.

[0084] Furthermore, any of the above functional units may be implemented by dividing one functional unit into multiple functional units. Alternatively, any two or more of the above functional units may be combined into a single functional unit. Figure 1 represents the functions of the information processing system 1 using functional blocks, and does not indicate, for example, that each functional unit is composed of separate program files or the like.

[0085] Furthermore, the information processing terminal 10, information processing terminal 20, or information processing device 30 may be a device implemented in a single enclosure, or it may be a system implemented from multiple devices connected via a network or the like. For example, the information processing terminal 10, information processing terminal 20, or information processing device 30 may implement some or all of its functions through other virtual devices, such as cloud services provided by a cloud computing system. In other words, the information processing terminal 10, information processing terminal 20, or information processing device 30 may implement at least one of the above-mentioned functional units in other devices.

[0086] Next, the method of installing the anchor will be explained using Figures 2 to 5. Figures 2 to 5 are diagrams illustrating the method of installing the anchor in the embodiment.

[0087] In Figure 2, the display screen 1000 is displayed on the display unit 12. The display screen 1000 includes an icon 1001 indicating the anchor setting screen, an anchor placement button 1002, an exit button 1003, a save button 1004, and a naming button 1005 for naming the playlist.

[0088] The display screen 1000 is a display screen shown on the information processing terminal 10, and displays an image captured by the camera of the information processing terminal 10. The anchor placement button 1002 is a button for placing an anchor in any of the shooting ranges displayed on the display screen 1000. For example, by operating the anchor placement button 1002, an anchor can be placed at the center of the display screen 1000, or at a cursor position not shown within the display screen 1000. By operating the anchor placement button 1002, the user can transition to the screen for placing anchors, as explained in Figure 3.

[0089] The Exit button 1003 is used to exit the settings shown on the display screen 1000, and the Save button 1004 is used to save the current settings. The Name button 1005 is used to name the playlist. By operating the Name button 1005, the user can enter a name from a text box or similar.

[0090] In Figure 3, the display screen 2000 is displayed on the display unit 12. The display screen 2000 has a trial placement button 2001. The display screen 2000 is an enlarged captured image obtained by moving the shooting range displayed on the display screen 1000. The trial placement button 2001 is a button for trial placement of the anchor. By operating the trial placement button 2001, the user can place the anchor approximately in the center of the display screen 2000. Note that trial placement means placing the anchor before determining the final placement position, and the user can experimentally experience the anchor they have placed and the sound source set to the anchor.

[0091] Figure 4 shows a third display example of a display device used in an information processing terminal according to the embodiment.

[0092] In Figure 4, the display screen 1000 is displayed on the display unit 12. The display screen 3000 has an anchor position correction button 3001, an anchor position determination button 3002, and an anchor 3003. The anchor position correction button 3001 is a button for correcting the position of the anchor 3003 that was placed using the trial placement button 2001. By operating the anchor position correction button 3001, the user can correct the position of the placed anchor 3003 to the back or front. When the position of the anchor 3003 is corrected to the back, the size of the anchor 3003 is displayed smaller, while when the position of the anchor 3003 is corrected to the front, the size of the anchor 3003 is displayed larger. The display size of the anchor 3003 may be changed by a setting operation not shown.

[0093] The anchor position determination button 3002 is used to determine the installation position of the anchor 3003. By operating the anchor position determination button 3002, the user can transition to the screen for completing the anchor installation, as explained in Figure 5.

[0094] In Figure 5, the display screen 1000 is displayed on the display unit 12. The display screen 4000 has an end button 4001 and an anchor 4002. The user completes the anchor installation by operating the end button 4001.

[0095] Next, we will explain sound reflection in space using Figure 6. Figure 6 is a diagram illustrating an example of sound reflection in space in an embodiment.

[0096] In Figure 6, the sound source is assumed to be located at anchor position O inside room R. Anchor position O is the spatial position of the anchor installed in the procedure described above. The acquisition unit 21 of the information processing terminal 20 scans the space and acquires the distance to objects present in the space. User position S is the position where the user hears the sound source, and is inside the geofence at anchor position O.

[0097] Sound emitted from a sound source at anchor position O is divided into direct sound that reaches the user's position S and reflected sound (reverberation) that reaches the user's position S after reflecting off the walls of room R. Figure 6 illustrates the reverberation that reaches the user's position S after reflecting once at reflection point F, but there are multiple paths for the reverberation to reach the user's position depending on the direction of sound diffusion from the sound source.

[0098] Direct sound reaches the user's position S at a distance L1. On the other hand, reverberation reaches the user's position S at a distance L2+L3. Therefore, the time difference until the sound reaches the user's position S changes depending on the distance to reach it. Furthermore, the reverberation reaching the user's position S changes depending on the acoustic characteristics inside room R. By playing a pulse sound source at anchor position O and measuring the change in sound (sound pressure, etc.) at the user's position S using an acoustic analyzer, the reverberation characteristics in room R can be measured. In addition, a dummy head with microphones placed at the ear positions of a human-shaped dummy can be used to measure reverberation characteristics. By using a dummy head, for example, it is possible to measure the acoustic characteristics that a human would perceive in the measurement space.

[0099] Next, the hardware configuration of the information processing terminal 10 will be explained using Figure 8. Figure 8 is a block diagram showing an example of the hardware configuration of the information processing terminal 10 in this embodiment. Note that the hardware configurations of the information processing terminal 20 and the information processing device 30 are the same as those of the information processing terminal 10, and therefore will not be explained.

[0100] The information processing terminal 10 includes a CPU (Central Processing Unit) 101, RAM (Random Access Memory) 102, ROM (Read Only Memory) 103, I / O device 104, and communication I / F (Interface) 105. The information processing terminal 10 is a device that executes the information processing program described in Figure 1.

[0101] The CPU 101 controls the user terminal by executing information processing programs stored in RAM 102 or ROM 103. The information processing programs are obtained, for example, from a recording medium on which the programs are stored, or from a program distribution server via a network, installed in ROM 103, read by the CPU 101, and executed.

[0102] The I / O device 104 has an operation input function and a display function (operation display function). The I / O device 104 is, for example, a touch panel. The touch panel enables users of the information processing terminal 10 to perform operation input using their fingertips or a stylus. In this embodiment, the case in which the I / O device 104 is a touch panel with an operation display function is described, but the I / O device 104 may also have a display device with a display function and an operation input device with an operation input function separately. In that case, the display screen of the touch panel can be performed as the display screen of the display device, and the operation of the touch panel can be performed as the operation of the operation input device. The I / O device 104 may be implemented in various forms such as a head-mounted type, glasses type, or wristwatch type display.

[0103] Communication I / F 105 is a communication interface. Communication I / F 105 performs short-range wireless communication such as wireless LAN, wired LAN, and infrared. The diagram shows only communication I / F 105 as a communication interface, but the information processing terminal 10 may have multiple communication interfaces for each communication method.

[0104] Next, we will explain the operation of the information processing system 1 using Figures 9 to 10.

[0105] Figure 9 is a flowchart illustrating a first example of the operation of the information processing system in the embodiment. The flowchart shown in the first example describes the operation related to the placement of anchors and sound sources.

[0106] In Figure 9, the information processing system 1 provides a UI for sound source settings from the display screen of the information processing terminal 10 (step S11). After executing the process in step S11, the information processing system 1 acquires scan data from a sensor that scans the space (step S12).

[0107] After executing the process in step S12, the information processing system 1 determines whether or not to place the anchor (step S13). The decision of whether or not to place the anchor can be made, for example, by whether or not the user has completed the anchor placement operation. If it is determined not to place the anchor (step S13: NO), the information processing system 1 repeats the processes in steps S12 to S13 and waits for the anchor to be placed.

[0108] If it is determined that an anchor has been placed (Step S13: YES), the information processing system 1 determines whether or not to correct the position of the anchor (Step S14). If it is determined that the position of the anchor should be corrected (Step S14: YES), the information processing system 1 corrects the position of the anchor (Step S15).

[0109] After executing the process in step S15, or if it is determined that the anchor position will not be corrected (step S14: NO), the information processing system 1 determines whether a content list has been selected (step S16). Selecting a content list also includes saving the selected content list. If the desired content does not exist in the content list, the information processing system 1 may provide a UI for acquiring new content and creating a content list. If it is determined that no content list has been selected (step S16: NO), the information processing system 1 repeats the process in step S16 and waits for a content list to be selected.

[0110] On the other hand, if it is determined that a content list has been selected (step S16: YES), the information processing system 1 provides a UI for inputting execution conditions for executing the content from the display screen of the information processing terminal 10 (step S17).

[0111] After executing the process in step S17, the information processing system 1 determines whether the input of the playback conditions has been completed (step S18). If it determines that the input has not been completed (step S18: NO), the information processing system 1 repeats the process in step S18 and waits for the input to be completed.

[0112] On the other hand, if it is determined that the input of the playback conditions has been completed (step S18: YES), the information processing system 1 provides the anchor information entered in steps S13 to S15 from the information processing terminal 10 to the information processing device 30 (step S19).

[0113] After executing the process in step S19, the information processing system 1 provides the information related to the sound source input in steps S16 to S18 from the information processing terminal 10 to the information processing device 30 (step S20).

[0114] Once the processing in step S20 is complete, the information processing system 1 stores the anchor information provided in step S19 in the information processing device 30 (step S21). After the processing in step S21 is complete, the information processing system 1 stores the sound source information provided in step S20 in the information processing device 30 (step S22), and then terminates the process shown in the flowchart.

[0115] Figure 10 is a flowchart illustrating a second example of the operation of the information processing system 1 in the embodiment. The flowchart shown in the second example describes the operation related to the playback of a sound source as content provision.

[0116] In Figure 10, the information processing system 1 provides a UI for playing sound sources from the display screen of the information processing terminal 20 (step S31). The UI for playing sound sources is, for example, an AR (Augmented Reality) application program that allows users to enjoy playing sound sources placed in space.

[0117] After executing the process in step S32, the information processing system 1 acquires scan data from the sensor that scans the space (step S32).

[0118] After executing the process in step S32, the information processing system 1 determines whether or not an anchor has been detected (step S33). If it determines that no anchor has been detected (step S33: NO), the information processing system 1 repeats the processes in steps S32 to S33 and waits for an anchor to be detected.

[0119] On the other hand, if it is determined that an anchor has been detected (step S33: YES), the information processing system 1 displays the detected anchor on the display screen of the information processing terminal 20 (step S34).

[0120] After executing the process in step S34, the information processing system 1 determines whether the regeneration conditions have been met (step S35). Whether the regeneration conditions have been met can be determined, for example, by whether the information processing terminal 20 has acquired regeneration condition information from the information processing device 30. The regeneration conditions may include whether the information processing terminal 20 is inside a geofence. The information processing system 1 can determine that the regeneration conditions have been met when the information processing terminal 20 enters the geofence.

[0121] If it is determined that the playback conditions are not met (step S35: NO), the information processing system 1 repeats the processing from steps S32 to S35 and waits for the playback conditions to be met.

[0122] On the other hand, if it is determined that the playback conditions have been met (step S35: YES), the information processing system 1 starts playback of the sound source (step S36). The playback of the sound source is, for example, the playback of an impulse sound source.

[0123] After executing the process in step S36, the information processing system 1 calculates the reverberation characteristics (step S37). As described above, the calculation of reverberation characteristics can be performed based on the measurement of the impulse response in the space of the object. After executing the process in step S36, the information processing system 1 adjusts the sound source and saves the adjusted settings (step S38). By calculating the reverberation characteristics and adjusting the sound source, it is possible to easily adjust the sound source according to the space and reduce the cost of providing the sound source.

[0124] After executing the process in step S38, the information processing system 1 determines whether or not to terminate playback (step S39). Whether or not to terminate playback depends on factors such as the terminal state no longer satisfying the playback conditions, the completion of playback of a pre-prepared sound source, or an explicit playback stop operation by the user. If it is determined not to terminate playback (step S39: NO), the information processing system 1 repeats the processes in steps S37 to S39 and waits for playback to finish. On the other hand, if it is determined to terminate playback (step S39: YES), the information processing system 1 terminates the operations shown in the flowchart.

[0125] The flowchart shown is an example of the operation and does not limit the possible operations.

[0126] Furthermore, the various processes described above in this embodiment may be performed by recording a program for realizing the functions of the device described in this embodiment onto a computer-readable recording medium, loading the program recorded on the recording medium into a computer system, and executing it. The term "computer system" here may include hardware such as an operating system and peripheral devices. Also, if a WWW system is used, the "computer system" shall also include the homepage provisioning environment (or display environment). Furthermore, "computer-readable recording medium" refers to a storage device such as a flexible disk, magneto-optical disk, ROM, flash memory, portable media such as a CD-ROM, or a hard disk built into a computer system.

[0127] Furthermore, "computer-readable recording media" includes volatile memory (e.g., DRAM (Dynamic Random Access Memory)) within computer systems that act as servers or clients when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, which retains the program for a certain period of time. In addition, the above program may be transmitted from the computer system that stores the program in a memory device, etc., to another computer system via a transmission medium or by transmission waves within the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium that has the function of transmitting information, such as a network such as the Internet or a communication line such as a telephone line. Furthermore, the above program may be for the purpose of realizing a part of the above-mentioned functions. In addition, it may be a so-called differential file (differential program) that realizes the above-mentioned functions in combination with a program already recorded in the computer system.

[0128] While embodiments of the present invention have been described above with reference to the drawings, the specific configuration is not limited to these embodiments, and various modifications are possible without departing from the spirit of the present invention. [Explanation of symbols]

[0129] 1. Information Processing System 10 Information Processing Terminals 11 Acquisition Department 12 Display section 13 Anchor placement section 14 Anchor Information Provision Department 15. Content Settings Section 16. Content Information Provision Department 20 Information Processing Terminals 21 Acquisition Department 22 Coordinate Information Calculation Unit 23 Coordinate information provision section 24 Content Execution Unit 30 Information Processing Devices 31 Anchor Information Storage Unit 32 Anchor detection unit 33 Acoustic Characteristics Memory Unit 34 Reverberation Characteristics Calculation Section 35 Sound source adjustment section 40 Content Execution Device 41 Content Execution Unit 101 CPU 102 RAM 103 ROM 104 I / O equipment 105 Communication I / F 1000 display screen 1001 Icons 1002 Anchor installation button 1003 Exit button 1004 Save button 1005 Naming Button 1005 2000 display screen 2001 Test placement button 3000 display screen 3001 Anchor Position Correction Button 3002 Anchor position determination button 3003 Anchor 4000 display screen 4001 Exit button< / clarity>

Claims

1. An information processing system for providing audio-related content to users, An acquisition unit, which is held by the user, scans the space and obtains the distance to objects present in that space, A coordinate information calculation unit calculates coordinate information including the point cloud coordinates of the object in the space and the position of the user relative to the point cloud coordinates of the object, based on the distance acquired by the acquisition unit. A detection unit detects the sound source location based on the coordinate information and the position information of the sound source location that has been virtually placed in the space in advance. A reverberation characteristic calculation unit calculates reverberation characteristics that indicate the characteristics of sound emitted from the sound source location, reflected off the surface of the object, and reaching the user, based on the coordinate information and the detected sound source location. Based on the reverberation characteristics calculated by the reverberation characteristics calculation unit, the sound source adjustment unit adjusts the sound source when providing content, An information processing system equipped with the following features.

2. The reverberation characteristic calculation unit calculates the characteristics of an impulse sound source until the perceived intensity of the sound decreases to a predetermined amount, as the reverberation characteristic. The information processing system according to claim 1, wherein the sound source adjustment unit adjusts the reverberation characteristics of the sound source itself based on the reverberation characteristics.

3. The reverberation characteristics calculation unit calculates the reverberation characteristics by reading the reverberation characteristics recorded in advance in accordance with the user's position, which is calculated based on the distance obtained by scanning the space, as described in claim 1 or 2 of the information processing system.

4. The reverberation characteristics calculation unit calculates the reverberation characteristics based on the measured values ​​of the impulse response to the impulse sound source, which was measured in advance at the user's position, calculated based on the distance obtained by scanning the space, according to the information processing system according to any one of claims 1 to 3.

5. The system further includes an acoustic characteristics storage unit that stores in advance the acoustic characteristics relating to the surface of the object, The reverberation characteristics calculation unit calculates the reverberation characteristics in a space based on the acoustic characteristics stored in the acoustic characteristics storage unit, according to any one of claims 1 to 4.

6. The reverberation characteristics calculation unit calculates the time centroid as the reverberation characteristics, according to any one of claims 1 to 5.

7. The reverberation characteristic calculation unit calculates the reverberation characteristic based on the energy ratio of the direct sound heard directly from the sound source position and the reverberation sound heard due to reflection from an object at the sound source position, according to any one of claims 1 to 6.

8. The reverberation characteristic calculation unit calculates the reverberation characteristic based on the energy ratio of the direct sound heard directly from the sound source position and the total tone which is the sum of the reverberation sound heard due to reflection from an object from the sound source position and the direct sound, according to any one of claims 1 to 7.

9. The information processing system according to any one of claims 1 to 8, further comprising: a content control unit that controls the provision of content related to the sound source, adjusted based on the reverberation characteristics, to the user when a start condition set in a geofence corresponding to the virtual sound source location is met.

10. An information processing method for providing audio content to users, The acquisition step involves the user holding data, scanning the space, and obtaining the distance to objects present in that space. A coordinate information calculation step, based on the distance obtained in the acquisition step, calculates coordinate information including the point cloud coordinates of the object in the space and the position of the user relative to the point cloud coordinates of the object; A detection step in which the sound source location is detected based on the coordinate information and the position information of the sound source location that has been virtually placed in the space in advance, A reverberation characteristics calculation step, which calculates reverberation characteristics that indicate the characteristics of sound emitted from the sound source location, reflected off the surface of the object, and reaching the user, based on the coordinate information and the detected sound source location, Based on the reverberation characteristics calculated in the reverberation characteristics calculation step, a sound source adjustment step is performed to adjust the sound source when providing content. Information processing methods, including those mentioned above.

11. On the computer, The data is held by the user and involves an acquisition process that scans the space to obtain the distance to objects present in that space. A coordinate information calculation process that calculates coordinate information including the point cloud coordinates of the object in the space and the position of the user relative to the point cloud coordinates of the object, based on the distance obtained in the acquisition process, A detection process for detecting the sound source location based on the coordinate information and the position information of the sound source location that has been virtually placed in the space in advance, A reverberation characteristics calculation process calculates reverberation characteristics that indicate the characteristics of sound emitted from the sound source location, reflected off the surface of the object, and reaching the user, based on the coordinate information and the detected sound source location. Based on the reverberation characteristics calculated in the aforementioned reverberation characteristics calculation process, a sound source adjustment process is performed to adjust the sound source when providing content. An information processing program to execute an action.

Images