Information processing system, information processing method, and information processing program

Abstract

To reduce the construction cost of a system that can execute an event arranged in space even in a situation where positioning by a positioning satellite is difficult, and has highly accurate start conditions for starting an event. [Solution] Scan data obtained by a sensor that scans space is acquired, and an image taken by a camera is displayed on a display. The user is provided with an operation to place a virtual anchor in the displayed image based on the acquired scan data. Anchor information related to the placed anchor is stored, and an event placement operation is provided to the user to associate and place an event with the stored anchor information, and a start condition setting operation is provided to the user to set a start condition for starting the event placed in the event placement operation. The placement information of the placed event and the start condition set in the start condition setting operation are stored, and the event is stored, and the event is executed when the start condition is met.

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there is a technology that sets an area within a predetermined distance from a geographical coordinate position as a geofence, and starts an event such as providing content to a mobile terminal when the mobile terminal, which can acquire position information by a positioning satellite such as a GPS (Global Positioning System), enters the set geofence. For example, Patent Document 1 discloses a technology in which a mobile client device has a GPS receiver, and provides print data such as an advertisement when the mobile client device enters within a threshold distance of a geographical position defined by the geofence.

[0003] There is also a technology for executing an event arranged in a space in VR (Virtual Reality) by superimposing an image on a video captured by a camera, etc. For example, Patent Document 2 discloses a technology related to a device for arranging a pre-registered object on an image captured by a camera and playing it back. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-208077 A [Patent Document 2] JP 2019-186921 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology of Patent Document 1, the accuracy of positioning varies depending on the reception state of radio waves from the positioning satellite, and for example, the accuracy of positioning by the positioning satellite may decrease or positioning may be impossible in places with poor visibility in the sky or indoors. Therefore, in a system in which an event is executed according to the positioning result by the positioning satellite, the accuracy of detection of whether or not the terminal has entered the geofence may decrease, causing the accuracy of the start of the event to decrease, such as a delay in the timing of the start of the event.

[0006] Furthermore, in Patent Document 2, in order to accurately detect the user's position, multiple sensors need to be installed, and the cost of building the system may increase in order to improve the accuracy of the start conditions for starting an event virtually placed in space.

[0007] The present invention has been made in consideration of the above-mentioned circumstances, and has as an object to provide an information processing system, an information processing method, and an information processing program that can execute events arranged in space even in situations where positioning by positioning satellites is difficult, and that can reduce the cost of building a system that has high accuracy in the starting conditions for starting an event. [Means for solving the problem]

[0008] In order to solve the above problems, the information processing system includes an acquisition unit that acquires scan data scanned by a sensor that scans a space, a display unit that displays a captured image captured by a camera on a display, an anchor placement unit that provides a user with an operation to place a virtual anchor in the captured image displayed on the display unit based on the scan data acquired by the acquisition unit, an anchor information storage unit that stores anchor information related to the anchor placed in the anchor placement unit, an event setting unit that provides a user with an event placement operation that associates an event with the anchor information stored in the anchor information storage unit and places it there, and a start condition setting operation that sets a start condition for starting the event placed in the event placement operation, an event information storage unit that stores placement information of the event placed in the event placement operation and the start condition set in the start condition setting operation, an event storage unit that stores the event, and an event providing unit that provides the event stored in the event storage unit when the start condition stored in the event information storage unit is satisfied.

[0009] In order to solve the above problems, the information processing terminal includes an acquisition unit that acquires scan data scanned by a sensor that scans a space, a display unit that displays a captured image captured by a camera on a display, an anchor placement unit that provides a user with an operation to place a virtual anchor in the captured image displayed on the display unit based on the scan data acquired by the acquisition unit, an anchor information providing unit that provides the information processing device with anchor information related to the anchor placed in the anchor placement unit, an event setting unit that provides a user with an event placement operation that associates an event with the anchor information provided to the information processing device and places it there, and a start condition setting operation that sets a start condition for starting the event placed in the event placement operation, and an event information providing unit that provides the information processing device with placement information of the event placed in the event placement operation and the start condition set in the start condition setting operation.

[0010] In order to solve the above problems, the information processing terminal includes an acquisition unit that acquires scan data scanned by a sensor that scans a space, a detection unit that detects anchors virtually placed in space based on the scan data acquired by the acquisition unit, a display unit that displays the anchors detected by the detection unit in an image captured by a camera, and an event execution unit that executes the event when a start condition for starting the event, which is set for an event placed in correspondence with the anchor detected by the detection unit, is satisfied. Effect of the Invention

[0011] According to one embodiment of the present invention, an information processing system acquires scan data scanned by a sensor that scans a space, displays a captured image captured by a camera on a display, provides a user with an operation to place a virtual anchor in the displayed captured image based on the acquired scan data, stores anchor information related to the placed anchor, and provides the user with an event placement operation to associate an event with the stored anchor information and place it thereon, and a start condition setting operation to set a start condition for starting the event placed in the event placement operation, stores placement information of the placed event and the start condition set in the start condition setting operation, stores an event, and executes the stored event when the stored start condition is satisfied, thereby making it possible to execute an event placed in space even in a situation where positioning by a positioning satellite is difficult, and reducing the construction cost of a system in which the start conditions for starting an event are highly accurate. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram illustrating an example of a system configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing a first display example of a display device used in an information processing terminal in an embodiment. [Diagram 3]FIG. 11 is a diagram showing a second display example of the display device used in the information processing terminal in the embodiment. [Figure 4] FIG. 13 is a diagram showing a third display example of a display device used in the information processing terminal in the embodiment. [Diagram 5] FIG. 13 is a diagram showing a fourth display example of the display device used in the information processing terminal in the embodiment. [Figure 6] FIG. 4 is a diagram illustrating a first example of a terminal state of the information processing terminal in the embodiment. [Figure 7] FIG. 11 is a diagram illustrating a second example of the terminal state of the information processing terminal in the embodiment. [Figure 8] 2 is a block diagram showing an example of a hardware configuration of an information processing terminal according to an embodiment. FIG. [Figure 9] 4 is a flowchart showing a first example of the operation of the information processing system in the embodiment. [Figure 10] 10 is a flowchart showing a second example of the operation of the information processing system in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] An information processing system, an information processing method, and an information processing program according to an embodiment of the present invention will be described in detail below with reference to the drawings.

[0014] First, the configuration of an information processing system will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the system configuration of an information processing system according to an embodiment.

[0015] In FIG. 1, an information processing system 100 includes an information processing terminal 10, an information processing terminal 20, and an information processing device 30.

[0016] The information processing terminal 10 is a terminal for setting an event that is virtually placed in a space and executed. The information processing terminal 20 is a terminal for executing an event that is virtually placed in a space. The information processing device 30 is a device that stores the settings of the event placed in the information processing terminal 10 and causes the information processing terminal 20 to execute the event based on the stored event settings.

[0017] An event in this embodiment is an operation of a device in the information processing system 100 that is executed by the movement of the information processing terminal 20. The device in the information processing system 100 is, for example, the information processing terminal 20 (or the information processing terminal 10), or an audio system, a display system, or an actuator (not shown).

[0018] The operation of the device is, for example, the reproduction of content, data communication, or the operation of an actuator. The reproduction of content is, for example, the reproduction of a sound source, the display of an image (moving image or still image), or the display of a 3D object. The data communication is, for example, communication between the information processing terminal 20 and the information processing device 30, and is communication that provides the information processing device 30 with terminal information of the information processing terminal 20 (for example, device ID, position information, acceleration sensor measurement value, or camera captured image). The data communication may include, for example, control of an audio device or a video device by data communication from the information processing terminal 20. The operation of the actuator is the operation of an actuator such as a motor, an air cylinder, or an air valve, and is, for example, the operation of moving an animal-shaped object or a door driven by a motor or the like, or inflating a balloon that is inflated by air.

[0019] In the following embodiment, the reproduction of a sound source in the information processing terminal 20 is exemplified as the execution of an event, but the execution of an event in the information processing system 100 is not limited to this.

[0020] The information processing terminal 10 and the information processing terminal 20 are devices carried by users, such as smartphones, tablet PCs, or head-mounted, glasses-type, or wristwatch-type wearable devices. The information processing device 30 is a device communicably connected to the information processing terminal 10 and the information processing terminal 20 via the network 9, such as a server device. Note that the user in this embodiment refers to the user of the information processing terminal 10, the information processing terminal 20, or the information processing device 30, and may be different users or the same user. The user may be a dedicated operator, or a general user other than a dedicated operator.

[0021] The information processing terminal 10 has an acquisition unit 11, a display unit 12, an anchor placement unit 13, an anchor information provision unit 14, an event setting unit 15, and an event information provision unit 16. The information processing terminal 20 has an acquisition unit 21, a display unit 22, a detection unit 23, a scan data provision unit 24, an event information acquisition unit 25, and an event execution unit 26. The information processing device 30 has an anchor information storage unit 31, an event information storage unit 32, an event provision unit 33, and an event storage unit 34. Each of the above-mentioned functional units of the information processing system 1 in this embodiment will be described as a functional module realized by an information processing program (software) in this embodiment.

[0022] The acquisition unit 11 and the acquisition unit 21 acquire scan data obtained by scanning a sensor that scans a space. The sensor that scans a space is a sensor that scans the shape of an object existing in the space, and is, for example, a distance measurement sensor that measures the distance to an object existing in the space. For example, a sensor using LiDAR (Light Detection And Ranging) technology can be used as the distance measurement sensor. For example, laser light is used in the LiDAR. The LiDAR may use ultraviolet light, infrared light, or near infrared light. The LiDAR can measure the shape of an object existing in the space in three dimensions by measuring (ranging) the distance from the sensor to the object. Note that the scan data obtained by scanning the space may be, for example, a captured image captured by a stereo camera (not shown) possessed by the information processing terminal 10 or the information processing terminal 20. In addition, the scan data obtained by scanning the space may include information acquired by a position sensor or acceleration sensor (not shown) possessed by the information processing terminal 10 or the information processing terminal 20.

[0023] The display unit 12 and the display unit 22 display images captured by the camera on a display device. For example, the display unit 12 and the display unit 22 display images captured by a smartphone camera on the display screen of the smartphone.

[0024] The anchor placement unit 13 provides the user with an operation of placing a virtual anchor in the captured image displayed on the display unit 12 based on the scan data acquired by the acquisition unit 11. A virtual anchor is a mark (marker) virtually installed (placed) in space, and has coordinate values ​​that indicate a specific position in the space. The anchor in this embodiment is a mark or the like that indicates a position where a sound source is placed. The shape of the mark is, for example, a sphere, a cube, or an icon image, and may be displayed on the display unit 12 and visually recognized by the user. The anchor placement unit 13 clarifies the shape of an object in the space measured based on the scan data acquired by the acquisition unit 11, and provides the user with a UI for performing an operation of placing an anchor in the captured image displayed on the display unit 12. The user can confirm the position where the anchor is placed while checking the captured image displayed on the display unit 12.

[0025] The anchor placement unit 13 further provides the user with an operation to correct the position at which the anchor is placed by using the captured image displayed on the display unit 12. Since the position at which the anchor is placed is determined while checking the two-dimensional captured image displayed on the display unit 12, the placement position may be unclear in the depth direction of the display screen. For example, the anchor placement unit 13 can correct the position at which the anchor is placed by moving the position of an anchor once placed a predetermined distance in the depth direction (e.g., 1 m or 50 cm, etc.). By correcting the position at which the anchor is placed, it is possible to improve the efficiency of anchor placement compared to the case where the anchor is re-placed.

[0026] The anchor information providing unit 14 provides the information processing device 30 with anchor information related to the anchors placed by the anchor placement unit 13. The anchor information related to the anchor includes, for example, coordinate information in space. The coordinate information in space is coordinate information based on scan data obtained by scanning the space, and is, for example, coordinate information indicating a relative position from an object with a large feature value existing in the space. The feature value of the space is, for example, a distribution (histogram) of color data existing in the space, the shape of the object, or vector data of color data. When recognizing the space, a place with similar feature values ​​can be recognized as the same place. Since the position of an object with a large feature value has a large difference in feature value from the surroundings, it is possible to reduce the error in the position of the object in the coordinate information. On the other hand, since the position of an object with a small feature value has a small difference in feature value from the surroundings, the error in the position of the object in the coordinate information is large. By using the relative position from an object with a large feature value existing in the space as coordinate information, it is possible to reduce the error in the position of the object in the coordinate information, for example, even when placing an anchor at the position of an object with a small feature value. The above-mentioned coordinate information is based on scan data obtained by scanning a space, and is therefore relative position information from an object existing in the space. However, the coordinate information may include, for example, coordinate information calculated from a relative distance from a reference point (beacon) that emits radio waves, absolute coordinate information consisting of longitude, latitude, and altitude calculated from radio waves obtained from a positioning satellite, etc. For example, by using coordinate information based on two calculation methods, the coordinate information can be double-checked, thereby improving the reliability of the coordinate information.

[0027] In this embodiment, the term "provision" may refer to either push-type transmission or pull-type transmission. In this embodiment, the term "acquisition" may refer to either pull-type reception or push-type reception.

[0028] The event setting unit 15 provides the user with an event placement operation for placing an event on the anchor information provided to the information processing device 30. The event setting unit 15 also provides the user with a start condition setting operation for setting a start condition for starting the event placed in the event placement operation.

[0029] The event placement operation is an operation of placing an event on anchor information, for example, an operation of assigning a content to be played to a placed anchor. The content allocation includes, for example, an operation of selecting and assigning (associating) a playlist of a sound source that has been created in advance or that has been created newly. A playlist is a method of determining a sound source to be played using one or more sound source files, and by giving a playlist name, it is possible to easily identify the sound source to be played. By providing the event placement operation to a user, for example, it becomes possible to change an event to be executed simply by changing the allocation of an event to an anchor, and it is possible to improve the operability of event placement. Note that the event to be placed may be content playback such as image playback, data communication, or actuator operation, as described above. In the event placement operation, for example, an event that operates an actuator that drives an animal-shaped object may be placed on an anchor placed near the object.

[0030] The start condition for executing an event is a condition for executing an event in the information processing terminal 20 or the like, and is, for example, the distance between the anchor and the information processing terminal 20, the shooting direction of the camera of the information processing terminal 20, the altitude of the information processing terminal 20 (such as the altitude difference relative to the anchor or the absolute altitude based on sea level), the display time of the anchor, or the stop time of the information processing terminal 20. The start condition may also include a condition such as whether the distance between the anchor and the information processing terminal 20 is getting closer or farther away. The start condition may also include the anchor being photographed by multiple information processing terminals 20. The start condition may also include the behavior of the user acquired from an acceleration sensor (not shown) or the like.

[0031] The start conditions may also include the execution mode of the event. For example, when the event is the reproduction of a sound source, the execution mode is the reproduction volume, the reproduction time, the reproduction speed, the emphasis of high or low tones, or the gradual increase or decrease of the volume of the sound source. The execution mode may include the above-mentioned changes in the execution mode of the event depending on the distance between the anchor and the information processing terminal 20, the shooting direction of the camera of the information processing terminal 20, the altitude of the information processing terminal 20, the display time of the anchor, or the stop time of the information processing terminal 20. By setting the reproduction conditions, it becomes possible to perform various performances using the sound source.

[0032] The event information providing unit 16 provides the information processing device 30 with the arrangement information of the event for which the event arrangement operation has been performed in the event setting unit 15, and the start condition set in the start condition setting operation. By providing the arrangement information and the start condition to the information processing device 30, it becomes possible to store this information in the information processing device 30. By storing this information in the information processing device 30, it becomes possible to read and use the stored information. Note that the arrangement of anchors or the setting of events may be executed in multiple information processing terminals 10. When the arrangement of anchors or the like is executed in multiple information processing terminals 10, the execution result of the arrangement of anchors or the like in one information processing terminal 10 is stored in the information processing device 30 and is reflected in the other information processing terminals 10.

[0033] The detection unit 23 detects an anchor virtually arranged in space based on the scan data acquired by the acquisition unit 21. The detection unit 23 acquires anchor information related to the anchor arranged by the anchor arrangement unit 13 from the information processing device 30. The anchor information includes position information of the anchor, etc. The detection of the anchor can be performed, for example, based on whether or not the above-mentioned feature amount of the space is similar. The detection unit 23 can detect the anchor based on whether or not the feature amount of the space is similar based on the scan data acquired by the acquisition unit 21 and the anchor information acquired from the information processing device 30. The detection unit 23 can detect the anchor based on three-dimensional scan data. However, the detection unit 23 may be configured to detect the anchor based on two-dimensional scan data. In addition, the detection unit 23 may be configured to detect the anchor further based on a measurement value acquired from a GNSS (Global Navigation Satellite System) sensor or an acceleration sensor (not shown).

[0034] In this embodiment, the detection unit 23 detects anchors virtually arranged in space based on the acquired scan data, thereby making it possible to set a geofence in three dimensions. In the conventional geofence, a range on a plane can be set as a geofence. On the other hand, in this embodiment, an anchor is detected based on scan data. The scan data differs depending on the three-dimensional position of the information processing terminal 10 (or the information processing terminal 20). For example, even if the information processing terminal 10 is at the same position on a plane, if the information processing terminal 10 is at a different height, the scan data will be different even if the information processing terminal 10 is scanned facing the same point in space. The detection unit 23 detects anchors based on different scan data, thereby making it possible to obtain three-dimensionally different detection results and set a geofence in three dimensions. In this embodiment, the detection unit 23 detects anchors virtually arranged in space based on the acquired scan data, thereby improving the accuracy of the start condition for starting an event arranged in space and making it possible to provide an event using a geofence set in three dimensions.

[0035] The scan data providing unit 24 provides the scan data acquired by the acquiring unit 21 to the information processing device 30. The information processing device 30 can determine whether or not the terminal state estimated by the information processing device 30 satisfies the start condition based on the scan data provided by the scan data providing unit 24. The terminal state is, for example, the position of the information processing terminal 20, the shooting direction of the camera, the distance to the anchor, the vibration of the information processing terminal 20, or the positional relationship with another information processing terminal 20. The information processing device 30 can determine whether or not the terminal state satisfies the playback condition. Whether or not the terminal state satisfies the start condition can be determined by, for example, comparing the terminal state information acquired from the information processing terminal 20 with the start condition stored in the event information storage unit 32.

[0036] In the present embodiment, the case where the detection unit 23 detects the anchor has been exemplified, but the detection of the anchor may be performed in the information processing device 30. By providing the scan data to the information processing device 30, it becomes possible for the information processing device 30 to detect the anchor instead of the detection unit 23 or simultaneously with the detection unit 23.

[0037] The event information acquiring unit 25 acquires from the information processing device 30 start condition information indicating that the terminal state estimated in the information processing device 30 based on the scan data provided by the scan data providing unit 24 satisfies the start condition.

[0038] The event execution unit 26 executes an event when a start condition for starting an event placed on the anchor detected by the detection unit 23 is satisfied. As described above, the execution of an event is, for example, the playback of a sound source. The playback of a sound source is executed, for example, by outputting the sound source provided from the information processing device 30 from a speaker, earphones, or other speakers attached to the information processing terminal 20. By executing an event such as the playback of a sound source when the start condition is satisfied, it becomes possible to perform a performance using the event.

[0039] In this embodiment, the sound source may be provided using an OSC (Open Sound Control) protocol. By using the OSC protocol in the event execution unit 26, it is possible to provide the sound source in real time. For example, if the sound source is a live sound source, by using the OSC protocol, it is possible to share the live sound source in real time among a plurality of information processing terminals 20. In addition, in this embodiment, a sound source format (e.g., Ambisonics) for realizing stereophonic sound may be used in providing the sound source. By using a sound source format for realizing stereophonic sound in the event execution unit 26, it is possible to easily provide a stereophonic sound source. In addition, in this embodiment, a platform for game development (e.g., Wwise, etc.) may be used in providing the sound source. By using a platform for game development in the event execution unit 26, it is possible to easily provide an event in a game. In addition, in this embodiment, the sound source may be reproduced using a head related transfer function (HRTF) in providing the sound source. The head related transfer function is a transfer function that expresses the change in sound caused by surrounding objects including the auricle, the human head, and shoulders. In the event execution unit 26, by quantifying the characteristics of the change in sound from the sound source to both ears using a head-related transfer function, it becomes possible to recognize the sense of direction of the sound source and the sense of distance to the sound source.

[0040] The anchor information storage unit 31 stores anchor information relating to the anchors arranged by the anchor arrangement unit 13. The anchor information storage unit 31 can provide the stored anchor information to the information processing terminal 20.

[0041] The event information storage unit 32 stores the arrangement information of the event arranged by the event arrangement operation in the event setting unit 15 and the start condition set in the start condition setting operation. The event information storage unit 32 stores the arrangement information or start condition input by the user of the information processing device 30. The event information storage unit 32 can execute the event in the information processing terminal 20 etc. based on the stored arrangement information and start information.

[0042] The event providing unit 33 provides the event stored in the event storage unit 34 when the start condition stored in the event information storage unit 32 is satisfied. The event providing unit 33 may provide the event in a predetermined event execution mode according to a preset start condition. For example, the event providing unit 33 may provide the event by changing the event execution mode according to the identification information of the information processing terminal 20 (pre-registered attribute information of the user such as gender, age, language used, nationality, or religion, or device information acquired from the information processing terminal 20, etc.). The identification information may be acquired in association with, for example, the user's account information. The event executing unit 26 executes a variety of events by executing the event changed according to the identification information.

[0043] The event storage unit 34 may store an event list in which a plurality of events are listed. For example, if the event is the reproduction of a sound source, the event list is a plurality of sound sources or a playlist created in advance. The events stored in the event storage unit 34 can be provided to the information processing terminal 20 and reproduced by the event execution unit 26. The events stored in the event storage unit 34 may be content data such as audio data or image data, or may be information indicating a storage destination or an acquisition method of the content data.

[0044] The above-mentioned functional units of the information processing system 1 are merely examples of functions, and do not limit the functions of the information processing system 1. For example, the information processing terminal 10, the information processing terminal 20, or the information processing device 30 does not need to have all the functional units described above, and may have only some of the functional units. The information processing system 1 may also have functions other than those described above.

[0045] For example, a functional unit of the information processing terminal 10 may be realized in the information processing device 30. Similarly, a functional unit of the information processing terminal 20 may be realized in the information processing device 30. Furthermore, a functional unit of the information processing device 30 may be realized in the information processing terminal 10 or the information processing terminal 20. Furthermore, the information processing terminal 10 and the information processing terminal 20 may be formed integrally.

[0046] As described above, each of the above functional units has been described as being realized by software, but at least one of the above functional units may be realized by hardware.

[0047] Also, any of the above functional units may be implemented by dividing one functional unit into a plurality of functional units. Also, any two or more of the above functional units may be implemented by integrating them into one functional unit. Fig. 1 shows the functions of the information processing system 1 in the form of functional blocks, and does not indicate, for example, that each functional unit is configured by a separate program file or the like.

[0048] Furthermore, the information processing terminal 10, the information processing terminal 20, or the information processing device 30 may be a device realized by one housing, or may be a system realized by a plurality of devices connected via a network or the like. For example, the information processing terminal 10, the information processing terminal 20, or the information processing device 30 may have some or all of its functions realized by another virtual device, such as a cloud service provided by a cloud computing system. That is, the information processing terminal 10, the information processing terminal 20, or the information processing device 30 may have at least one or more of the above-mentioned functional units realized by another device.

[0049] 2 to 5, a display example of the display device of the information processing terminal 10 will be described. In the following embodiment, a case will be exemplified in which a sound source is placed at an anchor and an event in which the sound source is played back in the information processing terminal 20.

[0050] FIG. 2 is a diagram showing a first display example of a display device used in an information processing terminal in the embodiment.

[0051] 2, a display screen 1000 is displayed on the display unit 12. The display screen 1000 has an icon 1001 indicating an anchor setting screen, an anchor installation button 1002, an exit button 1003, a save button 1004, and a naming button 1005 for naming the playlist.

[0052] Display screen 1000 is a display screen displayed on information processing terminal 10, and displays an image captured by the camera of information processing terminal 10. Anchor installation button 1002 is a button for installing an anchor anywhere in the shooting range displayed on display screen 1000. For example, by operating anchor installation button 1002, an anchor can be installed at the center position of display screen 1000 or at a cursor position (not shown) or the like within display screen 1000. By operating anchor installation button 1002, a user can transition to a screen for placing an anchor, which will be described in FIG. 3.

[0053] An exit button 1003 is a button for ending the settings shown on the display screen 1000, and a save button 1004 is a button for saving the current settings. A name button 1005 is a button for naming the playlist. By operating the name button 1005, the user can input a name in a text box or the like.

[0054] FIG. 3 is a diagram showing a second display example of the display device used in the information processing terminal in the embodiment.

[0055] In FIG. 3, a display screen 2000 is displayed on the display unit 12. The display screen 2000 has a trial placement button 2001. The display screen 2000 is a captured image that is enlarged by moving the shooting range displayed on the display screen 1000. The trial placement button 2001 is a button for trial placement of an anchor. A user can place an anchor substantially in the center of the display screen 2000 by operating the trial placement button 2001. Note that trial placement means placement of an anchor before the placement position is confirmed, and the user can trial experience the anchor that he or she placed and the sound source set to the anchor.

[0056] FIG. 4 is a diagram showing a third display example of the display device used in the information processing terminal in the embodiment.

[0057] In FIG. 4, the display screen 1000 is displayed on the display unit 12. The display screen 3000 has an anchor position correction button 3001, an anchor position determination button 3002, and an anchor 3003. The anchor position correction button 3001 is a button for correcting the position of the anchor 3003 set by the trial placement button 2001. The user can correct the position of the set anchor 3003 to the rear side or the front side by operating the anchor position correction button 3001. When the position of the anchor 3003 is corrected to the rear side, the size of the anchor 3003 is displayed smaller, whereas when the position of the anchor 3003 is corrected to the front side, the size of the anchor 3003 is displayed larger. The display size of the anchor 3003 may be changed by a setting operation not shown.

[0058] The anchor position determination button 3002 is a button for determining the installation position of the anchor 3003. By operating the anchor position determination button 3002, the user can transition to a screen for completing the installation of the anchor, which will be described with reference to FIG.

[0059] FIG. 5 is a diagram showing a fourth display example of the display device used in the information processing terminal 10 in the embodiment.

[0060] 5, a display screen 1000 is displayed on the display unit 12. A display screen 4000 has an end button 4001 and an anchor 4002. The user operates the end button 4001 to end the setting of the anchor.

[0061] Next, the terminal state of the information processing terminal 20 will be described with reference to FIGS.

[0062] Fig. 6 is a diagram showing a first example of the terminal state of the information processing terminal in the embodiment. Fig. 6 illustrates a case where the playback mode differs when the distance and direction between the information processing terminal 20 and the anchor are in a terminal state.

[0063] In Fig. 6, anchor position O is the position in space of the anchor installed in the above-mentioned procedure. The information processing terminal 20 at the position shown in Fig. 6 (A) is at a distance x from the anchor position O and is photographing the anchor. On the other hand, the information processing terminal 20 at the position shown in Fig. 6 (B) is at a distance y from the anchor position O and is photographing the anchor.

[0064] Here, the information processing terminal 20 detects an anchor with a sensor that scans a space, and plays back a sound source placed at anchor position O in a playback mode with a volume and direction according to the distance x from the information processing terminal 20 and the direction of the anchor position O. Similarly, the information processing terminal 20 detects an anchor with a sensor that scans a space, and plays back a sound source placed at anchor position O in a playback mode with a volume and direction according to the distance y from the information processing terminal 20 and the direction of the anchor position O.

[0065] For example, if the playback condition for the sound source is that the distance between the information processing terminal 20 and the anchor position O is “within distance x,” then the information processing terminal 20 at position (A) will play the sound source, but the information processing terminal 20 at position (B) will not play the sound source.

[0066] Fig. 7 is a diagram showing a second example of the terminal state of the information processing terminal in the embodiment. Fig. 7 explains the playback mode in the terminal state when there are two information processing terminals 20. The terminal state of the information processing terminal 20 in Fig. 7 is the distance of the information processing terminal 20 to the anchor, the direction from the information processing terminal 20 to the anchor, and the positional relationship with the other information processing terminals 20.

[0067] 7, the distance between information processing terminal 20a and information processing terminal 20b is z. That is, information processing terminal 20a is in a terminal state in which the distance between information processing terminal 20a and anchor position O is x and the distance between information processing terminal 20b and information processing terminal 20b is z, and information processing terminal 20b is in a terminal state in which the distance between information processing terminal 20a and anchor position O is y and the distance between information processing terminal 20a and information processing terminal 20b is z. The positional relationship between information processing terminal 20a and information processing terminal 20b is calculated by each information processing terminal 20 providing its positional relationship with anchor position O to information processing device 30.

[0068] FIG. 7 illustrates an example in which the playback mode of a sound source in one information processing terminal is determined by the position of another information processing terminal. The information processing terminal 20a plays back a sound source in a playback mode of both the sound source at the anchor position O and the sound source regarded as the information processing terminal 20b. Similarly, the information processing terminal 20b plays back a sound source in a playback mode of both the sound source at the anchor position O and the sound source regarded as the information processing terminal 20a. That is, the playback mode of a sound source in the information processing terminal 20a is determined by the distance z from the information processing terminal 20b and the position of the information processing terminal 20b. Similarly, the playback mode of a sound source in the information processing terminal 20b is determined by the distance z from the information processing terminal 20a and the position of the information processing terminal 20a.

[0069] Next, the hardware configuration of the information processing terminal 10 will be described with reference to Fig. 8. Fig. 8 is a block diagram showing an example of the hardware configuration of the information processing terminal 10 in the embodiment. Note that the hardware configurations of the information processing terminal 20 and the information processing device 30 are similar to that of the information processing terminal 10, and therefore descriptions thereof will be omitted.

[0070] The information processing terminal 10 includes a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, an I / O device 104, and a communication I / F (Interface) 105. The information processing terminal 10 is a device that executes the information processing program described in FIG.

[0071] The CPU 101 controls the user terminal by executing an information processing program stored in the RAM 102 or the ROM 103. The information processing program is obtained, for example, from a recording medium on which the program is recorded or a program distribution server via a network, and installed in the ROM 103, and is read out and executed by the CPU 101.

[0072] The I / O device 104 has an operation input function and a display function (operation display function). The I / O device 104 is, for example, a touch panel. The touch panel enables the user of the information processing terminal 10 to input operations using a fingertip, a touch pen, or the like. In this embodiment, the I / O device 104 is described as using a touch panel having an operation display function, but the I / O device 104 may have a display device having a display function and an operation input device having an operation input function separately. In that case, the display screen of the touch panel can be implemented as the display screen of the display device, and the operation of the touch panel can be implemented as the operation of the operation input device. The I / O device 104 may be realized in various forms such as a head-mounted type, a glasses type, or a wristwatch type display.

[0073] The communication I / F 105 is an I / F for communication. The communication I / F 105 executes short-distance wireless communication such as wireless LAN, wired LAN, infrared, etc. Although the figure illustrates only the communication I / F 105 as an I / F for communication, the information processing terminal 10 may have an I / F for each of a plurality of communication methods.

[0074] Next, the operation of the information processing system 1 will be described with reference to FIGS.

[0075] 9 is a flowchart showing a first example of the operation of the information processing system in the embodiment. The flowchart shown in the first example explains the operation related to the arrangement of anchors and sound sources.

[0076] 9, the information processing system 1 provides a UI for setting a sound source on the display screen of the information processing terminal 10 (step S11). After executing the process of step S11, the information processing system 1 acquires scan data from a sensor that scans a space (step S12).

[0077] After executing the process of step S12, the information processing system 1 judges whether or not to place an anchor (step S13). The judgment of whether or not to place an anchor can be made, for example, based on whether or not the user has completed the operation of setting the anchor. If it is judged that the anchor is not to be placed (step S13: NO), the information processing system 1 repeats the processes of steps S12 to S13 and waits for the anchor to be placed.

[0078] When it is determined that the anchor has been placed (step S13: YES), the information processing system 1 determines whether or not to modify the position of the anchor (step S14). When it is determined that the position of the anchor should be modified (step S14: YES), the information processing system 1 modifies the position of the anchor (step S15).

[0079] After executing the process of step S15, or when it is determined that the anchor position is not to be corrected (step S14: NO), the information processing system 1 determines whether or not a playlist has been selected (step S16). The selection of a playlist includes saving the selected playlist. If the desired playlist does not exist, the information processing system 1 may provide a UI for creating a new playlist. If it is determined that a playlist has not been selected (step S16: NO), the information processing system 1 repeats the process of step S16 and waits for a playlist to be selected.

[0080] On the other hand, if it is determined that a playlist has been selected (step S16: YES), the information processing system 1 provides a UI for inputting playback conditions on the display screen of the information processing terminal 10 (step S17). After executing the process of step S17, the information processing system 1 determines whether input of playback conditions has been completed (step S18). If it is determined that input has not been completed (step S18: NO), the information processing system 1 repeats the process of step S18 and waits for input to be completed.

[0081] On the other hand, when it is determined that the input of the playback conditions is completed (step S18: YES), the information processing system 1 provides the anchor information input in steps S13 to S15 from the information processing terminal 10 to the information processing device 30 (step S19).

[0082] After executing the process of step S19, the information processing system 1 provides the information relating to the sound source input in steps S16 to S18 from the information processing terminal 10 to the information processing device 30 (step S20).

[0083] When the process of step S20 is completed, the information processing system 1 stores the anchor information provided in step S19 in the information processing device 30 (step S21). After the process of step S21 is completed, the information processing system 1 stores the sound source information provided in step S20 in the information processing device 30 (step S22), and ends the process shown in the flowchart.

[0084] 10 is a flowchart showing a second example of the operation of the information processing system 1 in the embodiment. The flowchart shown in the second example explains the operation related to the reproduction of a sound source.

[0085] 10, the information processing system 1 provides a UI for playing a sound source on the display screen of the information processing terminal 20 (step S31). The UI for playing a sound source is, for example, an AR (Augmented Reality) application program for enjoying the playback of a sound source placed in a space.

[0086] After executing the process of step S32, the information processing system 1 acquires scan data from a sensor that scans a space (step S32).

[0087] After executing the process of step S32, the information processing system 1 determines whether or not an anchor is detected (step S33). If it is determined that an anchor is not detected (step S33: NO), the information processing system 1 repeats the processes of steps S32 to S33 and waits for an anchor to be detected.

[0088] On the other hand, if it is determined that an anchor has been detected (step S33: YES), the information processing system 1 causes the detected anchor to be displayed on the display screen of the information processing terminal 20 (step S34).

[0089] After executing the process of step S34, the information processing system 1 determines whether the playback conditions are satisfied (step S35). Whether the playback conditions are satisfied can be determined, for example, by whether the information processing terminal 20 acquires playback condition information from the information processing device 30. If it is determined that the playback conditions are not satisfied (step S35: NO), the information processing system 1 repeats the processes of steps S32 to S35 and waits until the playback conditions are satisfied.

[0090] On the other hand, when it is determined that the playback condition is satisfied (step S35: YES), the information processing system 1 starts playing the sound source (step S36). The information processing terminal 20 plays the sound source acquired from the information processing device 30, thereby executing the playback of the sound source.

[0091] After executing the process of step S36, the information processing system 1 checks the terminal state (step S37). As described above, the terminal state is the position of the information processing terminal 20, the shooting range of the information processing terminal 20, etc. After executing the process of step S36, the information processing system 1 changes the playback mode (step S38). For example, when the information processing terminal 20 is approaching an anchor, the information processing system 1 changes the playback mode to increase the playback volume of the sound source. The playback mode according to the terminal state can be set in advance. In addition, the playback mode according to the terminal state may be changed randomly.

[0092] After executing the process of step S38, the information processing system 1 judges whether or not to end the playback (step S39). The decision to end the playback may be made, for example, when the terminal state no longer satisfies the playback conditions, when playback of a previously prepared sound source has ended, or when the user explicitly stops playback. If it is determined not to end the playback (step S39: NO), the information processing system 1 repeats the processes of steps S37 to S39 and waits for the playback to end. On the other hand, if it is determined to end the playback (step S39: YES), the information processing system 1 ends the operation shown in the flowchart.

[0093] It should be noted that the illustrated flowchart shows only one example of the operation and does not limit the operation.

[0094] The above-mentioned various processes of the present embodiment may be performed by recording a program for realizing the functions constituting the device described in the present embodiment in a computer-readable recording medium, and reading and executing the program recorded in the recording medium into a computer system. The "computer system" referred to here may include hardware such as an OS and peripheral devices. In addition, if a WWW system is used, the "computer system" also includes a homepage providing environment (or display environment). In addition, the "computer-readable recording medium" refers to a storage device such as a flexible disk, a magneto-optical disk, a ROM, a writable non-volatile memory such as a flash memory, a portable medium such as a CD-ROM, or a hard disk built into a computer system.

[0095] Furthermore, the term "computer-readable recording medium" includes a memory that holds a program for a certain period of time, such as a volatile memory (e.g., DRAM (Dynamic Random Access Memory)) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. The above-mentioned program may be transmitted from a computer system that stores the program in a storage device or the like to another computer system via a transmission medium, or by a transmission wave in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above-mentioned program may also be a program for realizing part of the above-mentioned functions. Furthermore, the above-mentioned function may be realized in combination with a program already recorded in the computer system, that is, a so-called difference file (difference program).

[0096] Although an embodiment of the present invention has been described above with reference to the drawings, the specific configuration is not limited to this embodiment, and various modifications are also included within the scope that does not deviate from the spirit of the present invention. [Explanation of symbols]

[0097] 1. Information Processing Systems 10. Information processing terminal 11 Acquisition Department 12 Display section 13 Anchor placement section 14 Anchor Information Department 15 Event Settings Section 16 Event Information Department 20 Information processing terminal 20a Information processing terminal 20b Information processing terminal 21 Acquisition Department 22 Display section 23 Detection unit 24 Scan Data Providing Department 25 Event Information Acquisition Department 26 Event Organizing Department 30 Information processing device 31 Anchor information storage unit 32 Event information storage unit 33 Events Department 34 Event Memory Section 101 CPU 102 RAM 103 ROM 104 I / O equipment 105 Communication I / F 1000 display screen 1001 Icon 1002 Anchor installation button 1003 Exit button 1004 Save button 1005 Naming button 1005 2000 display screen 2001 Trial Placement Button 3000 display screen 3001 Anchor position correction button 3002 Anchor position determination button 3003 Anchor 4000 display screen 4001 Exit button

Claims

1. An acquisition unit that acquires scan data scanned by a LIDAR sensor that scans a space; a detection unit that detects an anchor based on whether a feature of a space based on the scan data acquired by the acquisition unit and a feature of a space that is virtually placed in the space and that is preset for the anchor are similar to each other; a display unit that displays the anchor detected by the detection unit on an image captured by a camera; an event execution unit that executes an event when a start condition for starting the event, which is set for the event arranged in association with the anchor detected by the detection unit, is satisfied; An information processing terminal comprising:

2. The information processing terminal according to claim 1 , wherein the event execution unit executes the event when a terminal state of the information processing terminal estimated based on the scan data acquired by the acquisition unit satisfies the start condition.

3. a scan data providing unit that provides the scan data acquired by the acquisition unit to an information processing device; and a start information acquisition unit that acquires from the information processing device start condition information indicating that the terminal state estimated in the information processing device based on the scan data provided by the scan data provision unit satisfies the start condition, 3. The information processing terminal according to claim 2, wherein said event executing section executes an event when the start condition information acquired by said start information acquiring section satisfies said start condition.

4. 4. The information processing terminal according to claim 1, wherein the event execution unit plays a sound source when a position and direction of the information processing terminal estimated based on the scan data acquired by the acquisition unit and a position of the anchor detected by the detection unit satisfy the start condition.

5. The information processing terminal according to claim 1 , wherein the event execution unit changes an execution mode of the event based on the scan data acquired by the acquisition unit.

6. The information processing terminal according to claim 5 , wherein the event execution section changes a volume of the sound reproduced during the event as the execution mode.

7. The information processing terminal according to claim 1 , wherein the event executing section changes an execution mode of the event based on a terminal state of another information processing terminal.

8. The information processing terminal according to claim 1 , wherein the event execution unit changes an execution mode of the event based on the captured image displayed on the display unit.

9. An acquisition step of acquiring scan data scanned by a LIDAR sensor that scans a space; a detection step of detecting an anchor based on whether or not a feature amount of a space based on the scan data acquired in the acquisition step is similar to a feature amount of a space that is virtually placed in the space and is preset for the anchor; a display step of displaying the anchor detected in the detection step on an image captured by a camera; an event execution step of executing the event when a start condition for starting the event, which is set for the event arranged in association with the anchor detected in the detection step, is satisfied; An information processing method executed by an information processing terminal, comprising:

10. On the computer, An acquisition function for acquiring scan data scanned by a LIDAR sensor that scans a space; a detection function for detecting an anchor based on whether or not a feature of a space based on the scan data acquired by the acquisition function and a feature of a space that is virtually placed in the space and is preset for the anchor are similar to each other; a display function for displaying the anchor detected by the detection function in an image captured by a camera; an event execution function that executes an event when a start condition for starting the event, which is set for the event arranged in association with the anchor detected by the detection function, is satisfied; An information processing program that realizes this.

Images