Information processing device, information processing method, and program

Abstract

An information processing device according to one aspect of this invention comprises a communication unit, a point group data management unit, a sound information analysis unit, an attribute change unit, and a virtual space construction unit. The communication unit communicates with a real-world information acquisition device that includes an environment sensor and an acoustic sensor. The point group data management unit stores 3D point group data of a real space, which is acquired from the real-world information acquisition device. The sound information analysis unit analyzes sound information data of ambient sounds acquired from the real-world information acquisition device, and obtains the type of the sound information data. The attribute change unit changes, on the basis of the type of the sound information data, the attribute information of the 3D point group data stored in the point group data management unit. The virtual space construction unit constructs a virtual space image on the basis of the 3D point group data of which the attribute information is changed.

Description

[Technical Field]

[0001] One aspect of the present invention relates to an information processing device, an information processing method, and a program used in a virtual space construction technology called XR (Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), etc.). [Background technology]

[0002] Technology that uses computers to create virtual spaces is attracting attention. Virtual spaces are constructed using CG (Computer Graphics) technology, for example, on a computer server that multiple people can log in to. Virtual Reality (VR) is a representative technology, and is used to represent virtual worlds such as scenes in quest games. In addition to this, in recent years, attempts have been made to faithfully reproduce real space (the real world) in a 3D virtual space based on measurement data from the real world.

[0003] 3D point cloud data obtained by measuring a real space is suitable for representing the real space in a virtual space. Furthermore, it is possible to reflect sound information from the real space in a virtual space. For example, by simultaneously measuring 3D point cloud data and sound information and reflecting the sound information in a virtual space constructed with the 3D point cloud data, a realistic digital twin can be created.

[0004] One of the first methods for reflecting sound information in a virtual space is to acoustically reproduce the collected sound information in the virtual space. However, if a person in the virtual space is hearing impaired or if there is a problem with the sound reproduction device, it is difficult to recognize the sound information in the virtual space. Therefore, methods are being considered in which the sound information is visualized by some means and visually superimposed on the virtual space. For example, Non-Patent Documents 1 and 2 are known for visualization of sound information.

[0005] Non-Patent Document 1 is an example of visualizing sound effects in online games. In online games, sound effect data and visualization data (CG) for visualizing this data are stored in advance on the game server. In the technology of Non-Patent Document 1, when the sound effect visualization function is turned on, visualization data representing the corresponding sound effect is superimposed on game world data (CG) and sent to the user terminal. The user terminal displays the data received from the game server on its monitor.

[0006] Non-Patent Document 2 is an example of converting voice to text. This device recognizes human voice (utterance) input from a microphone, converts it into text information, and displays the converted text as text on a terminal monitor. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] "[Visualization of Sound Effects] What has been enhanced in the update (Fortnite)?", [online], [Searched on May 2, 2022], Internet,<URL:https: / / hebi-pan.com / fn-visual / > [Non-patent document 2] "PokeTalk (registered trademark) mimi (SOURCENEXT)", [online], [searched May 2, 2022], Internet,<URL:https: / / pocketalkmimi.jp> Summary of the Invention [Problem to be solved by the invention]

[0008] Techniques for visualizing sound effects in online games, such as those described in Non-Patent Document 1, can only visualize pre-prepared sound effect data, making it difficult to visualize sound information acquired from real space in real time. This is because both the sound effects and visualization data are simply prepared in advance along with the game program, loaded into memory, and reused. Furthermore, since the visual data for games is a pre-prepared CG object, it is even more difficult to visualize and superimpose sound information on 3D point cloud data, which is a "collection of points."

[0009] Speech-to-text technology such as that described in Non-Patent Document 2 is not capable of visualizing artificial sounds such as car horns, natural sounds such as the sound of wind or rain, or ambient sounds (environmental sounds). This technology recognizes and transcribes human speech (conversational speech) in order to improve communication for people with hearing impairments, for example, and is not intended to visualize environmental sounds in a virtual space.

[0010] The present invention has been made in light of the above circumstances, and its object is to provide a technology that can reflect sounds in a real space in a virtual space. [Means for solving the problem]

[0011] An information processing device according to one aspect of the present invention includes a communication unit, a point cloud data management unit, a sound information analysis unit, an attribute change unit, and a virtual space construction unit. The communication unit communicates with a real world information acquisition device equipped with an environmental sensor and an acoustic sensor. The point cloud data management unit stores 3D point cloud data of real space acquired from the real world information acquisition device. The sound information analysis unit analyzes sound information data of ambient sounds acquired from the real world information acquisition device to determine the type of the sound information data. The attribute change unit changes attribute information of the 3D point cloud data stored in the point cloud data management unit based on the type of sound information data. The virtual space construction unit constructs a virtual space image based on the 3D point cloud data with the changed attribute information. [Effects of the Invention]

[0012] According to one aspect of the present invention, it is possible to provide a technology that can reflect sounds in a real space in a virtual space. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing an example of a system to which an information processing device according to an embodiment is applied. [Figure 2] FIG. 2 is a functional block diagram showing an example of the virtual space management server 1, the real world information acquisition device 2, and the user terminal 3. As shown in FIG. [Figure 3] FIG. 3 is a diagram showing the flow of data relating to the virtual space management server 1, the real world information acquisition device 2, and the user terminal 3. As shown in FIG. [Figure 4] FIG. 4 is a diagram showing the flow of data relating to the virtual space management server 1, the real world information acquisition device 2, and the user terminal 3. As shown in FIG. [Figure 5] FIG. 5 is a flowchart showing an example of a processing procedure of the processor 70 of the virtual space management server 1. DETAILED DESCRIPTION OF THE INVENTION

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the accompanying drawings. First, 3D point cloud data will be described. One method for measuring real space and constructing a virtual space that mimics it is to use 3D point cloud data. 3D point cloud data can be acquired, for example, using LiDAR (Light Detection and Ranging). LiDAR is a device that measures the distance, direction, position, and shape of an object by shining a laser beam at the object and measuring the time it takes for the beam to travel back and forth. The results of the laser beam scan are represented as points, and by stippling the countless points collected (point cloud), an image of an object in real space can be generated.

[0015] Generally, 3D point cloud data is simply a collection of points that have attribute information such as three-dimensional coordinate values ​​(X, Y, Z) and color information (R, G, B), and it is difficult to superimpose visualized information onto data of this nature. In other words, it is difficult to superimpose sound information onto 3D point cloud data. Below, we will explain a technology that can solve this problem.

[0016] 1 is a diagram showing an example of a system to which an information processing device according to an embodiment is applied. This system has a virtual space management server 1 as its core, which serves as an information processing device. The virtual space management server 1 can access a real-world information acquisition device 2 via a network 4, and a user terminal 3 can also access the virtual space management server 1 via the network 4.

[0017] In FIG. 1, a real-world information acquisition device 2 is installed, for example, at an intersection or a park, and acquires environmental information around the installation location. In the embodiment, 3D point cloud data and sound information data are assumed as the environmental information. The real-world information acquisition device 2 includes a measurement unit 21 as an environmental sensor and an ambient sound collection unit 22 as an acoustic sensor. The measurement unit 21 acquires 3D point cloud data of the real space. The ambient sound collection unit 22 acquires sound information data of the real space. The 3D point cloud data and sound information data are transmitted to a virtual space management server 1 via a network 4.

[0018] The virtual space management server 1 acquires the 3D point cloud data and sound information data via the network 4 and constructs a virtual space image from these data. The user terminal 3 accesses the virtual space management server 1 via the network 4 and acquires the virtual space image.

[0019] The virtual space management server 1 is an information processing device (computer) including a processor 70 and a memory 60. In addition, the virtual space management server 1 includes a storage 50 and a communication unit 80 that functions as a communication interface with the network 4.

[0020] The communication unit 80 communicates with the real world information acquisition device 2 via the network 4 to acquire the 3D point cloud data and sound information data. The acquired 3D point cloud data (reference numeral 51) is stored in the storage 50.

[0021] The storage 50 further stores a program 52. The program 52 is loaded into the memory 60 by the OS (Operating System) of the virtual space management server 1 and executed by the processor 70. The program 52 causes the processor 70 to function as a point cloud data management unit 10, a sound information analysis unit 20, a virtual space construction unit 30, and an attribute change unit 40.

[0022] The point cloud data management unit 10 stores the 3D point cloud data of the real space acquired from the real world information acquisition device 2 in, for example, a storage 50. The sound information analysis unit 20 analyzes the sound information data of the ambient sound acquired from the real world information acquisition device 2, and determines the type of the sound information data. The attribute change unit 40 changes the attribute information of the 3D point cloud data stored in the point cloud data management unit 10 based on the type of the sound information data. The virtual space construction unit 30 constructs a virtual space image based on the 3D point cloud data whose attribute information has been changed.

[0023] 2 is a functional block diagram showing an example of a virtual space management server 1, a real-world information acquisition device 2, and a user terminal 3. In Fig. 2, a measurement unit 21 of the real-world information acquisition device 2 is, for example, a LiDAR, which scans the real space with laser light to obtain 3D point cloud data.

[0024] The ambient sound collection unit 22 is, for example, a 360-degree microphone that collects ambient sounds around the location where the real world information acquisition device 2 is installed and generates sound information data. The 3D point cloud data and sound information data are transmitted to the virtual space management server 1 via a communication link formed between the communication unit 23 of the real world information acquisition device 2 and the communication unit 80 of the virtual space management server 1.

[0025] The user terminal 3 includes a video display unit 31 connected to a monitor 31a, an audio playback unit 32 connected to a speaker 32a, an I / O unit 33 connected to a keyboard (KB) 33a and the like, and a communication unit 34. The video display unit 31 displays a virtual space image based on 3D point cloud data received from the virtual space management server 1 on the monitor 31a. The audio playback unit 32 plays back sound from the speaker 32a. The I / O unit 33 accepts user input / output operations from the KB 33a.

[0026] The virtual space management server 1 includes a communication unit 80, a point cloud data management unit 10, a sound information analysis unit 20, a virtual space construction unit 30, and an attribute change unit 40. The point cloud data management unit 10 includes a point cloud data acquisition unit 10a and a position determination unit 10b. The point cloud data acquisition unit 10 a acquires 3D point cloud data from the measurement unit 21 of the real world information acquisition device 2 and stores it as 3D point cloud data 51 in the storage 50 . The position determination unit 10b acquires the installation position of the real world information acquisition device 2.

[0027] The sound information analysis unit 20 includes a sound information acquisition unit 20a, an analysis unit 20b, and a sound information analysis rule management unit 20c. The sound information acquisition unit 20 a acquires sound information data from the ambient sound collection unit 22 . The sound information analysis rule management unit 20c manages the analysis rules for sound information data. The analysis unit 20b analyzes the acquired sound information data based on the analysis rules of the sound information analysis rule management unit 20c, and calculates the type of the sound information data.

[0028] The attribute change unit 40 includes a point cloud data attribute change unit 40a and a point cloud data attribute change rule management unit 40b. The point cloud data attribute change rule management unit 40b manages the change rules for the attributes of 3D point cloud data. The point cloud data attribute change unit 40a changes the attribute information of the 3D point cloud data stored in the point cloud data management unit 10 based on the change rules of the point cloud data attribute change rule management unit 40b.

[0029] The virtual space construction unit 30 includes a point cloud data visualization management unit 30a and a point cloud data visualization unit 30b. The point cloud data visualization unit 30b visualizes the 3D point cloud data whose attribute information has been changed, and generates image data of the virtual space. The point cloud data visualization management unit 30a manages the visualization of 3D point cloud data by the point cloud data visualization unit 30b based on the installation position of the real world information acquisition device 2.

[0030] FIG. 3 is a diagram showing the flow of data relating to the virtual space management server 1, the real world information acquisition device 2, and the user terminal 3. As shown in FIG. In Figure 3, the real-world information acquisition device 2 transmits 3D point cloud data around the device acquired by the measurement unit 21 and sound information data around the device collected by the ambient sound collection unit 22 to the virtual space management server 1 from the communication unit 23.

[0031] The virtual space management server 1 sends the received 3D point cloud data to and manages the point cloud data management unit 10. The position determination unit 10b of the point cloud data management unit 10 extracts the position information of the real world information acquisition device 2 from the 3D point cloud data and passes it to the sound information analysis unit 20.

[0032] The sound information acquisition unit 20a of the sound information analysis unit 20 acquires sound information data and position information and passes them to the analysis unit 20b. The analysis unit 20b analyzes the sound information data based on the analysis rules of the sound information analysis rule management unit 20c and determines the type of the sound information data. In particular, the analysis unit 20b determines the type of sound information data in association with the position information. In other words, the type of sound information data is determined for each piece of position information, and the result is passed to the attribute change unit 40.

[0033] The point cloud data attribute change unit 40a of the attribute change unit 40 changes the attribute information of the 3D point cloud data according to the type of sound information data notified by the analysis unit 20b. At this time, the change rules of the point cloud data attribute change rule management unit 40b are referenced. The changes to the attribute information of the 3D point cloud data are notified to the virtual space construction unit 30 together with the position information.

[0034] The point cloud data visualization unit 30b of the virtual space construction unit 30 superimposes the 3D point cloud data whose attributes have been changed by the attribute change unit 40 on the original 3D point cloud data (with attributes unchanged) to construct a virtual space image. The construction of the virtual space image is managed by the point cloud data visualization management unit 30a based on the installation position of the real-world information acquisition device 2. The virtual space image after the attribute change is transmitted from the communication unit 80 to the user terminal 3.

[0035] The user terminal 3 functions based on instructions from the user given via the I / O unit 33, and displays the attribute-changed virtual space image received by the communication unit 34 on the monitor 31a. The audio playback unit 32 plays back sound information.

[0036] 4 is a diagram showing the flow of data related to the virtual space management server 1, real world information acquisition device 2, and user terminal 3. The 3D point cloud data and sound information data collected by the real world information acquisition device 2 are sent to the virtual space management server 1. The virtual space management server 1 analyzes the type of sound information data and changes the attributes of the 3D point cloud data based on this type. A virtual space image created based on the 3D point cloud data after the attribute change is sent to the user terminal 3 and displayed on the monitor 31a.

[0037] Fig. 5 is a flowchart showing an example of a processing procedure of the processor 70 of the virtual space management server 1. In Fig. 5, the processor 70 acquires 3D point cloud data of the real space from the real world information acquisition device 2 (step S1) and stores it in storage (step S2). The processor 70 also acquires sound information data of ambient sounds from the real world information acquisition device 2 (step S3), analyzes it, and calculates the type of the sound information data (step S4). Furthermore, the processor 70 changes the attribute information of the 3D point cloud data based on the type of sound information data (step S5), and constructs a virtual space image from the 3D point cloud data whose attribute information has been changed (step S6).

[0038] Next, the operation of the above configuration will be explained by dividing it into several operation examples. <Example 1> In this operation example, it is assumed that the real-world information acquisition device 2 is installed at an intersection. In this operation example, the virtual space management server 1 analyzes the artificial sounds at the intersection collected by the real-world information acquisition device 2.

[0039] (1) Acquisition of real-world information: First, the real world information acquisition device 2 is installed near an intersection in real space. The real world information acquisition device 2 senses its surroundings with a measurement unit 21 and acquires 3D point cloud data. In addition, an ambient sound collection unit 22 collects sound information around the real world information acquisition device 2. The acquired 3D point cloud data and sound information data are transmitted to the virtual space management server 1.

[0040] (2) Point cloud data storage: The virtual space management server 1 stores the 3D point cloud data received from the real world information acquisition device 2 in the storage 50, and the position determination unit 10b records the position information of the intersection that is the source of the 3D point cloud data.

[0041] (3) Analysis of sound information: The virtual space management server 1 analyzes the sound information data around the device received from the real world information acquisition device 2 together with the position information notified from the position determination unit 10b in the sound information analysis unit 20. The analysis rules used for the analysis are managed by the sound information analysis rule management unit 20c. The analysis rules in <Operation Example 1> are to determine whether the sound information is an artificial sound such as a car horn (rule (a)), and to determine the strength of the sound information (rule (b)).

[0042] In addition, in <Operation Example 1>, it is assumed that a loud horn was sounded at the intersection, and it was determined that (a: artificial sound) and (b: sound intensity: high).

[0043] (4) Rules for changing attributes of point cloud data: The point cloud data attribute change unit 40a of the virtual space management server 1 changes the attribute data of the 3D point cloud data based on the position information and the judgment type of the sound information. The content (rules) of the changes are defined in the point cloud data attribute change rule management unit 40b. For example, for "artificial sound," it is defined that "red" is "loud," "orange" is "medium," and "yellow" is "low."

[0044] (5) Changing the attributes of point cloud data: In <Operation Example 1>, the sound information data is determined to be (a: artificial sound) and (b: sound intensity: high). In response to this, the point cloud data attribute change unit 40a sends to the virtual space construction unit 30 [position information and attribute change content] that changes the color attribute information of the 3D point cloud data within a 3m range from the coordinates recorded in the position determination unit 10b to a value that becomes "red."

[0045] (6) Construction of virtual space image after attribute change: Based on the [location information and attribute change details] notified by the point cloud data attribute change unit 40a, the point cloud data visualization unit 30b of the virtual space construction unit 30 changes the attributes of the points at the corresponding coordinates in the 3D point cloud data read from the storage 50. Once the attribute changes for all relevant points are complete, the point cloud data visualization unit 30b creates virtual space video data using this new point cloud data. When this virtual space video data is visualized, the area around the intersection where the real world information acquisition device 2 is installed in the virtual space is displayed as a red point cloud.

[0046] (7) Display of virtual space image after attribute change: The virtual space image data after attribute change, in which the area around the intersection in the virtual space where the real-world information acquisition device 2 is installed is displayed as a red dot cloud, is transmitted to the user terminal 3. The user terminal 3 receives this data, visualizes it, and displays it on the monitor 31a. In this way, the user can visually recognize in the virtual space that a loud artificial sound has been played near the intersection.

[0047] <Example 2> In this operation example, it is assumed that the real world information acquisition device 2 is installed in a park. In this operation example 2, the virtual space management server 1 analyzes natural sounds in the park collected by the real world information acquisition device 2.

[0048] (1) Acquisition of real-world information: First, the real world information acquisition device 2 is installed in a park in real space. The real world information acquisition device 2 senses its surroundings with a measurement unit 21 and acquires 3D point cloud data. In addition, an ambient sound collection unit 22 collects sound information around the real world information acquisition device 2. The acquired 3D point cloud data and sound information data are transmitted to the virtual space management server 1.

[0049] (2) Point cloud data storage: The virtual space management server 1 stores the 3D point cloud data received from the real world information acquisition device 2 in the storage 50, and the position determination unit 10b records the position information of the park that is the source of the 3D point cloud data.

[0050] (3) Analysis of sound information: The virtual space management server 1 analyzes the sound information data around the device received from the real world information acquisition device 2 together with the position information notified from the position determination unit 10b in the sound information analysis unit 20. The analysis rules used for the analysis are managed by the sound information analysis rule management unit 20c. The analysis rules in <Operation Example 1> are to determine whether the sound information is a natural sound such as the sound of wind (rule (a)), and to determine its strength (rule (b)).

[0051] In addition, in <Operation Example 2>, it is assumed that the sound of wind, which feels like rustling, is collected in the park and is determined to be (a: natural sound) and (b: sound intensity: medium).

[0052] (4) Rules for changing attributes of point cloud data: The point cloud data attribute change unit 40a of the virtual space management server 1 changes the attribute data of the 3D point cloud data based on the position information of the sound information and the judgment type. The content (rules) of the changes are specified in the point cloud data attribute change rule management unit 40b. For example, for "natural sounds," it is specified that the position information of the point cloud data for "large" is "5m surroundings," "medium" is "3m surroundings," and "small" is "1m surroundings" is changed within 10%.

[0053] (5) Changing the attributes of point cloud data: In <Operation Example 2>, the sound information data is determined to be (a: natural sound) and (b: sound intensity: medium). In response to this, the point cloud data attribute change unit 40a sends to the virtual space construction unit 30 [position information and attribute change content] that changes the coordinates of the 3D point cloud data within a 3m range, for example, to values ​​within 10% of the coordinates recorded in the position determination unit 10b.

[0054] (6) Construction of virtual space image after attribute change: Based on the [position information and attribute change details] notified by the point cloud data attribute change unit 40a, the point cloud data visualization unit 30b of the virtual space construction unit 30 changes the attributes of the points at the corresponding coordinates in the 3D point cloud data read from the storage 50. Once the attribute changes for all relevant points are complete, the point cloud data visualization unit 30b creates virtual space video data using this new point cloud data. When this virtual space video data is visualized, the point cloud around the location in the park where the real-world information acquisition device 2 is installed in the virtual space is displayed with a "shaking" appearance.

[0055] (7) Display of virtual space image after attribute change: The virtual space image data after attribute change, which shows the surroundings of the location in the park in the virtual space where the real-world information acquisition device 2 is installed with a "shaking" feeling, is transmitted to the user terminal 3. The user terminal 3 receives this, visualizes it, and displays it on the monitor 31a. In this way, the user can visually recognize that a "medium-level natural sound" has been heard in the park.

[0056] <Example 3> In this operation example, it is assumed that the sound information analysis unit 20 of the virtual space management server 1 uses, for example, an external AI (Artificial Intelligence) system having a sound information analysis function. In Operation Examples 1 and 2, sound information data is analyzed by the sound information analysis unit 20 of the virtual space management server 1. Instead of this, it is expected that the performance of the sound information analysis function can be improved by linking with an external AI system, etc. To achieve this, for example, an interface function with the external AI system is implemented in the sound information analysis unit 20, and the sound information data is sent to the external AI system, and an analysis result including the type of this sound information data is received.

[0057] <Example 4> In this operation example, it is assumed that the virtual space management server 1 further has a text change function and also has a function of converting sound information into text. In Operation Examples 1 and 2, the point cloud data attributes are changed by the point cloud data attribute change unit 40a of the virtual space management server 1. In addition to this, to further enrich the visual expression in the virtual space, it is also possible to use a voice-to-text function, for example, to change the attributes of the point cloud data. For example, it is advisable to add a voice-to-text function to the sound information analysis unit 20, send the converted text (text) to the virtual space construction unit 30, and construct a virtual space image with superimposed text information.

[0058] <Example 5> In this operation example, it is assumed that the virtual space management server 1 further has a function of adding video data and also has a function of converting sound information into video data. In operation examples 1 and 2, the attributes of the point cloud data are changed by the point cloud data attribute change unit 40a of the virtual space management server 1. In addition to this, in order to enrich the visual expression on the virtual space side, a CG (Computer Graphics) image corresponding to the type of sound information is prepared and stored in storage 50 or the like. Then, in parallel with changing the attributes of the 3D point cloud data, the CG image is read out and superimposed on the virtual space video.

[0059] In this case, for example, a sound information CG storage function for storing CG images corresponding to sound information can be added to the sound information analysis unit 20, and the CG images corresponding to the type of sound information data can be sent to the virtual space construction unit 30, thereby constructing a virtual space image with the CG images superimposed.

[0060] <Example 6> In this operation example, it is assumed that the real world information acquisition device 2 is portable and can move freely.

[0061] In Operation Example 1, it is assumed that the real world information acquisition device 2 is installed at an intersection. In Operation Example 2, it is assumed that the real world information acquisition device 2 is installed in a park. In both cases, the real world information acquisition device 2 is installed in a fixed location.

[0062] Recently, mobile terminals (smartphones, etc.) equipped with a simple LiDAR measurement function have become known. Therefore, in the future, it may be possible to make the real-world information acquisition device 2 portable. In such a case, it is advisable to add a position estimation function to the position determination unit 10b in the virtual space management server 1, estimate the current position of the real-world information acquisition device 2, and then record the position.

[0063] As described above, in the embodiment, a real world information acquisition device 2 equipped with a 3D point cloud data measurement function and a microphone for collecting ambient sounds is installed at an intersection, a park, or the like as a real space information acquisition device.

[0064] The virtual space management server 1 also analyzes the sound information data received from the real world information acquisition device 2 and determines its type. Depending on the type of sound information data, the virtual space management server 1 changes the attributes of the received 3D point cloud data and constructs a virtual space image using the 3D point cloud data. For example, if the type of sound information data is determined to be an artificial sound (such as a car horn), the color information of the point at the corresponding position in the 3D point cloud data is changed to red. If the type of sound information data is determined to be a natural sound (such as the sound of the wind), the coordinate value information of the point at the corresponding position is changed within a certain range.

[0065] The generated virtual space image data is transmitted to the user terminal 3. The virtual space image based on the 3D point cloud data whose attributes have been changed based on the sound information data is displayed on the user terminal 3. This allows the user to experience a virtual space based on the 3D point cloud data in which the sound information of the real space is visually reflected.

[0066] According to the embodiment, a user can recognize ambient sound information as visualized information in a virtual space created by a 3D point cloud. This makes it possible to recognize ambient sound information in the real space even if there is an abnormality in the audio playback device in the virtual space or if a person in the virtual space is hearing impaired. From these points, according to the embodiment, it is possible to provide a technology that can reflect sounds in a real space in a virtual space.

[0067] It should be noted that the present invention is not limited to the above-described embodiment. For example, in <Operation Example 1>, whether or not the sound information data is an artificial sound is determined by the sound information analysis unit 20. If the performance of sound information analysis is further improved, it is expected that the sound information analysis unit 20 will be able to determine even the type of the artificial sound (whether it is a horn sound, a collision sound, etc.). If this becomes possible, it is conceivable that the number of change patterns in the point cloud data attribute change unit 40a will also increase in accordance with, for example, an increase in the number of distinguishable types of artificial sounds.

[0068] Moreover, in <Operation Example 2>, whether or not the sound information data is a natural sound is determined by the sound information analysis unit 20. If the performance of sound information analysis is further improved, it is expected that the sound information analysis unit 20 will be able to determine even the type of natural sound (whether it is the sound of wind, the sound of rain, etc.). If this becomes possible, it is conceivable that the number of change patterns in the point cloud data attribute change unit 40a will also increase in accordance with, for example, an increase in the number of distinguishable types of natural sounds.

[0069] Furthermore, in the implementation stage, the components of the present invention can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0070] 1...Virtual space management server 2...Real-world information acquisition device 3...User terminal 4. Network 10...Point cloud data management section 10a...Point cloud data acquisition unit 10b...Position determination part 20...Sound information analysis section 20a...Sound information acquisition unit 20b…Analysis department 20c…Sound Information Analysis Rules Management Department 21...Measuring part 22...Ambient sound collection unit 23…Communications Department 30...Virtual Space Construction Department 30a...Point cloud data visualization management unit 30b...Point cloud data visualization unit 31...Video display section 31a...Monitor 32...Audio playback section 32a...Speaker 33...I / O section 33a...keyboard 34…Communications Department 40...Attribute change section 40a…3 point cloud data attribute change section 40b…3 point cloud data attribute change rule management section 50…Storage 51...3D point cloud data 52...Program 60…Memory 70...Processor 80...Communications Department.

Claims

1. a communication unit that communicates with a real-world information acquisition device that includes an environmental sensor and an acoustic sensor; a point cloud data management unit that stores 3D point cloud data of the real space acquired by the environmental sensor together with position information of the real world information acquisition device; a sound information analysis unit that analyzes sound information data of the surrounding sound acquired by the acoustic sensor and determines a type of the sound information data in association with the position information; an attribute change unit that changes attribute information of the 3D point cloud data stored in the point cloud data management unit based on the position information and type of the sound information data; a virtual space construction unit that constructs a virtual space image by changing attributes of points at coordinates corresponding to the position information of the 3D point cloud data based on the content of the changed attributes.

2. The point cloud data management unit a point cloud data acquisition unit that acquires the 3D point cloud data from the environmental sensor; The information processing apparatus according to claim 1 , further comprising: a point cloud data storage unit that stores the acquired 3D point cloud data.

3. The sound information analysis unit a sound information acquisition unit that acquires the sound information data from the acoustic sensor; a sound information analysis rule management unit that manages analysis rules for the sound information data; The information processing apparatus according to claim 1 , further comprising: an analysis unit that analyzes the acquired sound information data based on the analysis rule and calculates a type of the sound information data.

4. The sound information analysis unit An interface function with an AI system equipped with a sound information analysis function; The information processing device according to claim 1 , further comprising a function of transmitting the acquired sound information data to the AI ​​system and receiving an analysis result of the sound information data by the AI ​​system.

5. The attribute change unit a point cloud data attribute change rule management unit that manages a change rule for the attributes of the 3D point cloud data; The information processing apparatus according to claim 1 , further comprising: a point cloud data attribute changing unit that changes attribute information of the 3D point cloud data stored in the point cloud data management unit based on the change rule.

6. The virtual space construction unit a point cloud data visualization unit that visualizes the 3D point cloud data whose attribute information has been changed to generate image data of a virtual space; The information processing device according to claim 1 , further comprising: a point cloud data visualization management unit that manages the visualization of the 3D point cloud data based on an installation position of the real-world information acquisition device.

7. An information processing method executed by a processor of an information processing device including a processor and a storage, the processor communicating with a real world information acquisition device that includes an environmental sensor and an acoustic sensor; a step of storing, by the processor, 3D point cloud data of the real space acquired by the environmental sensor in the storage together with position information of the real-world information acquisition device; a step in which the processor analyzes sound information data of the ambient sound acquired by the acoustic sensor and determines a type of the sound information data in association with the position information; a step of changing attribute information of the 3D point cloud data stored in the storage by the processor based on position information and type of the sound information data; and a step in which the processor changes attributes of points at coordinates corresponding to the position information of the 3D point cloud data based on the content of the changed attributes to construct a virtual space image.

8. A program that causes a computer to function as each of the units of the information processing device according to claim 1 .

Images