Augmented reality device and sound visualization method

Abstract

This invention proposes an augmented reality device for sound visualization, comprising a display, a sound sensor, and a processor. The sound sensor is used to sense sound signals. The processor identifies sound information of the sound signals, including the type or direction of the sound source. The processor also converts the sound information into image information and displays the image information on the display.

Description

Technical Field

[0001] This disclosure relates to an augmented reality device that combines auditory and visual elements, as well as a method for visualizing sound. Prior Technology

[0002] Existing Augmented Reality (AR) technology has been widely applied in the entertainment industry, providing users with rich immersive audio-visual interactive experiences. This technology primarily focuses on enhancing visual information, such as virtual object manipulation, stereoscopic image display, and AR game applications. However, with the rapid advancement of AR technology, its applications are gradually extending to fields such as medicine, industry, and education, showing particular potential in assisting workflows and providing real-time information. Nevertheless, most AR devices currently on the market emphasize visual enhancement, and real-time processing technology for auditory information remains lacking. Summary of the Invention

[0003] This disclosure proposes an augmented reality device and method for sound visualization, which can convert sound information into image information, thereby enhancing the perception of the environment on the augmented reality device.

[0004] This disclosure presents an augmented reality device for sound visualization, comprising a display, a sound sensor, and a processor. The sound sensor is used to sense sound signals. The processor is electrically connected to the display and the sound sensor to acquire the sound signals and identify sound information, including the type or direction of the sound source. The processor also converts the sound information into image information and displays the image information on the display.

[0005] In one embodiment of this disclosure, the augmented reality device further includes an inertial measurement unit for sensing attitude information. A processor is electrically connected to the inertial measurement unit to adjust image information based on the attitude information to generate updated image information, and displays the updated image information on a display.

[0006] In one embodiment of this disclosure, the augmented reality device further includes an image sensor for acquiring environmental images. A processor is electrically connected to the image sensor for identifying multiple objects in the environmental images. The processor also selects one of the objects based on sound information and sets image information to indicate the selected object.

[0007] In one embodiment of this disclosure, the aforementioned audio information further includes distance, specifically a vehicle. The processor is used to set the image information to include the distance and direction of the vehicle relative to the augmented reality device.

[0008] In one embodiment of this disclosure, the aforementioned audio information includes mechanical operating status. The processor is used to set the image information to include mechanical operating status.

[0009] From another perspective, embodiments of the present invention provide a sound visualization method for augmented reality devices. This sound visualization method includes: acquiring a sound signal through a sound sensor of the augmented reality device; identifying sound information of the sound signal, including the type or direction of the sound source; converting the sound information into image information; and displaying the image information on a display of the augmented reality device.

[0010] In one embodiment of this disclosure, the above-described sound visualization method further includes: sensing attitude information through the inertial measurement unit of an augmented reality device; adjusting image information according to the attitude information to generate updated image information; and displaying the updated image information on a display.

[0011] In one embodiment of this disclosure, the above-described sound visualization method further includes: acquiring an environmental image through an image sensor of an augmented reality device; identifying multiple objects in the environmental image; and selecting one of the objects based on sound information, and setting image information to indicate the selected object.

[0012] In one embodiment of this disclosure, the aforementioned sound information further includes distance, specifically a vehicle. The sound visualization method further includes setting the image information to include the distance and direction of the vehicle relative to the augmented reality.

[0013] In one embodiment of this disclosure, the aforementioned sound information further includes mechanical operating status. The sound visualization method further includes: setting the image information to include mechanical operating status.

[0014] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings for detailed explanation. Simple Explanation of the Diagram

[0015] Figure 1 is a schematic diagram illustrating an augmented reality device according to one embodiment. Figure 2 is a system block diagram illustrating an augmented reality device 100 according to one embodiment. Figure 3 is a schematic diagram illustrating the adjustment of image information based on posture information according to one embodiment. Figure 4 is a schematic diagram illustrating the matching of environmental images and sound information according to one embodiment. Figure 5 is a schematic diagram illustrating a road environment according to one embodiment. Figure 6 is a schematic diagram illustrating the mechanical operating state according to one embodiment. Figure 7 is a flowchart illustrating a sound visualization method according to one embodiment. Implementation

[0016] Some embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Component symbols used in the following description are considered identical or similar when they appear in different drawings. These embodiments are only a part of the present invention and do not disclose all possible implementations of the invention. More precisely, these embodiments are merely examples of systems and methods within the scope of the present invention's patent application.

[0017] The terms "first," "second," etc., used in this article do not specifically refer to order or sequence; they are merely used to distinguish elements or operations described using the same technical terms.

[0018] Figure 1 is a schematic diagram illustrating an augmented reality device according to one embodiment. In this embodiment, the augmented reality device 100 is implemented as a device with the shape of glasses, but in other embodiments, the augmented reality device 100 may also be a mobile phone, tablet computer, head-mounted device, etc. The present invention does not limit the shape of the augmented reality device 100.

[0019] Figure 2 is a system block diagram illustrating an augmented reality device 100 according to one embodiment. The augmented reality device 100 includes a processor 210, an image sensor 220, a communication module 230, an inertial measurement unit 240, a memory 250, a sound sensor 260, and a display 270, wherein the processor 210 is electrically connected to the other components.

[0020] Processor 210 may be a central processing unit, microprocessor, microcontroller, application-specific integrated circuit (ASIC), programmable logic device (PLD), etc. Image sensor 220 may include a charge-coupled device (CCD) sensor, complementary metal-oxide-semiconductor (CMOS) sensor, or other suitable photosensitive element. In some embodiments, image sensor 220 may also include dual cameras, structured light sensing device, laser, or any element capable of sensing scene depth. Communication module 230 may include circuitry capable of providing communication functions such as cellular networking (or mobile network), near-field communication, infrared communication, Bluetooth, and Wi-Fi. Inertial measurement unit 240 may include an accelerometer, gyroscope, magnetometer, etc. Memory 250 may include random access memory, flash memory, etc. The sound sensor 260 may include one or more microphones, which can form a microphone array when multiple microphones are present. The display 270 may be a transmissive or non-transmissive display. A transmissive display can overlay digital images onto real-world images, allowing the user to simultaneously view virtual information and the real world. A non-transmissive display may include a liquid crystal display panel, an organic light-emitting diode panel, an electronic paper panel, etc., and this disclosure is not limited thereto. For simplicity, not all components of the augmented reality device 100 are shown here; for example, the augmented reality device 100 may also include a global positioning system (GPS), etc.

[0021] The sound sensor 260 is used to sense sound signals, which are then acquired by the processor 210. The processor 210 can execute a suitable algorithm to identify sound information in the sound signal, including the type, direction, distance, and content of the sound source. For example, the processor 210 can use triangulation to calculate the direction of the sound source, and combine this with the Doppler effect to calculate the relative velocity of the sound source, thereby calculating the location and distance of the sound source. The processor 210 can also execute a machine learning algorithm to identify the type of sound source, which can include people, vehicles, any device, or any animal, etc., and is not limited thereto. The processor 210 can also execute a natural language processing algorithm to convert the sound signal (including human speech) into text. In some embodiments, the sound signal contains multiple sound sources, and the processor 210 can also execute an audio source separation algorithm to distinguish different sound sources and identify information such as the direction and distance of these sound sources.

[0022] Next, the processor 210 converts the sound information into image information. This image information can contain any text, numbers, graphics, animations, etc., and the present invention does not limit the content or size of this image information. The processor 210 displays this image information on the display 270. In this way, the user can see information about the sound on the display 270. In some embodiments, the augmented reality device 100 can be provided to hearing-impaired users, combining sound recognition technology, multi-sensor data processing, and augmented reality display to provide hearing-impaired users with an assistive system that provides real-time sound and image visualization. In other embodiments, the augmented reality device 100 can also be provided to hearing-free users; several embodiments will be given below to illustrate its application in different scenarios.

[0023] Figure 3 is a schematic diagram illustrating the adjustment of image information based on posture information according to one embodiment. Referring to Figure 3, in this embodiment, the image information includes illustration 310, which is an arrow indicating the direction of the sound source (left-hand side). As mentioned above, this sound source can be a person, animal, vehicle, device, etc., and this invention is not limited thereto. However, after identifying the direction of the sound source, the user may turn their head, causing the relative direction between the sound source and the user to change, so illustration 310 must be changed accordingly. Specifically, the aforementioned inertial measurement unit 240 is used to acquire posture information, which includes the direction of the user's face. When the rotation of the user's head is detected (e.g., turning 90 degrees to the right), the processor 210 adjusts the image information according to the posture information to generate updated image information and displays the updated image information on the display 270. This updated image information includes illustration 320, which indicates that the sound source is located behind the user. In other words, the inertial measurement unit 240 captures the rotation angle and posture changes of the user's head, thereby correcting the sound source localization results. This ensures that no matter how the user moves their head, the image information can accurately provide the corresponding directional guidance, improving the synchronization of visual and auditory information.

[0024] Figure 4 is a schematic diagram illustrating the matching of environmental image and sound information according to one embodiment. Referring to Figure 4, in this embodiment, multiple people are talking around the user, and it is necessary to mark which person is currently speaking. Specifically, the image sensor 220 is used to acquire an environmental image, which contains multiple objects (e.g., people). The processor 210 is used to execute an object detection algorithm to detect or identify objects in the environmental image. The processor 210 selects one of the objects based on the sound information and then sets image information to indicate the selected object. The processor 210 may additionally display an icon to mark the selected object, such as displaying an arrow, or adding some eye-catching markings to the selected object, such as adding different colored shadows, outlines, etc., but this invention is not limited thereto. In the example of Figure 4, the user sees objects 401-403 through the transparent display 270, and these objects 401-403 are also included in the environmental image. In addition to identifying objects 401-403 in the environmental image, processor 210 also identifies sound information such as the direction of the sound source. Based on the sound information, processor 210 selects the object 401 that matches the direction of the sound source, and therefore displays illustration 410 on display 270 to indicate the selected object 401. In some embodiments, processor 210 may also display the identity information of the sound source on display 270, such as the speaker's name. In some embodiments, processor 210 may also convert the speaker's speech into text and then display this text on display 270. In some embodiments, processor 210 may also execute beamforming algorithms to remove environmental noise and enhance the speaker's voice, ensuring highly accurate sound localization results. Through this integration of visual and auditory information, the system can help users perceive important sound events in their surrounding environment more comprehensively.

[0025] The above methods can also be applied to other scenarios. For example, in education and rehabilitation training, users must practice identifying the source of a sound. The above methods can help users practice, thereby improving learning efficiency and promoting sound perception training during the rehabilitation process. In employee education and training, users must operate with simultaneous sound and visual information. The above methods can visualize sound information, improving learners' learning outcomes and operational skills, thereby promoting a more efficient training process.

[0026] Figure 5 is a schematic diagram illustrating a road environment according to one embodiment. Referring to Figure 5, in the scenario of Figure 5, a user is moving on a road (walking, cycling, or riding a motorcycle). The system identifies vehicles in the road environment and provides corresponding image information on display 270. Specifically, processor 210 can identify the type of sound source as a vehicle and calculate the distance between the sound source and augmented reality device 100. Processor 210 can set the image information to include the distance and / or direction of the vehicle relative to augmented reality device 100. For example, in the case of Figure 5, processor 210 identifies a vehicle (e.g., a car) approaching from the user's right rear, and therefore displays illustration 510 on display 270 to remind the user to pay attention to the right lane. In some embodiments, processor 210 can also display a distance, such as "10 m," to remind the user of the current distance to the vehicle.

[0027] In some embodiments, also in a road environment, the processor 210 can identify whether the sound signal contains warning sounds, such as the sound of a fire truck or ambulance, or the sound of a horn, and then display information such as the type, direction, or distance of the sound source on the display 270. This can remind the user to react in time, thereby avoiding potential dangers.

[0028] Figure 6 is a schematic diagram illustrating the display of mechanical operating status according to one embodiment. Referring to Figure 6, the scenario depicts an industrial operating environment containing various devices. The processor 210 can identify the mechanical operating status of a particular device from sound signals and write setting image information containing this mechanical operating status. For example, in Figure 6, the environment includes device 610. When device 610 malfunctions and emits a specific sound (which could be an alarm or abnormal noise caused by a component failure), the processor 210 can identify the mechanical operating status as "abnormal" and thus display illustration 620 to indicate that device 610 requires maintenance or inspection. This approach helps operators monitor the operating status of device 610 in real time, improving operational accuracy and enhancing safety in the work environment.

[0029] Figure 7 is a flowchart illustrating a sound visualization method according to an embodiment. Referring to Figure 7, in step 701, a sound signal is acquired through the sound sensor of the augmented reality device. In step 702, the sound information of this sound signal is identified, including the type or direction of the sound source. In step 703, the sound information is converted into image information and displayed on the display of the augmented reality device. The steps in Figure 7 have been described in detail above and will not be repeated here. It is worth noting that each step in Figure 7 can be implemented as multiple pieces of code or circuits, and the present invention is not limited thereto. Furthermore, the method in Figure 7 can be used in conjunction with the above embodiments or used alone; in other words, other steps can be added between the steps in Figure 7.

[0030] The aforementioned augmented reality device and method combine sound processing technology and augmented reality technology, possessing irreplaceable market value for hearing-impaired individuals or users requiring real-time sound visualization. Because this system can present sound information as real-time, accurate images and intelligently adjust to the user's actual environment, compared to traditional sound devices, this invention provides a more comprehensive solution. It not only improves the user's quality of life but also has a wide range of applications, applicable from daily life to industrial operations.

[0031] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0032] 100: Augmented Reality Device 210: Processor 220: Image sensor 230: Communication Module 240: Inertial Measurement Unit 250: Memory 260: Sound sensor 270: Monitor 310, 320, 410, 510, 620: Illustration 401~403: Object 610: Equipment 701~703: Steps

Claims

1. An augmented reality device for sound visualization, comprising: a display; an image sensor for acquiring an environmental image; a sound sensor for sensing a sound signal; and a processor electrically connected to the display and the sound sensor for acquiring the sound signal and identifying sound information of the sound signal, the sound information including a type or direction of a sound source, wherein the processor converts the sound information into image information and displays the image information on the display, wherein the processor identifies a plurality of objects in the environmental image, selects one of the objects according to the sound information, sets the image information to indicate the selected object, and displays the identity information of the selected object on the display, wherein the processor executes a beamforming algorithm to remove environmental noise and amplify the sound of the selected object.

2. The augmented reality apparatus as described in claim 1, further comprising: An inertial measurement unit is used to sense attitude information, wherein the processor is electrically connected to the inertial measurement unit to adjust the image information according to the attitude information to generate updated image information and display the updated image information on the display.

3. The augmented reality device as claimed in claim 1, wherein the audio information further includes a distance, the type being a vehicle, and the processor is configured to set the image information to include the distance and direction of the vehicle relative to the augmented reality device.

4. The augmented reality device as claimed in claim 1, wherein the audio information includes a mechanical operating state, and wherein the processor is configured to set the image information to include the mechanical operating state.

5. A sound visualization method for an augmented reality device, the sound visualization method comprising: acquiring an environmental image through an image sensor of the augmented reality device; acquiring a sound signal through a sound sensor of the augmented reality device; identifying sound information of the sound signal, wherein the sound information includes a type or direction of a sound source; converting the sound information into image information and displaying the image information on a display of the augmented reality device; identifying a plurality of objects in the environmental image, selecting one of the objects according to the sound information, setting the image information to indicate the selected object, and displaying the identity information of the selected object on the display; and performing a beamforming algorithm to remove environmental noise and enhance the sound of the selected object.

6. The sound visualization method as described in claim 5 further includes: An attitude information is sensed through an inertial measurement unit of the augmented reality device; And adjust the image information according to the posture information to generate updated image information, and display the updated image information on the display.

7. The sound visualization method as described in claim 5, wherein the sound information further includes a distance, the type being a mode of transportation, and the sound visualization method further includes: The image information is set to include the distance and direction of the vehicle relative to the augmented reality device.

8. The sound visualization method as described in claim 5, wherein the sound information includes a mechanical operating state, the sound visualization method further comprising: The image information is set to include the machine's operating status.

Images