Augmented reality device for audio recognition and control method thereof

Abstract

This invention relates to an augmented reality device capable of audio recognition and a control method thereof. The augmented reality device includes: a see-through display unit configured to allow the user's eyes to see through and output virtual objects; an audio input unit for receiving audio signals generated within a preset distance from the display unit; and a control unit for recognizing event information corresponding to the audio signals and controlling the operation of the see-through display unit to output image information of the virtual objects corresponding to the recognized event information.

Description

Technical Field

[0001] This invention relates to an augmented reality device. More specifically, this invention relates to an augmented reality device capable of audio recognition and a control method thereof. Background Technology

[0002] Augmented Reality (AR) is a technology that overlays three-dimensional virtual images onto real images or backgrounds and displays them as a single image. In other words, AR is defined as a technology that expands a user's senses and perception by outputting image information corresponding to virtual objects to a glass layer through which a real image is presented.

[0003] Recently, the proliferation of mobile terminals and smartphones equipped with cameras and various sensors such as the Global Positioning System (GPS) is spreading, and various converged services using high-speed mobile internet are being introduced, thus rapidly expanding augmented reality services using mobile devices.

[0004] On the other hand, artificial intelligence technologies such as deep learning are also being applied to process audio information. The development of audio recognition technology, one of the audio-related processing technologies, aims to detect which entity generated the audio input and under what circumstances.

[0005] With the development of artificial intelligence, the accuracy of audio recognition is constantly improving. As the computing power of devices increases, the time required for audio recognition is reduced, so users can perform audio recognition on the input audio information in real time.

[0006] Therefore, various studies are underway to expand the user's sensory area by integrating audio recognition technology with augmented reality devices. Summary of the Invention

[0007] Technical issues

[0008] The purpose of this invention is to provide an augmented reality device and its control method that can intuitively notify users of nearby events by using audio recognition technology.

[0009] Furthermore, the present invention aims to provide an augmented reality device and its control method capable of recognizing audio information generated in the vicinity, enabling users to recognize audio generated outside the visible area.

[0010] Furthermore, the purpose of this invention is to provide an augmented reality device and its control method for real-time detection of labels and source points of nearby sounds, and outputting virtual objects accordingly.

[0011] Furthermore, the purpose of this invention is to provide an augmented reality device and its control method that expands the user's cognitive area by linking audio and image information to provide virtual objects.

[0012] Furthermore, the present invention aims to provide an augmented reality device that operates based on the properties of audio information generated in the vicinity.

[0013] Solution to the problem

[0014] The augmented reality device according to the present invention is characterized by comprising: a see-through display unit configured to allow the user's eyes to see through and output image information of a virtual object; an audio input unit for receiving an input of an audio signal generated within a preset distance from the display unit; and a control unit for recognizing event information corresponding to the audio signal and controlling the operation of the see-through display unit to output image information of a virtual object corresponding to the recognized event information.

[0015] Furthermore, in another embodiment of the augmented reality device according to the present invention, the image information includes at least one of text, image, and location information related to the audio signal, the control unit detects the location where the audio signal is generated, and controls the operation of the perspective display unit, such that the image information includes information related to the detected location.

[0016] The effects of the invention

[0017] According to the present invention, the user can accurately identify the sounds generated in the surrounding area by converting and outputting the sounds generated in areas not visible to the user.

[0018] Furthermore, according to the present invention, a hearing-impaired user can also have the same effect as seeing information related to various sounds generated nearby through his / her eyes.

[0019] Furthermore, according to the present invention, by combining nearby audio signals with virtual objects generated by augmented reality, the effect of maximizing the user's sensory area can be achieved. Attached Figure Description

[0020] Figure 1 A conceptual diagram illustrating a system that includes augmented reality devices.

[0021] Figure 2 Perspective view for augmented reality devices.

[0022] Figure 3 A block diagram illustrating the components of an augmented reality device.

[0023] Figure 4A flowchart illustrating the control method of the augmented reality device according to the present invention.

[0024] Figures 5 to 8 A diagram illustrating an embodiment of the augmented reality device according to the present invention. Detailed Implementation

[0025] The best way to implement an invention

[0026] The augmented reality device is characterized by including: a perspective display unit configured to allow the user's eyes to see through and output virtual objects; an audio input unit for receiving an input audio signal generated within a preset distance from the display unit; and a control unit for recognizing event information corresponding to the audio signal and controlling the operation of the perspective display unit to output image information of the virtual object corresponding to the recognized event information.

[0027] Methods of implementing the invention

[0028] In the following description, the embodiments disclosed herein will be described in detail with reference to the accompanying drawings. However, the technical terms used in this specification are only for describing specific embodiments and are not intended to limit the technical ideas disclosed herein.

[0029] As described above, users of augmented reality devices desire information about the functionality of interactive elements on the device. Therefore, the embodiments described herein relate to providing this information via a perspective display device. For example, as detailed below, the perspective display device is configured to provide information about the functionality of bidirectional elements of the augmented reality device as an augmented reality image displayed in the user's field of view.

[0030] Figure 1 A conceptual diagram illustrating a system including augmented reality devices. (Refer to...) Figure 1In one embodiment of the present invention, an input unit 100 receives an audio signal input, a control unit 200 identifies event information corresponding to the received audio signal, and can control a perspective display unit 300 to output image information of a virtual object corresponding to the identified event information. Specifically, an audio signal input can be received, and the relevant audio signal can be detected, identified, and matched with image information via an artificial neural network, and then visually transmitted to the user via the display unit 300. As will be described later, the display unit 300 can be provided in the form of smart glasses. Smart glasses, based on existing technology, can include: a frame portion supporting the shape of the glasses; a fixing portion fixing the smart glasses to the user; and a lens portion mounted on the frame, allowing the user to see and confirm augmented reality image information through the visible area. Smart glasses can also include a camera and various sensors, and can also include a microphone capable of detecting audio signals. According to one embodiment, an augmented reality device can be configured as an augmented reality display that allows audio signals and image information to interact in a user-intuitive manner. The augmented reality display is the aforementioned display unit 300, and can be either AR glasses or a head-mounted display. The lens section can be formed of a transparent or see-through material such as glass or plastic, allowing the user to clearly observe the actual scene when wearing the display unit 300. Furthermore, digital AR image information and other content can be projected through the lens section. Therefore, the user can simultaneously observe a real-world scene and projected digital image information while wearing the display unit 300.

[0031] Figure 2 An exemplary embodiment of an augmented reality device, showing a physical space including the augmented reality device, is illustrated as a perspective view. A user can see the visible area VA through the perspective display unit 300 in the augmented reality device. Figure 2 This embodiment is confirmed. The described perspective display unit 300 can be provided in the form of goggles, glasses, etc., allowing the user's hands to be free. The perspective display unit 300 includes: an AR output unit 310 configured to enhance visual information for the user to realize the appearance of physical space; and an information display unit 320 for guiding the direction and position of audio signals. By allowing light emitted from physical space to pass through the perspective display unit 300 via the AR output unit 310, the user can directly see the real physical space while simultaneously seeing one or more virtual objects overlapping with the real physical space. According to one embodiment, when the user's gaze is directly directed at the visible area VA through the AR output unit 310, the perspective display unit 300 displays one or more virtual objects in the form of information about one or more functions of the augmented reality device. The displayed information is obtained in an appropriate manner. For example, the displayed information can be received from a computing device via a network, from a remote computing device, or from the augmented reality device.

[0032] The perspective display unit 300 according to the present invention can be used as follows Figure 2 The goggles shown are in the form of [example goggles], but their implementation method is not limited and can be implemented in any other suitable form. (See reference [reference]). Figure 2 In a simplified description of one embodiment, a user can visually confirm the visible area VA in real time through the perspective display unit 300. When an audio signal is generated within the visible area VA, by detecting the location where the audio signal is generated, position information can be displayed in the direction of the relevant location in the information display unit 320, and the image information 10 of the virtual object is projected onto the location where the audio signal is generated in an overlapping manner. In detail, by visually projecting the information of the audio signal, that is, the type and sound of the audio signal, even a user who cannot confirm it audibly can visually confirm the audio signal.

[0033] In the following text, reference will be made to Figure 3 The augmented reality device of the present invention will be described in detail. Figure 3 A block diagram illustrating the components of an augmented reality device.

[0034] Reference Figure 3 According to one embodiment, the augmented reality device of the present invention may include: a perspective display unit 300 configured to allow the user's eyes to see through it and output image information 10 of a virtual object; an audio input unit 100 for receiving an input of an audio signal generated within a preset distance from the display unit 300; and a control unit 200 for recognizing event information corresponding to the audio signal and controlling the operation of the perspective display unit 300 to output image information 10 of a virtual object corresponding to the recognized event information.

[0035] In detail, the input unit 100 is configured to receive audio signals and image signals and transmit them to the control unit 200, and may include a microphone 110, a camera 120, and a transmitting unit 130. The microphone 110 is configured to receive audio signals generated within a predetermined distance from the display unit 300. When an audio signal is generated within the predetermined distance while the user is wearing the display unit 300, the relevant audio signal can be input through the microphone 110 and transmitted to the control unit 200 through the transmitting unit 130. At this time, the input unit 100 is mounted on a portion of the perspective display unit 300 and also includes a camera 120 for capturing images from one direction, thereby capturing an image of the front of the display unit 300 worn by the user. Specifically, the user's visible area VA as seen through the display unit 300 can be captured by the camera 120, and an image signal can be simultaneously transmitted to the control unit 200 through the transmitting unit 130.

[0036] The control unit 200 includes a detection unit 210, an identification unit 220, and a matching unit 230 that detect, identify, and match provided event information based on audio and image signals input and transmitted from the input unit 100. The detection unit 210 controls the operation of the perspective display unit 300 to include information related to the image information 10 by detecting the type of the audio signal transmitted from the input unit 100 and the location where the related audio signal was generated. Specifically, audio and image signals can be transmitted from the input unit 100 to the control unit 200, wherein the detection unit 210 can detect the type or condition of the audio signal, and detect the image signal corresponding to the audio signal from the image signal. According to one embodiment, as... Figure 2 As shown, when a baby cry is input as an audio signal via the microphone 110 of the input unit 100, and the crying baby is input as an image signal via the camera 120 and sent to the control unit 200, the detection unit 210 detects the distance to the location where the transmitted audio signal is generated, the magnitude of the audio signal, etc., via two or more audio sensors, thereby detecting the location where the relevant audio signal is generated. In other words, when a baby cry is input as an audio signal, the direction and location of the related cry can be confirmed and detected by the detection unit 210. Simultaneously, by detecting the image signal that generates the direction and location of the cry based on the input image signal, a more detailed analysis can be performed. The direction, location, and image signal of the audio signal detected by the detection unit 210 can be specifically identified by the recognition unit 220. Specifically, the recognition unit 220 identifies the type or condition of the audio signal using a pre-installed artificial neural network and identifies and extracts image information 10 corresponding to the type or condition of the related audio signal from the image signal. Specifically, according to... Figure 2By analyzing the wavelength, waveform, and frequency of audio signals, an artificial neural network can identify the relevant audio signals as a baby's cry. Furthermore, while identifying the audio signal, an image signal corresponding to the identified audio information can also be identified. That is, when an audio signal is identified as a baby's cry, the baby can be located and identified from the input image signal. Subsequently, when the audio signal and image signal are identified separately, these two pieces of information are matched, and the operation of the perspective display unit is controlled so that the image signal at the location where the audio signal is generated on the screen of the perspective display unit 300 overlaps with the image information 10 of the virtual object. In other words, when a baby's cry is input as an audio signal and a baby is also found in the image signal, the baby and the baby's cry can be matched, and the baby's cry information can be overlapped around the baby using augmented reality. The overlapping image information 10 can include at least one of text, image, and location information related to the identified audio signal. The text can be represented by letters, and the subject generating the audio signal, such as a person, object, or animal, can be displayed as an image. Furthermore, the location information of the location where the audio signal is generated can be provided as text, an image, or a direction / location representation. Image information 10 about the audio signal identified and matched by the control unit 200 can be output through the display unit 300. Specifically, the display unit 300 includes an AR output unit 310 and an information display unit 320. The AR output unit 310 is configured to overlay the image information 10 onto the visible area of ​​the user wearing the display unit 300 using augmented reality, and the information display unit 320 can be configured to represent the location or direction information of the place where the audio signal is generated along the circumferential direction of the display unit 300.

[0037] In the following text, reference will be made to Figure 4 A detailed description of the control method for the augmented reality device according to the present invention is provided. Figure 4 A flowchart illustrating the control method of the augmented reality device according to the present invention.

[0038] Reference Figure 4To explain, after the user wears the see-through display unit 300, the user can view the visible area VA through the AR output unit 310. At this time, when an audio signal is generated within a predetermined distance from the display unit 300, the relevant audio signal input can be received through the input unit 100. When an audio signal is input, event information corresponding to the relevant audio signal is identified, and image information 10 of a virtual object corresponding to the identified event information is output in the visible area VA through the AR output unit 310 and the information display unit 320. However, after identifying the event information, it is first determined whether the location where the audio signal is generated is within the visible area. If it is within the visible area, the image information 10 can be projected; if it is outside the visible area, location information can be output, allowing the user to move the location where the relevant audio signal is generated into the visible area. In detail, based on the direction in which a portion of the see-through display unit 300 worn by the user is pointing, the control unit 200 detects the direction of the input audio signal and can control the operation of the see-through display unit 300 so that the image information 10 includes information related to the detected direction. The image information 10 includes direction-related information, which can be displayed by the information display unit 320 to show the direction and location of the audio signal detection. The image information 10 can be overlaid using augmented reality at the audio signal detection location. Specifically, when the location where the audio signal is generated is outside the visible area of ​​the user through the perspective display unit 300, the control unit 200 can control the operation of the perspective display unit 300 to output the location-related image information 10. Furthermore, when the location where the audio signal is generated is within the visible area, the operation of the perspective display unit 300 can be controlled so that the image information 10 of a virtual object is overlaid on a portion of the location where the audio signal is generated, projected onto the screen of the perspective display unit.

[0039] In the following text, reference will be made to Figures 5 to 8 More detailed explanation will be provided through examples. Figures 5 to 8 A diagram illustrating an embodiment of the augmented reality device according to the present invention.

[0040] First, refer to Figure 5 When the audio signal is generated outside the visible area VA that the user can see through the perspective display unit 300, that is, in Figure 5 When a knocking sound is generated on the user-invisible left side door via AR output unit 310, image information 10 related to the location of the generated audio signal can be output via AR output unit 310. For details, refer to... Figure 5When no knocking sound is generated within the user's visible area VA, the location or direction of the audio signal generation is indicated, so that the user's visible area VA is switched to the location of the audio signal generation. That is, when a sound is generated from the left side door, it can be represented as a knocking sound, and location-related image information 10 is output using directional indicators and arrows, etc. More specifically, the object generating the audio signal can also be marked for output. At the same time, by displaying relevant directional information on multiple information display units 320 arranged along the periphery of the display unit 300, the user can be immediately notified that an audio signal has been generated in the relevant direction. At this time, the information display unit 320 can guide the user with directional information through LED notifications or vibration notifications, etc.

[0041] Figure 6 It is the user in Figure 5 After confirming the image information 10 related to direction or position in the visible area VA, the location of the audio signal generation point is brought into the screen within the visible area VA by turning the user's head to the left. For example... Figure 6 As shown, in Figure 5 When the location where the audio signal is generated is not included in the visible area VA, the system guides the user to the location and direction, allowing the user to switch the visible area VA to the location where the audio signal is generated. Therefore, as... Figure 6 As shown, when the location where the audio signal is generated is within the visible area, the operation of the perspective display unit 300 can be controlled so that the image information 10 of the virtual object is superimposed on a portion of the AR output unit 310 at the location where the audio signal is generated, projected onto the screen of the perspective display unit 300. That is, when a door makes a knocking sound, and the user moves the visible area VA towards the door, and the door is within the visible area VA, the type and sound of the audio signal can be marked by superimposing it with images or text to guide the user to knock on the door where the audio signal is generated. At this time, the superimposed image information 10 can be displayed differently depending on the type and object of the audio signal. Specifically, icons can be used to represent objects such as things, people, and animals, or the user can intuitively identify the sound object through differences in the shape and color of speech bubbles. In detail, this will be achieved through comparison... Figures 6 to 8 Let me explain.

[0042] Figure 6 An audio signal representing a knocking sound coming from the door, such as Figure 6 As shown, the type and sound of a sound can be labeled with text within a rounded rectangle representing the indicator line in the object from which the audio signal is to be generated. Furthermore, Figure 7The audio signal involving human dialogue can be superimposed and represent image information 10 about the audio signal when the people engaging in the dialogue enter the visible area VA. At this time, the dialogue can be superimposed on the relevant person through dialogue bubbles. Therefore, the user can communicate with people without listening. Figure 8 In an embodiment involving the input of animal sounds as audio signals, when a dog bark is input as an audio signal, curved indicator lines are matched around the relevant dog. These indicator lines are provided as image information 10, where an icon representing the dog is represented in the image information 10, thereby allowing for visual identification of the audio signal generated by the dog.

[0043] According to the embodiments described above, in this invention, after receiving an audio signal input, the audio signal of a relevant type is matched with the image information 10 by matching it with the image signal of the display unit 300, thereby providing audio signal information to the user's vision. At this time, the process of recognizing the audio signal, extracting the image signal corresponding to the type of the relevant audio signal from the image signal, and matching the audio signal and the image signal to represent it in the image information 10 can be implemented by an artificial neural network pre-installed in the control unit 200. As the user repeatedly uses the augmented reality device of this invention, the artificial neural network can be trained to provide more accurate and expanded information.

[0044] In this specification, the perspective display unit 300 has been described as an embodiment of smart glasses, but this is only one embodiment and can be implemented by image processing devices and audio analysis devices such as mobile phones, laptops and wearable devices.

[0045] The above invention relates to one embodiment, which is merely one example. Those skilled in the art can make various modifications and equivalent embodiments based on this. Therefore, the scope of the invention is not limited to the above embodiment and drawings.

[0046] Industrial availability

[0047] According to the present invention, the user can accurately identify the sounds generated in the surrounding area by converting and outputting the sounds generated in areas not visible to the user.

[0048] Furthermore, according to the present invention, a hearing-impaired user can also have the same effect as seeing information related to various sounds generated nearby through his / her eyes.

[0049] Furthermore, according to the present invention, by combining nearby audio signals with virtual objects generated by augmented reality, the effect of maximizing the user's sensory area can be achieved.

Claims

1. An augmented reality device, characterized in that, include: The perspective display unit is configured to allow the user's eyes to see through it and output virtual object image information. An audio input unit receives an input audio signal generated within a preset distance from the aforementioned display unit; A camera, positioned as part of the aforementioned perspective display unit, is used for one-way photography; and The control unit uses a pre-installed artificial neural network to identify event information corresponding to the aforementioned audio signal, and controls the operation of the aforementioned perspective display unit to output image information of virtual objects corresponding to the identified event information. The aforementioned image information includes at least one of the following: text, image, audio signal type, and location information related to the aforementioned audio signal. The aforementioned control unit detects the location where the audio signal is generated and controls the operation of the aforementioned perspective display unit so that the image information includes information related to the detected location. The aforementioned control unit utilizes the artificial neural network to identify and extract at least one object corresponding to the identified event information from the image signal captured by the aforementioned camera. When at least one object corresponding to the identified event information is identified from the image signal, the identified at least one object is matched with the event information of the aforementioned audio signal, such that the image signal at the location where the aforementioned audio signal is generated on the screen of the aforementioned perspective display unit overlaps with the image information of the virtual object. When the location where the aforementioned audio signal is generated is outside the visible area that the user can see through the aforementioned perspective display unit, the aforementioned control unit controls the operation of the aforementioned perspective display unit to output image information about the aforementioned location.

2. The augmented reality device according to claim 1, characterized in that, The control unit detects the input direction of the audio signal based on the direction in which a portion of the perspective display unit points, and controls the operation of the perspective display unit so that the image information includes information related to the detected direction.

3. The augmented reality device according to claim 1, characterized in that, When the location where the audio signal is generated is within the aforementioned visible area, the control unit controls the operation of the perspective display unit so that a portion of the location projected on the screen of the perspective display unit overlaps with the image information of the virtual object.

Images