Augmented reality device with audio recognition and method for controlling same

Abstract

The present invention relates to an augmented reality device capable of audio identification and a control method thereof, comprising: a see-through display unit configured to allow a user's eyes to see through and output a virtual object; an audio input unit receiving an input of an audio signal generated within a predetermined distance from the display unit; and a control unit identifying event information corresponding to the audio signal and controlling the operation of the see-through display unit to output video information of a virtual object corresponding to the identified event information.

Description

[Technical field]

[0001] The present invention relates to an augmented reality device, and more particularly to an augmented reality device capable of performing audio discrimination and a method for controlling the same. [Background technology]

[0002] Augmented Reality (AR) is a technology that displays a 3D virtual image overlaid on a real image or background in a single image. In other words, AR is defined as the extension of a user's senses and perception by outputting image information corresponding to virtual objects onto a transparent glass while the user is presented with a real image through the glass.

[0003] Recently, mobile terminals and smartphones equipped with various sensors such as cameras and GPS (Global Positioning System) have become more widespread, and various converged services using high-speed mobile Internet have been unveiled, and augmented reality services using mobile devices are rapidly spreading.

[0004] Meanwhile, artificial intelligence techniques such as deep learning are being applied to process audio information. Audio identification technology, which is one of the audio-related processing techniques, is developed for the purpose of detecting from which subject an audio input is generated and in which situation of the subject it is generated.

[0005] With the development of artificial intelligence, the accuracy of audio identification is increasing, and as the computing power of devices improves, the time required for audio identification is decreasing, allowing users to perform audio identification on input audio information in real time.

[0006] Accordingly, various researches are being conducted to expand the user's sensory range by combining audio recognition technology with augmented reality devices. Summary of the Invention [Problem to be solved by the invention]

[0007] The technical problem of the present invention is to provide an augmented reality device and a control method thereof that can intuitively inform a user of events occurring in the vicinity using audio recognition technology.

[0008] Another technical object of the present invention is to provide an augmented reality device and a control method thereof that can identify audio information generated in the vicinity so that a user can recognize audio generated outside the visible area.

[0009] Another technical objective of the present invention is to provide an augmented reality device and a control method thereof that detects the label and source point of a sound generated in the surroundings in real time and outputs a virtual object based on the detected label and source point.

[0010] Another technical object of the present invention is to provide an augmented reality device and a control method thereof that provides virtual objects so as to expand a user's cognitive range by linking audio information and video information.

[0011] Another technical problem of the present invention is to provide an augmented reality device that operates based on attributes of audio information generated in the surroundings. [Means for solving the problem]

[0012] The augmented reality device according to the present invention is characterized by including a see-through display unit that is formed so that a user's eyes can see through it and outputs virtual objects, an audio input unit that receives an input of an audio signal generated within a preset distance from the display unit, and a control unit that identifies event information corresponding to the audio signal and controls the operation of the see-through display unit so that image information of a virtual object corresponding to the identified event information is output.

[0013] In another embodiment of the augmented reality device of the present invention, the video information includes at least one of text, image, and location information related to the audio signal, and the control unit detects the location of the point where the audio signal was generated and controls the operation of the see-through display unit so that the video information includes information related to the detected location. Effect of the Invention

[0014] According to the present invention, a sound generated in an area that a user cannot see is converted into visual information and outputted, thereby enabling a user to accurately recognize sounds generated in the surroundings.

[0015] In addition, according to the present invention, even a user who has lost his / her hearing ability can experience the effect of visually seeing information related to various sounds generated in the surroundings.

[0016] In addition, according to the present invention, by combining audio signals generated in the surroundings with virtual objects generated by augmented reality, it is possible to obtain an effect of maximizing the user's sensory range. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a conceptual diagram illustrating a system including an augmented reality device. [Diagram 2] FIG. 1 is a perspective view of an augmented reality device. [Diagram 3] FIG. 2 is a block diagram illustrating components of an augmented reality device. [Figure 4] 4 is a flowchart showing a method for controlling an augmented reality device according to the present invention. [Diagram 5] FIG. 1 illustrates an embodiment of an augmented reality device according to the present invention. [Figure 6] FIG. 1 illustrates an embodiment of an augmented reality device according to the present invention. [Figure 7] FIG. 1 illustrates an embodiment of an augmented reality device according to the present invention. [Figure 8]FIG. 1 illustrates an embodiment of an augmented reality device according to the present invention. Best Mode for Carrying Out the Invention

[0018] An augmented reality device comprising: a see-through display unit configured to be seen through by a user's eyes and to output video information of a virtual object; an audio input unit configured to receive an input of an audio signal generated within a preset distance from the display unit; and a control unit configured to identify event information corresponding to the audio signal and control the operation of the see-through display unit so that video information of a virtual object corresponding to the identified event information is output. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the attached drawings. However, it should be noted that the technical terms used in this specification are merely used to describe specific embodiments and are not intended to limit the ideas of the technology disclosed in this specification.

[0020] As previously mentioned, users of augmented reality devices seek to obtain information regarding the functionality of interactive elements on the augmented reality device. Accordingly, embodiments described herein relate to providing such information via a see-through display device. For example, as specifically described below, the see-through display device is configured to provide information about the functionality of interactive elements of the augmented reality device as an augmented reality image displayed on the field of view of the user of the augmented reality device.

[0021] FIG. 1 is a conceptual diagram showing a system including an augmented reality device. Referring to FIG. 1, an embodiment of the present invention can be controlled such that an audio signal is input through an input unit 100, event information corresponding to the input audio signal is identified by a control unit 200, and image information of a virtual object corresponding to the identified event information is output through a see-through display unit 300. Specifically, an audio signal is input, and the audio signal is detected, identified, and matched with image information by an artificial neural network, and visually transmitted to a user through a display unit 300. Here, the display unit 300 can be provided in the form of smart glass as described below. The smart glass is based on conventional technology and may include a frame unit that supports the glass form, a fixing unit that fixes the smart glass to a user, and a lens unit that is attached on the frame and allows a user to view a visible area through the frame and check augmented reality image information. This may further include a camera and various sensors, and may further include a microphone that can detect an audio signal. According to an embodiment, the augmented reality device may be configured as an augmented reality display that allows a user to interact with an audio signal in an intuitive manner with image information. Here, the augmented reality display may be one of the above-mentioned display unit 300 and AR glasses or a head-mounted display. Here, the lens unit may be made of a transparent or see-through material such as glass or plastic so that a real scene can be clearly observed by a user when wearing the display unit 300. In addition, it may be configured to project content such as digital AR image information through the lens unit. Thus, a user can simultaneously observe a real-world scene and the projected digital image information while wearing the display unit 300.

[0022] FIG. 2 is a perspective view of an augmented reality device, showing an exemplary embodiment of a physical space including the augmented reality device. A user can see a visible area VA through a see-through display unit 300 in the augmented reality device, and such an embodiment can be seen in FIG. 2. The illustrated see-through display unit 300 may be provided in the form of goggles, glasses, etc., so that the user's hands are free. The see-through display unit 300 includes an AR output unit 310 configured to allow the user to visually augment the appearance of the physical space, and an information display unit 320 that guides the direction, position, etc. of an audio signal. The light emitted from the physical space through the AR output unit 310 passes through the see-through display unit 300, so that the user can see one or more virtual objects overlaid on the actual physical space while simultaneously looking directly at the actual physical space. According to an embodiment, when the user's gaze is directed straight at the visible area VA through the AR output unit 310, the see-through display unit 300 displays one or more virtual objects in the form of information about one or more functions of the augmented reality device. The information displayed here is obtained in an appropriate manner. For example, the displayed information may be received from the computing device, over a network, from a remote computing device, from an augmented reality device, or the like.

[0023] The see-through display unit 300 according to the present invention may be embodied in a goggle shape as shown in FIG. 2, but the embodiment is not limited thereto and may be embodied in any other suitable shape. Briefly describing an embodiment with reference to FIG. 2, a user can visually check the visible area VA in real time through the see-through display unit 300, and when an audio signal is generated within the visible area VA, the position of the point where the audio signal is generated can be detected and position information can be displayed in the direction of the position on the information display unit 320, and the image information 10 of the virtual object can be projected so as to overlap the position where the audio signal is generated. Specifically, information on the audio signal, i.e., the type and sound of the audio signal, can be visually projected to allow a user who cannot audibly check the audio signal to visually check the audio signal.

[0024] The augmented reality device of the present invention will now be described in detail with reference to Fig. 3. Fig. 3 is a block diagram showing components of the augmented reality device.

[0025] According to one embodiment of FIG. 3, the augmented reality device of the present invention may include a see-through display unit 300 that is formed so that a user's eyes can see through it and outputs video information 10 of a virtual object, an audio input unit 100 that inputs an audio signal generated within a preset distance from the display unit 300, and a control unit 200 that identifies event information corresponding to the audio signal and controls the operation of the see-through display unit 300 so that video information 10 of a virtual object corresponding to the identified event information is output.

[0026] Specifically, the input unit 100 is configured to receive an audio signal and a video signal and transmit them to the control unit 200, and may include a microphone 110, a camera 120, and a transmission unit 130. The microphone 110 is configured to receive an audio signal generated within a predetermined distance from the display unit 300. When an audio signal is generated within a predetermined distance while the user is wearing the display unit 300, the audio signal may be input through the microphone 110 and transmitted to the control unit 200 through the transmission unit 130. In this case, the input unit 100 further includes a camera 120 that is provided at a part of the see-through display unit 300 and captures an image in one direction, and the screen in the front direction of the display unit 300 worn by the user may be captured by the camera 120. Specifically, a visible area VA of the user that can be seen through the display unit 300 may be captured by the camera 120, and a video signal may be transmitted to the control unit 200 by the transmission unit 130 together with the image.

[0027] The control unit 200 includes a detection unit 210 that detects information based on an audio signal and a video signal input and transmitted from the input unit 100, and identifies and matches the provided event information, and an identification unit 220 and a matching unit 230. The detection unit 210 can detect the type of an audio signal transmitted from the input unit 100 and the location of a point where the audio signal is generated, and control the operation of the see-through display unit 300 to include information related to the video information 10. Specifically, an audio signal and a video signal can be transmitted from the input unit 100 to the control unit 200, and the detection unit 210 can detect the type or state of the audio signal and detect a video signal corresponding to the audio signal from the video signal. According to an embodiment, as shown in FIG. 2, when a baby cry is input to an audio signal through the microphone 110 of the input unit 100 and a crying baby is input to a video signal through the camera 120 and transmitted to the control unit 200, the detection unit 210 detects the distance to the point where the transmitted audio signal is generated, the volume of the audio signal, etc., using a plurality of audio sensors, and can detect the position of the point where the audio signal is generated. That is, when the baby cry is input to the audio signal, the direction and position where the crying sound is generated can be grasped and detected through the detection unit 210. At the same time, the video signal of the direction and position where the crying sound is generated can be detected based on the input video signal and analyzed more specifically. The direction and position of the audio signal and the video signal detected through the detection unit 210 can be specifically identified through the identification unit 220. Specifically, the identification unit 220 can identify the type or situation of the audio signal using a pre-installed artificial neural network, and identify and extract the video information 10 corresponding to the type or situation of the audio signal from the video signal. Specifically, as shown in Fig. 2, the wavelength, waveform, frequency, etc. of the audio signal are analyzed, and the audio signal is identified as a baby's cry by an artificial neural network. In addition, while identifying the audio signal, the video signal corresponding to the identified audio information can be identified. That is, once the audio signal is identified as a baby's cry, the baby can be found and identified from the input video signal.Then, when the audio signal and the video signal are respectively identified, the two pieces of information are matched, and the operation of the see-through display unit 300 can be controlled so that the video information 10 of the virtual object is overlapped on the video signal at the position where the audio signal is generated on the screen of the see-through display unit 300. That is, when a baby's cry is input as an audio signal and a baby is also found in the video signal, the baby and the baby's cry can be matched and the baby's crying information can be overlapped around the baby through augmented reality. Here, the overlapped video information 10 can include at least one of text, image, and location information related to the identified audio signal. The text can be written in characters, and the subject from which the audio signal is generated, such as a person, thing, or animal, can be displayed as an image. In addition, the location information of the location where the audio signal is generated can be provided as text, image, direction, location display, etc. The video information 10 regarding the audio signal identified and matched by the control unit 200 can be output through the display unit 300. Specifically, the display unit 300 includes an AR output unit 310 and an information display unit 320. The AR output unit 310 is configured to overlap and output the image information 10 through augmented reality in a visible area of ​​the user wearing the display unit 300, and the information display unit 320 may be configured to display position or direction information of a point where an audio signal is generated along a circumference direction of the display unit 300.

[0028] Hereinafter, the control method for an augmented reality device according to the present invention will be described in detail with reference to Fig. 4. Fig. 4 is a flowchart showing the control method for an augmented reality device according to the present invention.

[0029] 4, the user can check the visible area VA through the AR output unit 310 after wearing the see-through display unit 300. In this case, if an audio signal is generated within a predetermined distance from the display unit 300, the input unit 100 can receive the audio signal. When the audio signal is input, event information corresponding to the audio signal is identified, and image information 10 of a virtual object corresponding to the identified event information can be output to the visible area VA through the AR output unit 310 and the information display unit 320. However, after identifying the event information, it is first determined whether the audio signal generation position is within the visible area, and if it is within the visible area, position information can be output so that the user can move the audio signal generation position into the visible area. Specifically, the control unit 200 can detect the direction in which the audio signal is input based on the direction in which a part of the see-through display unit 300 worn by the user is pointed, and control the operation of the see-through display unit 300 so that the image information 10 includes information related to the detected direction. Here, the image information 10 may display the detected direction and position of the audio signal via the information display unit 320, and may overlap the image information 10 at the audio signal detection point in augmented reality. Specifically, when the position of the point where the audio signal is generated is outside a visible area visible to the user through the see-through display unit 300, the control unit 200 may control the operation of the see-through display unit 300 to output the image information 10 related to the position. Also, when the position of the point where the audio signal is generated is within the visible area, the control unit 200 may control the operation of the see-through display unit 300 to overlap the image information 10 of the virtual object on a portion of the screen of the see-through display unit where the position where the audio signal is generated is projected.

[0030] Hereinafter, the embodiment will be described in more detail with reference to Figures 5 to 8. Figures 5 to 8 are diagrams showing an embodiment of an augmented reality device according to the present invention.

[0031] First, referring to FIG. 5, when an audio signal is generated outside the visible area VA that is visible to the user through the see-through display unit 300, that is, when a knock sound is generated from a left door that is not visible to the user through the AR output unit 310 in FIG. 5, image information 10 related to the position of the point where the audio signal is generated can be output through the AR output unit 310. Specifically, referring to FIG. 5, when a knock sound is not generated within the visible area VA of the user, the position or direction of the point where the audio signal is generated can be displayed so that the visible area VA of the user can be switched to the position where the audio signal is generated. That is, when a sound is generated from the left door, a knock, which is a type of sound, can be displayed, and image information 10 related to the position can be output using a direction display and an arrow. More specifically, the object where the audio signal is generated can be displayed and output together. At the same time, the information display unit 320 provided along the outer periphery of the display unit 300 displays information in the corresponding direction, thereby immediately notifying the user that the audio signal is generated from the corresponding direction. At this time, the information display unit 320 can inform the user of the direction information through an LED notification or a vibration notification.

[0032] FIG. 6 shows a screen in which a user checks the video information 10 related to a direction or position in the visible area VA of FIG. 5, then turns his / her head to the left and switches the screen so that the position of the point where the audio signal is generated comes into the visible area VA. As shown in FIG. 6, when the position where the audio signal is generated is not included in the visible area VA in the screen of FIG. 5, the user can switch the visible area VA toward the position where the audio signal is generated by receiving the position and direction guidance. Therefore, as shown in FIG. 6, when the position where the audio signal is generated belongs to the visible area, the operation of the see-through display unit 300 can be controlled so that the video information 10 of the virtual object is overlapped on a part of the AR output unit 310 on which the position where the audio signal is generated is projected, among the screen of the see-through display unit 300. That is, when a knock sound is generated from the door and the user switches the visible area VA toward the door and the door belongs to the visible area VA, the type and sound of the audio signal can be overlapped and displayed as an image or a character, etc., to inform the user that a knock sound has occurred at the door where the audio signal is generated. At this time, the overlapped video information 10 can be displayed differently depending on the type and object of the audio signal. Specifically, icons are displayed according to the object, such as things, people, animals, etc., and the user can intuitively confirm the object of the sound by differences in the shape and color of the speech bubble, etc. Specifically, the following will be explained by comparing Figs. 6 to 8.

[0033] FIG. 6 shows an audio signal in which a knock sound is generated from a door, and the type and sound of the sound can be written in a rounded rectangular box with a guide line on the object generating the audio signal as shown in FIG. 6. FIG. 7 shows an audio signal in which people are talking, and when the person who is talking enters the visible area VA, the video information 10 about the audio signal can be overlapped and displayed. At this time, the person's conversation can be overlapped to match the person through a speech bubble. Therefore, the user can communicate with the person without hearing it. FIG. 8 shows an embodiment in which the sound of an animal is input to the audio signal, and when the sound of a dog barking is input to the audio signal, a curved guide line can be matched around the dog and provided as the video information 10, and at this time, an icon showing the dog is displayed in the video information 10, so that the user can intuitively confirm that the audio signal is generated by the dog.

[0034] As described above, according to the embodiment, in the present invention, after receiving an input of an audio signal, the audio signal is matched with a video signal of the display unit 300, and a corresponding type of audio signal is matched with the video information 10, thereby providing the audio signal as visual information to the user. At this time, the processes of identifying the audio signal, extracting a video signal corresponding to the type of the audio signal from among the video signals, and matching the audio signal with the video signal to display it as the video information 10 may be performed by an artificial neural network pre-installed in the control unit 200, and the artificial neural network may learn through the user's repeated use of the augmented reality device of the present invention, thereby providing more accurate and augmented information.

[0035] In this specification, the see-through display unit 300 has been described as an embodiment of smart glass, but this is only one embodiment and may be implemented via an image processing device and an audio analysis device of a mobile phone, a laptop, a wearable device, etc.

[0036] The present invention relates to one embodiment, which is merely an embodiment, and a person skilled in the art will be able to make various modifications and equivalent embodiments. Therefore, the scope of the rights of the present invention is not limited by the above-described embodiment and the accompanying drawings. Industrial Applicability

[0037] According to the present invention, a sound generated in an area that a user cannot see is converted into visual information and outputted, thereby enabling a user to accurately recognize sounds generated in the surroundings.

[0038] In addition, according to the present invention, even a user who has lost his / her hearing ability can experience the effect of visually seeing information related to various sounds generated in the surroundings.

[0039] In addition, according to the present invention, by combining audio signals generated in the surroundings with virtual objects generated by augmented reality, it is possible to obtain an effect of maximizing the user's sensory range.

Claims

1. a see-through display unit formed of a see-through material and outputting image information of a virtual object; an audio input unit for receiving an audio signal generated within a predetermined distance from the display unit; A control unit that identifies event information corresponding to the audio signal by a pre-installed artificial neural network and controls the operation of the see-through display unit so that image information of a virtual object corresponding to the identified event information is output, the video information includes at least one of text, image, type of audio signal, and location information related to the audio signal; The input unit includes: A camera is provided in a part of the see-through display unit and captures an image in one direction, The control unit is detecting a position of a point where the audio signal is generated, and controlling an operation of the see-through display unit so that the video information includes information related to the detected position; Identifying at least one object included in the video signal captured by the camera, and matching the identified at least one object with event information of the audio signal; An augmented reality device characterized by controlling the operation of the see-through display unit so that, when the position of the point where the audio signal is generated is outside the visible area visible to the user through the see-through display unit, image information related to the position is output.

2. The control unit is The augmented reality device of claim 1, further comprising: a display unit configured to detect a direction in which the audio signal is input based on a direction in which a portion of the see-through display unit is pointing; and a display unit configured to control an operation of the see-through display unit so that the video information includes information related to the detected direction.

3. The control unit is 2. The augmented reality device of claim 1, further comprising: a display unit configured to display a virtual object on a screen of the see-through display unit; a display unit configured to display a virtual object on a screen of the see-through display unit;

Images