Glasses-type display device

By designing a glasses-type display device, using the microphone and camera to receive and shoot live sounds and images, and comparing and outputting lines in real time, it solves the problem that hearing-impaired audiences find it difficult to see lines in stage play performances, real-time and correct subtitles display, and improves the audience experience.

CN120223870APending Publication Date: 2025-06-27QUANTA COMPUTER INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410051068.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-12-26
Filing Date
2024-01-12
Publication Date
2025-06-27

Smart Images

  • Figure CN120223870A_ABST
    Figure CN120223870A_ABST
Patent Text Reader

Abstract

The invention discloses a glasses type display device. The glasses type display device comprises a lens, a micro-projection device, a memory and a control unit, the control unit is used for reading the lines in the memory, comparing input voice with the lines to generate a comparison result, and deciding to output the lines according to the comparison result; and if it is determined that the line is output, controlling the micro-projection device to project the line to the lens.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a glasses-type display device, and particularly to a glasses-type display device for displaying subtitles. Background Art

[0002] When deaf or hard-of-hearing users watch a stage play, they may have a poor viewing experience because it is difficult to hear the lines spoken by the actors. To solve this problem, current theaters have staff operate a caption machine to display corresponding subtitles when they hear the actors speak their lines. However, this method requires additional human resources, and the staff may make mistakes. Therefore, how to correctly and instantaneously display the lines spoken by the actors is one of the problems to be solved in this field. Summary of the Invention

[0003] An embodiment of the present invention provides a glasses-type display device, including lenses, a micro-projection device, a memory, and a control unit. The control unit is configured to read the lines in the memory, compare the input voice with the lines to generate a comparison result, determine whether to output the lines according to the comparison result; and if it is determined to output the lines, control the micro-projection device to project the lines onto the lenses. Brief Description of the Drawings

[0004] Figure 1A is a schematic diagram of the glasses-type display device according to an embodiment of the present invention;

[0005] Figure 1B is a block diagram of the glasses-type display device according to an embodiment of the present invention;

[0006] Figure 2A is a schematic diagram of the glasses-type display device according to another embodiment of the present invention;

[0007] Figure 2B is a block diagram of the glasses-type display device according to another embodiment of the present invention;

[0008] Figure 3 is a flowchart of the method according to an embodiment of the present invention;

[0009] Figure 4 is a flowchart of the method according to an embodiment of the present invention;

[0010] Figure 5 is a flowchart of the method according to an embodiment of the present invention;

[0011] Figure 6A is a schematic diagram according to an embodiment of the present invention;

[0012] Figure 6B is a schematic diagram according to an embodiment of the present invention;

[0013] Figure 6C is a schematic diagram according to an embodiment of the present invention.

[0014] Symbol Explanation

[0015] 100: Eyeglass-type display device

[0016] 110: Frame

[0017] 120: Lens

[0018] 130: Microphone

[0019] 140: Camera

[0020] 150: Control unit

[0021] 151: Voice comparison module

[0022] 152: Position determination module

[0023] 160: Memory

[0024] 170: Micro projection device

[0025] 200: Electronic device

[0026] 300, 400, 500: Methods

[0027] 600: Lines

[0028] S310, S320, S330, S340, S410, S420, S430, S440, S510, S520, S530, S540, S550: Steps

[0029] A1~A4: Image regions

[0030] B1~B4: Lens regions

[0031] BLK1~BLK12: Boxes Detailed implementation manners

[0032] To make the objectives, features, and advantages of the present invention more obvious and understandable, specific embodiments of the present invention are hereinafter given, and detailed descriptions are made in conjunction with the accompanying drawings as follows.

[0033] Reference Figure 1A 、 Figure 1B , Figure 1A is a schematic diagram of the eyeglass-type display device 100 according to an embodiment of the present invention, Figure 1B is a block diagram of the eyeglass-type display device 100 according to an embodiment of the present invention. As Figure 1AAs shown, the glasses-type display device 100 includes a frame 110, lenses 120, a microphone 130, and a camera 140. The frame 110 supports and fixes the lenses 120, the microphone 130, and the camera 140. The lenses 120 are transparent. The microphone 130 and the camera 140 can be located, for example, on the frame 110 between the two lenses 120. However, the present invention is not limited thereto. The microphone 130 and / or the camera 140 can also both be on the left / right frames, or be respectively provided on the left / right frames. The microphone 130 and / or the camera 140 can also be mounted on the side of the glasses-type display device 100 (for example, on the temple). As Figure 1B As shown, the glasses-type display device 100 includes a microphone 130, a camera 140, a control unit 150, a memory 160, and a micro-projection device 170. In this embodiment, the control unit 150 is used to run / implement a voice comparison module 151 and a position determination module 152. The microphone 130, the camera 140, the control unit 150, the memory 160, and the micro-projection device 170 are connected to each other, and these components can exchange information with each other.

[0034] The microphone 130 is used to receive sound and convert the received sound into an electrical signal. In some embodiments, the microphone 130 is a directional microphone. A directional microphone can receive sound from a specific direction and ignore other sounds that do not come from that specific direction. Therefore, using a directional microphone can ignore the sounds of other audiences beside the user (wearing the glasses-type display device 100), achieving a better sound collection effect. The camera 140 is used to capture an image and convert the captured image into an electrical signal. The camera 140 is, for example, a Charge Coupled Device (CCD) or a Complementary Metal-Oxide Semiconductor (CMOS) imager.

[0035] The control unit 150 provides the ability to calculate and process data. The control unit 150 can be used to execute programs, software, firmware, modules. In addition, the control unit 150 is used to control the microphone 130, the camera 140, the memory 160, and the micro-projection device 170 (for example, through instructions). For example, the control unit 150 can include a general-purpose processor, a special-purpose processor, a central processing unit, and / or a microcontroller unit.

[0036] The voice comparison module 151 is used to receive the input voice from the microphone 130. The voice comparison module 151 determines the output line (i.e., string) according to the input voice. The position determination module 152 receives the input image from the camera 140. The position determination module 152 is used to determine the position where the line is displayed (projected) on the lens 120 according to the input image. The voice comparison module 151 and the position determination module 152 will be described in more detail below.

[0037] The memory 160 stores the data required for the operation of the control unit 150, including program codes and lines (strings). The control unit 150 can access / write to the memory 160. In some embodiments, program codes are stored in the memory 160, and these program codes can be read and executed by the control unit 150, so that the control unit 150 runs / implements the voice comparison module 151 and the position determination module 152, as well as the method of the embodiments of the present invention. The memory 160 may include non-volatile memories, such as read-only memories and flash memories. The memory 160 may also include volatile memories, such as dynamic random access memories and static random access memories.

[0038] The micro-projection device 170 can project a given line onto the specified position of the lens 120 under the control of the control unit 150. Therefore, the user can see the performance and the line through the lens 120 at the same time.

[0039] Reference Figure 2A 、 Figure 2B , Figure 2A is a schematic diagram of the glasses-type display device 100 according to another embodiment of the present invention, Figure 2B is a block diagram of the glasses-type display device 100 according to another embodiment of the present invention. As Figure 2A 、 Figure 2B shown, in this embodiment, the glasses-type display device 100 does not include the microphone 130. The microphone 130 is separated from the glasses-type display device 100. The microphone 130 can be set in front of the stage or on the stage to achieve better sound collection effect. In addition, in the case where the microphone 130 is set on the glasses-type display device 100 (as Figure 1A ), when the user turns his head, it will be difficult for the microphone 130 to receive the sound from the stage. Figure 2A 、 Figure 2B The embodiments shown can avoid this problem. Reference Figure 2B ​​, in this embodiment, the voice comparison module 151 is run by an electronic device 200 separated from the glasses-type display device 100. The electronic device 200 has the ability to compute and process data. For example, the electronic device 200 is a computer, a notebook computer, or a control box. In some embodiments, the electronic device 200 includes a processor and a memory. The microphone 130 transmits the received input voice (audio signal) to the electronic device 200. The voice comparison module 151 determines the lines (strings) according to the received input voice. The electronic device 200 then transmits the determined lines to the glasses-type display device 100. The glasses-type display device 100 determines the position where the received lines are to be displayed through the position determination module and displays the received lines.

[0040] In Figure 2A , Figure 2B the displayed embodiment, the control unit 150 only needs to display the received lines. Therefore, the control unit 150 can be implemented using relatively simple hardware, and thus the hardware cost can be reduced and the volume of the glasses-type display device 100 can be reduced. Any known communication technology can be used for communication between the microphone 130, the electronic device 200, and the glasses-type display device 100, such as Wi-Fi and Bluetooth. In some embodiments, the electronic device 200 transmits the lines to a mobile phone, and the mobile phone then transfers the lines to the glasses-type display device 100. This can further simplify the hardware circuit required for the glasses-type display device 100.

[0041] Reference Figure 3 , Figure 3 is a flowchart of the method 300 according to an embodiment of the present invention. The method 300 can be implemented in the Figure 1A , Figure 1B , Figure 2A , Figure 2B displayed embodiments. The method 300 starts from step S310. In step S310, the input voice is received through the microphone 130, and the input image is captured through the camera 140. Then the method 300 proceeds to step S320 and step S330. In some embodiments, step S320 and step S330 are executed simultaneously in parallel. In step S320, the lines are determined according to the input voice through the voice comparison module 151. The voice comparison module 151 outputs the determined lines. In step S330, the position where the lines are to be displayed (on the lens 120) is determined according to the input image through the position determination module 152. Then the method 300 proceeds to step S340. In step S340, the control unit 150 receives the lines output by the voice comparison module 151 and the position determined by the position determination module 152. The control unit 150 controls the micro projection device 170 to display the lines determined by the voice comparison module 151 at the position determined by the position determination module 152.

[0042] ReferenceFigure 4 , Figure 4 is a flowchart of method 400 according to an embodiment of the present invention. Method 400 can be implemented in the embodiments shown in Figure 1A , Figure 1B , Figure 2A , Figure 2B . Method 400 is a method for determining a line (string) according to the input voice in step S320. Therefore, method 400 can be executed by the voice comparison module 151. In step S410, the voice comparison module 151 receives the input voice from the microphone 130. The voice comparison module 151 can receive the input voice from the microphone 130 through wired ( Figure 1A , Figure 1B ) or wireless ( Figure 2A , Figure 2B ) communication. The input voice is an audio signal, and this audio signal represents, for example, at least one sentence spoken by a human. For example, the input voice is the line spoken by an actor performing live.

[0043] In step S420, the voice comparison module 151 reads the first line from among the multiple lines (strings / text files) stored in the memory. The multiple lines stored in the memory can be referred to as a line library. For example, when the voice comparison module 151 is run by the control unit 150, the voice comparison module 151 reads the lines stored in the memory 160. Or, when the voice comparison module 151 is run by the electronic device 200, the voice comparison module 151 reads the lines stored in the memory of the electronic device 200. In other embodiments, the voice comparison module 151 can also read the lines stored in the server. These lines are the lines of the script, and the actor will perform according to the script. In other words, the input voice will correspond to one of these line libraries. These lines are stored in the memory in advance (for example, before the performance starts). In some embodiments, these lines are stored in the memory before step S410 or step S420. In some embodiments, these lines are stored in the memory in the form of text files.

[0044] In step S430, the voice comparison module 151 compares the input voice and the voice corresponding to the first line of dialogue to generate a comparison result. Therefore, the voice comparison module 151 determines whether the first line of dialogue among multiple lines of dialogue matches the input voice. As described above, the first line of dialogue is a text document, and the voice corresponding to the first line of dialogue is an audio signal, which represents the pronunciation of the first line of dialogue. In some embodiments, the voice comparison module 151 uses voiceprint comparison technology to determine whether the input voice is the same as the voice corresponding to the first line of dialogue. In some embodiments, the memory storing the lines of dialogue also stores the voices (audio signals) corresponding to these lines of dialogue. The voice comparison module 151 reads the voices stored in the memory to determine whether these voices are the same as the input voice. In another specific embodiment, the voice comparison module 151 can use speech-to-text technology or speech recognition to convert the input voice into text, and then compare and determine whether the input voice is the same as the first line of dialogue, so that there is no need to store the voices (audio signals) corresponding to these lines of dialogue.

[0045] In some embodiments, the way of the comparison result is to calculate the accuracy rate. The input voice can be cut into multiple voice single words, and each voice single word is an audio signal representing the pronunciation of a character. The first line of dialogue can also be cut into multiple dialogue single words. The calculation method of the accuracy rate is:

[0046] The number of matching words / The number of all dialogue single words

[0047] The number of matching words is the number of pronunciations of the speech words that are the same as the corresponding pronunciations of the script words. In some embodiments, the speech comparison module 151 sequentially determines whether each speech word of the input speech has the same pronunciation as the corresponding script word of the first script. In other words, the speech comparison module 151 first determines whether the first speech word of the input speech (e.g., the first word at the beginning of the input speech) has the same pronunciation as the first script word of the first script (e.g., the first word at the beginning of the script), and then determines whether the second speech word of the input speech (the next speech word after the first speech word) has the same pronunciation as the second script word of the first script (the next script word after the first script word), and so on. The speech comparison module 151 repeatedly performs the above operations until all the speech words of the input speech or all the script words of the first script are determined. In some embodiments, when the speech comparison module 151 determines that the first speech word of the input speech has a different pronunciation from the first script word of the first script, the speech comparison module 151 skips the first speech word. After that, the speech comparison module 151 compares the second speech word of the input speech with the first script word of the first script to sequentially compare the pronunciation of the third speech word of the input speech (the next speech word after the second speech word) with the second script word of the first script (the next script word after the first script word). Optionally, the speech comparison module 151 can also skip the first script word of the first script and directly compare the first speech word of the input speech with the second script word of the first script to sequentially compare the second speech word of the input speech with the third script word of the first script.

[0048] In step S440, the speech comparison module 151 determines whether to output the first script according to the comparison result (correct rate). In some embodiments, when the speech comparison module 151 determines that the correct rate is greater than the first threshold, the speech comparison module 151 outputs the first script (e.g., outputs it to the control unit 150). The first threshold is, for example, 80%. Therefore, even if the lines spoken by the on-site actor are slightly different from the lines in the script, the glasses-type display device 100 can still display the correct lines. If the speech comparison module 151 determines that the first script does not match the input speech, for example, when the comparison result (correct rate) is less than the first threshold and greater than the second threshold, the speech comparison module 151 continues to compare the input speech with other scripts (i.e., the second script) in the script library. When the speech comparison module 151 determines that the correct rate is less than the second threshold, the speech comparison module 151 does not output the first script (e.g., outputs it to the control unit 150). The second threshold is, for example, 20%. Therefore, when the on-site actor forgets the lines, the glasses-type display device 100 does not display the lines.

[0049] Referring to Table 1 and Table 2, Table 1 and Table 2 illustrate the relationship between the input voice, the lines in the line library, and the lines output by the voice comparison module 151.

[0050]

[0051] Table 1

[0052]

[0053] Table 2

[0054] Therefore, the voice comparison module 151 in the above embodiments of the present invention does not convert the input voice into text, but determines whether the input voice conforms to known text. In other words, the voice comparison module 151 in the embodiments of the present invention does not perform semantic understanding and sentence generation. Compared with the method of using speech recognition (including semantic understanding and sentence generation), the voice comparison module 151 in the embodiments of the present invention requires less computational effort. Therefore, the voice comparison module 151 in the embodiments of the present invention can output the lines spoken by the actor in real time. In addition, the voice comparison module 151 in the embodiments of the present invention directly outputs the pre-stored text. Therefore, it is possible to avoid incorrect output due to incorrect content of the input voice. In particular, the voice comparison module 151 in the embodiments of the present invention can solve the problem of judging homophones and homophonic characters.

[0055] Reference Figure 5 , Figure 5 is a flowchart of the method 500 according to an embodiment of the present invention. The method 500 can be implemented in the embodiments shown in Figure 1A , Figure 1B , Figure 2A , Figure 2B The method 500 is a method for determining the position of the lines according to the input image in step S330. Therefore, the method 500 can be executed by the position determination module 152. In step S510, the position determination module 152 receives the input image captured by the camera 140. Since the camera 140 is installed on the glasses-type display device 100, the image captured by the camera 140 is generally the scene seen by the user. In step S520, the position determination module 152 marks the position of the human figure in the input image. The position determination module 152 can use any known image recognition or object recognition technology to identify the human figure in the input image. Then, the position determination module 152 marks the position of the identified human figure. Reference Figure 6A , Figure 6A is a schematic diagram of an embodiment of the present invention. Figure 6A Schematically shows the input image after being marked. As shown in Figure 6AAs shown, the position determination module 152 marks the positions of the human figures in the input image with boxes BLK1 to BLK12. Optionally, the whole body of the human figure is included within boxes BLK1 to BLK12.

[0056] In step S530, the position determination module 152 divides the input image into a plurality of regions (hereinafter referred to as image regions). The position determination module 152 can divide the input image in any manner. For example, the position determination module 152 can divide the input image into N×M image regions, where N and M are any positive integers. N can be equal to or not equal to M. In some embodiments, these regions are of equal size. In step S540, the position determination module 152 divides the lens 120 into a plurality of regions (hereinafter referred to as lens regions) in the same manner as it divides the input image. For example, the lens 120 is divided into N×M regions respectively. Thus, each lens region corresponds to an image region. In some embodiments, the position determination module 152 assigns coordinates (or numbers) to the image regions and the lens regions in the same manner. The corresponding image region and lens region have the same coordinates (or numbers). For example, referring to Figure 6B , Figure 6B is a schematic diagram of an embodiment of the present invention. In Figure 6B , the position determination module 152 divides both the input image and the lens 120 into 2×2 regions. The position determination module 152 assigns coordinates in sequence from left to right and from top to bottom. The coordinates of the image region A1 and the lens region B1 are (1,1), the coordinates of the image region A2 and the lens region B2 are (1,2), the coordinates of the image region A3 and the lens region B3 are (2,1), and the coordinates of the image region A4 and the lens region B4 are (2,2). Thus, the image region A1 corresponds to the lens region B1, the image region A2 corresponds to the lens region B2, the image region A3 corresponds to the lens region B3, and the image region A4 corresponds to the lens region B4.

[0057] In step S550, the position determination module 152 determines the position on the lens 120 where the lines are to be displayed. The position determination module 152 determines a first image region in the input image where there is no human figure based on the marked positions. Then, the position determination module 152 determines to display the lines in a first lens region in the lens 120, and the first lens region corresponds to the first image region. In some embodiments, the position determination module 152 can also select a plurality of regions for displaying the lines.

[0058] In other embodiments, the position determination module 152 may also not execute steps S510 to S540. In this embodiment, the position determination module 152 determines to display the lines at a preset position on the lens 120. The preset position is, for example, the uppermost, lowermost, leftmost, or rightmost position. Since the actor usually appears in the middle of the stage. By this method, the computational load can be reduced and the processing time can be decreased, so that the glasses-type display device 100 can display the lines more promptly.

[0059] Therefore, by using the position determination module 152, the glasses-type display device 100 according to the embodiment of the present invention can prevent the lines from blocking the actor in the user's field of view. Refer to Figure 6C , Figure 6C which schematically shows the scene seen by the user through the lens 120. As Figure 6C shown, since the actor is concentrated in the lower part, the position determination module 152 determines to display the line 600 above the lens 120.

[0060] In summary, the glasses-type display device according to the embodiment of the present invention can correctly display the lines corresponding to the received voice. And it can display the lines (captions) at a position that does not block the actor. Therefore, the glasses-type display device according to the embodiment of the present invention can display the correct lines at the correct time point to enhance the user experience of the user.

[0061] Although the present invention is disclosed in combination with the above embodiments, the above embodiments are not intended to limit the invention of this case. Any person skilled in this art can make various modifications and refinements according to the above embodiments without departing from the spirit and scope of this case. Therefore, the protection scope of this case should be defined by the appended claims.

Claims

1. A glasses-type display device, comprising: lens; Micro-projection device; Memory; as well as A control unit for: Read the lines in the memory; comparing the input speech and the line to generate a comparison result; Determine to output the line according to the comparison result; as well as If it is decided to output the lines, the micro-projection device is controlled to project the lines onto the lens.

2. The glasses-type display device according to claim 1, further comprising: A microphone is used to receive the input voice. The glasses-type display device as claimed in claim 2 , wherein the microphone is a directional microphone.

4. The eyeglass-type display device as claimed in claim 1, wherein the input speech is composed of a plurality of speech words, and the lines are composed of a plurality of line words; in, The control unit is also used to: Determining whether the pronunciation words match the pronunciations corresponding to the dialogue words; Calculate the number of matching words, which is the number of matching words between the speech words and the dialogue words; and Calculate the accuracy rate; The comparison result is the accuracy rate. 5 . The glasses-type display device as claimed in claim 4 , wherein when the control unit determines that the accuracy rate is greater than a first threshold, the lines are output. 6 . The glasses-type display device as claimed in claim 5 , wherein when the control unit determines that the accuracy rate is less than a second threshold, the lines are not output.

7. The glasses-type display device as claimed in claim 1, wherein the control unit further determines the position where the lines are displayed on the lens. 8 . The glasses-type display device as claimed in claim 7 , wherein the control unit displays the lines at the top or bottom of the lens.

9. The glasses-type display device according to claim 1, further comprising: A camera for capturing input images; The control unit is further used for: Marking a position of at least one person image in the input image; It is determined to display the lines in a first lens area of ​​the lens, where the first lens area does not include the at least one portrait.

10. The glasses-type display device as claimed in claim 2, wherein the microphone is disposed in front of or above the stage.