User Interaction Assistance Method, Device, Electronic Device, and Readable Storage Medium

Abstract

The present application discloses a user interaction assistance method, apparatus, electronic device, and readable storage medium, which are applied to AR glasses. The AR glasses are provided with bone conduction headphones. The user interaction assistance method includes: picking up the audio of the user wearing the AR glasses through the bone conduction headphones to obtain the picked-up audio; if the picked-up audio is valid audio, when a sign language action from the outside is detected, converting the sign language action into corresponding natural language information for playback to assist the wearing user in interacting with the outside; if the picked-up audio is not valid audio, when interaction information from the outside is detected, displaying each preset reply answer corresponding to the interaction information in the AR display screen of the AR glasses to assist the wearing user in interacting with the outside. This application solves the technical problem of the communication barrier between deaf-mute users and the outside world in the prior art.

Description

Technical Field

[0001] This application relates to the technical field of AR (Augmented Reality), and particularly to a user interaction assistance method, device, electronic device, and readable storage medium. Background Art

[0002] With the development of AR technology, science and technology have made more improvements in people's communication. However, deaf-mute users have deficiencies. Deaf-mute users usually need to communicate with the outside world through sign language. But if the communication target of a deaf-mute user has not learned sign language, the communication target of the deaf-mute user usually cannot understand what the deaf-mute user wants to express, and there are obstacles in the communication between deaf-mute users and the outside world. Summary of the Invention

[0003] The main purpose of this application is to provide a user interaction assistance method, device, electronic device, and readable storage medium, aiming to solve the technical problem of communication obstacles between deaf-mute users and the outside world in the prior art.

[0004] To achieve the above object, this application provides a user interaction assistance method, which is applied to AR glasses. The AR glasses are provided with bone conduction headphones. The user interaction assistance method includes:

[0005] Pick up the audio of the user wearing the AR glasses through the bone conduction headphones to obtain the picked-up audio;

[0006] If the picked-up audio is valid audio, when a sign language action from the outside is detected, convert the sign language action into corresponding natural language information for playback to assist the wearing user in interacting with the outside world;

[0007] If the picked-up audio is not valid audio, when interaction information from the outside is detected, display each preset reply answer corresponding to the interaction information on the AR display screen of the AR glasses to assist the wearing user in interacting with the outside world.

[0008] To achieve the above object, this application also provides a user interaction assistance device, which is applied to AR glasses. The AR glasses are provided with bone conduction headphones. The user interaction assistance device includes:

[0009] An audio pick-up module, configured to pick up the audio of the user wearing the AR glasses through the bone conduction headphones to obtain the picked-up audio;

[0010] A first interaction mode module, configured to, if the picked-up audio is valid audio, when a sign language action from the outside is detected, convert the sign language action into corresponding natural language information for playback to assist the wearing user in interacting with the outside world;

[0011] A second interaction mode module, configured to, if the picked-up audio is not valid audio, display each preset reply answer corresponding to the interaction information in the AR display screen of the AR glasses when detecting external interaction information, so as to assist the wearing user to interact with the outside world.

[0012] The present application further provides an electronic device, which is a physical device and includes: a memory, a processor, and a program of the user interaction assistance method stored on the memory and executable on the processor. When the program of the user interaction assistance method is executed by the processor, the steps of the user interaction assistance method as described above can be implemented.

[0013] The present application further provides a computer-readable storage medium, on which a program for implementing the user interaction assistance method is stored. When the program of the user interaction assistance method is executed by a processor, the steps of the user interaction assistance method as described above are implemented.

[0014] The present application further provides a computer program product, including a computer program, which when executed by a processor implements the steps of the user interaction assistance method as described above.

[0015] The present application provides a user interaction assistance method, device, electronic device and readable storage medium, which are applied to AR glasses. The AR glasses are provided with bone conduction earphones, that is, the audio of the wearing user of the AR glasses is picked up through the bone conduction earphones to obtain the picked-up audio. If the picked-up audio is valid audio, it proves that the wearing user can communicate with the outside world through bone conduction, and the deaf-mute degree of the wearing user is relatively low. Furthermore, when detecting a sign language action from the outside world, the sign language action from the outside world can be converted into corresponding natural language information for playback, so as to assist the wearing user to interact with the outside world. If the picked-up audio is not valid audio, it proves that the wearing user cannot communicate with the outside world through bone conduction, and the deaf-mute degree of the wearing user is relatively high. Furthermore, when detecting external interaction information, each preset reply answer corresponding to the interaction information is displayed in the AR display screen of the AR glasses to assist the wearing user to interact with the outside world. Thus, in the present application, an interaction assistance mode adapted to the wearing user is matched according to the deaf-mute degree of the wearing user, so that under the corresponding interaction assistance mode, the communication barrier between deaf-mute users with different deaf-mute degrees and the outside world is overcome. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.

[0017] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0018] Figure 1 Schematic flowchart of the first embodiment of the user interaction assistance method of the present application;

[0019] Figure 2 Schematic flowchart of the second embodiment of the user interaction assistance method of the present application;

[0020] Figure 3 Schematic diagram of the device structure of the hardware operating environment involved in the user interaction assistance method in the embodiments of the present application.

[0021] The realization of the purpose, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. Specific embodiments

[0022] To make the above objects, features and advantages of the present application more obvious and understandable, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present application belong to the scope of protection of the present application.

[0023] Embodiment 1

[0024] The embodiment of the present application provides a user interaction assistance method. In the first embodiment of the user interaction assistance method of the present application, it is applied to an AR glasses, and the AR glasses are provided with a bone conduction headset. The user interaction assistance method includes:

[0025] Step S10, pick up the audio of the user wearing the AR glasses through the bone conduction headset to obtain the picked-up audio;

[0026] Step S20, if the picked-up audio is valid audio, when a sign language action from the outside world is detected, convert the sign language action into corresponding natural language information for playback to assist the wearing user to interact with the outside world;

[0027] Step S30, if the picked-up audio is not valid audio, when interaction information from the outside world is detected, display each preset reply answer corresponding to the interaction information on the AR display screen of the AR glasses to assist the wearing user to interact with the outside world.

[0028] In this embodiment, it should be noted that the user interaction assistance method is applied to AR glasses. The user wearing the AR glasses can be a normal person who does not belong to the deaf-mute population. Among the deaf-mute population, there are some users who can hear external sounds through bone conduction and make corresponding feedback. Therefore, these users can interact with the outside world by means of bone conduction headphones. At the same time, among the deaf-mute population, there are also some users who cannot hear external sounds and make corresponding feedback even through bone conduction. These users cannot hear external sounds at all and cannot make any sound. Therefore, these deaf-mute people have a more serious communication barrier with the outside world and a higher degree of deafness. So there are deaf-mute users with different degrees of deafness in the deaf-mute population.

[0029] As an example, step S10 includes: picking up the audio emitted by the user wearing the AR glasses through the bone conduction headphones to obtain a picked-up audio, and then determining whether the picked-up audio is a valid audio.

[0030] As an example, the step of determining whether the picked-up audio is a valid audio includes:

[0031] Extracting the audio features of the picked-up audio, performing binary classification on the picked-up audio based on the audio features to obtain a binary classification label; and determining whether the picked-up audio is a valid audio based on the binary classification label. For example, when the binary classification label is set to 1, it indicates that the picked-up audio is a valid audio, and when the binary classification label is set to 0, it indicates that the picked-up audio is not a valid audio.

[0032] As an example, the step of determining whether the picked-up audio is a valid audio further includes:

[0033] Picking up the audio emitted by the user wearing the AR glasses through an air conduction microphone to obtain a second picked-up audio; determining whether the wearing user can emit a valid audio based on the picked-up audio and the second picked-up audio. If the wearing user can emit a valid audio, then execute the step: when a sign language action from the outside is detected, convert the sign language action into corresponding natural language information for playback to assist the wearing user in interacting with the outside world; if the wearing user cannot emit a valid audio, then execute the step: when interaction information from the outside is detected, display each preset reply answer corresponding to the interaction information in the AR display screen of the AR glasses to assist the wearing user in interacting with the outside world. In the embodiment of the present application, two channels of an air conduction microphone and a bone conduction microphone can be used to comprehensively determine whether the wearing user can emit a valid audio, improving the reliability and accuracy of the determination of valid audio.

[0034] As an example, determining whether the wearing user can emit valid audio based on the picked-up audio and the second picked-up audio includes:

[0035] Extract the audio features of the picked-up audio and the second picked-up audio respectively to obtain the first audio feature corresponding to the picked-up audio and the second audio feature corresponding to the second picked-up audio; classify the picked-up audio based on the first audio feature to obtain a first binary classification label, and classify the second picked-up audio based on the second audio feature to obtain a second binary classification label; determine whether the wearing user can emit valid audio according to the first binary classification label and the second binary classification label. For example, when the binary classification label is set to 1, it indicates that the picked-up audio is valid, and when the binary classification label is set to 0, it indicates that the picked-up audio is not valid. Furthermore, it can be set that when there is a preset binary label of 1 in the first binary classification label and the second binary classification label, it is determined that the wearing user can emit valid audio, and when there is no preset binary label of 1 in the first binary classification label and the second binary classification label, it is determined that the wearing user cannot emit valid audio.

[0036] As an example, steps S20 to S30 include: If the picked-up audio is valid audio, it proves that the wearing user can communicate with the outside world by means of bone conduction. Then, when a sign language action from the outside world is detected, the sign language action is converted into corresponding natural language information, and the natural language information is played to the wearing user through the bone conduction earphone to assist the wearing user in interacting with the outside world; If the picked-up audio is not valid audio, it proves that the wearing user cannot communicate with the outside world by means of bone conduction. Then, when interaction information from the outside world is detected, each preset reply answer corresponding to the interaction information is displayed in the AR display screen of the AR glasses, and then the target reply answer selected by the wearing user from each of the preset reply answers is determined, and the target reply answer is broadcast, so as to realize assisting the wearing user in interacting with the outside world. In the embodiment of the present application, different interaction modes are provided for deaf-mute users according to their different degrees of deaf-muteness. When the degree of deaf-muteness of the deaf-mute user is relatively low, through the sign language conversion mode, the sign language action is converted into natural language information, and the natural language information is played to the wearing user in a bone conduction manner; When the degree of deaf-muteness of the deaf-mute user is relatively high, through the fixed answer mode, the preset answers corresponding to the interaction information are directly displayed to the wearing user visually, so that the wearing user can interact with the outside world by selecting the target reply answer. Therefore, the user interaction efficiency and simplicity of the fixed answer mode are significantly lower than those of the sign language conversion mode. In the embodiment of the present application, by pre-determining the degree of deaf-muteness of the deaf-mute user, on the one hand, a sign language conversion mode with higher interaction efficiency can be automatically provided for deaf-mute users with a relatively low degree of deaf-muteness, while overcoming the communication barrier between the deaf-mute user and the outside world, ensuring the interaction efficiency. On the other hand, a fixed answer mode with higher interaction accuracy can be automatically provided for deaf-mute users with a relatively high degree of deaf-muteness, so that when the degree of deaf-muteness of the deaf-mute user is relatively high, the communication barrier between the deaf-mute user and the outside world can also be overcome.

[0037] Among them, the step of converting the sign language action into corresponding voice information for playing includes:

[0038] Step S21, performing sign language recognition on the sign language action to obtain a sign language recognition result;

[0039] Step S22, generating natural language information corresponding to the sign language action according to the sign language recognition result;

[0040] Step S23, playing the natural language information to the wearing user of the AR glasses.

[0041] As an example, steps S21 to S23 include: capturing the outside world through an AR glasses to obtain an outside image; identifying whether there is a sign language action in the outside image, and if so, determining that a sign language action is detected; segmenting the sign language action image from the outside image, classifying the sign language action image, identifying the sign language action in the sign language action image to obtain a sign language action label, and using the sign language action label as the sign language recognition result, where the sign language action label is a category label of the sign language action and is used to identify the semantic category of the sign language action; determining the natural language information corresponding to the sign language action label according to the mapping relationship between the sign language action label and the natural language information; and playing the natural language information to the user wearing the AR glasses, where the playing method can be in the form of voice broadcast or in the form of an AR display screen.

[0042] Wherein, the natural language information includes at least one of a broadcast voice and a text display screen, and the step of generating the natural language information corresponding to the sign language action according to the sign language recognition result includes:

[0043] Step A10, converting the sign language action into corresponding voice information according to the sign language recognition result, and generating a broadcast voice corresponding to the voice information;

[0044] In this embodiment, it should be noted that the number of the sign language actions can be one or more. The sign language recognition result can be one or more sign language action labels.

[0045] As an example, step A10 includes: determining the text word labels corresponding to the sign language action labels in the sign language recognition result according to the correspondence between the sign language action label and the text word label, where the text word label is a semantic label of a text word and is used to identify the semantics of the text word; sorting the text word labels according to the sign language word order jointly corresponding to the sign language actions to obtain a text word label sequence; determining the audio sequence data corresponding to the text word label sequence according to the semantic correspondence between the text word label and the audio, where the audio sequence data is the voice information corresponding to each sign language action, and specifically can be a feature sequence composed of audio features corresponding to each text word; and generating a real voice corresponding to the audio sequence data as the broadcast voice.

[0046] Step B10, converting the sign language action into corresponding text information according to the sign language recognition result, and generating a text display screen corresponding to the text information.

[0047] As an example, the text information may be a text sentence. Step A20 includes: determining text words corresponding to each sign language action tag in the sign language recognition result according to the correspondence between the sign language action tag and the text word tag; sorting the text words according to the sign language word order jointly corresponding to each sign language action to obtain a text sentence; and generating an AR display screen with the text sentence as the text display screen.

[0048] Wherein, the natural language information includes at least one of a broadcast voice and a text display screen. The step of playing the natural language information to the wearing user of the AR glasses includes:

[0049] Step C10, playing the broadcast voice to the wearing user through the bone conduction earphone;

[0050] Step D10, displaying the text display screen to the wearing user through the AR display screen of the AR glasses.

[0051] As an example, step D10 includes: determining a to-be-replaced image area corresponding in the AR display screen, replacing the to-be-replaced image area in the AR display screen with the text display screen to obtain a fused AR display screen, and displaying the fused AR display screen. Wherein, the to-be-replaced image area may be the entire AR display screen or a partial image area in the AR display screen, and specifically may be an image area in the AR display screen that does not affect the AR experience of the user, such as the edge area of the AR display screen, etc.

[0052] Wherein, the interaction information includes voice interaction information. The step of displaying each preset reply answer corresponding to the interaction information in the AR display screen of the AR glasses includes:

[0053] Step S31, performing speech recognition on the voice interaction information to obtain a speech recognition result;

[0054] Step S32, determining each preset reply answer corresponding to the voice interaction information according to the speech recognition result;

[0055] Step S33, displaying each preset reply answer in the AR display screen.

[0056] In this embodiment, it should be noted that the speech recognition result may be a semantic feature corresponding to the voice interaction information, and the semantic feature may be an embedding feature identifying the semantics of the voice interaction information, that is, embedding.

[0057] As an example, steps S31 to S33 include:

[0058] Extract semantic features from the voice interaction information to obtain the semantic features corresponding to the voice interaction information; determine the feature with the highest similarity to the semantic features among the preset semantic features as the target semantic feature; determine the preset reply answers corresponding to the target semantic feature according to the mapping relationship between the preset semantic features and the preset reply answers; display the preset reply answers corresponding to the target semantic feature in the AR display screen of the AR glasses, and detect the feedback information of the wearing user on the preset reply answers corresponding to the target semantic feature. Further, according to the feedback information, select a target reply answer from the preset reply answers corresponding to the target semantic feature, where the feedback information may be the click information of the wearing user on each displayed preset reply answer. For example, there are 3 preset reply answers A, B, and C. If the wearing user clicks on C, then C is the target reply answer; play the target reply answer through the microphone. In the embodiments of the present application, the purpose of assisting deaf-mute people to make corresponding feedback to external information is achieved. Even if the deaf-mute people cannot make any sound and cannot hear any sound at all, that is, there is a more serious communication barrier with the outside world, the embodiments of the present application can also assist deaf-mute users to make timely feedback on external information, so as to complete communication with the outside world, thereby overcoming the communication barrier between deaf-mute people and the outside world.

[0059] An embodiment of the present application provides a user interaction assistance method, which is applied to an AR glasses. The AR glasses are provided with a bone conduction earphone, that is, the audio of the wearing user of the AR glasses is picked up through the bone conduction earphone to obtain the picked-up audio; if the picked-up audio is valid audio, it proves that the wearing user can communicate with the outside world through bone conduction, and the deaf-mute degree of the wearing user is relatively low. Furthermore, when detecting a sign language action from the outside world, the sign language action from the outside world can be converted into corresponding natural language information for playback, so as to assist the wearing user to interact with the outside world; if the picked-up audio is not valid audio, it proves that the wearing user cannot communicate with the outside world through bone conduction, and the deaf-mute degree of the wearing user is relatively high. Furthermore, when detecting external interaction information, display the preset reply answers corresponding to the interaction information in the AR display screen of the AR glasses to assist the wearing user to interact with the outside world. Thus, in the present application, an interaction assistance mode adapted to the wearing user is matched according to the deaf-mute degree of the wearing user, so that under the corresponding interaction assistance mode, the communication barrier between deaf-mute users with different deaf-mute degrees and the outside world is overcome.

[0060] Embodiment 2

[0061] Further, refer to Figure 2, in another embodiment of the present application, for the same or similar content as the above embodiment, reference may be made to the above introduction and will not be elaborated hereinafter. On this basis, the interaction information includes sign language actions, and the steps of displaying each preset reply answer corresponding to the interaction information in the AR display screen of the AR glasses include:

[0062] Step E10, perform sign language recognition on the sign language actions to obtain a sign language recognition result;

[0063] Step E20, determine each preset reply answer corresponding to the sign language actions according to the sign language recognition result;

[0064] Step E30, display each of the preset reply answers in the AR display screen.

[0065] In this embodiment, it should be noted that the wearing user is a deaf-mute user, and the interaction target corresponding to the wearing user also belongs to the deaf-mute population. The interaction target shows sign language to the wearing user, but due to the wearing user's inability to understand sign language, there is a communication barrier between the wearing user and the interaction target.

[0066] As an example, the sign language recognition result may be a sign language semantic feature. Steps S31 to S33 include: taking a picture of the outside world through the AR glasses to obtain an outside image; identifying whether there are sign language actions in the outside image, and if so, determining that a sign language action is detected; segmenting the sign language action image from the outside image; extracting features from the sign language action image to extract the sign language semantic feature corresponding to the sign language action in the sign language action image, where the sign language semantic feature may be an embedded feature representing the sign language semantics of the sign language action; determining the feature with the highest similarity to the sign language semantic feature among each preset sign language semantic feature as the target sign language semantic feature; determining each preset reply answer corresponding to the target sign language semantic feature according to the mapping relationship between the preset sign language semantic feature and the preset reply answer; and displaying each preset reply answer corresponding to the target sign language semantic feature in the AR display screen of the AR glasses.

[0067] Wherein, after the step of displaying each of the preset reply answers in the AR display screen, the user interaction assistance method further includes:

[0068] Step E40, receive feedback information of the wearing user on each of the preset reply answers;

[0069] Step E50, select a target reply answer from each of the preset reply answers according to each of the feedback information;

[0070] Step E60, broadcast the target reply answer.

[0071] As an example, steps S34 to S36 include: receiving feedback information of the wearing user on each of the preset reply answers; according to the feedback information, selecting a target reply answer from each of the preset reply answers corresponding to the target sign language semantic feature, where the feedback information may be click information of the wearing user on each of the displayed preset reply answers. For example, there are 3 preset reply answers A, B, and C. If the wearing user clicks on C, then C is the target reply answer; playing the target reply answer through a microphone.

[0072] The embodiment of the present application provides a user interaction assistance method, that is, performing sign language recognition on the sign language action to obtain a sign language recognition result; determining each preset reply answer corresponding to the sign language action according to the sign language recognition result; displaying each of the preset reply answers in the AR display screen; receiving feedback information of the wearing user on each of the preset reply answers; selecting a target reply answer from each of the preset reply answers according to each of the feedback information; and broadcasting the target reply answer. Thus, in a visual display manner, the sign language information from the outside world is transmitted to the deaf-mute user, so that the deaf-mute user can correctly understand the meaning expressed by the sign language information from the outside world, thereby achieving the purpose of assisting the wearing user as a deaf-mute to communicate with the outside world and overcoming the communication barrier between the deaf-mute user and other deaf-mute users in the outside world.

[0073] Embodiment III

[0074] The present application further provides a user interaction assistance device, which is applied to an AR glasses, and the AR glasses are provided with a bone conduction earphone. The user interaction assistance device includes:

[0075] An audio pickup module, configured to pick up the audio of the wearing user of the AR glasses through the bone conduction earphone to obtain a picked-up audio;

[0076] A first interaction mode module, configured to, if the picked-up audio is a valid audio, when detecting an external sign language action, convert the sign language action into corresponding natural language information for playing to assist the wearing user to interact with the outside world;

[0077] A second interaction mode module, configured to, if the picked-up audio is not a valid audio, when detecting external interaction information, display each preset reply answer corresponding to the interaction information in the AR display screen of the AR glasses to assist the wearing user to interact with the outside world.

[0078] Optionally, the first interaction mode module is further configured to:

[0079] Perform sign language recognition on the sign language action to obtain a sign language recognition result;

[0080] Generate natural language information corresponding to the sign language action according to the sign language recognition result;

[0081] Play the natural language information to the user wearing the AR glasses.

[0082] Optionally, the natural language information includes at least one of a broadcast voice and a text display screen, and the first interaction mode module is further configured to:

[0083] Convert the sign language action into corresponding voice information according to the sign language recognition result, and generate a broadcast voice corresponding to the voice information; and / or

[0084] Convert the sign language action into corresponding text information according to the sign language recognition result, and generate a text display screen corresponding to the text information.

[0085] Optionally, the natural language information includes at least one of a broadcast voice and a text display screen, and the first interaction mode module is further configured to:

[0086] Play the broadcast voice to the user wearing the bone conduction headphones; and / or

[0087] Display the text display screen to the user wearing the AR glasses through the AR display screen of the AR glasses.

[0088] Optionally, the second interaction mode module is further configured to:

[0089] Perform sign language recognition on the sign language action to obtain a sign language recognition result;

[0090] Determine each preset reply answer corresponding to the sign language action according to the sign language recognition result;

[0091] Display each of the preset reply answers in the AR display screen.

[0092] Optionally, the user interaction assistance device is further configured to:

[0093] Receive feedback information of the user wearing the device on each of the preset reply answers;

[0094] Select a target reply answer from each of the preset reply answers according to each of the feedback information;

[0095] Broadcast the target reply answer.

[0096] Optionally, the interaction information includes voice interaction information, and the second interaction mode module is further configured to:

[0097] Perform voice recognition on the voice interaction information to obtain a voice recognition result;

[0098] Determine each preset reply answer corresponding to the voice interaction information according to the voice recognition result;

[0099] Display each of the preset reply answers in the AR display screen.

[0100] The user interaction assistance device provided by the present application adopts the user interaction assistance method in the above embodiment, and solves the technical problem that there is a communication barrier between deaf-mute users and the outside world. Compared with the prior art, the beneficial effects of the user interaction assistance device provided by the embodiments of the present application are the same as those of the user interaction assistance method provided by the above embodiment, and other technical features in the user interaction assistance device are the same as the features disclosed in the method of the above embodiment, and will not be elaborated here.

[0101] Embodiment 4

[0102] The embodiments of the present application provide an electronic device, which may be an AR glasses. The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the user interaction assistance method in Embodiment 1 above.

[0103] Next, refer to Figure 3 , which shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0104] As Figure 3 shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.), which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the electronic device are also stored. The processing device, ROM, and RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0105] Generally, the following systems can be connected to the I / O interface: input devices including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices including, for example, a magnetic tape, a hard disk, etc.; and a communication device. The communication device can allow the electronic device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows an electronic device having various systems, it should be understood that it is not required to implement or have all the systems shown. Instead, more or fewer systems can be implemented or had.

[0106] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0107] The electronic device provided by this application adopts the user interaction assistance method in the above embodiment, and solves the technical problem that there are communication barriers between deaf-mute users and the outside world. Compared with the prior art, the beneficial effects of the electronic device provided by the embodiment of this application are the same as those of the user interaction assistance method provided by the above embodiment, and other technical features in this electronic device are the same as those disclosed in the method of the above embodiment, and will not be elaborated here.

[0108] It should be understood that each part of the present disclosure can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0109] The above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by this application, and all should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0110] Embodiment Five

[0111] This embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon for executing the user interaction assistance method in the first embodiment above.

[0112] The computer-readable storage medium provided by the embodiments of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or in conjunction with an instruction execution system, device, or component. The program code contained on the computer-readable storage medium can be transmitted by any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0113] The above computer-readable storage medium may be included in an electronic device; or it may exist separately without being assembled into the electronic device.

[0114] The above computer-readable storage medium carries one or more programs. When the above one or more programs are executed by an electronic device, the electronic device: picks up the audio of the user wearing the AR glasses through the bone conduction headphones to obtain the picked-up audio; if the picked-up audio is valid audio, when a sign language action from the outside is detected, converts the sign language action into corresponding natural language information for playback to assist the wearing user in interacting with the outside; if the picked-up audio is not valid audio, when interaction information from the outside is detected, displays each preset reply answer corresponding to the interaction information in the AR display screen of the AR glasses to assist the wearing user in interacting with the outside.

[0115] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof. The above programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0117] The modules described in the embodiments of the present disclosure can be implemented in software or in hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0118] The computer-readable storage medium provided by the present application stores computer-readable program instructions for executing the above-mentioned user interaction assistance method, and solves the technical problem of communication barriers between deaf-mute users and the outside world. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the embodiments of the present application are the same as those of the user interaction assistance method provided by the above embodiments, and will not be elaborated herein.

[0119] Embodiment Six

[0120] The present application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the steps of the user interaction assistance method as described above.

[0121] The computer program product provided by the present application solves the technical problem of communication barriers between deaf-mute users and the outside world. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiments of the present application are the same as those of the user interaction assistance method provided by the above embodiments, and will not be elaborated herein.

[0122] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be included in the patent scope of the present application by the same token.

Claims

1. A user interaction assistance method, characterized in that, Applied to an AR glasses, the AR glasses are provided with bone conduction headphones, and the user interaction assistance method includes: Pick up the audio of the user wearing the AR glasses through the bone conduction headphones to obtain the picked-up audio; If the picked-up audio is valid audio, when a sign language action from the outside is detected, convert the sign language action into corresponding natural language information for playback to assist the wearing user in interacting with the outside; If the picked-up audio is not valid audio, when interaction information from the outside is detected, display each preset reply answer corresponding to the interaction information in the AR display screen of the AR glasses to assist the wearing user in interacting with the outside.

2. The user interaction assistance method according to claim 1, wherein The step of converting the sign language action into corresponding voice information for playback includes: Perform sign language recognition on the sign language action to obtain a sign language recognition result; Generate natural language information corresponding to the sign language action according to the sign language recognition result; Play the natural language information to the user wearing the AR glasses.

3. The user interaction assistance method according to claim 2, wherein The natural language information includes at least one of a broadcast voice and a text display screen. The step of generating natural language information corresponding to the sign language action according to the sign language recognition result includes: According to the sign language recognition result, convert the sign language action into corresponding voice information and generate a broadcast voice corresponding to the voice information; and / or According to the sign language recognition result, convert the sign language action into corresponding text information and generate a text display screen corresponding to the text information.

4. The user interaction assistance method according to claim 2, wherein The natural language information includes at least one of a broadcast voice and a text display screen. The step of playing the natural language information to the user wearing the AR glasses includes: Play the broadcast voice to the wearing user through the bone conduction headphones; and / or Display the text display screen to the wearing user through the AR display screen of the AR glasses.

5. The user interaction assistance method according to claim 1, wherein, The interaction information includes a sign language action. The step of displaying each preset reply answer corresponding to the interaction information in the AR display screen of the AR glasses includes: Perform sign language recognition on the sign language action to obtain a sign language recognition result; Determine each preset reply answer corresponding to the sign language action according to the sign language recognition result; Display each of the preset reply answers in the AR display screen.

6. The user interaction assistance method according to claim 5, wherein After the step of displaying each of the preset reply answers in the AR display screen, the user interaction assistance method further includes: Receive feedback information of the wearing user on each of the preset reply answers; Select a target reply answer from each of the preset reply answers according to each of the feedback information; Broadcast the target reply answer.

7. The user interaction assistance method according to claim 1, wherein The interaction information includes voice interaction information. The step of displaying each preset reply answer corresponding to the interaction information in the AR display screen of the AR glasses includes: Perform voice recognition on the voice interaction information to obtain a voice recognition result; Determine each preset reply answer corresponding to the voice interaction information according to the voice recognition result; Display each of the preset reply answers in the AR display screen.

8. A user interaction assistance device, characterized in that, Applied to an AR glasses, the AR glasses are provided with a bone conduction headset, and the user interaction assistance device includes: An audio pickup module, configured to pick up the audio of the user wearing the AR glasses through the bone conduction headset to obtain a picked-up audio; A first interaction mode module, configured to, if the picked-up audio is a valid audio, convert a sign language action detected from the outside into corresponding natural language information for playback when detecting a sign language action from the outside, so as to assist the user wearing the glasses to interact with the outside; A second interaction mode module, configured to, if the picked-up audio is not a valid audio, display each preset reply answer corresponding to the interaction information in the AR display screen of the AR glasses when detecting interaction information from the outside, so as to assist the user wearing the glasses to interact with the outside.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the steps of the user interaction assistance method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A program for implementing the user interaction assistance method is stored on the computer-readable storage medium, and the program for implementing the user interaction assistance method is executed by a processor to implement the steps of the user interaction assistance method according to any one of claims 1 to 7.

Images