Control method, device, equipment and storage medium

Abstract

The present application provides a control method, device, equipment and storage medium, which belongs to the field of augmented reality technology. The method is executed by an augmented reality AR device, and the method includes: collecting multiple types of information in the environment where the AR device is located; determining a first type and a second type that match the current working mode of the AR device, and the first type of information needs to be converted into the second type of information in the current working mode, and the current working mode is any one of the multiple working modes configured for the AR device; selecting the first type of information from the multiple types of information, and converting the first type of information into the second type of information; and controlling based on the converted second type of information. This method can meet the diverse needs of the object for the AR device.

Description

Technical Field

[0001] The present application relates to the field of augmented reality, and in particular to a control method, device, equipment and storage medium. Background Art

[0002] With the widespread application of smart devices, augmented reality (AR) devices have become a hot topic in current research. In recent years, a variety of AR devices have been produced, and different AR devices have different functions. However, there is still a lack of AR devices that can meet diverse needs in related technologies. Summary of the invention

[0003] The embodiments of the present application provide a control method, apparatus, device, and storage medium that can meet the diverse needs of subjects for AR devices. The technical solution is as follows:

[0004] According to one aspect of an embodiment of the present application, a control method is provided, which is performed by an augmented reality (AR) device, and the method includes:

[0005] Collecting various types of information about the environment in which the AR device is located;

[0006] Determine a first type and a second type that match a current working mode of the AR device, in which information of the first type needs to be converted into information of the second type, and the current working mode is any one of multiple working modes configured for the AR device;

[0007] Selecting the first type of information from the multiple types of information, and converting the first type of information into the second type of information;

[0008] Control is performed based on the converted second type of information.

[0009] In a possible implementation manner, determining the first type and the second type matching the current working mode of the AR device includes:

[0010] From the first matching relationship, searching for the first type and the second type that match the current working mode;

[0011] The first matching relationship includes a first type and a second type that match each working mode.

[0012] In a possible implementation, the method further includes:

[0013] Collecting object information, where the object information is used to describe an obstacle type of an object using the AR device;

[0014] From the second matching relationship, searching for a working mode that matches the obstacle type, and determining the found working mode as the current working mode;

[0015] The second matching relationship includes a working mode that matches each obstacle type.

[0016] In a possible implementation, the method further includes:

[0017] When an operating mode switching operation is detected, the current operating mode is switched to the operating mode indicated by the operating mode switching operation.

[0018] In a possible implementation, the working mode switching operation includes at least one of the following:

[0019] Trigger operation on the switch button;

[0020] Recognize that the voice data contains a switching keyword operation;

[0021] The target gesture operation is collected.

[0022] In one possible implementation,

[0023] In the hearing-impaired mode, the first type of information is voice information, and the second type of information is text information; or,

[0024] In the visual impairment mode, the first type of information is image information, and the second type of information is voice information; or,

[0025] In the speech impairment mode, the first type of information is sign language information, and the second type of information is voice information or text information; or

[0026] In the dialect translation mode, the first type of information is voice information containing a first language, the second type of information is voice information or text information containing a second language, and the first language is a branch language of the second language; or

[0027] In the language translation mode, the first type of information is voice information in a first language, and the second type of information is voice information or text information in a second language; or

[0028] In the external device control mode, the first type of information is brain wave information, voice information or image information, and the second type of information is external device control information.

[0029] In a possible implementation manner, the controlling based on the second type of information obtained by conversion includes:

[0030] In the hearing-impaired mode, displaying the converted text information; or,

[0031] In the visual impairment mode, playing the converted voice information; or,

[0032] In the speech barrier mode, playing the converted voice information, or displaying the converted text information; or,

[0033] In the dialect translation mode, the voice information obtained by conversion is played, or the text information obtained by conversion is displayed; or,

[0034] In the language translation mode, the converted voice information is played, or the converted text information is displayed; or,

[0035] In the external device control mode, the external device connected to the AR device is controlled based on the external device control information.

[0036] According to another aspect of an embodiment of the present application, a control device is provided, the device comprising:

[0037] A collection module, used to collect various types of information in the environment where the AR device is located;

[0038] a determination module, configured to determine a first type and a second type matching a current working mode of the AR device, wherein information of the first type needs to be converted into information of the second type in the current working mode, and the current working mode is any one of multiple working modes configured for the AR device;

[0039] a conversion module, configured to select the first type of information from the multiple types of information, and convert the first type of information into the second type of information;

[0040] A control module is used to perform control based on the second type of information obtained by conversion.

[0041] In a possible implementation manner, the determining module is used to:

[0042] From the first matching relationship, the first type and the second type matching the current working mode are searched; wherein the first matching relationship includes the first type and the second type matching each working mode.

[0043] In a possible implementation manner, the device further includes:

[0044] An object information acquisition module, used to acquire object information, where the object information is used to describe an obstacle type of an object using the AR device;

[0045] The working mode determination module is used to search for a working mode matching the obstacle type from the second matching relationship, and determine the found working mode as the current working mode; wherein the second matching relationship includes a working mode matching each obstacle type.

[0046] In a possible implementation manner, the device further includes:

[0047] The working mode switching module switches the current working mode to the working mode indicated by the working mode switching operation when a working mode switching operation is detected.

[0048] In a possible implementation, the working mode switching operation includes at least one of the following:

[0049] Trigger operation on the switch button;

[0050] Recognize that the voice data contains a switching keyword operation;

[0051] The target gesture operation is collected.

[0052] In one possible implementation,

[0053] In the hearing-impaired mode, the first type of information is voice information, and the second type of information is text information; or,

[0054] In the visual impairment mode, the first type of information is image information, and the second type of information is voice information; or,

[0055] In the speech impairment mode, the first type of information is sign language information, and the second type of information is voice information or text information; or

[0056] In the dialect translation mode, the first type of information is voice information containing a first language, the second type of information is voice information or text information containing a second language, and the first language is a branch language of the second language; or

[0057] In the language translation mode, the first type of information is voice information in a first language, and the second type of information is voice information or text information in a second language; or

[0058] In the external device control mode, the first type of information is brain wave information, voice information or image information, and the second type of information is external device control information.

[0059] In a possible implementation, the control module is used to:

[0060] In the hearing-impaired mode, displaying the converted text information; or,

[0061] In the visual impairment mode, playing the converted voice information; or,

[0062] In the speech barrier mode, playing the converted voice information, or displaying the converted text information; or,

[0063] In the dialect translation mode, the voice information obtained by conversion is played, or the text information obtained by conversion is displayed; or,

[0064] In the language translation mode, the converted voice information is played, or the converted text information is displayed; or,

[0065] In the external device control mode, the external device connected to the AR device is controlled based on the external device control information.

[0066] According to another aspect of an embodiment of the present application, an AR device is provided, wherein the terminal includes a processor and a memory, wherein at least one program code is stored in the memory, and the at least one program code is loaded and executed by the processor to implement the control method described in any possible implementation manner described above.

[0067] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the control method described in any possible implementation manner described above.

[0068] According to another aspect of an embodiment of the present application, a computer program product is provided, the computer program product comprising a computer program code, the computer program code being stored in a computer-readable storage medium, a processor reading the computer program code from the computer-readable storage medium, and the processor executing the computer program code to implement the control method as described in any one of the possible implementation modes described above.

[0069] In the embodiment of the present application, the AR device is configured with multiple working modes, and can also collect multiple types of information in the environment, providing diverse information support for the multiple working modes. When currently in any working mode, the first type of information in the environment can be converted into the second type of information, so as to perform control based on the second type of information. Then, with the support of the multiple working modes configured and the multiple types of information collected, different controls can be performed to meet the diverse needs of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0071] Figure 1 is a flow chart of a control method provided by an embodiment of the present application;

[0072] Figure 2 is a flow chart of another control method provided by an embodiment of the present application;

[0073] Figure 3 This is a schematic diagram of a hearing-impaired person communicating with a friend provided by an embodiment of the present application;

[0074] Figure 4 This is a schematic diagram of a visually impaired person communicating with a friend provided by an embodiment of the present application;

[0075] Figure 5 This is a schematic diagram of a speech-impaired person communicating with a friend provided by an embodiment of the present application;

[0076] Figure 6 is a schematic diagram of a function used by an AR device in different working modes provided by an embodiment of the present application;

[0077] Figure 7 is a structural block diagram of a control device provided in an embodiment of the present application;

[0078] Figure 8 It is a structural block diagram of an AR device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0080] It can be understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept.

[0081] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0082] It should be noted that the object information involved in this application (including but not limited to the object's personal information, object device information, etc.) is all information authorized by the object or fully authorized by all parties.

[0083] Figure 1 is a flow chart of a control method provided in an embodiment of the present application. The method is executed by an augmented reality AR device, see Figure 1 , the method comprises the following steps:

[0084] 101. The AR device collects various types of information from the environment in which the AR device is located.

[0085] Among them, AR devices include AR glasses, head-mounted AR displays, AR helmets, etc. The AR devices can display augmented reality images, which can include both real scene images and virtual elements, thereby integrating virtual elements with real scene images, bringing better visual experience to the object wearing the AR device.

[0086] Among them, the multiple types of information at least include but are not limited to at least one of image information, voice information, body movement information, text information, etc. The AR device in the embodiment of the present application has multiple types of acquisition units, which can respectively collect multiple types of information through the multiple types of acquisition units. The acquisition unit may at least include but is not limited to at least one of a camera, a microphone array, a millimeter wave radar, a 3D structured light camera, a multi-axis acceleration sensor, etc.

[0087] 102. The AR device determines a first type and a second type that match a current working mode of the AR device.

[0088] The current working mode is any one of the multiple working modes configured for the AR device, and the first type and the second type matched by different working modes may be the same or different.

[0089] The current working mode matches the first type and the second type, indicating that in the current working mode, the AR device needs to convert the first type of information into the second type of information. The first type is the type of information that needs to be processed among the multiple types of information collected by the AR device, and the second type is the type of information converted by the AR device. For example, for different application scenarios or different user groups, different types of information need to be collected and different types of information need to be output. Therefore, the appropriate working mode can be determined based on the current application scenario or user group, and the AR device can be controlled in this working mode.

[0090] Furthermore, the meaning of the second type of information is the same as that of the first type of information, but the type is different. In this way, the user who receives the second type of information can understand the meaning of the second type of information, that is, understand the meaning of the first type of information, and thus know the content that the user who outputs the first type of information needs to express, so that the user who receives the second type of information can communicate with the user who outputs the first type of information without any obstacles.

[0091] Among them, the multiple types of information collected by the AR device may include not only the first type of information, but also the second type of information. However, the collected second type of information may have a different meaning from the second type of information converted based on the first type of information. For example, when a user is standing next to a bus stop and talking, the AR device collects voice information and text information. The voice information includes the user's speech content, and the text information includes the text on the bus stop. At this time, the voice information is converted into text information, and the converted text information includes the user's speech content, but not the text on the bus stop.

[0092] Optionally, since the AR device is configured with multiple working modes, the first matching relationship includes the first type and the second type that match each working mode, so step 102 includes: the AR device searches for the first type and the second type that match the current working mode from the first matching relationship; for example, if the working mode in the first matching relationship is the hearing impairment mode, the first type that matches can be a voice type, and the second type can be a text type. In addition, the first matching relationship can be set on the AR device by a technician, or set on a server by a technician and downloaded from the server by the AR device, or can also be set by the object. In this way, the AR device can accurately and quickly find the first type and the second type that match the current working mode.

[0093] 103. The AR device selects first type of information from multiple types of information and converts the first type of information into second type of information.

[0094] The AR device converts the first type of information into the second type of information through an artificial intelligence (AI) function, and the AI ​​function includes at least but is not limited to at least one of speech recognition, motion recognition, object recognition, sign language recognition, speech synthesis, text content recognition, etc. The AR device can determine which AI function to use based on the first type and the second type that match the current working mode, and convert the first type of information into the second type of information.

[0095] For example, the first type of information is voice information, and the second type of information is text information. The AR device needs to convert the voice into text information, and the AR device converts the voice information into text information through a voice recognition function.

[0096] 104. The AR device is controlled based on the converted second type of information.

[0097] In an embodiment of the present application, the AR device performs a corresponding operation based on the second type of information obtained by conversion. Optionally, the AR device outputs the second type of information. For example, if the second type of information is text information, the AR device displays the text information; if the second type of information is voice information, the AR device plays the voice information. Alternatively, the AR device can also control other devices to perform corresponding operations based on the second type of information obtained by conversion, such as controlling an external device connected to the AR device to output the second type of information.

[0098] In the embodiment of the present application, the AR device is configured with multiple working modes, and can also collect multiple types of information in the environment, providing diverse information support for the multiple working modes. When currently in any working mode, the first type of information in the environment can be converted into the second type of information, so as to perform control based on the second type of information. Then, with the support of the multiple working modes configured and the multiple types of information collected, different controls can be performed to meet the diverse needs of the object.

[0099] Figure 2 is a flow chart of a control method provided in an embodiment of the present application. The method is executed by an AR device, see Figure 2 , the method comprises the following steps:

[0100] 201. The AR device collects object information.

[0101] Among them, the object information is used to describe the obstacle type of the object using the AR device. The obstacle type indicates the type of part of the object where the obstacle exists, indicating which part of the object cannot receive or output information normally. In addition, the object information can also describe other aspects of the object, such as the object's login account, usage habits, etc.

[0102] Optionally, the AR device obtains the object information by outputting voice information and text information and detecting the response of the object. For example, the AR device plays a piece of music, displays the text information "Can you hear the music?" and plays the voice information corresponding to the text information, and the AR device detects that the meaning of the object's response is "no", then it is determined that the object cannot hear the sound, and the obstacle type of the object is a hearing impairment type.

[0103] In the embodiment of the present application, multiple types of obstacles can be set. For example, if the subject has hearing impairment, the obstacle type is the hearing impairment type; if the subject has visual impairment, the obstacle type is the visual impairment type; if the subject has pronunciation impairment, the obstacle type is the speech impairment type. In this way, the AR device can more comprehensively understand the subject's situation by obtaining the subject's obstacle type, so as to provide the subject with a more suitable working mode and facilitate the subject to use the AR device smoothly.

[0104] 202. The AR device searches for a working mode that matches the obstacle type from the second matching relationship, and determines the found working mode as the current working mode.

[0105] Among them, the second matching relationship includes a working mode that matches each obstacle type. For example, if the obstacle type in the second matching relationship is a hearing impairment type, the matching working mode is a hearing impairment mode. Among them, the second matching relationship may include multiple obstacle types, each obstacle type matches one or more working modes, and the working modes matching different obstacle types may be the same or different. In addition, the second matching relationship can be set by a technician on the AR device, or by a technician on the server, and downloaded from the server by the AR device, or can also be set by the object.

[0106] In an embodiment of the present application, the AR device searches for a matching working mode based on the obstacle type of the object, and determines the found working mode as the current working mode, which simplifies the operational process of determining the current working mode and provides great convenience for the object.

[0107] Optionally, when the AR device finds a working mode that matches the obstacle type from the second matching relationship, the found working mode is determined as the current working mode. When the AR device finds multiple working modes that match the obstacle type from the second matching relationship, the multiple working modes are provided to the object for the object to select the working mode to be currently used from the multiple working modes, or select the working mode that was used last from the multiple working modes, or select the working mode that is used most times under the same obstacle type from the multiple working modes.

[0108] Among them, when providing the multiple working modes for the object, the provision method needs to match the obstacle type of the object. For example, if the obstacle type is a hearing impairment type, the voice information containing the multiple working modes cannot be played, but the selection buttons corresponding to the multiple working modes are displayed for the object to choose.

[0109] 203. The AR device collects various types of information in the environment where the AR device is located.

[0110] The specific implementation method of step 203 is the same as that of the above step 101, which will not be repeated here.

[0111] 204. The AR device determines a first type and a second type that match a current working mode of the AR device.

[0112] The current working mode is any one of the multiple working modes configured for the AR device, and the first type and the second type matched by different working modes may be the same or different.

[0113] The specific implementation method of step 204 is the same as that of the above step 102. The description of the first type, the second type, and step 204 can refer to step 102 and will not be repeated here.

[0114] 205. The AR device selects first type of information from multiple types of information and converts the first type of information into second type of information.

[0115] The specific implementation method of step 205 is the same as that of the above step 103, which will not be repeated here.

[0116] 206. The AR device performs control based on the converted second type of information.

[0117] The specific implementation method of step 206 is the same as that of the above step 104, which will not be repeated here.

[0118] It should be noted that the above embodiment describes the control process of the AR device in the current working mode, and the working mode can also be switched during specific use. Optionally, the method also includes: when the AR device detects the working mode switching operation, switching the current working mode to the working mode indicated by the working mode switching operation.

[0119] The object can perform the working mode switching operation according to its own usage needs, thereby choosing which working mode to switch to, making it convenient for the object to use the AR device to select the working mode, thereby improving the flexibility of the object in selecting the working mode.

[0120] Optionally, the working mode switching operation includes at least one of the following:

[0121] (1) Triggering operation of the switch button;

[0122] Among them, the AR device is provided with a switch button, which is used to switch the working mode. In the current working mode, the object performs a trigger operation on the switch button, such as a click operation, a long press operation, etc., and the AR device determines that the working mode switching operation is detected, and therefore switches the current working mode to the next working mode.

[0123] The AR device is configured with multiple working modes, and the multiple working modes are arranged in sequence. Then, the next working mode of the current working mode can be determined based on the sequence of the multiple working modes.

[0124] (2) recognizing an operation containing a switching keyword in the voice information;

[0125] Optionally, the AR device sets a switching keyword, which is used to indicate switching the current working mode to the next working mode. When the object speaks the switching keyword, the AR device collects voice information and recognizes that the voice information contains the switching keyword. It is determined that a working mode switching operation is detected, and therefore the current working mode is switched to the next working mode.

[0126] Alternatively, in order to improve flexibility, the AR device sets multiple switching keywords, and different switching keywords correspond to different working modes. The object speaks the corresponding switching keyword based on the working mode he wants to switch to. The AR device will collect voice information and recognize that the voice information contains the switching keyword, and then determine that the working mode switching operation is detected, so the current working mode is switched to the working mode indicated by the switching keyword. For example, if the switching keyword is "control external devices", the AR device switches the current working mode to the external device control mode.

[0127] (3) The operation of collecting the target gesture.

[0128] Optionally, the AR device sets a target gesture, which is used to indicate switching the current working mode to the next working mode. When the object makes the target gesture, the AR device will collect image information and recognize that the image information contains the target gesture. It is determined that the working mode switching operation is detected, and therefore the current working mode is switched to the next working mode.

[0129] Alternatively, in order to improve flexibility, the AR device sets multiple target gestures, and different target gestures correspond to different working modes. The object then makes corresponding target gestures based on the working mode he wants to switch to. The AR device will collect image information and recognize that the image information contains the target gesture. It will then determine that a working mode switching operation has been detected, and therefore, the current working mode will be switched to the working mode indicated by the target gesture.

[0130] Among them, one or more target gestures set by the AR device can be specific actions or specific sign language actions. The AR device can set the type of target gesture based on the obstacle type of the object. For example, if the obstacle type of the object is a speech obstacle type, the AR device sets the target gesture to a sign language action.

[0131] Optionally, the working mode switching operation includes at least two of the above three operations. When the subject wears the AR device, no matter which working mode switching operation is performed, the AR device can be controlled, providing the subject with multiple options.

[0132] In the embodiment of the present application, the AR device is configured with multiple working modes, and can also collect multiple types of information in the environment, providing diverse information support for the multiple working modes. When currently in any working mode, the first type of information in the environment can be converted into the second type of information, so as to perform control based on the second type of information. Then, with the support of the multiple working modes configured and the multiple types of information collected, different controls can be performed to meet the diverse needs of the object.

[0133] In addition, the AR device collects object information and obtains the obstacle type of the object, thereby finding a matching working mode based on the obstacle type of the object and setting the working mode as the current working mode. Then, by obtaining the obstacle type of the object, the AR device can have a more comprehensive understanding of the situation of the object, so as to provide a more suitable working mode for the object and facilitate the object to use the AR device smoothly.

[0134] Furthermore, when the AR device detects the working mode switching operation, it switches the current working mode to the working mode indicated by the working mode switching operation. In this way, the subject can choose which working mode to switch to according to his / her own usage needs, which improves the flexibility of the subject in selecting the working mode using the AR device. The diversity of the working mode switching operation meets the needs of the subject and provides the subject with multiple choices.

[0135] Based on the above embodiments, the processing flow and usage scenarios of the AR device in different working modes are illustrated below.

[0136] Optionally, in the hearing impairment mode, the above Figure 2 Steps 203 to 206 in the illustrated embodiment may be replaced by:

[0137] When the current working mode of the AR device is the hearing impairment mode, the AR device collects multiple types of information in the environment where the AR device is located, and the AR device determines that the first type matching the hearing impairment mode is the voice type, and the second type is the text type; the AR device selects voice information from the collected multiple types of information, converts the voice information into text information, and displays the converted text information.

[0138] For example, see Figure 3In the scenario where a hearing-impaired person communicates with a friend, by wearing an AR device, if the friend speaks, the AR device will collect the friend's voice information, and then display text information containing the friend's speech content. In this way, the hearing-impaired person can see what the friend said in the form of text, and thus communicate with the friend. Compared with the hearing-impaired person communicating with friends through sign language or text, communicating by wearing an AR device can reduce the difficulty of communication, and also realize real-time communication between the hearing-impaired person and friends, bringing great convenience to the lives of the hearing-impaired.

[0139] Optionally, in visual impairment mode, the above Figure 2 Steps 203 to 206 in the illustrated embodiment may be replaced by the following:

[0140] When the current working mode of the AR device is the visual impairment mode, the AR device collects multiple types of information in the environment where the AR device is located, and the AR device determines that the first type matching the visual impairment mode is the image type, and the second type is the voice type; then the AR device selects image information from the multiple types of information collected and converts it into voice information, and the AR device plays the converted voice information.

[0141] For example, see Figure 4 In the scenario where a visually impaired person communicates with a friend, the AR device collects image information by wearing the AR device, which includes the friend's appearance, height, gender, movements, etc., and converts the image information into voice information. By playing the voice information, the visually impaired person can fully understand the friend's information, thereby improving the communication efficiency between the visually impaired person and the friend. The AR device transmits the converted voice information to the visually impaired person through bone conduction, which can avoid interference caused by external noise and ensure that the visually impaired person obtains information accurately and timely.

[0142] Moreover, when a visually impaired person is traveling, the AR device collects image information, including road information and surrounding environment information at the location of the visually impaired person, by wearing the AR device. The AR device converts the image information into voice information and broadcasts the voice information in real time for the visually impaired person to listen to. In this way, the visually impaired person can understand the specific information of their location, making travel safer and more convenient for the visually impaired person.

[0143] Optionally, in speech impairment mode, the above Figure 2 Steps 203 to 206 in the illustrated embodiment may be replaced by the following:

[0144] When the current working mode of the AR device is the speech disorder mode, the AR device collects multiple types of information in the environment where the AR device is located, and the AR device determines that the first type matching the speech disorder mode is the sign language type, and the second type is the voice type or the text type; then the AR device selects the sign language information from the collected multiple types of information, and converts it into voice information or text information, and the AR device plays the converted voice information, or displays the converted text information.

[0145] For example, see Figure 5 In the scenario where a person with speech impairment communicates with a friend, by wearing an AR device, if the person with speech impairment makes a sign language gesture, the AR device collects the sign language information of the person with speech impairment, converts the sign language information into voice information, and plays the voice information. The friend can understand the meaning of the sign language gesture of the person with speech impairment, realizing barrier-free communication between the two. Moreover, the friend can communicate with the person with speech impairment without learning sign language, which reduces the difficulty of communication and brings great convenience to the life of the person with speech impairment. Figure 6 AR devices can also collect lip reading information from speech-impaired people and convert it into language information through lip reading recognition, which provides great convenience for speech-impaired people who do not have sign language skills.

[0146] Optionally, in dialect translation mode, the above Figure 2 Steps 203 to 206 in the illustrated embodiment may be replaced by the following:

[0147] When the current working mode of the AR device is the dialect translation mode, the AR device collects multiple types of information in the environment where the AR device is located, and the AR device determines that the first type matching the dialect translation mode is a voice type containing a first language, and the second type is a voice type or a text type containing a second language; then the AR device selects voice information containing the first language from the multiple types of information collected, and converts the voice information containing the first language into voice information or text information containing the second language, plays the voice information containing the second language, or displays the text information containing the second language.

[0148] The second language is one of the official languages, and the first language is one of the local languages ​​that is a variation of the official language. For example, the first language is Sichuanese and the second language is Mandarin. Figure 6 If the current first language is Sichuan dialect and the second language is Mandarin, the AR device will convert the voice information containing Sichuan dialect into voice information containing Mandarin through the dialect recognition function, which can realize barrier-free communication between people who speak the local language and people who speak the official language. It is of great help to some staff who often communicate with people who do not have the ability to speak the official language.

[0149] Optionally, in language translation mode, the above Figure 2 Steps 203 to 206 in the illustrated embodiment may be replaced by the following:

[0150] When the current working mode of the AR device is the language translation mode, the AR device collects multiple types of information in the environment where the AR device is located, and the AR device determines that the first type matching the language translation mode is a voice type belonging to the first language, and the second type is a voice type or a text type belonging to the second language; then the AR device selects the voice information belonging to the first language from the collected multiple types of information, converts it into voice information or text information belonging to the second language, plays the voice information belonging to the second language, or displays the text information belonging to the second language.

[0151] The first language and the second language are two different languages, representing two completely different language systems. For example, the first language is English and the second language is Chinese. Figure 6 , AR devices use voice recognition function to convert voice information in the first language into voice information or text information in the second language, which can achieve barrier-free communication between people from different countries.

[0152] Optionally, in external device control mode, the above Figure 2 Steps 203 to 206 in the illustrated embodiment may be replaced by the following:

[0153] When the current working mode of the AR device is the external device control mode, the AR device collects multiple types of information in the environment where the AR device is located, and the AR device determines that the first type that matches the external device control mode is the brain wave type, voice type or image type, and the second type is the external device control type; the AR device selects brain wave information, voice information or image information from the multiple types of information collected, and converts it into external device control information, and controls the external device connected to the AR device based on the external device control information.

[0154] Among them, the external device can be a wheelchair, a prosthetic manipulator, etc.

[0155] Optionally, the AR device collects brain wave information of the physically disabled person and converts it into external device control information, or the AR device collects voice information of the physically disabled person, determines the content contained in the voice information through voice recognition, and converts the content into external device control information, or the AR device collects image information of the physically disabled person, determines the gesture contained in the image information through image recognition, and converts the gesture into external device control information, etc. In this way, by wearing the AR device, the wheelchair can be moved and rotated without manual operation by the physically disabled person, which greatly facilitates the travel of the physically disabled person and improves the efficiency of the movement of the physically disabled person.

[0156] Of course, the first type of information may also be other types of information, which will not be described in detail in the embodiments of the present application.

[0157] Figure 7 is a structural block diagram of a control device provided in an embodiment of the present application. Figure 7 , the device comprises:

[0158] A collection module 701 is used to collect various types of information in the environment where the AR device is located;

[0159] A determination module 702 is used to determine a first type and a second type that match a current working mode of the AR device, where the first type of information needs to be converted into the second type of information in the current working mode, and the current working mode is any one of multiple working modes configured for the AR device;

[0160] A conversion module 703, configured to select a first type of information from multiple types of information and convert the first type of information into a second type of information;

[0161] The control module 704 is used to perform control based on the second type of information obtained by conversion.

[0162] In a possible implementation, the determining module 702 is used to:

[0163] From the first matching relationship, the first type and the second type matching the current working mode are searched; wherein the first matching relationship includes the first type and the second type matching each working mode.

[0164] In a possible implementation, the device further includes:

[0165] An object information acquisition module, used to acquire object information, where the object information is used to describe the obstacle type of the object using the AR device;

[0166] The working mode determination module is used to search for a working mode that matches the obstacle type from the second matching relationship, and determine the found working mode as the current working mode; wherein the second matching relationship includes a working mode that matches each obstacle type.

[0167] In a possible implementation, the device further includes:

[0168] The working mode switching module is used to switch the current working mode to the working mode indicated by the working mode switching operation when a working mode switching operation is detected.

[0169] In a possible implementation, the working mode switching operation includes at least one of the following:

[0170] Trigger operation on the switch button;

[0171] Recognize that the voice data contains a switching keyword operation;

[0172] The target gesture operation is collected.

[0173] In one possible implementation,

[0174] In the hearing-impaired mode, the first type of information is voice information, and the second type of information is text information; or,

[0175] In the visual impairment mode, the first type of information is image information, and the second type of information is voice information; or,

[0176] In the speech impairment mode, the first type of information is sign language information, and the second type of information is voice information or text information; or,

[0177] In the dialect translation mode, the first type of information is voice information containing a first language, the second type of information is voice information or text information containing a second language, and the first language is a branch language of the second language; or

[0178] In the language translation mode, the first type of information is voice information in a first language, and the second type of information is voice information or text information in a second language; or,

[0179] In the external device control mode, the first type of information is brain wave information, voice information or image information, and the second type of information is external device control information.

[0180] In a possible implementation, the control module 704 is used to:

[0181] In hearing-impaired mode, display the converted text information; or,

[0182] In the visual impairment mode, play the converted voice information; or,

[0183] In the speech impairment mode, the converted voice information is played, or the converted text information is displayed; or,

[0184] In the dialect translation mode, the converted voice information is played, or the converted text information is displayed; or,

[0185] In the language translation mode, the converted voice information is played, or the converted text information is displayed; or,

[0186] In the external device control mode, the external device connected to the AR device is controlled based on the external device control information.

[0187] In the embodiment of the present application, the AR device is configured with multiple working modes, and can also collect multiple types of information in the environment, providing diverse information support for the multiple working modes. When currently in any working mode, the first type of information in the environment can be converted into the second type of information, so as to perform control based on the second type of information. Then, with the support of the multiple working modes configured and the multiple types of information collected, different controls can be performed to meet the diverse needs of the object.

[0188] Regarding the control device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the relevant method, and will not be elaborated here.

[0189] An embodiment of the present application provides an AR device, which includes a processor and a memory. The memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the control method in the above embodiment.

[0190] Figure 8 800 is a structural block diagram of an AR device 800 provided according to an embodiment of the present application. The AR device 800 may be: AR glasses, a head-mounted AR display, or an AR helmet.

[0191] The AR device 800 includes a processor 801 and a memory 802 .

[0192] The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0193] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one program code, which is used to be executed by the processor 801 to implement the control method provided in the method embodiment of the present application.

[0194] In some embodiments, the AR device 800 may further optionally include: a peripheral device interface 803 and at least one peripheral device. The processor 801, the memory 802 and the peripheral device interface 803 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 803 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, a positioning assembly 808 and a power supply 809.

[0195] The peripheral device interface 803 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 801 and the memory 802. In some embodiments, the processor 801, the memory 802, and the peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 801, the memory 802, and the peripheral device interface 803 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0196] The radio frequency circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 804 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, an object identity module card, and the like. The radio frequency circuit 804 can communicate with other AR devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 804 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0197] The display screen 805 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 805 is a touch display screen, the display screen 805 also has the ability to collect touch signals on the surface or above the surface of the display screen 805. The touch signal can be input to the processor 801 as a control signal for processing. At this time, the display screen 805 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 805 can be one, and the front panel of the AR device 800 is set; in other embodiments, the display screen 805 can be at least two, which are respectively set on different surfaces of the AR device 800 or are folded; in other embodiments, the display screen 805 can be a flexible display screen, which is set on the curved surface or folded surface of the AR device 800. Even, the display screen 805 can also be set as a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 805 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0198] The camera assembly 806 is used to capture images or videos. Optionally, the camera assembly 806 includes a front camera and a rear camera. The front camera is arranged on the front panel of the AR device, and the rear camera is arranged on the back of the AR device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0199] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the object and the environment, and convert the sound waves into electrical signals and input them into the processor 801 for processing, or input them into the RF circuit 804 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged in different parts of the AR device 800. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert electrical signals from the processor 801 or the RF circuit 804 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 807 may also include a headphone jack.

[0200] The positioning component 808 is used to locate the current geographic location of the AR device 800 to implement navigation or LBS (Location Based Service). The positioning component 808 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system, or the European Union's Galileo system.

[0201] The power supply 809 is used to power various components in the AR device 800. The power supply 809 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 809 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0202] In some embodiments, the AR device 800 further includes one or more sensors 810 . The one or more sensors 810 include, but are not limited to, an acceleration sensor 811 , a gyroscope sensor 812 , a pressure sensor 813 , an optical sensor 814 , and a proximity sensor 815 .

[0203] The acceleration sensor 811 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the AR device 800. For example, the acceleration sensor 811 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 801 can control the display screen 805 to display the object interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 811. The acceleration sensor 811 can also be used to collect motion data of games or objects.

[0204] The gyro sensor 812 can detect the body direction and rotation angle of the AR device 800. The gyro sensor 812 can cooperate with the acceleration sensor 811 to collect the 3D action of the object on the AR device 800. The processor 801 can implement the following functions based on the data collected by the gyro sensor 812: motion sensing (such as changing the UI according to the tilt operation of the object), image stabilization during shooting, game control, and inertial navigation.

[0205] The pressure sensor 813 can be set on the side frame of the AR device 800 and / or the lower layer of the display screen 805. When the pressure sensor 813 is set on the side frame of the AR device 800, it can detect the object's holding signal of the AR device 800, and the processor 801 performs left and right hand recognition or shortcut operations according to the holding signal collected by the pressure sensor 813. When the pressure sensor 813 is set on the lower layer of the display screen 805, the processor 801 controls the operability controls on the UI interface according to the pressure operation of the object on the display screen 805. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0206] The optical sensor 814 is used to collect the ambient light intensity. In one embodiment, the processor 801 can control the display brightness of the display screen 805 according to the ambient light intensity collected by the optical sensor 814. Specifically, when the ambient light intensity is high, the display brightness of the display screen 805 is increased; when the ambient light intensity is low, the display brightness of the display screen 805 is reduced. In another embodiment, the processor 801 can also dynamically adjust the shooting parameters of the camera assembly 806 according to the ambient light intensity collected by the optical sensor 814.

[0207] The proximity sensor 815, also called a distance sensor, is disposed on the front panel of the AR device 800. The proximity sensor 815 is used to collect the distance between the object and the front of the AR device 800. In one embodiment, when the proximity sensor 815 detects that the distance between the object and the front of the AR device 800 is gradually decreasing, the processor 801 controls the display screen 805 to switch from the screen-on state to the screen-off state; when the proximity sensor 815 detects that the distance between the object and the front of the AR device 800 is gradually increasing, the processor 801 controls the display screen 805 to switch from the screen-off state to the screen-on state.

[0208] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the AR device 800, and may include more or less components than those shown in the figure, or combine certain components, or adopt a different component arrangement.

[0209] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one program code is stored, and the at least one program code is loaded and executed by a processor to implement the control method in the above embodiment. The computer-readable storage medium can be a memory. For example, the computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage terminal, etc.

[0210] In an exemplary embodiment, a computer program product is also provided, which includes a computer program code, the computer program code is stored in a computer-readable storage medium, a processor reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code to implement the control method in the above embodiment.

[0211] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0212] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A control method, characterized in that: The method is performed by an augmented reality (AR) device and includes: From the second matching relationship, searching for an operating mode that matches the obstacle type of the object using the AR device; If a working mode matching the obstacle type is found from the second matching relationship, the found working mode is determined as the current working mode; if multiple working modes matching the obstacle type are found from the second matching relationship, the multiple working modes found are provided to the object in a providing manner matching the obstacle type, so that the object can select the working mode to be currently used from the multiple working modes found, or select the working mode used last time from the multiple working modes found, or select the working mode with the most number of uses under the same obstacle type from the multiple working modes found; Collecting multiple types of information in the environment where the AR device is located, wherein the multiple types of information include at least one of image information, body movement information, and text information; Searching, from the first matching relationship, for a first type and a second type that match a current working mode of the AR device, where information of the first type needs to be converted into information of the second type in the current working mode, and the current working mode is any one of multiple working modes configured for the AR device, and the first matching relationship includes the first type and the second type that match each working mode; Based on the first type and the second type that match the current working mode, determine the artificial intelligence function to be used, select the first type of information from the multiple types of information, and convert the first type of information into the second type of information through the determined artificial intelligence function, wherein the artificial intelligence function includes at least one of speech recognition, action recognition, object recognition, sign language recognition, speech synthesis, and text content recognition; perform control based on the converted second type of information; when the current working mode is the speech impairment mode, the first type of information includes the sign language information or lip reading information of the object, and the second type of information includes voice information or text information, and the control refers to playing the voice information or displaying the text information on the AR device for acquisition by other users except the object; During the process of the subject using the AR device, the method further includes at least one of the following: Collecting voice information output by the object, if recognizing that the voice information contains a switching keyword, determining that a working mode switching operation is detected, switching the current working mode to the working mode indicated by the switching keyword, different switching keywords corresponding to different working modes; Image information of the object is collected. If a target gesture is identified in the image information, it is determined that a working mode switching operation is detected, and the current working mode is switched to the working mode indicated by the target gesture. Different target gestures correspond to different working modes.

2. The method according to claim 1, characterized in that: The method further comprises: Collecting object information, where the object information is used to describe an obstacle type of an object using the AR device; From the second matching relationship, a working mode matching the obstacle type is searched, and the found working mode is determined as the current working mode.

3. The method according to claim 1, characterized in that: The working mode switching operation also includes a triggering operation on a switching button.

4. The method according to claim 1, characterized in that: In the hearing-impaired mode, the first type of information is voice information, and the second type of information is text information; or, In the visual impairment mode, the first type of information is image information, and the second type of information is voice information; or, In the dialect translation mode, the first type of information is voice information containing a first language, the second type of information is voice information or text information containing a second language, and the first language is a branch language of the second language; or, In the language translation mode, the first type of information is voice information in a first language, and the second type of information is voice information or text information in a second language; or, In the external device control mode, the first type of information is brain wave information, voice information or image information, and the second type of information is external device control information.

5. The method according to claim 4, characterized in that The controlling based on the second type of information obtained by conversion includes: In the hearing-impaired mode, displaying the converted text information; or, In the visual impairment mode, playing the converted voice information; or, In the dialect translation mode, the voice information obtained by conversion is played, or the text information obtained by conversion is displayed; or, In the language translation mode, the converted voice information is played, or the converted text information is displayed; or, In the external device control mode, the external device connected to the AR device is controlled based on the external device control information.

6. A control device, characterized in that: The device comprises: a working mode determination module, configured to search for a working mode matching the obstacle type of the object using the AR device from the second matching relationship; if a working mode matching the obstacle type is found from the second matching relationship, the found working mode is determined as the current working mode; if multiple working modes matching the obstacle type are found from the second matching relationship, the multiple working modes found are provided to the object in a providing manner matching the obstacle type, so that the object can select a working mode to be currently used from the multiple working modes found, or select a working mode that was used last time from the multiple working modes found, or select a working mode that is used most times under the same obstacle type from the multiple working modes found; A collection module, used to collect various types of information in the environment where the AR device is located, wherein the various types of information include at least one of image information, body movement information, and text information; a determination module, configured to search, from the first matching relationship, for a first type and a second type that match a current working mode of the AR device, where information of the first type needs to be converted into information of the second type in the current working mode, and the current working mode is any one of multiple working modes configured for the AR device, and the first matching relationship includes the first type and the second type that match each working mode; a conversion module, configured to determine an artificial intelligence function to be used based on a first type and a second type matching the current working mode, and select information of the first type from the multiple types of information, and convert the information of the first type into information of the second type through the determined artificial intelligence function, wherein the artificial intelligence function includes at least one of speech recognition, action recognition, object recognition, sign language recognition, speech synthesis, and text content recognition; a control module, configured to perform control based on the second type of information obtained by conversion; when the current working mode is the speech impaired mode, the first type of information includes sign language information or lip reading information of the subject, and the second type of information includes voice information or text information, and the control refers to playing the voice information or displaying the text information on the AR device for acquisition by other users except the subject; During the process of the subject using the AR device, the apparatus further includes a working mode switching module, configured to perform at least one of the following: Collecting voice information output by the object, if recognizing that the voice information contains a switching keyword, determining that a working mode switching operation is detected, switching the current working mode to the working mode indicated by the switching keyword, different switching keywords corresponding to different working modes; Image information of the object is collected. If a target gesture is identified in the image information, it is determined that a working mode switching operation is detected, and the current working mode is switched to the working mode indicated by the target gesture. Different target gestures correspond to different working modes.

7. An augmented reality (AR) device, characterized in that: The AR device includes a processor and a memory, wherein at least one program code is stored in the memory, and the at least one program code is loaded and executed by the processor to implement the control method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: At least one program code is stored in the computer-readable storage medium, and the at least one program code is loaded and executed by the processor to implement the control method according to any one of claims 1 to 5.

Images