Augmented reality device control method, apparatus, device, and storage medium

Abstract

This invention provides a method, apparatus, device, and storage medium for controlling an augmented reality device, belonging to the field of device control technology. The method includes: acquiring an audio source to be analyzed; determining the coordinates of the sound source corresponding to the audio source; determining the positional relationship between the sound source coordinates and the field of view of the augmented reality device; determining and displaying elements on the display interface of the augmented reality device based on the positional relationship, wherein the elements displayed on the display interface are at least used to indicate the position of the object emitting the audio source. By displaying elements corresponding to the position of the sound-emitting object on the display interface of the augmented reality device, hearing-impaired users can directly view the location of the sound-emitting object on the display interface, transforming abstract audio into visual guidance and solving the technical problem that hearing-impaired users have difficulty distinguishing the direction of sound.

Description

Technical Field

[0001] This invention relates to the field of equipment control technology, and in particular to an augmented reality equipment control method, apparatus, device, and storage medium. Background Technology

[0002] For hearing-impaired users, assistive devices (such as hearing aids) are typically worn to amplify sound. However, in noisy or multi-sound-source environments, hearing-impaired users often struggle to discern the direction of sound, making it difficult for them to recognize vehicle horns behind them or other people calling out to them. Therefore, there is an urgent need for a technology that can address the challenge of hearing-impaired users discerning the direction of sound in noisy or multi-sound-source environments. Summary of the Invention

[0003] This invention provides an augmented reality device control method, apparatus, device, and storage medium, aiming to solve the technical problem that hearing-impaired users have difficulty distinguishing the direction of sound.

[0004] In a first aspect, embodiments of the present invention provide an augmented reality device control method, comprising: Acquire the audio to be analyzed and determine the coordinates of the sound source corresponding to the audio to be analyzed; Determine the positional relationship between the sound source coordinates and the field of view of the augmented reality device; The elements to be displayed on the augmented reality device display interface are determined based on the positional relationship, and the elements are displayed, wherein the elements displayed on the display interface are at least used to indicate the position of the sound-producing object of the audio to be analyzed.

[0005] Secondly, embodiments of the present invention also provide an augmented reality device control apparatus, comprising: The coordinate determination module is used to acquire the audio to be analyzed and determine the coordinates of the sound source corresponding to the audio to be analyzed. A positional relationship determination module is used to determine the positional relationship between the sound source coordinates and the field of view of the augmented reality device; An element display module is used to determine, based on the positional relationship, the elements to be displayed on the display interface of the augmented reality device, and to display the elements, wherein the elements displayed on the display interface are at least used to indicate the position of the sound-producing object of the audio to be analyzed.

[0006] Thirdly, embodiments of the present invention also provide an augmented reality device, the augmented reality device including a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for implementing communication between the processor and the memory, wherein when the computer program is executed by the processor, it implements the augmented reality device control method as described in the first aspect.

[0007] Fourthly, embodiments of the present invention also provide a storage medium for computer-readable storage, the storage medium storing one or more programs, the one or more programs being executable by one or more processors to implement the augmented reality device control method as described in the first aspect.

[0008] This invention provides an augmented reality device control method, apparatus, device, and storage medium. The invention acquires the audio to be analyzed, determines the coordinates of the sound source corresponding to the audio, determines the positional relationship between the sound source coordinates and the field of view of the augmented reality device, and determines and displays the elements on the display interface of the augmented reality device based on the positional relationship. The elements displayed on the display interface are at least used to indicate the position of the object emitting the audio to be analyzed. By displaying elements corresponding to the position of the sound-emitting object on the display interface of the augmented reality device, hearing-impaired users can directly view the location of the sound-emitting object on the display interface, transforming abstract audio into visual guidance. This solves the technical problem that hearing-impaired users have difficulty distinguishing the direction of sound in noisy or multi-sound-source environments. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating an augmented reality device control method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of an element displayed on the display interface of an augmented reality device according to an embodiment of the present invention; Figure 3 This is a schematic diagram of another element displayed on the display interface of an augmented reality device provided in an embodiment of the present invention; Figure 4 This is a schematic block diagram of the structure of an augmented reality device control device provided in an embodiment of the present invention; Figure 5 This is a schematic block diagram of the structure of an augmented reality device provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0013] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0014] The augmented reality device control method provided in this invention can be applied to augmented reality devices. Augmented reality devices may include augmented reality (AR) glasses, AR helmets, or other devices with augmented reality functionality.

[0015] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0016] Please see Figure 1 , Figure 1 This is a flowchart illustrating an augmented reality device control method provided in an embodiment of the present invention.

[0017] like Figure 1 As shown, the augmented reality device control method includes steps S101 to S103.

[0018] Step S101: Obtain the audio to be analyzed and determine the coordinates of the sound source corresponding to the audio to be analyzed.

[0019] Augmented reality (AR) devices include function buttons for activating hearing impairment display and visual impairment functions. These buttons can be physical or virtual. Users can activate these functions via voice, gestures, or touch. The AR device integrates at least one microphone array, comprising at least four pickup units distributed across different planes. For example, when the AR device is AR glasses, microphones are located on the bridge of the nose, the front side of the left temple, the rear side of the right temple, the bottom of the left temple, and the bottom of the right temple. The AR device uses this microphone array to collect ambient audio in real-time or periodically. Because the microphone array includes at least four pickup units distributed across different planes, the ambient audio collected by the AR device is omnidirectional, i.e., 360-degree ambient audio, which is the acquired audio to be analyzed. Once the augmented reality device (AR) acquires the audio to be analyzed, it can suppress ambient background noise using beamforming technology and calculate the source coordinates of the audio source based on the Time Difference of Arrival (TDOA) algorithm. These source coordinates are vector coordinates in three-dimensional space, including azimuth, pitch, and distance, where the distance is the distance between the sound-emitting object and the user wearing the AR device. For details on suppressing ambient background noise using beamforming technology and calculating source coordinates based on the TDOA algorithm, please refer to relevant technologies; this embodiment will not provide a detailed explanation.

[0020] Step S102: Determine the positional relationship between the sound source coordinates and the field of view of the augmented reality device.

[0021] Once the augmented reality (AR) device determines the coordinates of the sound source corresponding to the audio to be analyzed, it determines the positional relationship between these coordinates and the field of view. This relationship can be categorized into two types: one where the sound source coordinates are within the field of view, and another where they are not. The field of view refers to the area of ​​the augmented reality image that the user can see through the AR device, usually expressed in degrees (°). A larger field of view allows the user to see more virtual information, resulting in a stronger sense of visual immersion. The sound source coordinates indicate the location of the sound-emitting object. When the sound source coordinates fall within the corresponding frustum of the field of view, it indicates that the sound source coordinates are within the field of view; when they do not fall within this frustum, it indicates that the sound source coordinates are not within the field of view.

[0022] In one embodiment, during the acquisition of audio to be analyzed, the augmented reality device may acquire audio emitted by more than one sound-emitting object. Therefore, the coordinates of the sound sources whose positional relationship needs to be determined can be determined based on the distance in the sound source coordinates. This distance is the distance between the sound-emitting object and the augmented reality device. For example, the coordinates of sound sources with a distance less than a preset value can be determined as the coordinates of the sound sources whose positional relationship needs to be determined. The size of this preset value can be set according to specific needs, and this embodiment does not limit this. Alternatively, the acquired sound source coordinates can be sorted in ascending order of distance, and a preset number of sound source coordinates at the top can be determined as the coordinates of the sound sources whose positional relationship needs to be determined. This preset number can be 2, 3, or 5, etc. After determining the coordinates of the sound sources whose positional relationship needs to be determined, the coordinates of the sound sources whose positional relationship needs to be determined are marked as the target coordinates, and the augmented reality device determines the positional relationship between the target coordinates and the field of view. When there is more than one sound-emitting object, the coordinates of the sound sources whose positional relationship needs to be determined are determined by the distance in the sound source coordinates of each sound-emitting object. This avoids determining the positional relationship between the sound source coordinates of sound-emitting objects that are far away from the augmented reality device and the field of view, thus reducing the power consumption of the augmented reality device.

[0023] Step S103: Determine the elements to be displayed on the display interface of the augmented reality device based on the positional relationship, and display the elements, wherein the elements displayed on the display interface are at least used to indicate the position of the sound-producing object of the audio to be analyzed.

[0024] Once the augmented reality device determines the positional relationship between the sound source coordinates and its field of view, it determines and displays the elements on its display interface based on this relationship. These elements, at least, indicate the position of the object emitting the audio being analyzed. For example, such as... Figure 2 As shown, the sound source coordinates are within the field of view of the augmented reality device (AR device). Based on these coordinates, the AR device can determine the distance between the sound-emitting object and the user, i.e., the distance between the sound-emitting object and the AR device itself. The AR device's display interface shows the positional relationship between the user's location and the corresponding sound-emitting object. (See reference...) Figure 2 It is known that the sound-emitting object is 50 meters away from the user and is in front of the user. It is understandable that when multiple sound-emitting objects are present, the augmented reality device's display interface will show elements corresponding to multiple sound-emitting objects. For example, when the sound-emitting objects are two cars, the elements displayed on the interface will at least include information indicating the corresponding positions of these two cars. Figure 2In one embodiment, the user's location is displayed on the screen, allowing the user to clearly understand the positional relationship between themselves and the sound-emitting object. Alternatively, the user's location may not be displayed on the screen; instead, the center of the screen can be used as the user's location. Or, instead of displaying the element indicating the user's location, the screen can directly indicate the positional relationship between the sound-emitting object and the user. For example, when the sound-emitting object is identified, relevant information can be displayed next to the corresponding virtual element, such as "30m to the upper right," indicating that the sound-emitting object is to the upper right of the user, 30 meters away.

[0025] In one embodiment, the elements displayed on the display interface may further include dynamic sound waves to clearly inform the wearer that the element displayed on the display interface is used to indicate the location of the sound-emitting object of the audio to be analyzed.

[0026] Understandably, when an augmented reality device (AR) is equipped with both hearing and visual impairment detection functions, after determining the positional relationship between the sound source coordinates and its field of view, the AR device determines whether the hearing or visual impairment detection function is currently active. If the hearing detection function is active, the AR device determines the element to display on its screen based on the positional relationship and displays that element. If the visual impairment detection function is active, the AR device determines the target audio output based on the positional relationship. This target audio at least indicates the location of the sound-emitting object. For example, the target audio could be "sound-emitting object 50 meters ahead," or "car sound, 45 meters ahead."

[0027] This embodiment acquires the audio to be analyzed, determines the coordinates of the sound source corresponding to the audio, determines the positional relationship between the sound source coordinates and the field of view of the augmented reality device, and determines the elements to be displayed on the display interface of the augmented reality device based on the positional relationship, and displays the elements. The elements displayed on the display interface are at least used to indicate the position of the sound-producing object in the audio to be analyzed. By displaying the elements corresponding to the position of the sound-producing object on the display interface of the augmented reality device, hearing-impaired users can directly view the position of the sound-producing object on the display interface, transforming abstract audio into visual guidance, and solving the technical problem that hearing-impaired users have difficulty distinguishing the direction of sound in noisy or multi-sound-source environments.

[0028] In one embodiment, the step of determining the elements to be displayed on the augmented reality device display interface based on the positional relationship includes: obtaining the object type of the sound-emitting object and a virtual image of the user wearing the augmented reality device; and determining the elements to be displayed on the display interface based on the positional relationship, the object type, and the virtual image.

[0029] In one embodiment, the method for determining the object type of the sound-emitting object can refer to the description of the following embodiments, and will not be repeated here. The object type includes vehicles, people, or animals, etc., or the object type can be further subdivided into ambulances, police cars, ordinary vehicles, family members, strangers, dogs, or cats, etc. The virtual image of the wearer can be preset, such as a photo of the wearer, or a preset virtual graphic, such as a five-pointed star or something similar. Figure 2 The image shows a line figure, etc. Once the augmented reality (AR) device acquires the object type of the sound-emitting object and the virtual image of the user, it can determine the elements to be displayed on its interface based on the positional relationship, object type, and virtual image. It can be understood that the positional relationship determines the sound source coordinates, or in other words, the positional relationship includes the sound source coordinates. The AR device can determine the distance between the sound-emitting object and the user based on these sound source coordinates, and based on this positional relationship, it can determine the directional information between the sound-emitting object and the user. After acquiring the object type of the sound-emitting object, the AR device acquires the type identifier corresponding to that object type, which is pre-stored in the AR device. Therefore, the elements displayed on the interface at this time can include the distance between the sound-emitting object and the user, the directional information between the sound-emitting object and the user, the type identifier corresponding to the sound-emitting object, and the virtual image of the user. (Reference) Figure 2 The display interface shows a vehicle icon, indicating that the sound-emitting object is a vehicle, and the location information between the wearer and the sound-emitting object is that the sound-emitting object is 50m (meters) in front of the wearer.

[0030] In one embodiment, the augmented reality device can also obtain the object type of the sound-emitting object, and then determine the elements to be displayed on the display interface based on the positional relationship and the type identifier of the object type, that is, not display the virtual image of the wearer on the display interface.

[0031] In this embodiment, when determining the elements to be displayed on the augmented reality device's display interface, the object type of the sound-emitting object is obtained. Based on the object type and positional relationship of the sound-emitting object, the elements to be displayed on the display interface are determined. That is, the type identifier corresponding to the object type of the sound-emitting object is also displayed on the display interface. This not only allows the user, i.e., the hearing-impaired user, to directly view the location of the sound-emitting object on the display interface, but also to know what the sound-emitting object is. This solves the technical problem that hearing-impaired users have difficulty distinguishing the direction of sound and improves the intelligence of the augmented reality device in the hearing-impaired display function.

[0032] In some embodiments, step S103 includes: Step a: If the coordinates of the sound source are within the field of view, then in the display interface of the augmented reality device, a first virtual element is displayed at the position corresponding to the sound-emitting object.

[0033] If the augmented reality device determines that the sound source coordinates are within the field of view, it displays a first virtual element on its display interface at the position corresponding to the sound-emitting object. This first virtual element includes at least the orientation information of the sound-emitting object relative to the user, which indicates the position of the sound-emitting object relative to the user. Figure 2 As shown, the location information is "the sound-emitting object is 50m (meters) away from the user and is in front of the user." The first virtual element may also include at least one of the following: the type identifier corresponding to the sound-emitting object and the urgency level corresponding to the audio to be analyzed. The urgency level is determined by at least one of the frequency, volume, and object type of the audio to be analyzed. It is understood that the higher the frequency of the audio to be analyzed, the higher the urgency level; the higher the volume of the audio to be analyzed, the higher the urgency level. The urgency levels of different object types can be preset, such as the urgency level of police cars and ambulances being higher than that of ordinary vehicles, and the urgency level of people being higher than that of animals. In one embodiment, the urgency level can also be determined by the distance between the sound-emitting object and the user; the closer the distance between the sound-emitting object and the user, the higher the urgency level. When the urgency level is determined by multiple factors, such as the volume of the audio to be analyzed and the distance between the sound-emitting object and the user, a weight can be set for each factor to obtain the final urgency level. It is understood that, in addition to displaying the first virtual element, the display interface can also display a virtual image of the user and other information.

[0034] Step b: If the sound source coordinates are not within the field of view, then a second virtual element corresponding to the sound-emitting object is displayed at the target position on the display interface based on the sound source coordinates, wherein the distance between the target position and the target edge of the display interface is less than a preset distance, and the target edge is the edge corresponding to the sound source coordinates.

[0035] If the sound source coordinates are not within the field of view, the augmented reality device displays a second virtual element corresponding to the sound-emitting object at the edge of the display interface based on the sound source coordinates. This edge is the side closer to the sound-emitting object; for example, if the sound-emitting object is on the right side of the user, then this edge is the right edge of the display interface. Figure 3As shown; if the sound-emitting object is behind the user, then the edge is the bottom edge of the display interface. The specific position of the second virtual element displayed on this edge is determined by the sound source coordinates of the sound-emitting object. These coordinates provide the directional information between the sound-emitting object and the user. For example, if the user's location is defined as the center of the display interface, and sound-emitting object A is located to the upper right of the user, with a specific directional information of (30, 20), it indicates that sound-emitting object A is 30m horizontally and 20m vertically away from the user. If sound-emitting object B is also to the upper right of the user, with a specific directional information of (30, 30), then in the display interface, sound-emitting object B is farther away from the user than sound-emitting object A. It is understood that the target position is the edge of the display interface; therefore, the distance between the target position and the edge of the display interface is less than a preset distance. The size of the preset distance can be set according to specific needs; this embodiment does not impose a specific limitation on the size of the preset distance. The target edge is determined by the location of the sound-emitting object, that is, by the coordinates of the sound source. For example, if the augmented reality device determines that the sound-emitting object is to the right of the user based on the coordinates of the sound source, then the target edge is the right edge of the display interface.

[0036] This embodiment displays a second virtual element corresponding to the sound source in the display interface when the sound source coordinates are not within the field of view. The edge corresponding to the sound source coordinates is used to indicate the direction of the sound source. This enables the user to determine the location of the sound source even if it is not within the field of view, thus transforming abstract audio into visual guidance and solving the problem of hearing-impaired users' lack of perception of sound sources behind and to the side.

[0037] In some embodiments, the second virtual element includes at least a directional guide element, and the display parameters of the second virtual element are determined by at least one of the frequency, volume, or urgency of the audio to be analyzed. The display parameters include at least one of color and blink frequency; The degree of urgency is determined by at least one of the frequency, volume, and object type of the audio being analyzed.

[0038] In one embodiment, the second virtual element includes at least a directional guide element, which is used to indicate the corresponding position of the sound-emitting object, such as... Figure 3As shown, the directional guidance element includes an indicator arrow and a distance. The indicator arrow indicates the location of the sound-emitting object, and the distance is the distance between the user and the sound-emitting object. Based on this directional guidance element, the sound-emitting object is located 30 meters to the right of the user. The second virtual element may also include a type identifier corresponding to the sound-emitting object. The display parameters of the second virtual element are determined by at least one of the frequency, volume, and urgency level of the audio to be analyzed. The display parameters include at least one of color and flashing frequency. For example, the higher the frequency of the audio to be analyzed, the darker the display color of the second virtual element; the higher the volume and / or the higher the urgency level of the audio to be analyzed, the higher the flashing frequency of the second virtual element. The process for determining the urgency level has been described in the above embodiments and will not be repeated here.

[0039] This embodiment categorizes and sets the display parameters of the second virtual element. For example, when the urgency of the audio to be analyzed is high, the display color of the second virtual element is red; when the urgency of the audio to be analyzed is low, the display color of the second virtual element is blue. This allows the wearer to easily understand the type of the sound-producing object and the urgency of the audio to be analyzed based on the second virtual element displayed on the interface. This enables the wearer to react promptly based on the second virtual element, thereby improving the intelligence of the augmented reality device in the hearing impairment display function.

[0040] In some embodiments, the method further includes: analyzing the audio to be analyzed to obtain the voiceprint features corresponding to the audio to be analyzed; comparing the voiceprint features with pre-stored voiceprint features in the feature library, and determining the type corresponding to the pre-stored voiceprint feature with the highest similarity to the voiceprint features as the object type of the sound-emitting object.

[0041] Once the augmented reality device acquires the audio to be analyzed, it analyzes the audio to obtain the corresponding voiceprint features. During the analysis, the augmented reality device can perform time-domain and / or frequency-domain analysis to extract Mel-frequency cepstral coefficients (MFCCs), which are then used as voiceprint features. It is understood that voiceprint features can also be linear frequency cepstral coefficients (LFCCs); this embodiment does not limit the type of voiceprint features. The augmented reality device pre-stores voiceprint features for different object types in a voiceprint feature library. It compares the voiceprint features of the audio to be analyzed with the pre-stored features and determines the type corresponding to the pre-stored voiceprint feature with the highest similarity as the object type of the sound-producing object. In one embodiment, to improve the accuracy of object type recognition, the augmented reality device records the pre-stored voiceprint feature with the highest similarity to the voiceprint feature of the audio to be analyzed as the target feature. If the similarity between the target feature and the voiceprint feature of the audio to be analyzed is greater than or equal to a preset similarity, the object type corresponding to the target feature is determined as the object type of the sound-emitting object. If the similarity between the target feature and the voiceprint feature of the audio to be analyzed is less than the preset similarity, the augmented reality device determines that the object type of the sound-emitting object is an unknown sound source. In this embodiment, a specific type identifier can be set for the unknown sound source, or if the sound-emitting object is determined to be an unknown sound source during the process of displaying the first virtual element or the second virtual element, the text "unknown sound source" can be directly displayed on the display interface.

[0042] In one embodiment, to reduce the power consumption of the augmented reality device, the voiceprint feature library can be stored in other smart terminals or cloud servers. When the augmented reality device acquires the audio to be analyzed, it transmits the audio to other smart terminals or cloud servers, which then perform voiceprint analysis on the audio to determine the type of object corresponding to the audio.

[0043] In some embodiments, the method further includes: acquiring the head posture of a user wearing the augmented reality device; and acquiring an object image corresponding to the sound-emitting object when the head posture meets preset conditions. Determine the image coordinates of the sound-emitting object based on the object image; The display position of the sound-emitting object in the display interface is determined based on the sound source coordinates and the image coordinates, and the element corresponding to the sound-emitting object is displayed in the display interface based on the display position.

[0044] In one embodiment, if the augmented reality device (ARD) determines that the sound source coordinates are not within the field of view, the ARD acquires the user's head posture, such as through an IMU (Inertial Measurement Unit) within the ARD. After acquiring the user's head posture, the ARD determines whether the head posture meets preset conditions. For example, if the head posture indicates that the user's head rotation angular velocity is greater than a preset angular velocity, the head posture is determined to meet the preset conditions; if the head posture indicates that the user's head rotation angular velocity is less than or equal to the preset angular velocity, the head posture is determined not to meet the preset conditions. In another embodiment, the head posture can also be determined to meet the preset conditions if the user's head rotation angle is greater than a preset angle, and not if the user's head rotation angle is less than or equal to the preset angle. In yet another embodiment, the head posture can also be determined to meet the preset conditions if the user's head rotation distance is greater than a preset rotation distance, and not if the user's head rotation distance is less than or equal to the preset rotation distance. The preset angular velocity, preset angle, and preset rotation distance can be set according to specific needs, and this embodiment does not impose specific limitations on them. In one embodiment, the preset angular velocity, preset angle, and preset rotation distance can also be determined based on the sound source coordinates. For example, the farther the sound-emitting object is from the user based on the sound source coordinates, the higher the preset angular velocity, preset angle, and preset rotation distance, thereby ensuring that the augmented reality device can acquire an image of the sound-emitting object. When it is determined that the head posture does not meet the preset conditions, the augmented reality device continues to determine whether the head posture meets the preset conditions until it can successfully acquire an image of the object corresponding to the sound-emitting object.

[0045] When the user's head posture meets preset conditions, the augmented reality device can acquire an image of the sound-emitting object through its camera. Then, using visual-inertial odometry (VIO), it obtains image coordinates based on the object image and data acquired by the IMU. The method of obtaining image coordinates using VIO based on the object image and IMU data can be found in related technologies and will not be elaborated upon in this embodiment. In one embodiment, after acquiring the object image, the augmented reality device can identify the object type of the sound-emitting object using a pre-trained object detection model, such as YOLO or MobileNet, to improve the accuracy of object type recognition.

[0046] During the process of the augmented reality device acquiring an object image, the augmented reality device can determine whether the object type of the acquired object image is the same as the object type of the sound-emitting object in the second virtual element. If the object types are the same, the augmented reality device acquires the object image; if the object types are different, the augmented reality device does not acquire the object image.

[0047] Once the augmented reality device obtains the image coordinates and sound source coordinates, it compensates for the spatial deviation caused by the sound source coordinates using the image coordinates. In other words, it determines the display position of the sound-emitting object on the display interface based on the sound source coordinates and image coordinates. It's understandable that the sound source coordinates and image coordinates are based on a unified coordinate system. If the coordinate systems of the sound source coordinates and image coordinates are different, a coordinate system transformation is required to convert them to the same system. In determining the display position of the sound-emitting object on the display interface based on the sound source coordinates and image coordinates, the midpoint between the sound source coordinates and image coordinates can be used as the display position of the sound-emitting object; alternatively, weights can be assigned to the sound source coordinates and image coordinates. For example, if the sound source coordinates are labeled W1 with a weight of a1, and the image coordinates are labeled W2 with a weight of a2, then the display position of the sound-emitting object on the display interface would be W1. a1+W2 a2.

[0048] In one embodiment, in order to improve the accuracy of acquiring object images, when the augmented reality device determines that the head posture meets the preset conditions, the augmented reality device determines the rotation direction of the user's head. If the rotation direction is consistent with the direction of the orientation guide element in the second virtual element, the augmented reality device activates the camera and acquires the object image of the sound-emitting object through the camera.

[0049] In this embodiment, when the user's head posture meets preset conditions, an object image corresponding to the sound-emitting object is acquired. The image coordinates of the sound-emitting object are determined based on the object image. The display position of the sound-emitting object in the display interface is determined based on the sound source coordinates and the image coordinates. Instead of relying solely on the sound source coordinates to determine the position of the sound-emitting object, the method integrates the sound source coordinates and the image coordinates. By using the spatial overlap between the sound source coordinates and the image coordinates, i.e., through a dual mechanism of "sound positioning first, image calibration later," the spatial deviation caused by the sound source coordinates obtained based on the microphone array is corrected. This avoids the problem of interference from reflected sound when relying solely on the microphone array for positioning, thereby improving the accuracy of determining the position of the sound-emitting object.

[0050] In some embodiments, after step S103, the method further includes: when a change in the location of the sound-emitting object is detected, adjusting the elements displayed on the display interface based on the changed location.

[0051] In one embodiment, when the augmented reality device detects a change in the location of the sound-emitting object, it adjusts the elements displayed on its display interface based on the changed location of the sound-emitting object. For example, when the sound-emitting object changes from a location 50m away from the user to a location 30m away from the user, the distance between the sound-emitting object and the user displayed on the augmented reality device's display interface changes from 50m to 30m. This achieves real-time updates of the UI elements of the display interface based on the movement of the sound-emitting object, realizing a visually enhanced effect of sound following the image.

[0052] Please see Figure 4 , Figure 4 This is a schematic block diagram of the structure of an augmented reality device control device provided in an embodiment of the present invention, such as... Figure 4 As shown, the augmented reality device control unit 400 includes: The coordinate determination module 410 is used to acquire the audio to be analyzed and determine the coordinates of the sound source corresponding to the audio to be analyzed. The positional relationship determination module 420 is used to determine the positional relationship between the sound source coordinates and the field of view of the augmented reality device; The element display module 430 is used to determine the elements to be displayed on the display interface of the augmented reality device based on the positional relationship, and to display the elements, wherein the elements displayed on the display interface are at least used to indicate the position of the sound-producing object of the audio to be analyzed.

[0053] In some embodiments, the element display module 430 is further configured to: if the sound source coordinates are within the field of view, display a first virtual element at the position corresponding to the sound-emitting object in the display interface of the augmented reality device; if the sound source coordinates are not within the field of view, display a second virtual element corresponding to the sound-emitting object at a target position in the display interface based on the sound source coordinates, wherein the distance between the target position and the target edge of the display interface is less than a preset distance, and the target edge is the edge corresponding to the sound source coordinates.

[0054] In some embodiments, the augmented reality device control device 400 further includes: The posture acquisition module is used to acquire the head posture of the user wearing the augmented reality device; The image acquisition module is used to acquire the object image corresponding to the sound-emitting object when the head posture meets the preset conditions. The coordinate determination module 410 is also used to determine the image coordinates of the sound-emitting object based on the object image; A position determination module is used to determine the display position of the sound-emitting object in the display interface based on the sound source coordinates and the image coordinates; The element display module 430 is also used to display the element corresponding to the sound-emitting object in the display interface based on the display position.

[0055] In some embodiments, the second virtual element includes at least a directional guide element, and the display parameters of the second virtual element are determined by at least one of the frequency, volume, or urgency of the audio to be analyzed. The display parameters include at least one of color and blink frequency; The degree of urgency is determined by at least one of the frequency, volume, and object type of the audio being analyzed.

[0056] In some embodiments, the augmented reality device control device 300 further includes: The audio analysis module is used to analyze the audio to be analyzed and obtain the voiceprint features corresponding to the audio to be analyzed. The object type determination module is used to compare the voiceprint features with pre-stored voiceprint features in the feature library, and determine the type corresponding to the pre-stored voiceprint feature with the highest similarity to the voiceprint features as the object type of the sound-emitting object.

[0057] In some embodiments, the element display module 430 is further configured to adjust the elements displayed on the display interface based on the changed position after detecting a change in the location of the sound-emitting object.

[0058] In some embodiments, the element display module 430 is further configured to acquire the object type of the sound-emitting object and the virtual image of the user wearing the augmented reality device; and determine the elements to be displayed on the display interface based on the positional relationship, the object type, and the virtual image.

[0059] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the augmented reality device control device described above can be referred to the corresponding process in the aforementioned embodiments of the augmented reality device control method, and will not be repeated here.

[0060] Please see Figure 5 , Figure 5 This is a schematic block diagram of the structure of an augmented reality device provided in an embodiment of the present invention.

[0061] like Figure 5 As shown, the augmented reality device 500 includes a processor 501 and a memory 502, which are connected via a bus 503, such as an I2C (Inter-integrated Circuit) bus. The augmented reality device includes at least two display units (not shown in the figure).

[0062] Specifically, processor 501 provides computing and control capabilities to support the operation of the entire augmented reality device. Processor 501 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0063] Specifically, the memory 502 can be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a portable hard drive, etc.

[0064] Those skilled in the art will understand that Figure 5 The structures shown are merely block diagrams of some structures related to the embodiments of the present invention, and do not constitute a limitation on the augmented reality devices on which the embodiments of the present invention are applied. Specific augmented reality devices may include more or fewer components than those shown in the figures, or combine certain components, or have different component arrangements.

[0065] The processor 501 is used to run a computer program stored in the memory 502, and implements any of the augmented reality device control methods provided in the embodiments of the present invention when executing the computer program.

[0066] In one embodiment, the processor 501 is configured to run a computer program stored in a memory, and when executing the computer program, to perform the following steps: Acquire the audio to be analyzed and determine the coordinates of the sound source corresponding to the audio to be analyzed; Determine the positional relationship between the sound source coordinates and the field of view of the augmented reality device; The elements to be displayed on the augmented reality device display interface are determined based on the positional relationship, and the elements are displayed, wherein the elements displayed on the display interface are at least used to indicate the position of the sound-producing object of the audio to be analyzed.

[0067] In some embodiments, when the processor 501 determines the element to be displayed on the augmented reality device display interface based on the positional relationship, and displays the element, it is configured to: If the coordinates of the sound source are within the field of view, then a first virtual element is displayed in the display interface of the augmented reality device at the position corresponding to the sound-emitting object; If the sound source coordinates are not within the field of view, a second virtual element corresponding to the sound-emitting object is displayed at the target position on the display interface based on the sound source coordinates. The distance between the target position and the target edge of the display interface is less than a preset distance, and the target edge is the edge corresponding to the sound source coordinates.

[0068] In some embodiments, after the step of displaying a second virtual element corresponding to the sound-emitting object at a target position on the display interface based on the sound source coordinates if the sound source coordinates are not within the field of view, the processor 501 is further configured to implement: Acquire the head posture of the user wearing the augmented reality device; When the head posture meets the preset conditions, the object image corresponding to the sound-emitting object is obtained; Determine the image coordinates of the sound-emitting object based on the object image; The display position of the sound-emitting object in the display interface is determined based on the sound source coordinates and the image coordinates, and the element corresponding to the sound-emitting object is displayed in the display interface based on the display position.

[0069] In some embodiments, the second virtual element includes at least a directional guide element, and the display parameters of the second virtual element are determined by at least one of the frequency, volume, or urgency of the audio to be analyzed. The display parameters include at least one of color and blink frequency; The degree of urgency is determined by at least one of the frequency, volume, and object type of the audio being analyzed.

[0070] In some embodiments, the processor 501 is further configured to implement: The audio to be analyzed is analyzed to obtain the voiceprint features corresponding to the audio to be analyzed. The voiceprint feature is compared with the pre-stored voiceprint features in the feature library, and the type corresponding to the pre-stored voiceprint feature with the highest similarity to the voiceprint feature is determined as the object type of the sound-producing object.

[0071] In some embodiments, after the step of determining the elements to be displayed on the augmented reality device display interface based on the positional relationship, the processor 501 is further configured to implement: When a change in the location of the sound-emitting object is detected, the elements displayed on the display interface are adjusted based on the changed location.

[0072] In some embodiments, when the processor 501 determines the elements to be displayed on the augmented reality device display interface based on the positional relationship, it is configured to: Obtain the object type of the sound-emitting object, and the virtual image of the user wearing the augmented reality device; The elements displayed on the display interface are determined based on the positional relationship, the object type, and the virtual image.

[0073] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the augmented reality device described above can be referred to the corresponding process in the aforementioned embodiments of the augmented reality device control method, and will not be repeated here.

[0074] This invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs that can be executed by one or more processors to implement any of the augmented reality device control methods provided in the specification of this invention.

[0075] The storage medium can be volatile or non-volatile. It can be an internal storage unit of the augmented reality device described in the foregoing embodiments, such as the hard drive or memory of the augmented reality device. Alternatively, it can be an external storage device of the augmented reality device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card.

[0076] Those skilled in the art will understand that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware embodiments, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0077] It should be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0078] The sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The above descriptions are only specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for controlling an augmented reality device, characterized in that, The method includes: Acquire the audio to be analyzed and determine the coordinates of the sound source corresponding to the audio to be analyzed; Determine the positional relationship between the sound source coordinates and the field of view of the augmented reality device; The elements to be displayed on the augmented reality device display interface are determined based on the positional relationship, and the elements are displayed, wherein the elements displayed on the display interface are at least used to indicate the position of the sound-producing object of the audio to be analyzed.

2. The augmented reality device control method according to claim 1, characterized in that, The step of determining the element to be displayed on the augmented reality device display interface based on the positional relationship, and displaying the element, includes: If the coordinates of the sound source are within the field of view, then a first virtual element is displayed in the display interface of the augmented reality device at the position corresponding to the sound-emitting object; If the sound source coordinates are not within the field of view, a second virtual element corresponding to the sound-emitting object is displayed at the target position on the display interface based on the sound source coordinates. The distance between the target position and the target edge of the display interface is less than a preset distance, and the target edge is the edge corresponding to the sound source coordinates.

3. The augmented reality device control method according to claim 2, characterized in that, The step of displaying a second virtual element corresponding to the sound-emitting object at the target position on the display interface based on the sound source coordinates if the sound source coordinates are not within the field of view further includes: Acquire the head posture of the user wearing the augmented reality device; When the head posture meets the preset conditions, the object image corresponding to the sound-emitting object is obtained; Determine the image coordinates of the sound-emitting object based on the object image; The display position of the sound-emitting object in the display interface is determined based on the sound source coordinates and the image coordinates, and the element corresponding to the sound-emitting object is displayed in the display interface based on the display position.

4. The augmented reality device control method according to claim 2, characterized in that, The second virtual element includes at least a directional guide element, and the display parameters of the second virtual element are determined by at least one of the frequency, volume, or urgency of the audio to be analyzed. The display parameters include at least one of color and blink frequency; The degree of urgency is determined by at least one of the frequency, volume, and object type of the audio being analyzed.

5. The augmented reality device control method according to claim 4, characterized in that, The method further includes: The audio to be analyzed is analyzed to obtain the voiceprint features corresponding to the audio to be analyzed. The voiceprint feature is compared with the pre-stored voiceprint features in the feature library, and the type corresponding to the pre-stored voiceprint feature with the highest similarity to the voiceprint feature is determined as the object type of the sound-producing object.

6. The augmented reality device control method according to claim 1, characterized in that, After the step of determining the elements to be displayed on the augmented reality device display interface based on the positional relationship, the method further includes: When a change in the location of the sound-emitting object is detected, the elements displayed on the display interface are adjusted based on the changed location.

7. The augmented reality device control method according to any one of claims 1 to 6, characterized in that, The step of determining the elements to be displayed on the augmented reality device display interface based on the positional relationship includes: Obtain the object type of the sound-emitting object, and the virtual image of the user wearing the augmented reality device; The elements displayed on the display interface are determined based on the positional relationship, the object type, and the virtual image.

8. An augmented reality device control device, characterized in that, The device includes: The coordinate determination module is used to acquire the audio to be analyzed and determine the coordinates of the sound source corresponding to the audio to be analyzed. A positional relationship determination module is used to determine the positional relationship between the sound source coordinates and the field of view of the augmented reality device; An element display module is used to determine, based on the positional relationship, the elements to be displayed on the display interface of the augmented reality device, and to display the elements, wherein the elements displayed on the display interface are at least used to indicate the position of the sound-producing object of the audio to be analyzed.

9. An augmented reality device, characterized in that, The augmented reality device includes a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for establishing communication between the processor and the memory, wherein when the computer program is executed by the processor, it implements the steps of the augmented reality device control method as described in any one of claims 1 to 7.

10. A storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the method for controlling an augmented reality device as described in any one of claims 1 to 7.

Images