Audio processing method and device, electronic equipment, chip and medium

Abstract

The invention provides an audio processing method and device, electronic equipment, a chip and a medium, and relates to the technical field of intelligent wearable hardware and computer application. The audio processing method comprises the steps that the electronic equipment obtains the sight line direction of a first object and displays identity icons of one or more candidate objects, and the electronic equipment is used for outputting audio streams related to the one or more candidate objects; according to the line-of-sight direction, detecting that an identity icon of a second object in the one or more candidate objects and the line-of-sight direction meet a preset relationship, and determining a target audio stream related to the second object; and processing the audio stream of the at least one candidate object based on the target audio stream. Through the technical scheme provided by the invention, the problem that the sound of wearable virtual reality and other electronic equipment cannot be focused is solved, the sound attention effect of the target user is enhanced, and the user experience is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of smart wearable hardware and computer application technology, and in particular to an audio processing method, device, electronic device, chip and medium. Background Art

[0002] Wearable virtual reality devices, also known as augmented reality (AR) / virtual reality (VR) devices, are immersive viewing devices that have emerged in recent years. With the development of related applications, users can watch movies / make video calls / perform 3D design through AR / VR devices. When conducting video conferences through wearable virtual reality devices, since the spatial directionality is fixed and the psychoacoustic effect is not considered, the application does not perform spatial sound rendering and sound attention cannot be focused. Summary of the invention

[0003] The present disclosure provides an audio processing method, device, electronic device, chip and medium to solve the problem that the sound attention of wearable virtual reality devices cannot be focused in video conferencing scenarios. By combining eye tracking and sound focusing technology, the psychoacoustic effect of spatial acoustics is rendered, the effect of sound focusing with the eyes is achieved, and the user experience is improved.

[0004] A first aspect of the present disclosure provides an audio processing method, the method comprising:

[0005] The electronic device acquires the sight line direction of the first object and displays identity icons of one or more candidate objects, and the electronic device is used to output audio streams related to the one or more candidate objects;

[0006] According to the sight line direction, it is detected that the identity icon of a second object among the one or more candidate objects meets a preset relationship with the sight line direction, and a target audio stream related to the second object is determined;

[0007] Based on the target audio stream, an audio stream of at least one candidate object is processed.

[0008] In one embodiment of the present disclosure, before the electronic device acquires the sight line direction of the first object and displays the identity icons of one or more candidate objects, the electronic device further includes:

[0009] Get the first sight range of each candidate object;

[0010] According to the first sight range of each candidate object, the second sight range of the first object for each candidate object is calibrated, and the second sight range is used for: when the sight direction falls into the second sight range, the candidate object corresponding to the second sight range is the second object.

[0011] In one embodiment of the present disclosure, according to the sight direction, detecting that the identity icon of a second object among one or more candidate objects satisfies a preset relationship with the sight direction, and determining a target audio stream related to the second object includes:

[0012] Determine a spatial angle between a sight line direction and a line connecting an identity icon of the second object and the first object;

[0013] Determine the target audio stream based on the sight direction and spatial angle.

[0014] In one embodiment of the present disclosure, determining a target audio stream according to a sight line direction and a spatial angle includes:

[0015] Determine the time the spatial angle stays within the second visual range;

[0016] Determine an original target audio stream of the second object according to the sight direction, the second sight range, the spatial angle, and the dwell time;

[0017] The target audio stream is obtained by processing the original target audio stream through head-related transformation.

[0018] In one embodiment of the present disclosure, determining a target audio stream according to a sight line direction and a spatial angle includes:

[0019] When the spatial angle is within the second sight range, the sight movement trajectory formed by the sight direction is recorded;

[0020] Compare the sight movement trajectory with the preset trajectory for similarity;

[0021] If the similarity is greater than a first threshold, determining the original target audio stream of the second object;

[0022] The target audio stream is obtained by processing the original target audio stream through head-related transformation.

[0023] In one embodiment of the present disclosure, determining the original target audio stream of the second object according to the sight direction, the second sight range, the spatial angle, and the dwell time includes:

[0024] If the stay time is greater than or equal to the first duration threshold, at least one of the one or more candidate objects within the second visual range is determined as the second object, and at least one of the one or more candidate objects outside the second visual range is determined as the third object;

[0025] If the stay time is less than the first duration threshold, at least one of the one or more candidate objects within the second visual range is determined as the third object.

[0026] In one embodiment of the present disclosure, based on the target audio stream, processing the audio stream of at least one object among the one or more candidate objects includes:

[0027] According to the first weight, processing the target audio stream as the first audio stream; and / or

[0028] According to the second weight, processing the audio stream of at least one object among the one or more candidate objects excluding the second object into the second audio stream;

[0029] The first audio stream and the second audio stream are merged.

[0030] A second aspect of the present disclosure provides an audio processing device, the device comprising:

[0031] An acquisition module, configured for the electronic device to acquire the sight line direction of the first object and display identity icons of one or more candidate objects, and the electronic device to output audio streams related to the one or more candidate objects;

[0032] A determination module, configured to detect, based on the sight line direction, that an identity icon of a second object among the one or more candidate objects satisfies a preset relationship with the sight line direction, and determine a target audio stream related to the second object;

[0033] The processing module is used to process the audio stream of at least one candidate object based on the target audio stream.

[0034] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods in the first aspect embodiment of the present disclosure.

[0035] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable a computer to execute the method in the first aspect embodiment of the present disclosure.

[0036] The fifth aspect embodiment of the present disclosure proposes a computer program product, characterized in that it includes a computer program, and when the computer program is executed by a processor, it implements any method in the first aspect embodiment of the present disclosure.

[0037] The sixth aspect embodiment of the present disclosure proposes a chip, comprising at least one processor and a communication interface; the communication interface is used to receive signals input into the chip or signals output from the chip, the processor communicates with the communication interface and implements any one of the methods in the first aspect embodiment of the present disclosure through logic circuits or execution code instructions.

[0038] In summary, according to the audio processing method proposed in the present invention, the electronic device obtains the line of sight direction of the first object and displays the identity icons of one or more candidate objects. The electronic device is used to output audio streams related to one or more candidate objects, providing a directional basis for sound focusing; according to the line of sight direction, it is detected that the identity icon of the second object among the one or more candidate objects satisfies the preset relationship with the line of sight direction, the target audio stream related to the second object is determined, and the target audio is directionally acquired; based on the target audio stream, the audio stream of at least one candidate object is processed, and the sound is focused according to the sound attention level, thereby improving the user experience.

[0039] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0041] Figure 1 A schematic diagram of a basic conference scenario in the related art;

[0042] Figure 2 is a flow chart of an audio processing method according to an embodiment of the present disclosure;

[0043] Figure 3 A flowchart of obtaining a sight line direction of a first object and calibrating a spatial angle between a wearable virtual reality device and a second object according to an embodiment of the present disclosure;

[0044] Figure 3a A schematic diagram of a first visual range and a second visual range according to an embodiment of the present disclosure;

[0045] Figure 4 A flowchart of determining a target audio stream of a second object according to a sight line direction according to an embodiment of the present disclosure;

[0046] Figure 5 This is a flow chart of determining a target audio stream according to a sight line direction and a spatial angle according to an embodiment of the present disclosure;

[0047] Figure 6 This is a flow chart of determining a target audio stream according to a sight line direction and a spatial angle according to an embodiment of the present disclosure;

[0048] Figure 7 A flowchart of determining an original target audio stream of a second object according to a sight direction, a second sight range, a spatial angle, and a dwell time according to an embodiment of the present disclosure;

[0049] Figure 8 A flowchart of processing an audio stream of at least one candidate object based on a target audio stream according to an embodiment of the present disclosure;

[0050] Fig. 9 A schematic diagram of an audio focusing process following the line of sight according to an embodiment of the present disclosure;

[0051] Fig.10 is a structural schematic diagram of an audio processing device according to an embodiment of the present disclosure;

[0052] Fig.11 is a block diagram of an electronic device for implementing the audio processing method disclosed herein according to an exemplary embodiment;

[0053] Fig.12 It is a schematic diagram of the structure of the chip of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout identify the same or similar originals or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0055] Virtual reality technology is a technology that uses computers to simulate a three-dimensional virtual world, providing users with visual and other sensory simulations, making users feel as if they are actually there, and can observe things in the three-dimensional space instantly and without restrictions. When the user moves, the computer can immediately perform complex calculations to adjust the user's perspective to the new position, making the user feel that he is really moving.

[0056] Virtual reality technology has a wide range of applications, such as games, education, medical treatment, military, etc. In games, virtual reality technology can make players more immersed in the game world; in education, virtual reality technology can make students understand knowledge more intuitively; in medical treatment, virtual reality technology can help doctors perform surgery better; in the military, virtual reality technology can be used to simulate battlefield environments.

[0057] In the field of virtual reality technology, wearable virtual reality devices can provide a deep immersive experience. The technical principle of wearable virtual reality devices is to guide users to feel like they are in a virtual environment by closing their vision and hearing to the outside world. Wearable displays usually have two displays, and the two displays are driven by computers to provide different images to the two eyes. The two images are then fused through the human brain to obtain depth perception, thereby producing a three-dimensional image.

[0058] Figure 1FIG. 1 is a schematic diagram of a basic conference scene in the related art. Figure 1 As shown in the figure, in a video conference scenario, multiple users join the conference through virtual reality devices. The current users are Speaker 1, Speaker 2, Speaker 3 and the user. The user can see the virtual image / card avatar / human figure of the other party on the canvas. In terms of sound effects, when a participant speaks, the voices of all participants are recorded at the same time, which are audio stream 1, audio stream 2, and audio stream 3, and are played in the headphones of the virtual reality device after being mixed. The factors of psychoacoustic perception of users are not considered.

[0059] The method proposed in the present disclosure can be applied to electronic devices such as wearable virtual reality devices, and can also be applied to video conferencing, game scenarios and video call scenarios. The application scenarios are not limited in the embodiments of the present disclosure.

[0060] The audio processing method provided in this application is described in detail below with reference to the accompanying drawings.

[0061] Figure 2 Flow chart of an audio processing method according to an embodiment of the present disclosure. The audio processing method includes:

[0062] Step 201: The electronic device obtains the sight line direction of the first object and displays identity icons of one or more candidate objects. The electronic device is used to output audio streams related to the one or more candidate objects.

[0063] In this embodiment, the first object is a wearer of a wearable virtual reality device having a tracking module, and may also be a user of other electronic devices having a tracking module, such as a TV, a tablet, a mobile phone, etc. In order to facilitate understanding of the disclosed solution, the present disclosure specifically uses a wearable virtual reality device to explain and illustrate the method shown in the present disclosure.

[0064] In the disclosed embodiment, the line of sight direction of the first object is analyzed by the line of sight tracking module of the wearable virtual reality device, and the current line of sight direction of the first object can be obtained. The wearable device can receive and output audio data (audio stream) of one or more candidate objects. Exemplarily, the candidate objects may be other users wearing wearable virtual reality devices who participate in the current audio / video conference with the first object, and the user is at least one person. During the user's participation in the meeting, each user will be displayed in the form of an identity icon, which can be in the form of a user avatar or a virtual character image specified by the user.

[0065] Step 202: According to the sight line direction, it is detected that the identity icon of a second object among the one or more candidate objects satisfies a preset relationship with the sight line direction, and a target audio stream related to the second object is determined.

[0066] According to the line of sight direction, detecting that the identity icon of a second object among one or more candidate objects satisfies a preset relationship with the line of sight direction, and determining the target audio stream of the second object can include: determining the target audio stream of the second object according to a preset spatial relationship between the line of sight direction and the position of the second object icon.

[0067] In this embodiment, the second object is a participant in the line of sight of the first object among the candidate objects. The second object may be a user wearing a wearable virtual reality device. According to the determined line of sight direction, it is detected whether the second object and the line of sight direction satisfy a preset relationship. If the preset relationship is satisfied, the target audio stream of the second object may be obtained according to the second object. In other words, the sound data of the second object in the line of sight direction is obtained.

[0068] In an embodiment of the present disclosure, the preset relationship may indicate a spatial position relationship between a sight line direction and an identity marker.

[0069] Step 203: Processing the audio stream of at least one candidate object based on the target audio stream.

[0070] In this embodiment, after obtaining the target audio stream data of the second object, the first object processes the audio of the second object and other users in the candidate objects, focuses on the sound according to the sound attention, thereby improving the user experience.

[0071] In one embodiment of the present disclosure, processing the audio stream of at least one candidate object based on the target audio stream may include keeping the output parameters of the target audio stream unchanged and processing the audio streams other than the target audio stream. The output parameters include volume and sound effects, etc. The processing includes turning up the volume or switching the sound effect mode, etc. For example, keeping the volume of the target audio stream unchanged and reducing the volume of the audio streams other than the target audio stream.

[0072] In one embodiment of the present disclosure, processing the audio stream of at least one candidate object based on the target audio stream may include processing output parameters of the target audio stream, and keeping the output parameters of the audio streams other than the target audio stream unchanged. For example, increasing the volume of the target audio stream, and keeping the volume of the audio streams other than the target audio stream unchanged. It can be understood that in this embodiment, the target audio stream also belongs to the audio streams in the candidate object.

[0073] In one embodiment of the present disclosure, processing the audio stream of at least one candidate object based on the target audio stream may include, according to a preconfiguration, adjusting parameters of the target audio stream and the audio streams of other candidate objects. For example, increasing the volume of the target audio stream and reducing the volume of audio streams related to other candidate objects. In one embodiment of the present disclosure, processing the audio stream of at least one candidate object based on the target audio stream may include, according to the target audio stream, adjusting the mixing weights of one or more candidate objects. For example, increasing the mixing weight of the target audio stream and reducing the mixing weights of other candidate objects. Alternatively, reducing the mixing weight of the target audio stream and increasing the mixing weights of other candidate objects.

[0074] In summary, according to the audio processing method proposed in the present invention, the electronic device obtains the line of sight direction of the first object and displays the identity icons of one or more candidate objects. The electronic device is used to output audio streams related to one or more candidate objects, providing a directional basis for sound focusing; according to the line of sight direction, it is detected that the identity icon of the second object among the one or more candidate objects satisfies a preset relationship with the line of sight direction, the target audio stream related to the second object is determined, and the target audio is directionally acquired; based on the target audio stream, the audio stream of at least one candidate object is processed, the sound is focused, the sound attention effect on the second object is enhanced, and the user experience is improved.

[0075] Figure 3 The present invention is a flowchart of obtaining a sight line direction of a first object to calibrate a spatial angle between a wearable virtual reality device and a second object according to an embodiment of the present invention. Figure 3 for Figure 2 The steps before step 201 are based on Figure 3 The embodiment shown comprises the following steps:

[0076] Step 301: Obtain a first sight range of each candidate object.

[0077] In this embodiment, the first visual range is a visual range preset by the electronic device for each candidate. When all candidates wearing wearable virtual reality devices join the conference, the wearable virtual reality device or a server communicating with the device will assign a respective visual range to each candidate to identify the area where the candidate is located in the conference system.

[0078] Step 302: calibrate the second sight range of the first object for each candidate object according to the first sight range of each candidate object, the second sight range being used for: when the sight direction falls into the second implementation range, the candidate object corresponding to the second sight range is the second object.

[0079] Before the electronic device obtains the line of sight direction of the first object and displays the identity icons of one or more candidate objects, the first object is calibrated for each candidate object according to the first line of sight range assigned to the candidate object. The second line of sight range is the specific position area of ​​the candidate object in the overall field of view of the current first object determined based on the viewing angle of the first object, and is expressed in angles.

[0080] Figure 3a The schematic diagram of a first sight range and a second sight range of an embodiment of the present disclosure. In this embodiment, A is the first object, B, C, and D are candidate objects, AB is the sight direction of the first object, arc EBF is the first sight range of candidate object B, and the sight range is the sight range assigned to object B by the wearable virtual reality device or the server communicating with the device, arc FCG is the first sight range of candidate object C, and arc GDH is the first sight range of candidate object D. Taking candidate object B as an example, the endpoints of its first sight range are E and F, and the angle ∠EAF formed by sight AE and sight AF is the second sight range of the first object A for candidate object B. The sight direction of the current first object A is ray AP, the line connecting the first object and candidate object B is AB, and the angle θ (i.e. ∠BAP) between the sight direction AP and AB is a spatial angle, because the maximum spatial angle that can be formed in the clockwise direction of the spatial angle in the first sight range of candidate object B is β (i.e. ∠BAF), and the maximum spatial angle that can be formed in the counterclockwise direction in the first sight range of candidate object B is α (i.e. ∠BAE). With the clockwise direction as the positive direction and the counterclockwise direction as the negative direction, the second sight range of the first object A for the candidate object B is ∠EAF, which can also be expressed as [-α, β]. Similarly, through the above calibration method, ∠FAG in the figure is the second sight range of the first object A for the candidate object C, and ∠GAH in the figure is the second sight range of the first object A for the candidate object D. The second sight range can be combined with the spatial angle determined by the sight direction to determine the second object from the candidate objects. As shown in the figure, due to the angle θ between the current sight direction AP and AB, that is, the spatial angle θ∈[-α, β], it means that the sight direction AP is in the second sight range ∠EAF of the first object A for the candidate object B, so the current candidate object B is the second object.

[0081] In this embodiment, by calibrating the candidate objects, the second sight range of the first object for each candidate object is obtained, and the second object is determined based on the calibrated second sight range. After that, the first object can determine the target audio stream data of the second object according to the spatial angle.

[0082] Figure 4 The present invention is a flowchart of determining a target audio stream of a second object according to a line of sight direction according to an embodiment of the present disclosure. Figure 4 Yes Figure 2 Step 202 is further described based on Figure 4 The embodiment shown comprises the following steps:

[0083] Step 401, determining a spatial angle between a sight line direction and a line connecting an identity icon of a second object and a first object.

[0084] In this embodiment, according to the position of the first object and the position of the second object, the line between the two is used as the reference line. The positions of the first object and the second object can be determined by the wearable virtual reality device, or by a server communicating with the wearable virtual reality device. The angle formed by the sight line direction of the first object and the reference line is used as the spatial angle.

[0085] Step 402: Determine the target audio stream according to the sight direction and the spatial angle.

[0086] In this embodiment, according to the sight direction of the first object and the spatial angle with the candidate object, it is determined whether it belongs to the second sight range of the candidate object, so as to determine whether the current candidate object is the second object, thereby determining the target audio stream. The specific method for determining the target audio stream is as follows.

[0087] Figure 5 The present invention is a flowchart of determining a target audio stream according to a sight line direction and a spatial angle according to an embodiment of the present invention. Figure 5 Yes Figure 4 Step 402 is specifically described based on Figure 5 The embodiment shown comprises the following steps:

[0088] Step 501, determining the residence time of the spatial angle within the second visual range.

[0089] In this embodiment, the time that the spatial angle of the first object stays in the second visual range is recorded to confirm whether the first object quickly passes through the second visual range unintentionally or stares at the second visual range for a long time.

[0090] Step 502: Determine the original target audio stream of the second object according to the sight direction, the second sight range, the spatial angle, and the dwell time.

[0091] In this embodiment, based on the line of sight direction of the first object, the second line of sight range, the spatial angle of the line of sight direction of the first object, and the stay time of the spatial angle within the second line of sight range, it can be confirmed whether the candidate object currently within the second line of sight range is the second object. If it is the second object, the original target audio stream of the second object can be extracted.

[0092] Step 503: Process the original target audio stream through head-related transformation to obtain a target audio stream.

[0093] In this embodiment, after the original target audio stream of the second object is obtained, in order to obtain a sound effect with a spatial stereoscopic sense, a head related transfer function (HRTF) is performed on the original target audio stream to obtain an audio stream with a spatial sense.

[0094] In this embodiment, by determining the second object, the original target audio stream of the second object is obtained, and then the original target audio stream is processed into a target audio stream, and the target audio stream to be mixed is determined, providing an audio stream data source with a sense of space for the mixing process of the participant audio stream. The target audio stream can also be determined by the following embodiment.

[0095] Figure 6 The present invention is a flowchart of determining a target audio stream according to a sight line direction and a spatial angle according to an embodiment of the present invention. Figure 6 Yes Figure 4 Step 402 is specifically described based on Figure 6 The embodiment shown comprises the following steps:

[0096] Step 601, when the spatial angle is within the second visual range, record the visual movement trajectory formed by the visual direction.

[0097] In this embodiment, the line of sight movement trajectory is obtained by tracking and recording the line of sight changes when the spatial angle of the first object is within the second line of sight, and the line of sight tracking module of the wearable virtual reality device is tracking the line of sight, which is a continuous coordinate point.

[0098] Step 602: Compare the sight line movement trajectory with the preset trajectory for similarity.

[0099] In this embodiment, the preset trajectory is the trajectory of the line of sight formed when the first object locks the user, which is obtained by laboratory recording and analysis. The line of sight movement trajectory is compared with the preset trajectory for similarity, such as by comparing the degree of overlap of the curves, to analyze the similarity between the trajectory of the line of sight locking the user and the preset trajectory.

[0100] Step 603: If the similarity is greater than the first threshold, determine the original target audio stream of the second object.

[0101] In this embodiment, after comparing the similarity of the line of sight movement trajectory with the preset trajectory, if the preset trajectory has a similarity with the line of sight movement trajectory greater than a first threshold, the candidate object is determined as the second object, and the original target audio stream of the second object is extracted.

[0102] Step 604: Process the original target audio stream through head-related transformation to obtain a target audio stream.

[0103] In this embodiment, after the original target audio stream of the second object is obtained, in order to obtain a sound effect with a spatial stereoscopic sense, a head-related transform HRTF is performed on the original target audio stream to obtain an audio stream with a spatial sense.

[0104] In this embodiment, by determining the second object, the original target audio stream of the second object is obtained, and then the original audio stream is processed into the target audio stream, and the target audio stream to be mixed is determined, providing an audio stream data source with a sense of space for the mixing process of the audio streams of the participants. Among them, determining the second object is particularly critical, which can be achieved through the following embodiments.

[0105] Figure 7 This is a flowchart of an embodiment of the present disclosure for determining an original target audio stream of a second object according to a sight direction, a second sight range, a spatial angle, and a dwell time. Figure 7 Yes Figure 5 Step 502 is specifically described based on Figure 7 The embodiment shown comprises the following steps:

[0106] Step 701: If the dwell time is greater than or equal to the first duration threshold, at least one of the one or more candidate objects within the second visual range is determined as the second object, and at least one of the one or more candidate objects outside the second visual range is determined as the third object.

[0107] In this embodiment, the second object is the target object currently focused on by the first object. When the duration of stay in the second line of sight according to the line of sight direction of the first object and the spatial angle determined in any other candidate object is greater than or equal to the first duration threshold, one or more candidate objects in the second line of sight are confirmed as the second object, thereby focusing the audio stream of the second object. At the same time, one or more candidate objects outside the second line of sight are confirmed as third objects. The third object is at least one object other than the second object among the one or more candidate objects. Optionally, the first duration threshold is 3 seconds.

[0108] Step 702: If the stay time is less than the first duration threshold, at least one object among the one or more candidate objects within the second visual range is determined as a third object.

[0109] In this embodiment, when the duration of the spatial angle obtained according to the sight direction of the first object within the second sight range is less than the first duration threshold, at least one of the one or more candidate objects within the second sight range is taken as the third object.

[0110] In this implementation, the second object and the third object are determined by comparing the spatial angle of the sight line direction of the first object with the second sight line range, thereby providing processing objects for focusing processing of the audio stream.

[0111] Figure 8 The present invention is a flowchart of processing an audio stream of at least one candidate object based on a target audio stream according to an embodiment of the present disclosure. Figure 8 Yes Figure 2 Step 203 and Figure 5 To explain in detail, based on Figure 8 The embodiment shown comprises the following steps:

[0112] Step 801: Process a target audio stream as a first audio stream according to a first weight; and / or process an audio stream of at least one object among one or more candidate objects excluding a second object as a second audio stream according to a second weight.

[0113] The first weight is a weight parameter of the target audio stream of the second object, and at least one object excluding the second object from one or more candidate objects is taken as the third object, and the second weight is a weight parameter of the audio stream of the third object. The first audio stream is obtained by multiplying the target audio stream by the first weight, and the second audio stream is obtained by multiplying the audio stream of the third object by the second weight.

[0114] In one implementation of this embodiment, the target audio stream is processed as the first audio stream by the first weight. For example, the first weight is set to 1.5 only for the second object, and the audio stream of the third object is not processed, and the weight is 1 by default, thereby enhancing the audio stream of the second object.

[0115] In one implementation of this embodiment, the audio stream of the third object is processed as the second audio stream by the second weight. For example, the second weight is set to 0.25 only for the third object, and the first weight is 1 by default, so as to weaken the audio stream of the third object.

[0116] In one implementation of this embodiment, the target audio stream is processed as the first audio stream by the first weight, and the audio stream of the third object is processed as the second audio stream according to the second weight. For example, the first weight is 1.75 and the second weight is 0.25, so that the audio stream of the second object is enhanced and the audio stream of the third object is weakened.

[0117] Step 802: Merge the first audio stream and the second audio stream.

[0118] The first audio stream and the second audio stream can be added together to serve as the output audio of the wearable virtual reality device worn by the first object, thereby focusing on the sound of the second object, increasing the attention to the sound of the second object, and improving the user experience.

[0119] Fig. 9This is a schematic diagram of an audio focusing process following the line of sight of an embodiment of the present disclosure. In this embodiment, after the user puts on the AR / VR device and headphones, the video conference is started. Participants appear on the screen as cards / human figures / anthropomorphic images (speaker1, speaker2, speaker3). The spatial angle θ corresponding to each participant is automatically calibrated, and the audio stream of each user is processed through HRTF for spatial sound rendering. The line of sight tracking module embedded in the AR / VR device tracks the direction of the user's line of sight and returns the corresponding angle θ1 (taking speaker1 as an example). The AR / VR device accepts the angle value returned by the line of sight tracking module. And according to the angle value, the audio stream corresponding to the speaker is found, the gain Gain1 is increased for this audio stream, and the gains Gain2 and Gain3 are reduced for the audio streams corresponding to speakers in other directions. Thereby, the voice of the target user who is speaking is enhanced, and the voices of others are weakened, so as to achieve the purpose of focusing the sound.

[0120] Through the audio processing method proposed in the present invention, the electronic device obtains the line of sight direction of the first object and displays the identity icons of one or more candidate objects. The electronic device is used to output audio streams related to the one or more candidate objects, providing a directional basis for sound focusing; according to the line of sight direction, it is detected that the identity icon of the second object among the one or more candidate objects satisfies a preset relationship with the line of sight direction, the target audio stream related to the second object is determined, and the target audio is directionally acquired; based on the target audio stream, the audio stream of at least one candidate object is processed, and the sound is focused according to the sound attention level, thereby improving the user experience.

[0121] Corresponding to the methods provided in the above-mentioned embodiments, the present disclosure also provides an audio processing device. Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation method of the method is also applicable to the device provided in this embodiment and will not be described in detail in this embodiment.

[0122] Fig.10 FIG. 1 is a schematic diagram of the structure of an audio processing device 1000 according to an embodiment of the present disclosure. The device is applied to wearable virtual reality devices, such as Fig.10 As shown, the audio processing device comprises:

[0123] The acquisition module 1010 is used by the electronic device to acquire the sight line direction of the first object and display the identity icons of one or more candidate objects, and the electronic device is used to output the audio stream related to the one or more candidate objects.

[0124] The determination module 1020 is configured to detect, based on the sight line direction, that an identity icon of a second object among one or more candidate objects satisfies a preset relationship with the sight line direction, and determine a target audio stream related to the second object.

[0125] The processing module 1030 is configured to process the audio stream of at least one candidate object based on the target audio stream.

[0126] In some embodiments, before the electronic device acquires the sight line direction of the first object and displays the identity icons of one or more candidate objects, the acquisition module 1010 is further configured to:

[0127] Get the first sight range of each candidate object;

[0128] According to the first sight range of each candidate object, the second sight range of the first object for each candidate object is calibrated, and the second sight range is used for: when the sight direction falls into the second sight range, the candidate object corresponding to the second sight range is the second object.

[0129] In some embodiments, the determination module 1020 is configured to:

[0130] Determine a spatial angle between a sight line direction and a line connecting an identity icon of the second object and the first object;

[0131] Determine the target audio stream based on the sight direction and spatial angle.

[0132] In some embodiments, the determination module 1020 determines the target audio stream according to the sight line direction and the spatial angle in the following manner:

[0133] Determine the time the spatial angle stays within the second visual range;

[0134] Determine an original target audio stream of the second object according to the sight direction, the second sight range, the spatial angle, and the dwell time;

[0135] The target audio stream is obtained by processing the original target audio stream through head-related transformation.

[0136] In some embodiments, the determination module 1020 further determines the target audio stream according to the sight line direction and the spatial angle in the following manner:

[0137] When the spatial angle is within the second sight range, the sight movement trajectory formed by the sight direction is recorded;

[0138] Compare the sight movement trajectory with the preset trajectory for similarity;

[0139] If the similarity is greater than a first threshold, determining the original target audio stream of the second object;

[0140] The target audio stream is obtained by processing the original target audio stream through head-related transformation.

[0141] In some embodiments, the determination module 1020 determines the original target audio stream of the second object according to the sight direction, the second sight range, the spatial angle, and the dwell time in the following manner:

[0142] If the stay time is greater than or equal to the first duration threshold, at least one of the one or more candidate objects within the second visual range is determined as the second object, and at least one of the one or more candidate objects outside the second visual range is determined as the third object;

[0143] If the stay time is less than the first duration threshold, at least one of the one or more candidate objects within the second visual range is determined as the third object.

[0144] In some embodiments, the processing module 1030 is used to:

[0145] According to the first weight, processing the target audio stream as the first audio stream; and / or

[0146] processing, according to the second weight, an audio stream of at least one object among the one or more candidate objects as a second audio stream;

[0147] The first audio stream and the second audio stream are merged.

[0148] In summary, through the audio processing device, the electronic device obtains the line of sight direction of the first object and displays the identity icons of one or more candidate objects. The electronic device is used to output audio streams related to one or more candidate objects, providing a directional basis for sound focusing; according to the line of sight direction, it is detected that the identity icon of the second object among the one or more candidate objects satisfies the preset relationship with the line of sight direction, and the target audio stream related to the second object is determined, and the target audio is directionally acquired; based on the target audio stream, the audio stream of at least one candidate object is processed, and the sound is focused according to the sound attention, thereby improving the user experience.

[0149] In the embodiments provided in the present application, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the functions in the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. A function of the functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.

[0150] Fig.11 is a block diagram of an electronic device 1100 for implementing the above audio processing method according to an exemplary embodiment.

[0151] For example, the electronic device 1100 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0152] Reference Fig.11 , the electronic device 1100 may include one or more of the following components: a processing component 1102 , a memory 1104 , a power component 1106 , a multimedia component 1108 , an audio component 1110 , an input / output (I / O) interface 1112 , a sensor component 1114 , and a communication component 1116 .

[0153] The processing component 1102 generally controls the overall operation of the electronic device 1100, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1102 may include one or more processors 1120 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 1102 may include one or more modules to facilitate the interaction between the processing component 1102 and other components. For example, the processing component 1102 may include a multimedia module to facilitate the interaction between the multimedia component 1108 and the processing component 1102.

[0154] The memory 1104 is configured to store various types of data to support operations on the electronic device 1100. Examples of such data include instructions for any application or method operating on the electronic device 1100, contact data, phone book data, messages, pictures, videos, etc. The memory 1104 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0155] The power supply component 1106 provides power to various components of the electronic device 1100. The power supply component 1106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1100.

[0156] The multimedia component 1108 includes a screen that provides an output interface between the electronic device 1100 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1108 includes a front camera and / or a rear camera. When the electronic device 1100 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0157] The audio component 1110 is configured to output and / or input audio signals. For example, the audio component 1110 includes a microphone (MIC), and when the electronic device 1100 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1104 or sent via the communication component 1116. In some embodiments, the audio component 1110 also includes a speaker for outputting audio signals.

[0158] I / O interface 1112 provides an interface between processing component 1102 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0159] The sensor assembly 1114 includes one or more sensors for providing various aspects of status assessment for the electronic device 1100. For example, the sensor assembly 1114 can detect the open / closed state of the electronic device 1100, the relative positioning of components, such as the display and keypad of the electronic device 1100, and the sensor assembly 1114 can also detect the position change of the electronic device 1100 or a component of the electronic device 1100, the presence or absence of user contact with the electronic device 1100, the orientation or acceleration / deceleration of the electronic device 1100, and the temperature change of the electronic device 1100. The sensor assembly 1114 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1114 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1114 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0160] The communication component 1116 is configured to facilitate wired or wireless communication between the electronic device 1100 and other devices. The electronic device 1100 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio) or a combination thereof. In an exemplary embodiment, the communication component 1116 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component A16 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0161] In an exemplary embodiment, the electronic device 1100 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0162] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1104 including instructions, and the instructions can be executed by a processor 1120 of the electronic device 1100 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0163] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the audio processing method described in the above embodiments of the present disclosure.

[0164] The embodiments of the present disclosure further provide a computer program product, including a computer program, which executes the audio processing method described in the above embodiments of the present disclosure when a processor is used to execute the computer program.

[0165] Fig.12 is a schematic structural diagram of a chip 1200 for implementing the above audio processing method according to an exemplary embodiment.

[0166] Reference Fig.12The chip 1200 includes at least one communication interface 1201 and a processor 1202; the communication interface 1201 is used to receive a signal input to the chip 1200 or a signal output from the chip 1200, and the processor 1202 communicates with the communication interface 1201 and implements the audio processing method described in the above embodiment through a logic circuit or executing code instructions.

[0167] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0168] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0169] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0170] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0171] It should be understood that the various parts of the embodiments of the present disclosure can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0172] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0173] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0174] Although the embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present disclosure. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. An audio processing method, characterized in that: The method comprises: The electronic device acquires the sight line direction of the first object and displays identity icons of one or more candidate objects, and the electronic device is used to output audio streams related to the one or more candidate objects; According to the sight line direction, detecting that an identity icon of a second object among the one or more candidate objects satisfies a preset relationship with the sight line direction, and determining a target audio stream related to the second object; Based on the target audio stream, an audio stream of at least one of the candidate objects is processed.

2. The method according to claim 1, characterized in that Before the electronic device acquires the sight line direction of the first object and displays the identity icons of one or more candidate objects, the electronic device further includes: Get the first sight range of each candidate object; According to the first sight range of each candidate object, the second sight range of the first object for each candidate object is calibrated, and the second sight range is used for: when the sight direction falls into the second sight range, the candidate object corresponding to the second sight range is the second object.

3. The method according to claim 1, characterized in that The detecting, according to the sight line direction, that an identity icon of a second object among the one or more candidate objects satisfies a preset relationship with the sight line direction, and determining a target audio stream related to the second object includes: Determine a spatial angle between the sight line direction and a line connecting the identity icon of the second object and the first object; The target audio stream is determined according to the sight line direction and the spatial angle.

4. The method according to claim 3, characterized in that The determining the target audio stream according to the sight line direction and the spatial angle includes: Determining the residence time of the spatial angle within the second visual range; Determining an original target audio stream of the second object according to the sight direction, the second sight range, the spatial angle, and the dwell time; The target audio stream is obtained by performing head-related transformation on the original target audio stream.

5. The method according to claim 3, characterized in that: The determining the target audio stream according to the sight line direction and the spatial angle includes: When the spatial angle is within the second line of sight range, recording a line of sight movement trajectory formed by the line of sight direction; Comparing the sight line movement trajectory with a preset trajectory for similarity; If the similarity is greater than a first threshold, determining the original target audio stream of the second object; The target audio stream is obtained by performing head-related transformation on the original target audio stream.

6. The method according to claim 4, characterized in that The determining the original target audio stream of the second object according to the sight direction, the second sight range, the spatial angle, and the dwell time comprises: If the stay time is greater than or equal to a first duration threshold, determining at least one of the one or more candidate objects within the second visual range as the second object, and determining at least one of the one or more candidate objects outside the second visual range as a third object; If the stay time is less than the first duration threshold, at least one of the one or more candidate objects within the second visual range is determined as the third object.

7. The method according to claim 4, characterized in that The processing of the audio stream of at least one object among the one or more candidate objects based on the target audio stream comprises: According to the first weight, processing the target audio stream as a first audio stream; and / or processing, according to a second weight, an audio stream of at least one object among the one or more candidate objects excluding the second object as a second audio stream; The first audio stream and the second audio stream are merged.

8. An audio processing device, characterized in that: The device comprises: An acquisition module, configured for the electronic device to acquire the sight line direction of the first object and display identity icons of one or more candidate objects, and the electronic device to output audio streams related to the one or more candidate objects; A determination module, configured to detect, based on the sight line direction, that an identity icon of a second object among the one or more candidate objects satisfies a preset relationship with the sight line direction, and determine a target audio stream related to the second object; A processing module is used to process an audio stream of at least one of the candidate objects based on the target audio stream.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

11. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.

12. A chip, characterized in that: It includes at least one processor and a communication interface; the communication interface is used to receive a signal input to the chip or a signal output from the chip, and the processor communicates with the communication interface and implements the method according to any one of claims 1-7 through a logic circuit or executing code instructions.

Images