Sound pickup processing method and device thereof, wearable sound pickup device, and storage medium

Abstract

The application provides a pickup processing method and device, a wearable pickup device and a computer readable storage medium. The pickup processing method comprises: performing hand pointing detection based on a hand device connected to the wearable pickup device to obtain a first pointing direction; performing gesture pointing detection based on a camera of the wearable pickup device to obtain a second pointing direction; determining a specified pickup direction of the wearable pickup device based on the first pointing direction and the second pointing direction; establishing a pickup beam with the specified pickup direction as a beam direction to process a sound signal located in the specified pickup direction by using the pickup beam. The application can reduce the problem of easily mis-enhancing non-target speaker sound in a noisy environment, and improve the stability and anti-interference ability of pickup.

Description

Technical Field

[0001] This application relates to the field of sound pickup processing technology, specifically to a sound pickup processing method, apparatus, wearable sound pickup device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of wearable audio pickup devices such as smart glasses and hearing aids, directional audio pickup technology is increasingly being used in these devices. In traditional directional audio pickup scenarios, multi-microphone arrays estimate the spatial location of the sound source and form a directional beam in that direction, effectively enhancing the target speech signal and improving speech clarity. However, in real-world environments, when multiple people are speaking simultaneously, there is strong background noise, or significant reverberation, the sound source direction estimation of the microphone array is easily interfered with, making it difficult to accurately distinguish between the target sound source and interfering sound sources. This can cause the beamformer to incorrectly amplify the voice of non-target speakers, or even amplify noise and reverberation components, thereby reducing the stability and anti-interference capability of the audio pickup. Summary of the Invention

[0003] This application provides a sound pickup processing method, apparatus, wearable sound pickup device, and computer-readable storage medium, which can reduce the problem of accidental amplification of non-target speaker voices in noisy environments and improve the stability and anti-interference ability of sound pickup.

[0004] Firstly, the sound pickup processing method provided in this application includes: The first pointing direction is obtained by detecting the hand pointing using a hand device connected to a wearable microphone. The second pointing direction is obtained by detecting the gesture pointing based on the camera of the wearable microphone device. Based on the first pointing direction and the second pointing direction, the specified sound pickup direction of the wearable sound pickup device is determined; A pickup beam is established with the specified pickup direction as the beam direction, so as to process the sound signal located in the specified pickup direction using the pickup beam.

[0005] Secondly, the sound pickup processing device provided in this application includes: The first detection unit is used to perform hand pointing detection based on a hand device connected to the wearable microphone to obtain a first pointing direction; The second detection unit is used to perform gesture pointing detection based on the camera of the wearable audio pickup device to obtain a second pointing direction; The processing unit is configured to determine a specified pickup direction of the wearable microphone based on the first pointing direction and the second pointing direction; The processing unit is further configured to establish a pickup beam with the specified pickup direction as the beam direction, so as to process the sound signal located in the specified pickup direction using the pickup beam.

[0006] Thirdly, the wearable audio pickup device provided in this application includes a camera, a microphone array, a processor, and a memory. The memory stores a computer program, and when the processor calls the computer program in the memory, it executes any of the audio pickup processing methods provided in this application.

[0007] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to execute the aforementioned sound pickup processing method.

[0008] In this application, firstly, a first pointing direction is obtained by detecting hand pointing based on a hand device connected to the wearable microphone; a second pointing direction is obtained by detecting hand pointing based on the camera of the wearable microphone; and a specified microphone direction is determined based on the first and second pointing directions. Since both the first and second pointing directions directly reflect the user's explicit intent, the microphone direction can be determined using the user's explicit intent, thereby achieving highly accurate and stable directional microphone pickup in noisy environments with multiple users. This can reduce the problem of erroneously amplifying the voice of non-target speakers in noisy environments to a certain extent, and improve the stability and anti-interference capability of microphone pickup. Secondly, by comprehensively determining the specified pointing direction using the first and second pointing directions, a dual-modal redundancy backup can be formed, reducing the microphone pickup interruption problem caused by the failure of a single sensor channel, and improving the reliability and robustness of the microphone pickup direction. Attached Figure Description

[0009] Figure 1 This is a schematic block diagram of a wearable microphone provided in an embodiment of this application; Figure 2 This is a schematic flowchart of a sound pickup processing method provided in an embodiment of this application; Figure 3 This is a schematic flowchart of one embodiment of step 201 in this application; Figure 4 This is a schematic flowchart of one embodiment of step 202 in this application; Figure 5 This is a schematic flowchart of an embodiment of step 203 in this application; Figure 6 This is a schematic diagram of an embodiment of the sound pickup processing device provided in this application. Detailed Implementation

[0010] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0011] This application provides a sound pickup processing method, a sound pickup processing device, a wearable sound pickup device, and a computer-readable storage medium. The sound pickup processing device can be integrated into the wearable sound pickup device. The wearable sound pickup device can be smart glasses, smart helmets, etc. Smart glasses can be AR (augmented reality) glasses, VR (virtual reality) glasses, MR (mixed reality) glasses, XR (eXtended Reality) glasses, etc., and smart helmets can be AR helmets, etc.

[0012] The entity executing the sound pickup processing method in this application embodiment can be the sound pickup processing device provided in this application embodiment, or a wearable sound pickup device that integrates the sound pickup processing device. The sound pickup processing device can be implemented in hardware or software.

[0013] Figure 1 This is a schematic block diagram of a wearable audio pickup device provided in an embodiment of this application.

[0014] like Figure 1 As shown, the wearable audio pickup device 100 includes a processor 101, a memory 102, a camera 103, and a microphone array 104. The processor 101, memory 102, camera 103, and microphone array 104 are connected via a bus, such as an I2C (Inter-integrated Circuit) bus.

[0015] Specifically, processor 101 provides computing and control capabilities to support the operation of the entire wearable microphone 100. Processor 101 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0016] Specifically, the memory 102 can be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a portable hard drive, etc.

[0017] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the embodiments of this application, and does not constitute a limitation on the wearable audio pickup device to which the embodiments of this application are applied. A specific wearable audio pickup device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0018] In some embodiments, the wearable sound pickup device 100 is provided with a microphone array 104, which can be used to identify the direction of the sound source.

[0019] In some embodiments, the wearable microphone 100 is equipped with a camera 103, which is used to acquire images, such as when acquiring images for processing such as gesture pointing direction detection and position tracking of the target object.

[0020] In some embodiments, the wearable microphone 100 is provided with a wireless communication module.

[0021] In some embodiments, a sound pickup processing module (not shown in the figure) is integrated on the processor 101. The sound pickup processing module is used to execute any of the sound pickup processing methods provided in the embodiments of this application.

[0022] In some embodiments, the processor 101 is configured to run a computer program stored in the memory 102, and implement any of the sound pickup processing methods provided in the embodiments of this application when executing the computer program. For example, the processor 101 is configured to run a computer program stored in the memory 102, and may implement the following steps when executing the computer program: A first pointing direction is obtained by detecting hand pointing based on a hand device connected to the wearable audio pickup device; a second pointing direction is obtained by detecting hand pointing based on the camera of the wearable audio pickup device; a specified pickup direction of the wearable audio pickup device is determined based on the first and second pointing directions; an audio pickup beam is established with the specified pickup direction as the beam direction, so as to process the sound signal located in the specified pickup direction using the audio pickup beam.

[0023] For example, the processor 101 is used to run a computer program stored in the memory 102, and when executing the computer program, it can perform the following steps: The wearable sound pickup device is used to detect the direction of the sound source to obtain the sound source detection direction; based on the device motion information corresponding to the first pointing direction, a first fusion weight of the first pointing direction is obtained; based on the recognition image corresponding to the second pointing direction and the hand detection confidence of the recognition image, a second fusion weight of the second pointing direction is obtained; based on the device environment information corresponding to the sound source detection direction, a third fusion weight of the sound source detection direction is obtained; based on the first fusion weight, the second fusion weight, and the third fusion weight, the first pointing direction, the second pointing direction, and the sound source detection direction are fused to obtain the specified sound pickup direction.

[0024] For example, the processor 101 is used to run a computer program stored in the memory 102, and when executing the computer program, it can perform the following steps: Based on the measured distance between the wearable microphone and the hand device, the preliminary spatial position of the hand device relative to the wearable microphone is obtained; based on the initial posture data of the hand device, the first posture data of the hand device relative to the wearable microphone is determined; based on the preliminary spatial position and the first posture data, the hand pointing vector corresponding to the hand device is determined as the first pointing direction.

[0025] For example, the processor 101 is used to run a computer program stored in the memory 102, and when executing the computer program, it can perform the following steps: The first, second, and third reference positions of the wearable microphone device are measured to the distances between themselves and the hand device, respectively, to obtain the first, second, and third measurement distances. The first, second, and third reference positions are three pre-calibrated, non-collinear fixed points. A first reference sphere is obtained with the first reference position as the center and the first measurement distance as the radius. A second reference sphere is obtained with the second reference position as the center and the second measurement distance as the radius. A third reference sphere is obtained with the third reference position as the center and the third measurement distance as the radius. The intersection of the first, second, and third reference spheres is obtained as the preliminary spatial position.

[0026] For example, the processor 101 is used to run a computer program stored in the memory 102, and when executing the computer program, it can perform the following steps: Based on the first posture data, the preliminary spatial position is corrected to obtain the corrected spatial position; based on the corrected spatial position and the first posture data, the hand pointing vector is determined as the first pointing direction.

[0027] For example, the processor 101 is used to run a computer program stored in the memory 102, and when executing the computer program, it can perform the following steps: The system identifies the target object for pickup in the specified pickup direction; once the pickup beam is established, it controls the wearable pickup device to enter a directional pickup mode with the target object as the pickup target; while the wearable pickup device is in the directional pickup mode, the beam direction of the pickup beam is adjusted based on the real-time tracking position of the target object, so as to enhance the sound signal located in the direction of the real-time tracking position using the pickup beam.

[0028] For example, the processor 101 is used to run a computer program stored in the memory 102, and when executing the computer program, it can perform the following steps: The device acquires the degree of occlusion, signal attenuation, and / or tracking confidence of the target audio pickup; updates the locking confidence of the target audio pickup based on the degree of occlusion, the degree of signal attenuation, and / or the tracking confidence; and controls the wearable audio pickup device to exit the directional audio pickup mode when the locking confidence is less than a preset confidence threshold.

[0029] For example, the processor 101 is used to run a computer program stored in the memory 102, and when executing the computer program, it can perform the following steps: The system acquires the locked voiceprint features of the target audio pickup; acquires the voiceprint features of each speaker in the environment where the wearable audio pickup device is located; if the locked voiceprint features do not match the voiceprint features of each speaker, the system controls the wearable audio pickup device to exit the directional audio pickup mode.

[0030] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the wearable audio pickup device described above can be referred to the corresponding process in the following audio pickup processing method embodiments, and will not be repeated here.

[0031] The following will be based on Figure 1 Taking the wearable audio pickup device shown as the execution subject of the audio pickup processing method as an example, the audio pickup processing method provided in this application embodiment will be described in detail. For the sake of simplification and ease of description, the execution subject will be omitted in the subsequent method embodiments.

[0032] Please see Figure 2 , Figure 2 This is a schematic flowchart of a sound pickup processing method provided in an embodiment of this application. The sound pickup processing method includes steps 201-204, wherein: 201. Based on the hand device connected to the wearable microphone, hand pointing is detected to obtain the first pointing direction.

[0033] Wearable audio pickup devices can include smart glasses, smart helmets, etc.

[0034] The hand-held device can be a ring, watch, etc., and it is equipped with a wireless ranging module or wireless direction finding module (such as UWB or Bluetooth AoA), an inertial measurement unit (IMU), and a wireless communication module. Both the wearable microphone and the hand-held device have built-in wireless communication modules, which enables the establishment of a communication connection between them.

[0035] To better understand the embodiments of this application, some of the names involved in the embodiments of this application will be introduced below: 1. First reference coordinate system: refers to the reference coordinate system of the wearable microphone. The first reference coordinate system can be a three-dimensional coordinate system.

[0036] 2. Second reference coordinate system: refers to the reference coordinate system of the hand device. The second reference coordinate system can be a three-dimensional coordinate system.

[0037] 3. Global reference coordinate system: This is a three-dimensional unified reference coordinate system that is independent of hand devices (such as rings) and wearable microphone devices (such as AR glasses). For example, it can be a world coordinate system (such as a fixed coordinate system based on the earth) or a global reference coordinate system preset by the system (such as a fixed coordinate system with the initial wearing position of the wearable microphone device as the origin).

[0038] The first pointing direction is the hand pointing direction obtained by hand pointing detection based on the hand device, and the first pointing direction is specifically the direction vector in the first reference coordinate system.

[0039] There are multiple ways to implement step 201, including, for example: (1) In some embodiments, such as Figure 3 As shown, step 201 may specifically include the following steps 2011A to 2013A: 2011A. Based on the measured distance between the wearable microphone and the hand device, obtain the preliminary spatial position of the hand device relative to the wearable microphone.

[0040] The preliminary spatial position is the spatial position of the hand device relative to the wearable microphone based on the measured distance between the wearable microphone and the hand device. The preliminary spatial position is the coordinate position of the hand device in the first reference coordinate system.

[0041] In some embodiments, the wearable microphone calculates a preliminary spatial position based on three reference positions. In this case, step 2011A may specifically include: obtaining the measured distances between the first reference position, the second reference position, and the third reference position of the wearable microphone and the hand device, respectively, to obtain a first measured distance, a second measured distance, and a third measured distance; obtaining a first reference sphere with the first reference position as the center and the first measured distance as the radius; obtaining a second reference sphere with the second reference position as the center and the second measured distance as the radius; obtaining a third reference sphere with the third reference position as the center and the third measured distance as the radius; obtaining the intersection point of the first reference sphere, the second reference sphere, and the third reference sphere as the preliminary spatial position.

[0042] Among them, the first reference position, the second reference position, and the third reference position are the coordinate positions of three pre-marked, non-collinear fixed points on the wearable audio pickup device (such as smart glasses) in the first reference coordinate system (such as the end point of the left temple, the end point of the right temple, and the point in the frame of the smart glasses).

[0043] For example, firstly, wearable audio pickup devices (such as smart glasses) and hand devices (such as rings) establish a connection through a built-in wireless communication module; after the wearable audio pickup device and the hand device establish a communication connection, the wearable audio pickup device sends ranging signals to the hand device with three reference positions as signal transmission starting points respectively. After receiving the ranging signal from each reference position, the hand device feeds back a response signal through the wireless communication module. The wearable microphone calculates the straight-line distance between the three reference positions and the hand device based on at least one ranging technology, such as TDoA (Time Difference of Arrival), RToF (Round-Trip Time of Flight), or AoA (Angle of Arrival). For example, a first ranging signal is sent to the hand device from the first reference position as the signal transmission starting point. After receiving the first ranging signal, the hand device feeds back a first response signal to the wearable microphone. The wearable microphone calculates the straight-line distance between the first reference position and the hand device based on the received first response signal and the ranging technology, and uses this distance as the first measured distance between the first reference position and the hand device. A second ranging signal is sent to the hand device from a second reference position as the signal transmission starting point. Upon receiving the second ranging signal, the hand device sends a second response signal back to the wearable microphone. Based on the received second response signal, the wearable microphone calculates the straight-line distance between the second reference position and the hand device using ranging technology, and uses this as the second measured distance between the two. A third ranging signal is sent to the hand device from a third reference position as the signal transmission starting point. Upon receiving the third ranging signal, the hand device sends a third response signal back to the wearable microphone. Based on the received third response signal, the wearable microphone calculates the straight-line distance between the third reference position and the hand device using ranging technology, and uses this as the third measured distance between the two.

[0044] Then, in the first reference coordinate system, a sphere is drawn with the coordinates of the first reference position as the center and the first measured distance as the radius. This sphere is the first reference sphere, which mathematically represents the set of all points in space whose distance from the first reference position is equal to the first measured distance. Therefore, the actual position of the hand device will fall on this sphere. Similarly, a second reference sphere is drawn with the second reference position as the center and the second measured distance as the radius. The actual position of the hand device also falls on this sphere. Therefore, the actual position of the hand device should be a point on the intersection line (space circle) of the first and second reference spheres. Theoretically, a third reference sphere is constructed with the third reference position as its center and the third measurement distance as its radius. The intersection of this third reference sphere with the aforementioned spatial circle is the unique position that simultaneously satisfies the condition that the distances to the three reference positions (i.e., the first reference position, the second reference position, and the third reference position) are equal to the corresponding measurement distances (i.e., the first measurement distance, the second measurement distance, and the third measurement distance), respectively (after excluding redundant intersections caused by measurement errors, valid intersections are obtained through algorithmic filtering). The coordinates of this intersection point are the preliminary spatial position Pring(xr, yr, zr) of the hand device relative to the wearable microphone. In this way, by using the intersection of three reference spheres (i.e., the first reference sphere, the second reference sphere, and the third reference sphere) for positioning, three-dimensional positioning of the hand device can be achieved based on spatial geometry principles. Compared with two-dimensional positioning or two-point positioning methods, its positioning accuracy is higher, and it can effectively avoid the positional ambiguity problems existing in planar positioning, providing a reliable positional reference for subsequently determining the accurate hand pointing direction by combining the hand device's posture data. For example, when a user wears smart glasses and points using the ring, the ring's precise location in space can be accurately determined by the intersection of these three reference spheres, ensuring that the subsequently calculated pointing direction matches the user's actual pointing intention. Figure 1 This provides accurate directional input for directional sound pickup.

[0045] In some embodiments, the wearable microphone is used for positioning based on three reference positions combined with the posture data of the hand device. In this case, step 2011A may specifically include: obtaining the measured distances between the first reference position, the second reference position, and the third reference position of the wearable microphone and the hand device, respectively, to obtain a first measured distance, a second measured distance, and a third measured distance; determining the first posture data of the hand device relative to the wearable microphone based on the initial posture data of the hand device; obtaining a first reference sphere with the first reference position as the center and the first measured distance as the radius; and using the second reference position... Using the first reference sphere as the center and the second measured distance as the radius, obtain a second reference sphere; using the third reference position as the center and the third measured distance as the radius, obtain a third reference sphere; obtain candidate intersection points of the first reference sphere, the second reference sphere, and the third reference sphere (including two or more redundant intersection points caused by measurement errors); determine the pointing space range of the hand device based on the first attitude data (for example, the forward space area corresponding to the heading angle of the hand device when the user performs a pointing operation); filter the intersection points located within the pointing space range from the candidate intersection points to use as the preliminary spatial position.

[0046] The specific implementation of "obtaining the measurement distances between the first reference position, the second reference position, and the third reference position of the wearable microphone and the hand device, respectively, to obtain the first measurement distance, the second measurement distance, and the third measurement distance" can be found in the relevant description above, and will not be repeated here.

[0047] The specific implementation of "determining the first posture data of the hand device relative to the wearable microphone based on the initial posture data of the hand device" can be found in the relevant part of step 2012A, and will not be repeated here.

[0048] For example, in the first reference coordinate system, the intersection of three reference spheres may generate multiple candidate intersection points (ideally one under error-free conditions, but in actual measurements, due to signal interference, equipment errors, etc., two or more redundant intersection points may occur). When a user wears a handheld device to perform pointing operations, the first attitude data has a clear regularity (for example, the heading angle usually points directly forward, and the pitch and roll angles are within a reasonable range for a natural arm raise). Based on the first attitude data, the pointing space range of the handheld device can be defined. By matching the candidate intersection point positions with this pointing space range, redundant intersection points that exceed the reasonable range can be quickly eliminated, and valid intersection points that match the user's operating intentions can be selected. The coordinates of this valid intersection point are the preliminary spatial position of the handheld device relative to the wearable microphone.

[0049] This positioning method, based on three reference positions (i.e., the first, second, and third reference positions) combined with first attitude data, can further optimize the intersection point selection logic by utilizing the hand device's attitude characteristics. It effectively avoids redundant intersection point interference caused by measurement errors. Compared to positioning methods that simply rely on the intersection of three reference spheres, it has stronger anti-interference capabilities and better accuracy and stability of the initial spatial position. For example, in complex environments where slight interference to the ranging signal leads to multiple candidate intersection points on the three reference spheres, the effective intersection point can be accurately locked using the pointing space range defined by the first attitude data. This avoids subsequent deviations in hand pointing direction calculations due to incorrect intersection point selection, ensuring the reliability of directional sound input.

[0050] 2012A. Based on the initial posture data of the hand device, determine the first posture data of the hand device relative to the wearable microphone device.

[0051] The initial attitude data is the attitude parameters of the hand device relative to the second reference coordinate system, which are collected by the inertial measurement unit (IMU) built into the hand device. These parameters include three core attitude parameters: pitch angle, roll angle, and yaw angle. The pitch angle is used to characterize the degree of tilt of the hand device about the horizontal axis of the second reference coordinate system (e.g., the angle at which the front of the ring tilts upward when the user raises their hand). The roll angle is used to characterize the degree of rotation of the hand device about the vertical axis of the second reference coordinate system (e.g., the rotation angle of the ring when the user flips their wrist). The yaw angle is used to characterize the pointing orientation of the hand device on the horizontal plane of the second reference coordinate system (e.g., the angle at which the front of the ring is aligned with the front, left, or right).

[0052] The first attitude data refers to the attitude parameters of the hand device relative to the first reference coordinate system, including the pitch angle, roll angle and yaw angle of the hand device relative to the first reference coordinate system.

[0053] The second attitude data refers to the attitude parameters of the hand device relative to the global reference coordinate system, including the pitch angle, roll angle, and yaw angle of the hand device relative to the global reference coordinate system.

[0054] For example, the inertial measurement unit (IMU) of the handheld device can continuously collect its own motion state data. By parsing this motion state data, three core attitude parameters of the handheld device relative to the second reference coordinate system—pitch angle, roll angle, and yaw angle—can be obtained as initial attitude data. After obtaining the initial attitude data by parsing its own motion state data, the handheld device sends the initial attitude data to the wearable microphone. Upon receiving the initial attitude data, the wearable microphone first transforms the initial attitude data from the second reference coordinate system to the global reference coordinate system based on a preset transformation rule between the second reference coordinate system and the global coordinate system, obtaining the second attitude data of the handheld device in the global reference coordinate system. Then, referring to the preset transformation rule between the first reference coordinate system and the global reference coordinate system, it transforms the second attitude data in the global reference coordinate system back to the first reference coordinate system, obtaining the first attitude data in the first reference coordinate system. Thus, by calculating the pitch angle, roll angle and yaw angle together, the orientation deviation caused by mapping a single attitude parameter or two attitude parameters can be eliminated, and the actual pointing of the hand device in three-dimensional space can be accurately determined, providing a comprehensive and accurate attitude basis for subsequent calculation of the hand pointing vector.

[0055] 2013A. Based on the preliminary spatial position and the first posture data, determine the hand pointing vector corresponding to the hand device as the first pointing direction.

[0056] Among them, the hand pointing vector refers to the spatial direction in which the hand device is aligned with the first reference coordinate system when the user makes a pointing action by wearing the hand device. This allows the hand pointing vector to directly reflect the user's explicit sound pickup intention.

[0057] For example, firstly, the initial direction vector is determined with the origin of the first reference coordinate system as the starting point and the preliminary spatial position as the ending point. For instance, assuming that the origin O (0,0,0) of the first reference coordinate system is the center point of the microphone array of the wearable sound pickup device, and the preliminary spatial position determined in step 2011A is Pring (xr,yr,zr), then with the origin O as the starting point and Pring as the ending point, the initial direction vector V0=(xr-0,yr-0,zr-0)=(xr,yr,zr) is obtained. Then, a rotation matrix [R_roll, R_pitch, R_yaw] is constructed based on the first attitude data, where R_roll, R_pitch, and R_yaw represent the roll, pitch, and yaw angles of the hand device relative to the first reference coordinate system, respectively. The initial direction vector V0 is multiplied by the rotation matrix [R_roll, R_pitch, R_yaw] to correct the pointing deviation caused by the hand attitude rotation, resulting in the corrected vector V_corr = R_roll × R_pitch × R_yaw × V0. Finally, the corrected vector V_corr is normalized, such as by converting it to a unit vector, to eliminate the influence of distance on direction. The final result is the hand pointing vector of the hand device in the first reference coordinate system, which serves as the first pointing direction.

[0058] By combining preliminary spatial location and initial posture data, the calculation method can convert the user's physical pointing action into explicit spatial direction data (i.e., the first pointing direction). Compared to calculation methods that rely solely on location or posture, this method offers higher pointing accuracy and can accurately capture the user's explicit audio pickup intention, providing precise directional input for the subsequent establishment of directional audio beams. For example, in a meeting scenario, when a user wearing smart glasses and a ring points at the speaker opposite them, the first pointing direction can be accurately calculated, ensuring that the subsequent audio beam can be precisely aimed at the speaker, enhancing their voice signal.

[0059] (2) In some embodiments, step 201 may specifically include the following steps 2011B to 2014B: 2011B. Based on the measured distance between the wearable microphone and the hand device, obtain the preliminary spatial position of the hand device relative to the wearable microphone.

[0060] 2012B. Based on the initial posture data of the hand device, determine the first posture data of the hand device relative to the wearable microphone device.

[0061] The implementation of steps 2011B~2012B is similar to that of steps 2011A~2012A. For details, please refer to the relevant explanations above, which will not be repeated here.

[0062] 2013B. Based on the first attitude data, the preliminary spatial position is corrected to obtain the corrected spatial position.

[0063] For example, based on preset ergonomic parameters (wherein the ergonomic parameters are used to indicate the fixed straight-line distance L from the wearing position of the hand device to the fingertip, such as 8cm) and the first posture data, the fingertip offset vector ΔP(Δx,Δy,Δz) is derived and calculated through spatial set operations. For example, when the pitch angle Pitch_r corresponding to the first posture data is 30°, ΔP(0,L×sin30°,L×cos30°) is calculated. Then, the preliminary spatial position Pring(xr,yr,zr) is added to the fingertip offset vector ΔP to obtain the corrected spatial position P_finger(xr+Δx,yr+Δy,zr+Δz). Thus, by correcting the initial spatial position based on the first posture data, the corrected spatial position can reflect the position of the fingertip in the first reference coordinate system. This can compensate for the inconsistency between the fingertip position and the wearing position of the hand device, thereby compensating for the error of the initial spatial position (i.e., compensating for the wireless ranging error). As a result, the hand pointing vector calculated subsequently is closer to the direction of the user's intention, thereby improving the accuracy of the specified sound pickup direction.

[0064] 2014B. Based on the corrected spatial position and the first posture data, determine the hand pointing vector as the first pointing direction.

[0065] The implementation of step 2014B is similar to that of step 2013A. The difference is that step 2014B uses the corrected spatial position for calculation, while step 2013A uses the preliminary spatial position for calculation. For details, please refer to the relevant explanations above, which will not be repeated here.

[0066] 202. Based on the camera of the wearable audio pickup device, perform gesture pointing detection to obtain a second pointing direction.

[0067] The second pointing direction is obtained by detecting the gesture pointing based on the camera of the wearable audio pickup device. Specifically, the second pointing direction can be a direction vector in the first reference coordinate system, which is functionally equivalent to the first pointing direction and can be used independently to represent the user's explicit audio pickup intention.

[0068] For example, such as Figure 4 As shown, step 202 may specifically include the following steps 2021A to 2024A: 2021A. Real-time image frames are captured through the camera of the wearable audio pickup device.

[0069] For example, after the wearable audio pickup device enters the directional audio pickup preparation state, the camera automatically starts and continuously acquires real-time image frames in front of the user. During the acquisition process, the timestamps of the image frames are recorded synchronously to ensure that the data timing is consistent with that of other modules. The acquired image frames need to cover a reasonable range of the user's hand movements (such as a field of view area of ​​±60° horizontally and ±45° vertically centered on the camera lens) to avoid missing hand pointing actions due to field of view limitations.

[0070] 2022A. Based on a preset hand key point detection algorithm, identify pointing gestures in the real-time image frame.

[0071] Among them, pointing gestures refer to specific postures by which users express pointing intentions through hand movements, including but not limited to the standard pointing posture of "index finger straight, other fingers bent" (such as the index finger extending in a straight line with the palm, the middle finger, ring finger, and little finger bent and close to the palm, and the thumb naturally bent and close to the base of the index finger).

[0072] For example, hand keypoint detection algorithms (such as MediaPipe Hand, YOLO-Pose, etc.) can be used to parse the preprocessed image frames and extract key coordinate points of the hand, including wrist keypoints, palm keypoints, fingertip keypoints, and finger joint keypoints. Based on the extracted keypoint coordinates, it is verified whether the preset pointing gesture determination rules are met: for example, the slope of the line connecting the index fingertip and the wrist keypoint is calculated to determine whether the index finger is in a straight state; the angle between the fingertip of the other fingers (middle finger, ring finger, little finger) and the corresponding finger joint keypoint is calculated to determine whether they are in a bent state; if the feature threshold of "index finger straight and other fingers bent" is met (e.g., index finger straightness ≥ 90%, other fingers bending ≥ 60%), then the image frame is determined to contain a pointing gesture; if not, it is determined to be a non-pointing gesture, and the next frame image is collected.

[0073] 2023A. Extract the wrist spatial position, fingertip spatial position, and finger direction vector corresponding to the pointing gesture.

[0074] For example, if a pointing gesture is detected in an image frame, target key points are selected from the extracted hand key points: the coordinates of the wrist key point are used as the wrist position (Wx, Wy), and the coordinates of the index fingertip key point are used as the fingertip position (Fx, Fy). Based on camera intrinsic parameters and a preset pixel-space mapping relationship, the wrist position (Wx, Wy) and fingertip position (Fx, Fy) are converted from the two-dimensional pixel coordinates of the image frame to three-dimensional spatial coordinates in the first reference coordinate system, resulting in the wrist spatial position (Wx3d, Wy3d, Wz3d) and fingertip spatial position (Fx3d, Fy3d, Fz3d). Simultaneously, with the wrist spatial position as the starting point and the fingertip spatial position as the ending point, the finger direction vector (Fx-Wx, Fy-Wy) in the image plane is calculated, and the initial direction of the finger in space can be further deduced by combining the three-dimensional spatial coordinates, providing a basis for subsequent direction calculations.

[0075] 2024A. Based on the wrist spatial position, the fingertip spatial position, and the finger direction vector, the second pointing direction in the first reference coordinate system is calculated.

[0076] For example, firstly, taking the origin of the first reference coordinate system (such as the center of the microphone array of a wearable audio pickup device) as the starting point and the fingertip spatial position (Fx3d, Fy3d, Fz3d) as the ending point, the initial gesture pointing vector V_gesture0=(Fx3d-0, Fy3d-0, Fz3d-0) is obtained. Then, combining the relative relationship between the wrist spatial position and the fingertip spatial position, the initial gesture pointing vector is corrected (for example, obtaining the wrist three-dimensional spatial position and fingertip three-dimensional spatial position corresponding to the same frame of gesture image, using the wrist position as the hand reference endpoint and the fingertip position as the extension endpoint, constructing the relative vector from the wrist to the fingertip; then, obtaining the spatial angle between the initial gesture pointing vector and the relative vector; with the goal of reducing the spatial angle, adjusting the three-dimensional components corresponding to the initial gesture pointing vector to achieve the correction of the initial gesture pointing vector), resulting in the corrected gesture pointing vector V_gesture1. Finally, V_gesture1 is normalized and converted into a unit vector (magnitude = 1) to eliminate the influence of distance on direction, and finally the gesture pointing direction in the first reference coordinate system is obtained, which is the second pointing direction.

[0077] In this way, the user's gesture pointing action can be transformed into a clear spatial direction vector (second pointing direction) in the first reference coordinate system. This second pointing direction directly reflects the user's explicit sound pickup intention and is in the same coordinate system as the first pointing direction, providing a unified reference for the determination of the specified sound pickup direction in the subsequent step 203.

[0078] 203. Based on the first pointing direction and the second pointing direction, determine the specified sound pickup direction of the wearable sound pickup device.

[0079] There are several ways to implement step 203, including, for example: (1) In some embodiments, step 203 may specifically include the following steps 2031A to 2033A: 2031A. Based on the device motion information corresponding to the first pointing direction, obtain the first fusion weight of the first pointing direction.

[0080] 2032A. Based on the recognition image corresponding to the second pointing direction and the hand detection confidence of the recognition image, obtain the second fusion weight of the second pointing direction.

[0081] The implementation of steps 2031A to 2032A is similar to that of steps 2031B to 2032B. For details, please refer to the relevant explanations below. They will not be repeated here.

[0082] 2033A. Based on the first fusion weight and the second fusion weight, the first pointing direction and the second pointing direction are fused to obtain the specified pickup direction.

[0083] For example, the first pointing direction (D1) and the second pointing direction (D2) can both be converted into unit vectors in the first reference coordinate system, and then weighted and summed according to the first fusion weight and the second fusion weight. The result is used as the specified pickup direction Dtarget.

[0084] (2) In some embodiments, the sound pickup processing method further includes: detecting the direction of the sound source based on the wearable sound pickup device to obtain the direction of sound source detection. For example Figure 5 As shown, step 203 may specifically include the following steps 2031B to 2034B: 2031B. Based on the device motion information corresponding to the first pointing direction, obtain the first fusion weight of the first pointing direction.

[0085] Among them, the device motion information refers to the motion information of the hand device when the hand is pointed in the first pointing direction, including but not limited to the motion speed, motion acceleration, motion stability (such as whether there is shaking), and attitude change rate of the hand device. This information can be obtained by parsing the motion state data collected by the inertial measurement unit (IMU) built into the hand device.

[0086] For example, the IMU of the hand device continuously collects three-axis acceleration and three-axis angular velocity data. The raw data is smoothed by filtering algorithms (such as complementary filtering and Kalman filtering) to remove noise interference, and then the motion speed of the hand device (based on acceleration integral), motion acceleration (the effective value of the filtered three-axis acceleration), and attitude change rate (the attitude angle change amplitude calculated based on angular velocity) are calculated.

[0087] The first fusion weight is used to quantify the reliability of the first pointing direction, with a value ranging from 0 to 1. A higher weight indicates a more reliable first pointing direction. Quantifying the first fusion weight using device motion information objectively reflects the stability and reliability of hand pointing actions. For example, when a user completes a pointing action slowly and smoothly, the first fusion weight is close to 1, indicating high accuracy of the first pointing direction. If the user's hand shakes violently or swings rapidly to complete the pointing action, the first fusion weight is low, suggesting a possible deviation in the pointing direction, thus providing a reliable weight basis for subsequent fusion. The specific method for obtaining the first fusion weight is as follows: preset the scoring thresholds and weight ratios for each parameter of the device motion information, quantify and score each parameter, and then sum them by weight to obtain the first fusion weight. For example: Motion stability score (weight 0.4): If the hand device's posture change rate is ≤1° / frame (frame rate 10fps), it is judged as stable and gets 1 point; 1° / frame < posture change rate ≤3° / frame, gets 0.6 points; posture change rate >3° / frame, gets 0.2 points; Speed ​​score (weight 0.3): 1 point for speed ≤ 0.1m / s (slow pointing, high stability); 0.7 points for speed ≤ 0.3m / s; 0.3 points for speed > 0.3m / s. Acceleration score (weight 0.3): If the effective value of acceleration is ≤0.5g (g is the acceleration due to gravity), 1 point is awarded; if 0.5g < effective value of acceleration is ≤1g, 0.6 points are awarded; if the effective value of acceleration is >1g, 0.2 points are awarded. First fusion weight = motion stability score × 0.4 + motion speed score × 0.3 + acceleration score × 0.3.

[0088] 2032B. Based on the recognition image corresponding to the second pointing direction and the hand detection confidence of the recognition image, obtain the second fusion weight of the second pointing direction.

[0089] The image to be recognized refers to the image obtained when the gesture pointing in the second pointing direction is detected, such as the real-time image frame that was successfully recognized in step 202.

[0090] Among them, the hand detection confidence score is used to indicate the confidence level of the detected hand in the recognition image. For example, it can be the confidence score of the hand detection box output by the hand key point detection algorithm for the recognition image (the value range is 0~1). The higher the confidence score, the higher the recognition accuracy of the hand region in the image and the more reliable the determination of the pointing gesture.

[0091] The second fusion weight is used to quantify the reliability of the second pointing direction, and its value ranges from 0 to 1. It can be obtained through weighted calculation of multi-dimensional scores, for example: Hand detection confidence score (weight 0.5): The hand detection confidence score is directly output by the algorithm (e.g., if the confidence score is 0.8, the score is 0.8). Image sharpness score (weight 0.3): Sharpness is quantified by calculating the grayscale gradient value or sharpness value of the image. If the gradient value is ≥ the preset threshold (e.g., 30), 1 point is awarded; if the gradient value is 15 ≤ gradient value < 30, 0.6 points are awarded; if the gradient value is < 15, 0.2 points are awarded (low sharpness may lead to deviation in the extraction of key points of the hand). Pointing gesture integrity score (weight 0.2): Based on the completeness of the extraction of key points of the hand in the recognition image, if the core key points such as the index finger and wrist are completely extracted (extraction rate ≥ 90%), 1 point is awarded; 70% ≤ extraction rate < 90% is awarded 0.7 points; extraction rate < 70% is awarded 0.3 points (missing key points may lead to errors in the calculation of pointing direction). The second fusion weight = hand detection confidence score × 0.5 + recognition image clarity score × 0.3 + pointing gesture integrity score × 0.2.

[0092] 2033B. Based on the device environment information corresponding to the sound source detection direction, obtain the third fusion weight of the sound source detection direction.

[0093] Among them, the device environment information refers to the environmental information of the wearable sound pickup device when the sound source direction is detected, including but not limited to the environmental noise intensity (dB), the signal-to-noise ratio (SNR) of the sound source signal, the environmental reverberation degree (reverberation time RT60), and the number of sound sources (the number of effective sound sources identified by the microphone array). This information is collected and analyzed by the microphone array and environmental sensors of the wearable sound pickup device.

[0094] The third fusion weight is used to quantify the reliability of the sound source detection direction, with a value ranging from 0 to 1. The sound source detection direction can be used as an auxiliary direction (with lower priority than the first and second directional directions), and its upper limit can be preset to 0.5 (to ensure the dominance of explicit directional intent). It can be obtained through multi-dimensional scoring weighted calculation, for example: Signal-to-noise ratio (SNR) score (weight 0.4): If SNR ≥ 15dB (sound source is clearly identifiable), get 1 point; 8dB ≤ SNR < 15dB, get 0.7 points; SNR < 8dB (severe noise interference), get 0.3 points. Environmental noise intensity score (weight 0.3): 1 point for noise intensity ≤ 55dB (low noise environment); 0.6 points for noise intensity ≤ 70dB (55dB < noise intensity); 0.2 points for noise intensity > 70dB (high noise environment). Sound source quantity score (weight 0.3): 1 point is awarded if a single sound source is detected; 0.6 points are awarded if 2-3 sound sources are detected; and 0.3 points are awarded if more than 3 sound sources are detected (dense sound source environment) (multiple sound sources can easily lead to confusion in direction localization). The third fusion weight = (SNR score × 0.4 + environmental noise intensity score × 0.3 + number of sound sources score × 0.3) × 0.5 (weight upper limit coefficient).

[0095] 2034B. Based on the first fusion weight, the second fusion weight, and the third fusion weight, the first pointing direction, the second pointing direction, and the sound source detection direction are fused to obtain the specified pickup direction.

[0096] The display pointing direction refers to either the first pointing direction or the second pointing direction.

[0097] For example, firstly, it is detected whether the deviation between the sound source detection direction and the explicit pointing direction is within a third preset angle range. If the deviation between the sound source detection direction and the explicit pointing direction is within the third preset angle range, such as ≤30°, then the first pointing direction (D1), the second pointing direction (D2), and the sound source detection direction (D3) can all be converted into unit vectors in the first reference coordinate system, and weighted summed according to the first fusion weight, the second fusion weight, and the third fusion weight to obtain the fusion direction vector D_fusion: D_fusion=W1×D1+W2×D2+W3×D3. Then, calculate the angular deviations Δθ1 and Δθ2 between the fusion direction vector D_fusion and the first and second pointing directions: If the angular deviation between the fusion direction vector D_fusion and the first pointing direction is within the first preset angle range, and the angular deviation between the fusion direction vector D_fusion and the second pointing direction is within the first preset angle range, such as Δθ1≤10° and Δθ2≤10° (the fusion direction is consistent with the explicit pointing direction), directly normalize the fusion direction vector D_fusion and use it as the specified pickup direction Dtarget; If the angular deviation between the fusion direction vector D_fusion and the first pointing direction is within the second preset angle range, and the angular deviation between the fusion direction vector D_fusion and the second pointing direction is within the second preset angle range, such as Δθ1>15° or Δθ2>15° (the fusion direction deviates from the explicit pointing direction), then increase the weight ratio of the explicit pointing direction (e.g., increase W1 and W2 by 10% each, and decrease W3 by 20%), recalculate the fusion vector as the specified pickup direction Dtarget, and ensure that the user's explicit intent dominates.

[0098] If the deviation between the sound source detection direction and the explicit pointing direction is outside the third preset angle range, such as a deviation of >30°, then W3 is set to 0. Based solely on the fusion of D1 and D2, the first pointing direction (D1) and the second pointing direction (D2) are converted into unit vectors in the first reference coordinate system. The first fusion weight and the second fusion weight are used to perform a weighted summation, and the result is used as the specified pickup direction Dtarget to avoid the sound source direction interfering with the explicit intent.

[0099] In this way, the reliability of each direction (i.e., the first directional direction, the second directional direction, and the sound source detection direction) is reflected by the first fusion weight, the second fusion weight, and the third fusion weight, while adhering to the principle of explicit directional priority, ensuring that the specified pickup direction is both accurate and stable, and providing a reliable directional reference for the subsequent pickup beam establishment.

[0100] 204. Establish a pickup beam with the specified pickup direction as the beam direction, so as to process the sound signal located in the specified pickup direction using the pickup beam.

[0101] The pickup beam refers to the spatially directional signal receiving area formed by the microphone array of a wearable audio device through a specific algorithm. Its core feature is to amplify and enhance the sound signal from the specified pickup direction, while suppressing and attenuating interference signals such as environmental noise and other speakers' voices from directions other than the specified pickup direction, thereby achieving a directional sound pickup effect. The specified pickup direction is the direction vector in the first reference coordinate system, which is directly used as the center pointing direction of the pickup beam, ensuring that the beam is accurately aligned with the user's intended sound pickup target.

[0102] For example, based on a specified pickup direction and preset microphone array hardware parameters (such as the number of microphones, array topology, and unit spacing), the core parameters of the beamforming algorithm can be configured. The main lobe width (e.g., ±5°~±10°) can be set centered on the specified pickup direction, and the sidelobe suppression threshold (e.g., ≤-20dB) can be set. The signal delay and gain of each microphone unit can then be calibrated. Next, the microphone array acquires multi-channel audio signals. After noise reduction, delay alignment, and phase adjustment, the signals are weighted and summed using an algorithm to synchronously enhance the signals from the specified direction, while signals from non-target directions are attenuated by phase cancellation. Finally, the enhanced single-channel target speech signal is output, achieving directional pickup and ensuring that the user clearly hears the target speaker's voice.

[0103] Furthermore, the sound pickup processing method also includes the following steps A1~A3: A1. Determine the target object for the specified pickup direction.

[0104] Among them, the target of sound pickup lock refers to the subject that the wearable sound pickup device intends to pick up in the specified sound pickup direction. It is usually the speaker (or a specific sound-producing object). Its determination requires a combination of visual detection and acoustic verification to ensure that the locked target is accurately matched with the specified sound pickup direction.

[0105] For example, the front-facing camera of the wearable audio pickup device executes a face / body detection algorithm within a corresponding spatial area based on a specified pickup direction to extract face or body targets within that area (e.g., filtering valid targets through facial features and body contours; for ease of description and understanding, valid targets filtered by face / body detection algorithms are collectively referred to as "visual targets"). Simultaneously, the microphone array collects audio signals from the specified pickup direction and verifies whether the audio signal originates from the detected face / body target area by combining the temporal continuity of the sound source detection direction. If a single visual target is detected and the acoustic signal source is consistent, the visual target is directly identified as the audio pickup target. If multiple visual targets are detected, the distance between the visual target and the specified pickup direction (the closest one takes priority) and whether the visual target is in a vocalizing state (judged by lip movements or speech energy) can be combined to select a unique visual target as the audio pickup target, ensuring that the locked target matches the user's explicit audio pickup intention.

[0106] A2. After the sound pickup beam is established, the wearable sound pickup device is controlled to enter the directional sound pickup mode with the sound pickup target as the sound pickup target.

[0107] Among them, the directional sound pickup mode refers to the working mode of wearable sound pickup devices focusing on the target and continuously performing voice enhancement and tracking. After entering the directional sound pickup mode, the user does not need to keep pointing, and the device automatically maintains the sound pickup focus on the target.

[0108] For example, after the pickup beam established in step 204 is accurately aligned with the pickup target, the directional pickup mode switching is automatically triggered to start the continuous monitoring mechanism of the pickup target: the initial visual features (such as facial feature vectors and human contour parameters) and initial acoustic features (such as speech spectrum features) of the pickup target are recorded, and the tracking algorithm and dynamic beam adjustment logic are loaded for target tracking and signal enhancement to ensure the continuity and stability of the pickup effect after the directional pickup mode switching.

[0109] A3. When the wearable audio pickup device is in directional audio pickup mode, the beam direction of the audio pickup beam is adjusted based on the real-time tracking position of the audio pickup target, so as to use the audio pickup beam to enhance the sound signal located in the direction of the real-time tracking position.

[0110] Among them, real-time tracking position refers to the dynamic three-dimensional coordinates of the sound-locked object in the first reference coordinate system, which can be obtained through visual tracking and acoustic positioning to ensure the real-time and accuracy of position updates.

[0111] For example, the camera of the wearable audio pickup device captures image frames of the target object in real time, and continuously updates the visual position of the target object through a target tracking algorithm. Simultaneously, the microphone array monitors the direction of the sound source in real time, and, combined with the initial acoustic characteristics of the target object, verifies whether the sound source is the position after the target object has moved, thereby correcting the visual tracking results and obtaining the real-time tracking position. Based on the deviation between the real-time tracking position and the original specified pickup direction, the core parameters of the beamforming algorithm (such as the main lobe center angle and delay time) are dynamically adjusted to ensure that the pickup beam is always aligned with the target object. For example, when the target object moves laterally, causing the real-time tracking position to shift by 8° in the horizontal angle, the main lobe of the beam shifts synchronously by 8°, ensuring that the target speech signal is continuously superimposed and enhanced, achieving dynamic adaptation of "target movement - beam following," and ensuring that the user can still clearly obtain the target sound even in scenarios where the target is moving.

[0112] Furthermore, the sound pickup processing method also includes the following steps B1~B3: B1. Obtain the degree of obstruction, signal attenuation, and / or tracking confidence of the target object being picked up.

[0113] The occlusion level indicates the visual occlusion of the target object from the wearer's perspective (e.g., complete occlusion, partial occlusion, no occlusion). This can be detected by the wearable device's camera and is quantized using a range of 0 to 1 (1 for no occlusion, 0 for complete occlusion). The signal attenuation level indicates the attenuation of the target object's sound signal. This is achieved by collecting the target object's sound signal through a microphone array and calculating the ratio of the current signal strength to the initial signal strength. The tracking confidence level is a value (0 to 1) output by the visual tracking algorithm, representing the stability of tracking the target object. These three metrics can be obtained individually or in combination, providing a basis for updating the tracking confidence level.

[0114] B2. Update the locking confidence of the sound pickup target based on the degree of obstruction, the degree of signal attenuation, and / or the tracking confidence.

[0115] Lock-in confidence is a metric that quantifies the reliability of tracking a target object (e.g., represented by 0 to 1). For example, the weighted proportions of occlusion level, signal attenuation level, and tracking confidence can be preset (e.g., occlusion level 0.4, signal attenuation level 0.3, tracking confidence 0.3), and the weighted sum is used to obtain the updated lock-in confidence. For instance, no occlusion (0.4 points), no signal attenuation (0.3 points), and stable tracking (0.3 points) result in a lock-in confidence of 1; partial occlusion (0.2 points), slight signal attenuation (0.2 points), and relatively stable tracking (0.25 points) result in a lock-in confidence of 0.65.

[0116] B3. When the lock confidence level is less than the preset confidence threshold, control the wearable sound pickup device to exit the directional sound pickup mode.

[0117] The specific value of the preset confidence threshold (e.g., 0.3) can be set according to the actual business scenario requirements; there is no restriction on the specific value of the preset confidence threshold here. The updated lock confidence is compared with the preset confidence threshold in real time. If the lock confidence is consistently lower than the preset confidence threshold (e.g., for 3 consecutive frames), it indicates that the target being tracked can no longer be stably tracked. The wearable microphone is automatically controlled to exit directional microphone mode and return to standby mode, waiting for the user to point again, ensuring that the microphone performance does not decrease due to target loss. If the lock confidence is greater than or equal to the preset confidence threshold, the wearable microphone is controlled to continue maintaining directional microphone mode.

[0118] Furthermore, the sound pickup processing method also includes the following steps C1~C3: C1. Obtain the locked voiceprint features of the sound pickup lock object.

[0119] Among them, the locked voiceprint feature is the unique voice biometric feature of the voice-locked object, which is used for identity differentiation. The microphone array of the wearable voice-locking device collects the clear voice signal of the voice-locked object. After noise reduction and endpoint detection processing, core voiceprint features such as Mel frequency cepstral coefficients and fundamental frequency are extracted to form the locked voiceprint feature and stored in the memory as the benchmark for subsequent identity matching.

[0120] C2. Obtain the voiceprint features of each speaker in the environment where the wearable audio pickup device is located.

[0121] For example, in directional sound pickup mode, the microphone array continuously collects ambient audio signals and separates speech segments from different speakers using sound source localization and speech segmentation algorithms. Each speech segment undergoes the same feature extraction process as in step C1 to obtain the voiceprint features corresponding to each speaker, forming an environmental voiceprint feature set.

[0122] C3. If the locked voiceprint feature does not match the voiceprint features of each speaker, then control the wearable sound pickup device to exit the directional sound pickup mode.

[0123] For example, the similarity sim (Si, Starget) between the locked voiceprint feature and the voiceprint features of each speaker in the environment is calculated. If the similarity between the locked voiceprint feature and the voiceprint features of each speaker in the environment is lower than the preset similarity threshold (e.g., 0.8), it is considered that the locked voiceprint feature does not match the voiceprint features of each speaker, indicating that the voice pickup target has stopped speaking or left and there is no matching sound source. At this time, the wearable voice pickup device is controlled to exit the directional voice pickup mode and return to the standby state to avoid accidentally locking other speakers and ensure the consistency of the voice pickup target.

[0124] As can be seen from the above, firstly, by detecting hand pointing using a hand device connected to the wearable microphone, a first pointing direction is obtained; by detecting hand pointing using the camera of the wearable microphone, a second pointing direction is obtained. Based on the first and second pointing directions, the designated microphone direction of the wearable microphone is determined. Since both the first and second pointing directions directly reflect the user's explicit intent, the microphone direction can be determined using the displayed expression of the user's intent. This enables highly accurate and stable directional microphone pickup in noisy environments with multiple users, reducing the problem of erroneously amplifying the voice of non-target speakers in noisy environments and improving the stability and anti-interference capability of microphone pickup. Secondly, by comprehensively determining the designated pointing direction using the first and second pointing directions, a dual-modal redundancy backup can be formed, reducing the microphone pickup interruption problem caused by the failure of a single sensor channel and improving the reliability and robustness of the microphone pickup direction.

[0125] Furthermore, to better implement the sound pickup processing method in the embodiments of this application, based on the sound pickup processing method, the embodiments of this application also provide a sound pickup processing device, such as... Figure 6 The diagram shown is a structural schematic of one embodiment of the sound pickup processing device provided in this application. The sound pickup processing device 600 includes: The first detection unit 601 is used to perform hand pointing detection based on a hand device connected to the wearable microphone to obtain a first pointing direction; The second detection unit 602 is used to perform gesture pointing detection based on the camera of the wearable audio pickup device to obtain a second pointing direction; Processing unit 603 is configured to determine a specified pickup direction of the wearable microphone based on the first pointing direction and the second pointing direction; The processing unit 603 is further configured to establish a pickup beam with the specified pickup direction as the beam direction, so as to process the sound signal located in the specified pickup direction using the pickup beam.

[0126] In some embodiments, the sound pickup processing device 600 further includes a third detection unit (not shown in the figure), the third detection unit being used for: The direction of the sound source is detected based on the wearable sound pickup device. In some embodiments, the processing unit 603 is used to: Based on the device motion information corresponding to the first pointing direction, obtain the first fusion weight for the first pointing direction; Based on the recognition image corresponding to the second pointing direction and the hand detection confidence of the recognition image, a second fusion weight for the second pointing direction is obtained; Based on the device environment information corresponding to the sound source detection direction, the third fusion weight of the sound source detection direction is obtained; Based on the first fusion weight, the second fusion weight, and the third fusion weight, the first pointing direction, the second pointing direction, and the sound source detection direction are fused to obtain the specified pickup direction.

[0127] In some embodiments, the first detection unit 601 is used for: Based on the measured distance between the wearable microphone and the hand device, the preliminary spatial position of the hand device relative to the wearable microphone is obtained; Based on the initial posture data of the hand device, the first posture data of the hand device relative to the wearable microphone is determined; Based on the preliminary spatial position and the first posture data, the hand pointing vector corresponding to the hand device is determined as the first pointing direction.

[0128] In some embodiments, the first detection unit 601 is used for: The first reference position, the second reference position, and the third reference position of the wearable microphone are obtained, and the measured distances between them and the hand device are obtained. The first measured distance, the second measured distance, and the third measured distance are obtained. The first reference position, the second reference position, and the third reference position are three pre-calibrated non-collinear fixed points. A first reference sphere is obtained with the first reference position as the center and the first measured distance as the radius; Using the second reference position as the center and the second measured distance as the radius, obtain the second reference sphere; Using the third reference position as the center and the third measured distance as the radius, obtain a third reference sphere; The intersection point of the first reference sphere, the second reference sphere, and the third reference sphere is obtained as the preliminary spatial position.

[0129] In some embodiments, the first detection unit 601 is used for: Based on the first attitude data, the preliminary spatial position is corrected to obtain the corrected spatial position; Based on the corrected spatial position and the first posture data, the hand pointing vector is determined as the first pointing direction.

[0130] In some embodiments, the processing unit 603 is used to: Determine the target object for pickup in the specified pickup direction; Once the sound pickup beam is established, the wearable sound pickup device is controlled to enter a directional sound pickup mode with the sound pickup target as the sound pickup target; When the wearable audio pickup device is in directional audio pickup mode, the beam direction of the audio pickup beam is adjusted based on the real-time tracking position of the target being locked, so as to enhance the sound signal located in the direction of the real-time tracking position using the audio pickup beam.

[0131] In some embodiments, the processing unit 603 is used to: Obtain the degree of occlusion, signal attenuation, and / or tracking confidence of the target object being locked; The locking confidence of the pickup target is updated based on the degree of obstruction, the degree of signal attenuation, and / or the tracking confidence. When the lock confidence level is less than a preset confidence threshold, the wearable audio pickup device is controlled to exit the directional audio pickup mode.

[0132] In some embodiments, the processing unit 603 is used to: Obtain the locked voiceprint features of the sound pickup target; Acquire the voiceprint features of each speaker in the environment where the wearable audio pickup device is located; If the locked voiceprint feature does not match the voiceprint features of each speaker, then the wearable sound pickup device is controlled to exit the directional sound pickup mode.

[0133] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous embodiment of the sound pickup processing method, which will not be repeated here.

[0134] Those skilled in the art will understand that all or part of the steps in the above-described sound pickup processing method can be accomplished by instructions, or by controlling related hardware through instructions. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0135] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs, which can be loaded by a processor to execute any of the sound pickup processing methods provided in embodiments of this application.

[0136] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0137] In the above embodiments of the sound pickup processing device, computer-readable storage medium, and wearable sound pickup device, the descriptions of each embodiment have different focuses. Parts not described in detail in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes and beneficial effects of the sound pickup processing device, computer-readable storage medium, wearable sound pickup device, and their corresponding units described above can be referred to the description of the sound pickup processing method in the above embodiments, and will not be repeated here.

[0138] The above provides a detailed description of a sound pickup processing method, apparatus, wearable sound pickup device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A sound pickup processing method, characterized in that, The method includes: The first pointing direction is obtained by detecting the hand pointing using a hand device connected to a wearable microphone. The second pointing direction is obtained by detecting the gesture pointing based on the camera of the wearable microphone device. Based on the first pointing direction and the second pointing direction, the specified sound pickup direction of the wearable sound pickup device is determined; A pickup beam is established with the specified pickup direction as the beam direction, so as to process the sound signal located in the specified pickup direction using the pickup beam.

2. The sound pickup processing method according to claim 1, characterized in that, The method further includes: The direction of the sound source is detected based on the wearable sound pickup device. Determining the specified pickup direction of the wearable microphone based on the first and second pointing directions includes: Based on the device motion information corresponding to the first pointing direction, obtain the first fusion weight for the first pointing direction; Based on the recognition image corresponding to the second pointing direction and the hand detection confidence of the recognition image, a second fusion weight for the second pointing direction is obtained; Based on the device environment information corresponding to the sound source detection direction, the third fusion weight of the sound source detection direction is obtained; Based on the first fusion weight, the second fusion weight, and the third fusion weight, the first pointing direction, the second pointing direction, and the sound source detection direction are fused to obtain the specified pickup direction.

3. The sound pickup processing method according to claim 1, characterized in that, The method of detecting hand pointing based on a hand device connected to a wearable microphone to obtain a first pointing direction includes: Based on the measured distance between the wearable microphone and the hand device, the preliminary spatial position of the hand device relative to the wearable microphone is obtained; Based on the initial posture data of the hand device, the first posture data of the hand device relative to the wearable microphone is determined; Based on the preliminary spatial position and the first posture data, the hand pointing vector corresponding to the hand device is determined as the first pointing direction.

4. The sound pickup processing method according to claim 3, characterized in that, The step of obtaining the preliminary spatial position of the hand device relative to the wearable microphone based on the measured distance between the wearable microphone and the hand device includes: The first reference position, the second reference position, and the third reference position of the wearable microphone are obtained, and the measured distances between them and the hand device are obtained. The first measured distance, the second measured distance, and the third measured distance are obtained. The first reference position, the second reference position, and the third reference position are three pre-calibrated non-collinear fixed points. A first reference sphere is obtained with the first reference position as the center and the first measured distance as the radius; Using the second reference position as the center and the second measured distance as the radius, obtain the second reference sphere; Using the third reference position as the center and the third measured distance as the radius, obtain a third reference sphere; The intersection point of the first reference sphere, the second reference sphere, and the third reference sphere is obtained as the preliminary spatial position.

5. The sound pickup processing method according to claim 3, characterized in that, The step of determining the hand pointing vector corresponding to the hand device, based on the preliminary spatial position and the first posture data, as the first pointing direction, includes: Based on the first attitude data, the preliminary spatial position is corrected to obtain the corrected spatial position; Based on the corrected spatial position and the first posture data, the hand pointing vector is determined as the first pointing direction.

6. The sound pickup processing method according to claim 1, characterized in that, The method further includes: Determine the target object for pickup in the specified pickup direction; Once the sound pickup beam is established, the wearable sound pickup device is controlled to enter a directional sound pickup mode with the sound pickup target as the sound pickup target; When the wearable audio pickup device is in directional audio pickup mode, the beam direction of the audio pickup beam is adjusted based on the real-time tracking position of the target being locked, so as to enhance the sound signal located in the direction of the real-time tracking position using the audio pickup beam.

7. The sound pickup processing method according to claim 6, characterized in that, The method further includes: Obtain the degree of occlusion, signal attenuation, and / or tracking confidence of the target object being locked; The locking confidence of the pickup target is updated based on the degree of obstruction, the degree of signal attenuation, and / or the tracking confidence. When the lock confidence level is less than a preset confidence threshold, the wearable audio pickup device is controlled to exit the directional audio pickup mode.

8. A sound pickup processing device, characterized in that, The sound pickup processing device includes: The first detection unit is used to perform hand pointing detection based on a hand device connected to the wearable microphone to obtain a first pointing direction; The second detection unit is used to perform gesture pointing detection based on the camera of the wearable audio pickup device to obtain a second pointing direction; The processing unit is configured to determine a specified pickup direction of the wearable microphone based on the first pointing direction and the second pointing direction; The processing unit is further configured to establish a pickup beam with the specified pickup direction as the beam direction, so as to process the sound signal located in the specified pickup direction using the pickup beam.

9. A wearable audio pickup device, characterized in that, The device includes a camera, a microphone array, a processor, and a memory, wherein the memory stores a computer program, and the processor executes the sound pickup processing method as described in any one of claims 1 to 7 when it invokes the computer program in the memory.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to execute the sound pickup processing method according to any one of claims 1 to 7.

Images