Head tracking sound effect adaptive adjustment method and device based on three-dimensional scene perception

Abstract

A head-tracking adaptive audio effect adjustment method and device based on 3D scene perception, relating to the field of audio equipment technology, defines volume, channel balance, reverberation depth, sound field width, and audio delay compensation as five audio effect adjustment targets for audio devices. It utilizes a 3D ToF camera and a scene semantic segmentation network to detect the user's head region and calculate the movement distance of the head center point and the distance between the head center point and the audio device. This allows for the calculation of adjustment values ​​for the five audio effect adjustment targets. The five audio effect adjustment targets are then adjusted based on these calculated values. When the user manually corrects the audio effect adjustment targets, a loss value is fitted using calculated parameters, and a gradient descent algorithm is employed to optimize the five audio effect adjustment coefficients. The method and device provided by this invention are applicable to scenarios where users are present in the presence of audio equipment.

Description

Technical Field

[0001] This invention relates to audio equipment technology, and in particular to a method and apparatus for adaptive adjustment of head tracking sound effects based on three-dimensional scene perception. Background Technology

[0002] There are two main types of existing adaptive sound adjustment technologies: one is simple adaptation based on distance sensors, and the other is sound adjustment based on visual tracking.

[0003] Simple adaptation technologies based on distance sensors measure the straight-line distance between the user and the audio device using devices such as ultrasonic and infrared sensors. This technology only adjusts the volume and ignores the influence of head movement direction and scene environment on the sound effect, resulting in poor adaptability.

[0004] Visual tracking-based sound effect adjustment technologies all use 2D cameras to extract head contour features. This technology is greatly affected by lighting and background interference, and cannot obtain three-dimensional spatial position information, resulting in large errors in judging the direction of movement (usually ≥10cm). Moreover, sound effect adjustment is mostly limited to channel balance and does not involve core parameters such as reverberation and sound field width. Summary of the Invention

[0005] To address the shortcomings of the existing technology, the technical problem to be solved by the present invention is to provide a head tracking sound effect adaptive adjustment method and device based on three-dimensional scene perception, which has strong anti-interference ability, high calculation accuracy, and comprehensive sound effect adjustment dimensions.

[0006] To address the aforementioned technical problems, this invention provides a head-tracking sound effect adaptive adjustment method based on three-dimensional scene perception, characterized by the following specific steps:

[0007] Step 1: Define 5 sound effect adjustment coefficients , , , , And set the initial values ​​for the five sound effect adjustment coefficients, defining volume, channel balance, reverberation depth, sound field width, and audio delay compensation as the five sound effect adjustment targets for the audio device;

[0008] Step 2: Use a 3D ToF camera to acquire the 3D point cloud P and depth image I_depth of the surrounding environment;

[0009] Step 3: Use the scene semantic segmentation network to segment the 3D point cloud P and the depth image I_depth acquired in Step 2, and segment the user head region and the background region, and extract the head point cloud P_head and the background point cloud P_bg of the user head region.

[0010] Step 4: Detect the center points of the ears, top of the head, and chin from the head point cloud P_head, and calculate the 3D coordinates of the head center points. The specific calculation formula is as follows:

[0011]

[0012]

[0013]

[0014]

[0015] In the formula, t is the current frame. Let be the three-dimensional coordinates of the head center point in frame t. Let X be the X-axis coordinate of the head center point in frame t. Let Y be the Y-coordinate of the head center point in frame t of the image. Let Z be the Z-axis coordinate of the head center point in frame t. , Let X and Y be the X-axis coordinates of the center points of both ears in frame t of the image. Let X be the X-axis coordinate of the center point of the top of the head in frame t. Let X be the X-axis coordinate of the mandibular center point in frame t. , These are the Y-axis coordinates of the center points of both ears in frame t of the image. Let Y be the Y-coordinate of the center point of the top of the head in frame t. Let Y be the Y-axis coordinate of the mandibular center point in frame t. , These are the Z-axis coordinates of the center points of both ears in frame t of the image. Let Z be the Z-axis coordinate of the center point of the top of the head in frame t. Here are the Z-axis coordinates of the mandibular center point in frame t of the image;

[0016] Step 5: Calculate the distance the head center point moves and the distance between the head center point and the audio device. The specific calculation formula is as follows:

[0017]

[0018]

[0019] In the formula, t is the current frame, and t-1 is the frame preceding frame t. The distance moved by the center point of the head. The distance between the center point of the head and the audio device. The X-axis coordinate of the center point of the audio device. The Y-coordinate of the center point of the audio device. The Z-axis coordinate of the center point of the audio device. , , A pre-set fixed value;

[0020] Step 6, if If the value is greater than 0, proceed to step 7; otherwise, proceed to step 9.

[0021] Step 7: Calculate the adjustment values ​​for the five sound effect adjustment targets of the audio device. The specific calculation formula is as follows:

[0022]

[0023] In the formula, This refers to the volume adjustment value for audio devices. The preset audio device volume reference value, For the preset reference distance, The maximum compensation distance is preset.

[0024]

[0025] In the formula, W is the channel balance adjustment value for the audio device, and W is the preset channel spacing of the audio device.

[0026]

[0027] In the formula, This is the reverb depth adjustment value for audio devices. The preset reverberation depth reference value for audio devices, The maximum effective distance is preset.

[0028]

[0029] In the formula, This is the sound field width adjustment value for audio devices. H is the preset reference value for the sound field width of the audio device, and H is the preset effective distance in the vertical direction of the audio device.

[0030]

[0031] In the formula, This is the audio delay compensation adjustment value for audio devices. Speed ​​of sound;

[0032] Step 8, adjust the audio device volume to The audio device channel balance is adjusted to The reverb depth of the audio device is adjusted to The audio device's sound field width is adjusted to Audio device audio latency compensation adjusted to ;

[0033] Step 9: Check whether the user has manually corrected the five sound effect adjustment targets of the audio device;

[0034] If the user has manually adjusted the sound effects settings of the audio device, proceed to step 10; otherwise, proceed to step 2.

[0035] Step 10, calculate the parameter fitting loss value. The specific calculation formula is as follows:

[0036]

[0037] In the formula, Loss is the parameter fitting loss value. The adjustment value for the i-th sound effect adjustment target of the audio device. The final setting value for the user's i-th sound effect adjustment target for the audio device;

[0038] Step 11: Use the gradient descent algorithm to adjust the five sound effect coefficients. , , , , After optimizing the value, proceed to step 2;

[0039] The formula for optimizing the five sound effect adjustment coefficients using the gradient descent algorithm is as follows:

[0040]

[0041]

[0042] In the formula, This represents the optimized value of the j-th sound effect adjustment coefficient after the (n+1)-th iteration. This represents the optimized value of the j-th sound effect adjustment coefficient after the nth iteration. For the pre-set learning rate, The loss value after the nth iteration. The number of iterations is a pre-defined limit;

[0043] During the gradient descent algorithm iteration, in each iteration, the adjustment values ​​of the five sound effect adjustment targets of the audio device are recalculated using the formula in step 7, and the parameter fitting loss value is recalculated using the formula in step 10 before proceeding to the next iteration.

[0044] Furthermore, during the gradient descent algorithm iteration process, if the difference between the loss values ​​of two consecutive iterations is less than or equal to a pre-set loss value convergence threshold, the iteration ends.

[0045] Furthermore, during the gradient descent algorithm iteration process, if If the gradient convergence threshold is less than or equal to the preset threshold, the iteration ends.

[0046] The present invention also provides a head tracking sound effect adaptive adjustment device based on three-dimensional scene perception, characterized in that it includes an audio device, a three-dimensional ToF camera, a scene semantic segmentation network, a head tracking module, a sound effect adjustment module, and a user preference calibration module;

[0047] The 3D ToF camera is used to acquire 3D point cloud and depth images of the surrounding environment and then send them to the scene semantic segmentation network.

[0048] The input port of the scene semantic segmentation network is connected to the output port of the 3D ToF camera, which is used to segment the user's head region and background region from the received 3D point cloud and depth image, and extract the head point cloud of the user's head region and send it to the head tracking module.

[0049] The input port of the head tracking module is connected to the output port of the scene semantic segmentation network. The head tracking module is equipped with a software program that can execute steps 4 and 5 of the above method. It is used to calculate the three-dimensional coordinates of the user's head center point, the movement distance of the user's head center point, and the distance between the user's head center point and the audio device based on the received head point data, and send the calculation results to the sound effect adjustment module.

[0050] The input port of the sound effect adjustment module is connected to the output port of the head tracking module. The sound effect adjustment module stores the values ​​of 5 sound effect adjustment coefficients and is equipped with a software program that can execute step 7 of the above method. The software program is used to calculate the adjustment values ​​of the 5 sound effect adjustment targets of the audio device based on the calculation results of the head tracking module, and adjust the 5 sound effect adjustment targets of the audio device based on the calculation results. The 5 sound effect adjustment targets of the audio device are volume, channel balance, reverberation depth, sound field width, and audio delay compensation.

[0051] The user preference calibration module is used to collect the manual correction values ​​of five sound effect adjustment targets of the audio device. The user preference calibration module is equipped with a software program that can execute steps 10 and 11 in the above method. It is used to calculate the parameter fitting loss value based on the collected manual correction values ​​of the five sound effect adjustment targets, and optimize the values ​​of the five sound effect adjustment coefficients using the gradient descent algorithm. Then, it updates the five sound effect adjustment coefficients stored in the sound effect adjustment module with the optimized values ​​of the five sound effect adjustment coefficients.

[0052] The present invention provides a head-tracking audio effect adaptive adjustment method and device based on three-dimensional scene perception. It uses a three-dimensional ToF camera and a semantic segmentation network to monitor the movement direction and distance of the user's head center point, as well as the distance between the head center point and the audio device. It has strong anti-interference ability, can eliminate environmental background interference, and is not affected by changes in lighting. It is applicable to indoor, in-vehicle, VR and other scenarios. It has the characteristics of high calculation accuracy, and the audio effect adjustment dimensions are comprehensive, covering five core parameters: volume, channel balance, reverberation depth, sound field width and audio delay compensation. It adapts to the multi-dimensional effects of three-dimensional motion and can dynamically optimize parameter mapping according to user preference calibration mechanism to fit the listening habits of different users. Detailed Implementation

[0053] The technical solution of the present invention will be further described in detail below with reference to specific embodiments. However, these embodiments are not intended to limit the present invention. Any similar structures and similar variations of the present invention should be included in the protection scope of the present invention. The commas in the present invention all indicate the relationship between and. The English letters in the present invention are case-sensitive.

[0054] The head-tracking sound effect adaptive adjustment method based on three-dimensional scene perception provided in this invention is applicable to scenarios where there are users in the presence of audio equipment. The specific steps of this method are as follows:

[0055] Step 1: Define 5 sound effect adjustment coefficients , , , , And set the initial values ​​for the five sound effect adjustment coefficients, among which... Distance compensation coefficient ( The value range is 0.05. 0.1), Balance coefficient ( The value range is 0.8 1.0), Reverberation coefficient ( The value range is 0.3. 0.6), The sound field width factor ( The value range is 0.2. 0.4), The delay factor ( The value range is 0.9 1.1);

[0056] Volume, channel balance, reverberation depth, sound field width, and audio delay compensation are defined as five sound effect adjustment targets for audio devices;

[0057] Step 2: Use a 3D ToF (Time-of-Flight) camera to acquire 3D point cloud P and depth image I_depth of the surrounding environment;

[0058] Step 3: Use the scene semantic segmentation network to segment the 3D point cloud P and the depth image I_depth acquired in Step 2, and segment the user head region and the background region, and extract the head point cloud P_head and the background point cloud P_bg of the user head region.

[0059] The three coordinate axes of the three-dimensional coordinate system of the 3D ToF camera are the X-axis, Y-axis, and Z-axis, where the X-axis is the horizontal axis in the left and right direction, the Y-axis is the vertical axis, and the Z-axis is the horizontal axis in the front and back direction.

[0060] During scene semantic segmentation, the network first outputs a semantic segmentation mask M, and then extracts the head point cloud P_head={P(x,y,z)|M(x,y)=1} based on the semantic segmentation mask M. In this equation, P represents a point in the head point cloud P_head, x is the X-axis coordinate of point P, y is the Y-axis coordinate of point P, and z is the Z-axis coordinate of point P. The semantic segmentation mask M(x,y)=1 represents the user's head region, and M(x,y)=0 represents the background region. Then, the mean vector (cluster centers) and covariance matrix of the head point cloud P_head are calculated. The matrix reflects the geometric distribution characteristics of the head point cloud (the head point cloud is approximately ellipsoidal and its distribution is regular). Then, the Mahalanobis distance from each point in the head point cloud to the mean vector is calculated (the Mahalanobis distance can combine the distribution characteristics of the point cloud and avoid the defect of Euclidean distance ignoring distribution differences). Points in the head point cloud that exceed the Mahalanobis distance threshold (pre-set according to the distribution compactness of the head point cloud) are judged as isolated noise points (these points do not conform to the distribution pattern of the head body and are likely to be background points or sensor noise that were mistakenly included during segmentation). All isolated noise points are removed, and the valid point cloud that conforms to the geometric distribution of the head is retained.

[0061] The 3D point cloud P can provide the 3D spatial coordinate information of all objects in the environment, and the depth image I_depth can provide the depth information of each pixel (the distance from the object to the camera). This helps the scene semantic segmentation network to accurately distinguish between the foreground (user's head) and the background. The depth values ​​of the head region in the depth image are relatively concentrated (closer to the camera), while the depth values ​​of the background region are scattered or farther away. By constraining the depth features, the interference of lighting changes and complex backgrounds on the segmentation can be reduced, making the segmentation of the head region and the background region more accurate.

[0062] Step 4: Detect the center points of the ears, top of the head, and chin from the head point cloud P_head, and calculate the 3D coordinates of the head center points. The specific calculation formula is as follows:

[0063]

[0064]

[0065]

[0066]

[0067] In the formula, t is the current frame. Let be the three-dimensional coordinates of the head center point in frame t. Let X be the X-axis coordinate of the head center point in frame t. Let Y be the Y-coordinate of the head center point in frame t of the image. Let Z be the Z-axis coordinate of the head center point in frame t. , Let X and Y be the X-axis coordinates of the center points of both ears in frame t of the image. Let X be the X-axis coordinate of the center point of the top of the head in frame t. Let X be the X-axis coordinate of the mandibular center point in frame t. , These are the Y-axis coordinates of the center points of both ears in frame t of the image. Let Y be the Y-coordinate of the center point of the top of the head in frame t. Let Y be the Y-axis coordinate of the mandibular center point in frame t. , These are the Z-axis coordinates of the center points of both ears in frame t of the image. Let Z be the Z-axis coordinate of the center point of the top of the head in frame t. Here are the Z-axis coordinates of the mandibular center point in frame t of the image;

[0068] Based on the geometric shape features of the head point cloud P_head (the head is approximately a symmetrical ellipsoid, and key parts are coordinate extreme values ​​or symmetrical clustered regions), when detecting the center points of the ears, top of the head, and chin from the head point cloud P_head, the coordinates of all points in the head point cloud P_head are first normalized using the mean vector of the head point cloud P_head as the origin to eliminate the influence of position offset.

[0069] When detecting the top center point from the head point cloud P_head, find the maximum cluster center of the head point cloud in the Y-axis direction (the top of the head is the highest point of the head, and the point with the largest Y-axis coordinate will form a dense cluster), and define the mean point of the maximum cluster center as the top center point;

[0070] When detecting the mandibular center point from the head point cloud P_head, the minimum cluster center of the head point cloud is found in the Y-axis direction (the mandible is the lowest point of the head, and the point with the smallest Y-axis coordinate will form a dense cluster), and the mean point of the minimum cluster center is defined as the mandibular center point.

[0071] When detecting the center point of the two ears from the head point cloud P_head: find two symmetrical maximum cluster centers of the head point cloud in the X-axis direction (the two ears are located on the left and right sides of the head, and the X-axis coordinates are the extreme values ​​in the positive and negative directions, and the two clusters are symmetrical about the YZ plane). The mean point of the left maximum cluster center is defined as the center point of the left ear, and the mean point of the right maximum cluster center is defined as the center point of the right ear.

[0072] Step 5: Calculate the distance the head center point moves and the distance between the head center point and the audio device. The specific calculation formula is as follows:

[0073]

[0074]

[0075] In the formula, t is the current frame, and t-1 is the frame preceding frame t. The distance moved by the center point of the head. The distance between the center point of the head and the audio device. The X-axis coordinate of the center point of the audio device. The Y-coordinate of the center point of the audio device. The Z-axis coordinate of the center point of the audio device. , , A pre-set fixed value;

[0076] Step 6, if If the value is greater than 0, proceed to step 7; otherwise, proceed to step 9.

[0077] Step 7: Calculate the adjustment values ​​for the five sound effect adjustment targets of the audio device. The specific calculation formula is as follows:

[0078]

[0079] In the formula, This refers to the volume adjustment value for audio devices. The preset audio device volume reference value, For the preset reference distance, The maximum compensation distance is preset.

[0080]

[0081] In the formula, This is the channel balance adjustment value for the audio device (the calculated value ranges from -1). 1, where a value of -1 represents the left channel being tuned to its maximum value, and a value of 1 represents the right channel being tuned to its maximum value; W is the preset channel spacing of the audio device.

[0082]

[0083] In the formula, This is the reverb depth adjustment value for audio devices. The preset reverberation depth reference value for audio devices, The maximum effective distance is preset.

[0084]

[0085] In the formula, This is the sound field width adjustment value for audio devices. H is the preset reference value for the sound field width of the audio device, and H is the preset effective distance in the vertical direction of the audio device.

[0086]

[0087] In the formula, This is the audio delay compensation adjustment value for audio devices. Speed ​​of sound;

[0088] Step 8, adjust the audio device volume to The audio device channel balance is adjusted to The reverb depth of the audio device is adjusted to The audio device's sound field width is adjusted to Audio device audio latency compensation adjusted to ;

[0089] Step 9: Detect whether the user has manually corrected the five sound effect adjustment targets of the audio device (i.e., detect whether the current values ​​of the five sound effect adjustment targets have changed relative to the values ​​before this step).

[0090] If the user has manually adjusted the sound effects settings of the audio device, proceed to step 10; otherwise, proceed to step 2.

[0091] Step 10, calculate the parameter fitting loss value. The specific calculation formula is as follows:

[0092]

[0093] In the formula, Loss is the parameter fitting loss value. The adjustment value for the i-th sound effect adjustment target of the audio device (calculated in step 7). The final setting value for the user's i-th sound effect adjustment target for the audio device;

[0094] Step 11: Use the gradient descent algorithm to adjust the five sound effect coefficients. , , , , After optimizing the value, proceed to step 2;

[0095] The formula for optimizing the five sound effect adjustment coefficients using the gradient descent algorithm is as follows:

[0096]

[0097]

[0098] In the formula, This represents the optimized value of the j-th sound effect adjustment coefficient after the (n+1)-th iteration. The optimized value of the j-th sound effect adjustment coefficient after the nth iteration ( The value is the one before the iteration started. (value) For the pre-set learning rate ( The value ranges from 0.01 to 0.05, and in this embodiment, the value is 0.03. The Loss value after the nth iteration ( The value is the Loss value before the iteration begins. The number of iterations is set to a pre-defined limit (50 times in this embodiment);

[0099] During the gradient descent algorithm iteration, in each iteration, the adjustment values ​​of the five sound effect adjustment targets of the audio device are recalculated using the formula in step 7, and the parameter fitting loss value is recalculated using the formula in step 10 before proceeding to the next iteration.

[0100] During the gradient descent algorithm iteration, if the difference between the loss values ​​of two consecutive iterations is less than or equal to a pre-set loss convergence threshold (in this embodiment, the loss convergence threshold is 1e-5), or If the gradient convergence threshold is less than or equal to the preset threshold (in this embodiment, the gradient convergence threshold is 1e-6), then the iteration ends.

[0101] This invention also provides a head-tracking sound effect adaptive adjustment device based on three-dimensional scene perception, characterized in that it includes an audio device, a three-dimensional ToF camera, a scene semantic segmentation network, a head-tracking module, a sound effect adjustment module, and a user preference calibration module;

[0102] The 3D ToF camera is used to acquire 3D point cloud and depth images of the surrounding environment and then send them to the scene semantic segmentation network.

[0103] The input port of the scene semantic segmentation network is connected to the output port of the 3D ToF camera, which is used to segment the user's head region and background region from the received 3D point cloud and depth image, and extract the head point cloud of the user's head region and send it to the head tracking module.

[0104] The input port of the head tracking module is connected to the output port of the scene semantic segmentation network. The head tracking module is equipped with a software program that can execute steps 4 and 5 of the above method. It is used to calculate the three-dimensional coordinates of the user's head center point, the movement distance of the user's head center point, and the distance between the user's head center point and the audio device based on the received head point data, and send the calculation results to the sound effect adjustment module.

[0105] The input port of the sound effect adjustment module is connected to the output port of the head tracking module. The sound effect adjustment module stores the values ​​of 5 sound effect adjustment coefficients and is equipped with a software program that can execute step 7 of the above method. The software program is used to calculate the adjustment values ​​of the 5 sound effect adjustment targets of the audio device based on the calculation results of the head tracking module, and adjust the 5 sound effect adjustment targets of the audio device based on the calculation results. The 5 sound effect adjustment targets of the audio device are volume, channel balance, reverberation depth, sound field width, and audio delay compensation.

[0106] The user preference calibration module is used to collect the manual correction values ​​of five sound effect adjustment targets of the audio device. The user preference calibration module is equipped with a software program that can execute steps 10 and 11 in the above method. It is used to calculate the parameter fitting loss value based on the collected manual correction values ​​of the five sound effect adjustment targets, and optimize the values ​​of the five sound effect adjustment coefficients using the gradient descent algorithm. Then, it updates the five sound effect adjustment coefficients stored in the sound effect adjustment module with the optimized values ​​of the five sound effect adjustment coefficients.

[0107] The sampling frequency of the three-dimensional ToF camera used in this embodiment of the invention is ≥30fps, the distance measurement accuracy is ≤2cm, and the three-dimensional spatial resolution is ≥640×480; the scene semantic segmentation network is an existing lightweight semantic segmentation network.

Claims

1. A head-tracking sound effect adaptive adjustment method based on 3D scene perception, characterized in that, The specific steps are as follows: Step 1: Define 5 sound effect adjustment coefficients , , , , And set the initial values ​​for the five sound effect adjustment coefficients, defining volume, channel balance, reverberation depth, sound field width, and audio delay compensation as the five sound effect adjustment targets for the audio device; Step 2: Use a 3D ToF camera to acquire the 3D point cloud P and depth image I_depth of the surrounding environment; Step 3: Use the scene semantic segmentation network to segment the 3D point cloud P and the depth image I_depth acquired in Step 2, and segment the user head region and the background region, and extract the head point cloud P_head and the background point cloud P_bg of the user head region. Step 4: Detect the center points of the ears, top of the head, and chin from the head point cloud P_head, and calculate the 3D coordinates of the head center points. The specific calculation formula is as follows: In the formula, t is the current frame. Let be the three-dimensional coordinates of the head center point in frame t. Let X be the X-axis coordinate of the head center point in frame t. Let Y be the Y-coordinate of the head center point in frame t. Let Z be the Z-axis coordinate of the head center point in frame t of the image. , Let X and Y be the X-axis coordinates of the center points of both ears in frame t of the image. Let X be the X-axis coordinate of the center point of the top of the head in frame t. Let X be the X-axis coordinate of the mandibular center point in frame t. , These are the Y-axis coordinates of the center points of both ears in frame t of the image. Let Y be the Y-coordinate of the center point of the top of the head in frame t. Let Y be the Y-axis coordinate of the mandibular center point in frame t. , These are the Z-axis coordinates of the center points of both ears in frame t of the image. Let Z be the Z-axis coordinate of the center point of the top of the head in frame t. Here are the Z-axis coordinates of the mandibular center point in frame t of the image; Step 5: Calculate the distance the head center point moves and the distance between the head center point and the audio device. The specific calculation formula is as follows: In the formula, t is the current frame, and t-1 is the frame preceding frame t. The distance moved by the center point of the head. The distance between the center point of the head and the audio device. The X-axis coordinate of the center point of the audio device. The Y-coordinate of the center point of the audio device. The Z-axis coordinate of the center point of the audio device. , , A pre-set fixed value; Step 6, if If the value is greater than 0, proceed to step 7; otherwise, proceed to step 9. Step 7: Calculate the adjustment values ​​for the five sound effect adjustment targets of the audio device. The specific calculation formula is as follows: In the formula, This refers to the volume adjustment value for audio devices. The preset audio device volume reference value, For the preset reference distance, The maximum compensation distance is preset. In the formula, W is the channel balance adjustment value for the audio device, and W is the preset channel spacing of the audio device. In the formula, This is the reverb depth adjustment value for audio devices. The preset reverberation depth reference value for audio devices, The maximum effective distance is preset. In the formula, This is the sound field width adjustment value for audio devices. H is the preset reference value for the sound field width of the audio device, and H is the preset effective distance in the vertical direction of the audio device. In the formula, This is the audio delay compensation adjustment value for audio devices. Speed ​​of sound; Step 8, adjust the audio device volume to The audio device channel balance is adjusted to The reverb depth of the audio device is adjusted to The audio device's sound field width is adjusted to Audio device audio latency compensation adjusted to ; Step 9: Check whether the user has manually corrected the five sound effect adjustment targets of the audio device; If the user has manually adjusted the sound effects settings of the audio device, proceed to step 10; otherwise, proceed to step 2. Step 10, calculate the parameter fitting loss value. The specific calculation formula is as follows: In the formula, Loss is the parameter fitting loss value. The adjustment value for the i-th sound effect adjustment target of the audio device. The final setting value for the user's i-th sound effect adjustment target for the audio device; Step 11: Use the gradient descent algorithm to adjust the five sound effect coefficients. , , , , After optimizing the value, proceed to step 2; The formula for optimizing the five sound effect adjustment coefficients using the gradient descent algorithm is as follows: In the formula, This represents the optimized value of the j-th sound effect adjustment coefficient after the (n+1)-th iteration. This represents the optimized value of the j-th sound effect adjustment coefficient after the nth iteration. For the pre-set learning rate, The loss value after the nth iteration. The number of iterations is a pre-defined limit; During the gradient descent algorithm iteration, in each iteration, the adjustment values ​​of the five sound effect adjustment targets of the audio device are recalculated using the formula in step 7, and the parameter fitting loss value is recalculated using the formula in step 10 before proceeding to the next iteration.

2. The head-tracking sound effect adaptive adjustment method based on three-dimensional scene perception according to claim 1, characterized in that: During the gradient descent algorithm iteration, if the difference between the loss values ​​of two consecutive iterations is less than or equal to the pre-set loss value convergence threshold, the iteration ends.

3. The head-tracking sound effect adaptive adjustment method based on three-dimensional scene perception according to claim 1, characterized in that: During the gradient descent algorithm iteration, if If the gradient convergence threshold is less than or equal to the preset threshold, the iteration ends.

4. A head-tracking sound effect adaptive adjustment device based on three-dimensional scene perception, characterized in that: This includes audio equipment, a 3D ToF camera, a scene semantic segmentation network, a head tracking module, a sound effect adjustment module, and a user preference calibration module; The 3D ToF camera is used to acquire 3D point cloud and depth images of the surrounding environment and then send them to the scene semantic segmentation network. The input port of the scene semantic segmentation network is connected to the output port of the 3D ToF camera, which is used to segment the user's head region and background region from the received 3D point cloud and depth image, and extract the head point cloud of the user's head region and send it to the head tracking module. The input port of the head tracking module is connected to the output port of the scene semantic segmentation network. The head tracking module is equipped with a software program that can execute steps 4 and 5 of the method described in claim 1. The software program is used to calculate the three-dimensional coordinates of the user's head center point, the movement distance of the user's head center point, and the distance between the user's head center point and the audio device based on the received head point data, and then send the calculation results to the sound effect adjustment module. The input port of the sound effect adjustment module is connected to the output port of the head tracking module. The sound effect adjustment module stores the values ​​of 5 sound effect adjustment coefficients and is equipped with a software program that can execute step 7 of the method described in claim 1. The software program is used to calculate the adjustment values ​​of the 5 sound effect adjustment targets of the audio device based on the calculation results of the head tracking module, and adjust the 5 sound effect adjustment targets of the audio device based on the calculation results. The 5 sound effect adjustment targets of the audio device are volume, channel balance, reverberation depth, sound field width, and audio delay compensation. The user preference calibration module is used to collect the manual correction values ​​of five sound effect adjustment targets of the audio device. The user preference calibration module is equipped with a software program that can execute steps 10 and 11 in the method of claim 1. It is used to calculate the parameter fitting loss value based on the collected manual correction values ​​of the five sound effect adjustment targets, and optimize the values ​​of the five sound effect adjustment coefficients using the gradient descent algorithm. Then, it updates the five sound effect adjustment coefficients stored in the sound effect adjustment module with the optimized values ​​of the five sound effect adjustment coefficients.

Images