Method for adjusting sound signal and arithmetic device for adjusting sound signal

Abstract

The invention provides a method for adjusting a sound signal and an arithmetic device for adjusting the sound signal. And measuring the current attitude data of the current time interval. And determining to-be-evaluated data according to an angle error between the current attitude data and the previously predicted data. And inputting to-be-evaluated data to the prediction model to generate future prediction data in a future time interval. A sound characteristic of the sound signal is adjusted to a predicted rotation angle corresponding to a future time interval. Therefore, the auditory experience can be improved.

Description

Technical Field

[0001] This invention relates to a sound signal processing method, and more particularly to a sound signal adjustment method and a computing device for sound signal adjustment. Background Technology

[0002] Spatial audio transfers sound signals to a surround sound field created by multiple virtual speakers, adjusting the response and delay of virtual sound signals from different directions to create a three-dimensional sound field. It's worth noting that in practical applications, users' heads may turn, causing current spatial audio systems to experience transmission delays, which can reach over 204 milliseconds. Summary of the Invention

[0003] The present invention relates to a method for adjusting sound signals and a computing device for adjusting sound signals, which can reduce the time delay caused by head rotation.

[0004] According to an embodiment of the present invention, a method for adjusting a sound signal includes (but is not limited to) the following steps: measuring current posture data in the current time interval, wherein the current posture data includes a measured rotation angle of a target part in the current time interval; determining the data to be evaluated based on the angle error between the current posture data and previously predicted data, wherein the previously predicted data includes a predicted rotation angle of the target part in the current time interval predicted in a previous time interval, the angle error being the error between the measured rotation angle and the predicted rotation angle; and using the comparison result of the angle error and an error threshold to select posture data from at least one of a plurality of time intervals to the data to be evaluated, wherein closer comparison results correspond to selecting posture data from more time intervals, and farther comparison results correspond to selecting posture data from more time intervals. The method involves selecting a limited number of time intervals of attitude data, where the attitude data for these time intervals includes the measured rotation angle of the target body part and the changes in the measured rotation angle within these time intervals; generating future prediction data for future time intervals by inputting the data to be evaluated into a prediction model, wherein the prediction model is trained and learns the attitude changes of the target body part using a machine learning algorithm, and the future prediction data includes the predicted rotation angle of the target body part in the future time interval predicted in the current time interval, and the previous prediction data is the prediction data predicted by the prediction model corresponding to the current time interval; and adjusting the acoustic features of the sound signal to correspond to the predicted rotation angle in the future time interval, wherein the acoustic features are related to at least one of the amplitude and phase of the sound signal.

[0005] According to an embodiment of the present invention, a computing device for adjusting a sound signal includes (but is not limited to) a memory and a processor. The memory stores program code. The processor is coupled to the memory. The processor is configured to load the program code to execute: measuring current attitude data in a current time interval, wherein the current attitude data includes a measured rotation angle of a target part in the current time interval; determining data to be evaluated based on the angle error between the current attitude data and previously predicted data, wherein the previously predicted data includes a predicted rotation angle of the target part in the current time interval predicted in a previous time interval, the angle error being the error between the measured rotation angle and the predicted rotation angle; and using a comparison result of the angle error and an error threshold to select attitude data from at least one of a plurality of time intervals to the data to be evaluated, wherein closer comparison results correspond to selecting attitude data from more time intervals, and farther comparison results correspond to selecting fewer time intervals. The method includes: attitude data across multiple time intervals, including measured rotation angles of the target body part and changes in these rotation angles; generating future prediction data for future time intervals by inputting the data to be evaluated into a prediction model, wherein the prediction model is trained and learns the attitude changes of the target body part using a machine learning algorithm; the future prediction data includes a predicted rotation angle of the target body part in the future time interval predicted in the current time interval, and the previous prediction data is the prediction data corresponding to the current time interval predicted by the prediction model; and adjusting the acoustic features of an acoustic signal to correspond to the predicted rotation angle in the future time interval, wherein the acoustic features are related to at least one of the amplitude and phase of the acoustic signal.

[0006] Based on the above, the sound signal adjustment method and the computing device for sound signal adjustment in this embodiment of the invention compare the rotation angle error between the currently measured posture data and the posture data predicted for the current time interval in the previous time interval, select posture data from one or more time intervals as the data to be evaluated, determine the posture data corresponding to this data to be evaluated in the future time interval through a prediction model, and adjust the sound characteristics of the sound signal accordingly. Thus, the rotation angle of the next time interval can be predetermined, and the output delay of the sound player can be reduced. Attached Figure Description

[0007] The accompanying drawings are included to further illustrate the invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0008] Figure 1A This is a component block diagram of a system according to an embodiment of the present invention;

[0009] Figure 1B This is a schematic diagram illustrating an application scenario according to an embodiment of the present invention;

[0010] Figure 2 This is a flowchart of a method for adjusting a sound signal according to an embodiment of the present invention;

[0011] Figure 3 This is a schematic diagram illustrating the posture according to an embodiment of the present invention;

[0012] Figure 4 This is a flowchart of a method for identifying rotation angle according to an embodiment of the present invention;

[0013] Figure 5 This is a flowchart for determining the data to be evaluated according to an embodiment of the present invention;

[0014] Figure 6 This is a schematic diagram of a combination of a Convolutional Neural Network (CNN) and a Long Short-Term Memory (LSTM) network according to an embodiment of the present invention.

[0015] Figure 7 This is a schematic diagram of training samples according to an embodiment of the present invention.

[0016] Explanation of icon numbers

[0017] 10: Sound playback device;

[0018] 30: Sensors;

[0019] 50: Arithmetic unit;

[0020] 51: Memory;

[0021] 52: Processor;

[0022] H: Head;

[0023] S210~S240, S410~S430, S510~S540: Steps;

[0024] α H Yaw angle;

[0025] β H Pitch angle;

[0026] γ H Roll angle;

[0027] 610: CNN;

[0028] 611, 614: One-dimensional convolution operation;

[0029] 612, 615: Linear correction;

[0030] 613, 616: Max pooling;

[0031] 620: LSTM network;

[0032] 621: Storage unit;

[0033] 630: Comprehensive calculation;

[0034] TP: Time interval;

[0035] Measure rotation angle sequence;

[0036] The sequence of differences in rotation angles was measured.

[0037] A sequence of changes in the difference in measurement rotation angle between adjacent time intervals. Detailed Implementation

[0038] Reference will now be made in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same component reference numerals are used in the drawings and description to denote the same or similar parts.

[0039] Figure 1A This is a component block diagram of a system according to an embodiment of the present invention. Please refer to... Figure 1A Please refer to Figure 1A The system includes a sound playback device 10, a sensor 30, and a computing device 50.

[0040] The sound playback device 10 can be headphones or a wearable playback device. Figure 1B This is a schematic diagram illustrating an application scenario according to an embodiment of the present invention. Please refer to... Figure 1B The sound playback device 10 is designed to be worn on a user's head H. The speaker unit (in-ear or in-ear type) of the sound playback device 10 is oriented towards the ears on the head H. In one embodiment, the sound playback device 10 is used to play sound signals.

[0041] Sensor 30 can be a camera, video camera, or circuitry or device with image acquisition capabilities. Please refer to [reference needed]. Figure 1B Sensor 30 is a built-in or external image capture device 31. The lens of image capture device 31 can be directed towards the head H. In one embodiment, image capture device 31 is used to capture images. Figure 1B For example, image capture device 31 captures a picture of the head and generates a head image (i.e., acquires an image of the head H). Alternatively, sensor 30 may be an accelerometer, gyroscope, inertial sensor, or a component, circuit, or device with motion detection capabilities. In one embodiment, sensor 30 is used to acquire motion sensing data. For example, motion sensing data related to velocity, angular velocity, acceleration, and / or orientation.

[0042] The computing device 50 can be a smartphone, tablet computer, desktop computer, laptop computer, smart assistant device, wearable device, smart TV, or other electronic device. The computing device 50 is communicatively connected to the sound playback device 10 and the sensor 30. For example, it may be equipped with a USB, UART, or other wired transmission interface (not shown), or Wi-Fi, Bluetooth, or other wireless communication transceiver circuitry (not shown), and thereby transmit or receive signals. For example, the sensor 30 may transmit an image-carrying signal to the computing device 50, or the computing device 50 may transmit an audio signal to the sound playback device 10.

[0043] The computing device 50 includes (but is not limited to) a memory 51 and a processor 52.

[0044] The memory 51 can be any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), or similar component. In one embodiment, the memory 51 is used to store program code, software modules, configuration settings, data (e.g., audio signals, head images, or algorithm parameters), or files, as will be described in detail later.

[0045] Processor 52 is coupled to memory 51. Processor 52 may be a central processing unit (CPU), a graphics processing unit (GPU), or other programmable general-purpose or special-purpose microprocessor, digital signal processor (DSP), programmable controller, field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), neural network accelerator, or other similar components or combinations thereof. In one embodiment, processor 52 is used to execute all or part of the operations of computing device 50, and can load and execute various program codes, software modules, files, and data stored in memory 51. In one embodiment, processor 52 can control image capture device 31 to capture or acquire sensing data from sensor 30. In another embodiment, processor 52 can control the playback functions of sound playback device 10 (e.g., play, pause, switch tracks, fast forward, or rewind). In some embodiments, the functions of processor 52 may be implemented by software or a chip.

[0046] For application scenarios, Figure 1B For example, the computing device 50 is a laptop computer, and the head H is facing the laptop computer's display. However, the user's position and / or orientation may vary.

[0047] The methods described in this embodiment of the invention will be explained in conjunction with the components and modules in the sound playback device 10, sensor 30, and computing device 50. The various processes of this method may be adjusted according to the implementation situation, and are not limited thereto.

[0048] Figure 2 This is a flowchart of a method for adjusting a sound signal according to an embodiment of the present invention. Please refer to... Figure 2 The processor 52 measures the current posture data within the current time interval (step S210). Specifically, the current time interval is a time interval corresponding to the current time point. The time interval in this document is, for example, 15, 30, or 60 milliseconds, that is, 7.5, 15, or 30 milliseconds before and after the current time point, but its length can still be adjusted according to actual needs. The current posture data includes the rotation angle of the target body part measured within the current time interval. The target body part can be the head, ears, or other body parts. In one embodiment, the head is used to wear the sound playback device 10. Figure 1BAs shown, head H is fitted with over-ear headphones (i.e., an example of sound playback device 10). Rotation of head H will cause a change in posture. The change in posture includes the rotation angle of the head from a first orientation to a second orientation. For example, at time t, the head is facing the first orientation, and at time t+1, the head is facing the second orientation.

[0049] Figure 3 This is a schematic diagram illustrating the posture according to an embodiment of the present invention. Please refer to... Figure 3 The rotation angle of the head H includes yaw angles (α) corresponding to the three axes. H Pitch angle β H and roll angle (Roll) γ H .

[0050] In one embodiment, the processor 52 can identify the pose changes of the target part based on the captured image. Figure 4 This is a flowchart of a method for identifying rotation angle according to an embodiment of the present invention. Please refer to... Figure 4 The processor 52 can acquire the captured image through the image capture device 31 (step S410). For example... Figure 1B As shown, the head H is located in front of the image capture device 31, and the field of view of the lens of the image capture device 31 covers the head H. Image features of the captured image can be used to identify pose changes. Image features include, for example, histogram of Oriented Gradient (HOG), scale-invariant feature transform (SIFT), Haar, or speeded-up robust features (SURF). Image features may also be feature maps obtained through machine learning models.

[0051] The image capture device 31 captures images of the head as it rotates from a first orientation to a second orientation. The image capture device 31 can continuously capture images of the head. The image capture frequency can be 24, 60, or 120 frames per second, and is not limited to this. The image capture device 31 may also trigger the image capture function based on predetermined conditions (e.g., user operation or sound).

[0052] Processor 52 can identify target parts (e.g., head or face) in the captured image (step S420). Identification can be based on object detection techniques. For example, processor 52 can apply neural network-based algorithms (e.g., YOLO (You Only Look Once), Region Based Convolutional Neural Networks (R-CNN), or Fast R-CNN)) or feature matching-based algorithms (e.g., feature matching using Histogram of Oriented Gradients (HOG), Scale Invariant Feature Transform (SIFT), Haar, or Speeded Robust Feature (SURF)).

[0053] The processor 52 can also identify organs (e.g., eyes, mouth, or nose) in the captured images. It should be noted that when the lens of the image capture device 31 is fixed, the head may not be able to capture all facial organs in some poses.

[0054] Processor 52 can define feature points for the captured image. For example, feature points may be located at the corners of the mouth, the tip of the nose, the upper edge of the ear, or the eyes, and are not limited to these.

[0055] Processor 52 can determine the rotation angle based on the position of feature points of the target area in the captured image (step S430). Processor 52 can track the position of one or more feature points in multiple consecutive captured images. Changes in the pose of the target area (e.g., the head) will be reflected in changes in the position of these feature points. For example,

[0056]

[0057] β H =RP′ nose-x -RP nose-x …(2)

[0058] γ H =RP nose-y -RP′ nose-y …(3)

[0059] RP L-eye-y RP represents the position of the left eye feature point on the vertical axis of the captured image. R-eye-y RP represents the position of the right eye feature point on the vertical axis of the captured image. L-eye-x RP represents the position of the left eye feature point on the horizontal axis in the captured image. R-eye-x Let RP′ be the position of the right eye feature point on the horizontal axis in the captured image. nose-x RP represents the position of the nasal feature point on the horizontal axis in the captured image when the head is in its second orientation. nose-xRP′ represents the position of the nasal feature point on the horizontal axis in the captured image when the head is in its first orientation. nose-y RP represents the position of the nasal feature point on the vertical axis in the captured image when the head is in its second orientation. nose-y The position of the nasal feature point on the vertical axis in the captured image when the head is in its first orientation.

[0060] In other embodiments, processor 52 may also apply neural network-based algorithms (e.g., YOLO, region-based convolutional neural networks (R-CNN), or Fast R-CNN) or feature matching-based algorithms (e.g., feature matching using Histogram of Oriented Gradients (HOG), Scale-Invariant Feature Transform (SIFT), Haar, or Speed-Up Robust Feature (SURF)). For example, the neural network is trained to learn the associations between multiple reference poses / rotation angles and image features. Alternatively, a lookup table records the associations between multiple reference poses / rotation angles and image features. Yet another example is a transformation function that records the associations between multiple reference poses / rotation angles and image features.

[0061] In another embodiment, sensor 30 is a motion sensor (e.g., a gyroscope, accelerometer, or inertial measurement unit). The sensing data from the motion sensor can be used to analyze attitude changes.

[0062] Please refer to Figure 2 The processor 52 determines the data to be evaluated based on the angle error between the current attitude data and the previously predicted data (step S220). Specifically, the previously predicted data includes the predicted rotation angle of the target part in the current time interval predicted in the previous time interval. The previous time interval is a time interval earlier than the current time interval. For example, it is earlier than 20, 30, or 50 milliseconds, but is not limited thereto. In a certain previous time interval, the processor 52 can predict the predicted data (e.g., the predicted rotation angle of the target part in the current time interval) corresponding to the current time interval (i.e., a future time interval relative to the previous time interval, and the future time interval is later than the previous time interval) based on the attitude data of one or more previous time intervals. The generation of the predicted data will be described in detail later.

[0063] Angular error is the error between the measured rotation angle and the predicted rotation angle. For example, the yaw angle α measured in the current time interval. H Pitch angle β H and roll angle γ H The yaw angle α of the predicted current time interval H Pitch angle β H and roll angle γ H The error between them can be taken as the rotation angle of the three axes (i.e., the yaw angle α). H Pitch angle βH and roll angle γ H The root mean square or other statistical value of the error is used as a representative of the angular error. The error can be calculated by subtracting the measured rotation angle from the predicted rotation angle. For example, the mathematical expression for the current time interval n is:

[0064]

[0065] E θ (n) represents the rotation angle error (i.e., the aforementioned angle error) estimated from the current time interval measurement and the previous time interval (i.e., the previous time interval separated from the current time interval by one time interval), θ H (n) represents the measured rotation angle in the measured attitude data within the current time interval n, and This represents the predicted rotation angle for the current time interval n from the previously predicted data. Similarly, the mathematical expression for the previous time interval n-1 (which is one time interval apart from the current time interval) is: Where E θ (n-1) represents the rotation angle error (i.e., the aforementioned angle error) estimated from the previous time interval measurement and the previous time interval (i.e., the previous time interval separated from the previous time interval n-1 by one time interval), θ H (n-1) represents the measured rotation angle from the previous time interval n-1 of the measured attitude data, and This represents the predicted rotation angle for the previous time interval n-1 from the previously predicted data. The mathematical expression for the angle error for the remaining time intervals can be deduced similarly, and will not be elaborated here.

[0066] The comparison between the angle error and the error threshold is used to select at least one of the attitude data from multiple time intervals to the data to be evaluated. Closer comparison results correspond to selecting attitude data from more time intervals. That is, the smaller the angle error or the closer the current attitude data is to the previously predicted data (e.g., closer in the feature coordinate system), the more time intervals of attitude data are selected. Multiple time intervals include the current time interval, a previous time interval separated by one time interval, a previous time interval separated by two time intervals, ..., and a previous time interval separated by N time intervals. N is a positive integer greater than zero. N is, for example, 9, 10, or 15, and may be related to the length of the time intervals, but is not limited to this.

[0067] On the other hand, the further away the comparison result is, the fewer time intervals of pose data are selected. That is, the larger the angle error of the comparison result or the further the current pose data is from the previous prediction data (e.g., the greater the distance in the feature coordinate system), the fewer time intervals of pose data are selected.

[0068] The current time interval and each previous time interval include their respective attitude data. That is, if more time intervals of attitude data are selected for the data to be evaluated, the number of time intervals corresponding to the attitude data included in the data to be evaluated will be greater. If fewer time intervals of attitude data are selected for the data to be evaluated, the number of time intervals corresponding to the attitude data included in the data to be evaluated will be smaller.

[0069] The attitude data for each of the multiple time intervals includes the measured rotation angle of the target part within that time interval and the change in the measured rotation angle. The determination of the measured rotation angle can be referred to the explanation in step S210, and will not be repeated here. The change in the measured rotation angle can be the difference in the measured rotation angle between adjacent time intervals (for example, it can be obtained by subtracting the values ​​of two measured rotation angles). For example, the mathematical expression for the difference in the measured rotation angle corresponding to the current time interval n is:

[0070] Δθ H (n)=θ H (n)-θ H (n-1)…(5)

[0071] Δθ H (n) represents the difference in measurement rotation angle between the current time interval n and the previous time interval n-1 (which is one time interval apart from the current time interval), θ H (n) is the same as the measured rotation angle in the measured attitude data of the current time interval n defined by equation (4), and θ H (n-1) represents the measured rotation angle from the attitude measurement data in the previous time interval n-1. Similarly, the mathematical expression corresponding to the previous time interval n-1 is Δθ. H (n-1)=θ H (n-1)-θ H (n-2), where θ H (n-2) represents the measured rotation angle in the attitude measurement data of the previous time interval n-2 (which is two time intervals away from the current time interval n and one time interval away from the previous time interval n-1). The mathematical expression for the difference in the measured rotation angles corresponding to the other time intervals can be deduced by analogy, and will not be elaborated here.

[0072] The change in the measured rotation angle can also be the difference between the measured rotation angles in adjacent time intervals (i.e., the change in the difference, which can be obtained, for example, by subtracting the values ​​of the two differences). For example, the mathematical expression corresponding to the current time interval n is:

[0073] Δ 2 θ H (n)=Δθ H (n)-Δθ H (n-1)…(6)

[0074] Δ 2 θ H (n) represents the difference between the measured rotation angles of the adjacent current time interval n and the previous time interval n-1 (which is separated from the current time interval by one time interval), Δθ. H (n) is the same as the difference in the measured rotation angle between the current time interval n and the previous time interval n-1 as defined by equation (2), and Δθ H (n-1) represents the difference in measurement rotation angle between the previous time interval n-1 and another previous time interval n-2 (which is two time intervals away from the current time interval n and one time interval away from the previous time interval n-1). Similarly, the mathematical expression corresponding to the previous time interval n-1 is Δ. 2 θ H (n-1)=Δθ H (n-1)-Δθ H (n-2), where Δ 2 θ H (n-1) represents the difference between the measured rotation angles of adjacent previous time intervals n-1 and n-2, Δθ H (n-2) represents the difference in the measured rotation angle between the previous time interval n-2 and another previous time interval n-3 (which is three time intervals away from the current time interval n, two time intervals away from the previous time interval n-1, and one time interval away from the previous time interval n-2). The mathematical expression for the difference (i.e., the change in difference) between the measured rotation angles corresponding to the remaining time intervals can be deduced by analogy, and will not be elaborated here.

[0075] Alternatively, the change in the measured rotation angle can be a combination of the above differences (e.g., a combination of the measured rotation angle and its change corresponding to the current time interval n [Δθ]). H (n),Δ 2 θ H (n)]). At this time, the attitude data for each time interval is a combination of the measured rotation angle of the current time interval and the above differences (for example, the combination of the measured rotation angle and its changes corresponding to the current time interval n [θ]). H (n),ΔθH (n),Δ 2 θ H [(n)], corresponding to the combination of the measured rotation angle and its variation in the previous time interval n-1 [θ] H (n-1),Δθ H (n-1),Δ 2 θ H (n-1)], and so on.

[0076] Figure 5 This is a flowchart illustrating the determination of data to be evaluated according to an embodiment of the present invention. Please refer to... Figure 5 The error thresholds used for comparison with the angle error include an upper error limit and / or a lower error limit. The processor 52 can compare the angle error with the lower error limit and / or compare the angle error with the upper error limit (step S510). The upper error limit is, for example, 15, 20, or 25 degrees, and the lower error limit is, for example, 8, 10, or 12 degrees, but is not limited thereto.

[0077] Since the angular error between the current attitude data and the previously predicted data is less than the lower limit of error, the processor 52 can select all the measured rotation angles and changes in the measured rotation angles from multiple time intervals to the data to be evaluated (step S520). Specifically, the processor 52 can define the length of the time window, and this length is the number of these time intervals. For example, if the length of the time window is 10, then it includes ten time intervals. That is, the multiple time intervals include the current time interval, the previous time interval that is one time interval away from the current time interval, the previous time interval that is two time intervals away from the current time interval, ..., and the previous time interval that is nine time intervals away from the current time interval.

[0078] Furthermore, the processor 52 can define the data to be evaluated as For the measurement rotation angle (e.g., θ) in the data to be evaluated corresponding to the current time interval n for one or more time intervals. H (n), θ H (n-1), or θ H A sequence or vector of (n-2) This refers to the difference in the measured rotation angle (e.g., Δθ) in the data to be evaluated for one or more time intervals corresponding to the current time interval n. H (n), Δθ H (n-1), or Δθ H A sequence or vector of (n-2) and This refers to the difference / variation (e.g., Δ) between the difference in measured rotation angles for one or more time intervals in the data to be evaluated corresponding to the current time interval n and the difference in measured rotation angles for adjacent time intervals. 2 θH (n), Δ 2 θ H (n-1), or Δ 2 θ H A sequence or vector of (n-2).

[0079] Since the comparison results show that the smaller the angle error or the closer the current attitude data is to the previously predicted data, the rotation of the target part still conforms to the inertial trajectory, and the inertial trajectory can be referenced for more of the previous attitude data. For example, suppose the length of the time window corresponding to the data to be evaluated is 10. If all the measured rotation angles and the changes in the measured rotation angles from multiple time intervals are selected and applied to the data to be evaluated, then the data to be evaluated is... In That And its

[0080] Since the angle error between the current attitude data and the previously predicted data is between the lower and upper limits of the error (i.e., the angle error is greater than the lower limit and less than the upper limit), the processor 52 can select a portion of the time intervals (i.e., a portion of multiple time intervals) to apply the measured rotation angle and the change in the measured rotation angle to the data to be evaluated (step S530). Specifically, if the length of the time window is I, the processor 52 can select J time intervals of attitude data, where I is a positive integer greater than two and J is a positive integer less than I. For example, assume the length of the time window corresponding to the data to be evaluated is 10. If the measured rotation angle and the change in the measured rotation angle of a portion of multiple time intervals are applied to the data to be evaluated, then the data to be evaluated is... In That And its In other words, for measuring rotation angles, only the rotation angles of the current time interval n and the previous time interval n-1 are selected; for differences in measured rotation angles, only the differences in measured rotation angles of the current time interval n are selected; for differences between adjacent measured rotation angles (i.e., changes in differences), the differences between the differences in measured rotation angles of any time interval and the differences in measured rotation angles of their adjacent time intervals are prohibited / not selected. For unselected time intervals, the processor 52 can set their corresponding value in the data to be evaluated to zero or other initial values. Thus, compared to step S520, which selects attitude data from all time intervals, step S530 selects attitude data from fewer time intervals for the data to be evaluated.

[0081] If the angular error between the current attitude data and the previously predicted data exceeds the upper limit of error, the processor 52 can select the measured rotation angle of the current time interval to the data to be evaluated (step S540). Specifically, a sudden rotation of the target part will cause the angular error to increase excessively (i.e., exceed the upper limit of error). Therefore, the amount of inertial data that can be used for reference (i.e., attitude data of one or more time intervals) becomes less. For example, suppose the length of the time window corresponding to the data to be evaluated is 10. If the measured rotation angle of the current time interval n is selected to the data to be evaluated, then the data to be evaluated is... In That And its In other words, for the measured rotation angle, only the measured rotation angle of the current time interval n is selected; for the difference in measured rotation angle, the difference in measured rotation angle of any time interval is prohibited / not selected (e.g., the corresponding values ​​of the sequence or vector in the data to be evaluated are all zero or initial values); for the difference between the differences of adjacent measured rotation angles, the difference between the difference of measured rotation angle of any time interval and the difference between the measured rotation angle of its adjacent time interval is prohibited / not selected (e.g., the corresponding values ​​of the sequence or vector in the data to be evaluated are all zero or initial values). That is, all measured rotation angles and changes in measured rotation angles from previous time intervals are prohibited / not selected. Therefore, compared to step S530 which selects attitude data from all time intervals, step S540 selects attitude data from fewer time intervals for the data to be evaluated.

[0082] It should be noted that in other embodiments, the error threshold is not limited to the upper and lower limits of error, and the number of time intervals selected for each threshold can be adjusted according to actual needs.

[0083] Please refer to Figure 2 The processor 52 generates future prediction data for future time intervals by inputting the data to be evaluated into the prediction model (step S230). Specifically, the prediction model is trained and learns the pose changes of the target part through a machine learning algorithm. The machine learning algorithm is, for example, a Multiple Layer Perception (MLP), a Convolutional Neural Network (CNN), a Long Short-Term Memory (LSTM) network, or a Temporal Convolutional Network (TCN) (e.g., Conv-TasNet), but is not limited thereto.

[0084] For example, Figure 6This is a schematic diagram illustrating a combination of a Convolutional Neural Network (CNN) and a Long Short-Term Memory (LSTM) network according to an embodiment of the present invention. Please refer to... Figure 6 The trajectory data of the target part (taking the head as an example) (i.e., the pose changes of the target part, such as the data to be evaluated mentioned above) can be used as input data for the prediction model. The input data is sequentially processed by the CNN 610 through one-dimensional convolution operation 611, linear correction 612 (e.g., rectified linear unit function (ReLU)), maximum pooling 613, one-dimensional convolution operation 614, linear correction 615 (e.g., ReLU), and maximum pooling 616, and the features obtained from the training samples are output to the LSTM network 620. The LSTM network 620 includes multiple memory cells 621 (or LSTM blocks). The LSTM network 620 can store numerical values ​​of indefinite duration. Each memory cell 612 has a gate that can determine whether the input data is important enough to be remembered and whether it can be output. The dropout operation refers to the random discarding of a certain proportion of neurons from the original network during each iteration in the LSTM network 620. Then, by combining the multiple output data from the LSTM network 620 (e.g., a dense model or a fully connected layer), future prediction data (e.g., the predicted trajectory for a future time interval) can be generated.

[0085] Machine learning algorithms can train predictive models to understand labeled samples (i.e., pose data with labeled results, i.e., pose data in the current time interval (and previous time intervals)), such as pose data in the determined next time interval (i.e., future time intervals), and establish a correlation between the data to be evaluated (i.e., the model's input) and the future predicted data (i.e., the model's output). For example, during the learning phase of the predictive model, the parameters of the predictive model are recursively updated by minimizing an error function (related to the error between the model's output and the labeled results). Methods for updating parameters include, for example, gradient descent, but are not limited to, this. The predictive model can have three degrees of freedom (DOF) (e.g., corresponding to...). Figure 3 The trajectory motion model (rotation in three directions) is used to obtain the spatial and temporal characteristics of the motion trajectory of the target part.

[0086] Figure 7 This is a schematic diagram of training samples according to an embodiment of the present invention. Please refer to... Figure 7These training samples are labeled samples. Assume the time window length is 10. A training sample includes pose data corresponding to 10 time intervals TP, with corresponding labeling results (i.e., labels). Taking training sample A as an example, the pose data for 10 time intervals TP... Measurement rotation angle sequence Its measurement rotation angle difference sequence Furthermore, the sequence of changes in the measured rotation angles between adjacent time intervals. Furthermore, the label A of the training sample A is the measured rotation angle θ corresponding to the future time interval n+1 (the next time interval relative to the current time interval n). H (n+1), Measure the difference in rotation angle Δθ H The variation Δ of the measured rotation angle in (n+1) and adjacent time intervals 2 θ H (n+1).

[0087] A predictive model is a model constructed after learning, which can be used to infer the future prediction data corresponding to the data to be evaluated (e.g., attitude data of one or more time intervals to be evaluated).

[0088] Future forecast data includes predicted rotation angles of the target location in the current time interval for future time intervals. For example, yaw angles α corresponding to the three axes. H Pitch angle β H and roll angle γ H Furthermore, the previously predicted data used in step S220 is the predicted data corresponding to the current time interval predicted by the prediction model. That is, for the previous time interval (i.e., the previous time interval that is one time interval away from the current time interval), the data to be evaluated for this previous time interval is determined (refer to the description of step S220), and future predicted data for the future time interval relative to this previous time interval (i.e., the current time interval in step S210) is generated by inputting the data to be evaluated for this previous time interval into the prediction model.

[0089] Please refer to Figure 2 The processor 52 adjusts the acoustic characteristics of the sound signal to correspond to the predicted rotation angle for a future time interval (step S240). Specifically, the sound signal is the signal that the computing device 50 intends to transmit to the sound playback device 10 and play through the sound playback device 10. The content of the sound signal can be music, a speech, a lecture, or a broadcast, and is not limited thereto.

[0090] Sound features relate to at least one of the amplitude and phase of a sound signal. In one embodiment, sound features include a frequency response. The frequency response is the response of the sound signal in the frequency domain, or it may be the amplitude of the sound signal corresponding to multiple frequencies. Processor 52 can measure the frequency response of the sound signal. For example, its response in the frequency domain can be measured by inputting an impulse response, but it is not limited thereto.

[0091] In one embodiment, the sound features also include signal delay. Signal delay is the time difference of the sound signal between two channels (e.g., left and right channels). For example, the cross-correlation between the sound signals of the two channels is calculated, and the delay amount (as signal delay) is determined based on the peak value of the cross-correlation function.

[0092] It is worth noting that sound waves can propagate along different paths due to obstruction or interference from objects. For example, the surface of the auricle of the ear comprises curved surfaces with multiple degrees of curvature. Sound waves from a distance can be reflected by the auricle into the ear canal. Alternatively, sound waves can enter the ear canal directly. Sound waves from different directions also exhibit different frequency distribution characteristics. The frequency response reflects these distribution characteristics. In other words, sound waves from different directions may correspond to different frequency responses, with the amplitude / intensity of the response possibly differing at certain frequencies.

[0093] On the other hand, the propagation paths of sound signals to the left and right ears, whether directly or via reflection, may differ, and the propagation times of multiple paths may also differ. That is, the time it takes for a sound signal originating from a single source to reach the left and right ears, whether directly or via reflection, may be different. This difference in propagation / arrival time (i.e., signal delay) can affect the phase of the sound signal. Furthermore, sound waves from different directions may correspond to different signal delays in two channels.

[0094] In one embodiment, the processor 52 can configure corresponding spatial sound effects for multiple orientations of the target area. In another embodiment, the processor 52 can set spatial sound effects or other sound effects via an equalizer. The parameters of the equalizer can be corresponding gain / power (used to increase or decrease the response of the corresponding frequency / band) at multiple frequencies / bands. Different parameters can be configured for different orientations to provide spatial sound effects or other sound effects. Taking spatial sound effects as an example, the processor 52 can transfer the two-channel audio signal to a surround sound field equipped with multiple virtual speakers, adjust the frequency response and / or phase from different directions based on the Head Related Transfer Functions (HRTF) theory, and then transfer the adjusted audio signal back to the two-channel stereo sound field signal.

[0095] In one embodiment, processor 52 can adjust the frequency response of the sound signal via a first parameter of the equalizer. The first parameter corresponds to the spatial sound effect of the predicted rotation angle. The sound signal is recorded from a sound source located at a position in the direction of the sound source. That is, the microphone is located at a reference center, and the direction of the sound source is the direction of the sound source relative to the reference center. The direction of the sound source can include a horizontal direction and / or a vertical direction. The sound source can be a person, musical instrument, animal, loudspeaker, device, wind, or water, and is not limited thereto. For example, a person sings in front of a microphone, the microphone records the human voice, and generates a sound signal accordingly. The distance between the sound source and the reference center can be 20 cm, 50 cm, or 100 cm, and is not limited thereto. The spatial sound effect can set the direction of the sound source so that the listener perceives the sound as originating from this direction. Assuming the position of the sound source is fixed, the direction of the sound source relative to the target part will change in response to the rotation of the target part (i.e., change the direction of the sound source). Therefore, the corrected orientation (i.e., the orientation after the predicted rotation angle) corresponds to the first parameter of the equalizer. The first parameter has a corresponding gain / power at one or more frequencies / bands.

[0096] In one embodiment, processor 52 can adjust the signal delay of the two channels of the audio signal to a corrected delay. This corrected delay corresponds to the spatial sound effect of the predicted rotation angle. The corrected delay is the delay corresponding to the orientation of the target part after rotating by the predicted rotation angle. As explained above, the time it takes for a sound wave from a sound source to directly reach the left and right ears may be different (the difference being the time delay). In spatial sound effect processing, the time delay corresponding to different orientations may be different. Processor 52 can delay at least one of the two channels of the audio signal so that the signal delay of the two channels of the audio signal is the same as the corrected delay (i.e., the time delay corresponding to the orientation after rotating by the predicted rotation angle). For example, the time delay of the audio signal can be achieved by using a buffer or delay circuit.

[0097] In one embodiment, the future time interval includes a first interval and a second interval. The first interval precedes the second interval, and the second interval follows the end of the first interval. Assuming the length of the time interval is 30 milliseconds, the lengths of both the first and second intervals are 15 milliseconds. However, the length of the sub-intervals can still be adjusted according to actual needs. To avoid discomfort caused by instantaneous changes in the sound field due to excessively rapid rotation of the target part, the predicted rotation angle of each time interval can be divided into multiple (e.g., positive integers greater than one) sub-intervals before adjusting the sound signal.

[0098] Taking two sub-intervals as an example, processor 52 can determine that the new predicted rotation angle corresponding to the first interval is the average of the predicted rotation angle of the current time interval and the predicted rotation angle of the future time interval, and can determine that the new predicted rotation angle corresponding to the second interval is the predicted rotation angle of the future time interval:

[0099] .

[0100] For the new predicted rotation angle in the future time interval n+1, Let be the predicted rotation angle for the current time interval n (i.e., the predicted rotation angle for the current time interval n predicted in the previous time interval n-1), and This is the predicted rotation angle for the future time interval n+1 (i.e., the predicted rotation angle generated by the prediction model in step S230). The first interval is a transition zone, determined by the predicted rotation angle of the target location predicted in the previous time interval n-1. and the predicted rotation angle of the target location within the current time interval n. The composition (e.g., average or other weighted values). The second interval represents the rotation angle of the primary target location, directly corresponding to the predicted rotation angle of the target location in the future time interval n+1 generated by the prediction module. In other words, the adjustment of the sound signal in the first interval will be based on the adjustment corresponding to the new predicted rotation angle of the first interval; the adjustment of the sound signal in the second interval will be based on the adjustment corresponding to the new predicted rotation angle of the second interval.

[0101] It should be noted that in other embodiments, the future time interval can be divided into more sub-intervals. The predicted rotation angles in the new predicted rotation angles of these sub-intervals... With predicted rotation angle The proportions may differ. For example, the predicted rotation angle in the new predicted rotation angle of the sub-interval closer to the current time interval. The proportion is relatively high, and the more distant the sub-interval is from the current time interval, the higher the predicted rotation angle is from the new predicted rotation angle. The proportion is relatively low.

[0102] In summary, in the sound signal adjustment method and the computing device for sound signal adjustment in the embodiments of the present invention, suitable posture data is evaluated and selected as the evaluation data for the prediction model by using previously predicted data from a previous time interval and current posture data measured in the current time interval. The prediction model generates future predicted data corresponding to the evaluation data, and the sound characteristics of the sound signal are adjusted accordingly. This results in spatial sound effects with less delay. The evaluation data can be dynamically and immediately adjusted according to the degree of rotational change of the target part. Furthermore, by decomposing the time interval into multiple sub-intervals of new predicted rotation angles before adjusting the sound signal, uncomfortable auditory experiences caused by instantaneous sound field changes can be avoided.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for adjusting a sound signal, characterized in that, include: Measure the current attitude data for the current time interval, wherein the current attitude data includes the measured rotation angle of the target part in the current time interval; The data to be evaluated is determined based on the angle error between the current attitude data and the previously predicted data. The previously predicted data includes the predicted rotation angle of the target part in the current time interval predicted in the previous time interval. The angle error is the error between the measured rotation angle and the predicted rotation angle. The comparison result of the angle error and the error threshold is used to select the attitude data of at least one of multiple time intervals to the data to be evaluated. The closer the comparison result is, the more attitude data of the time intervals are selected. The farther the comparison result is, the fewer attitude data of the time intervals are selected. The attitude data of the time interval includes the measured rotation angle of the target part in the time interval and the change of the measured rotation angle. By inputting the data to be evaluated into the prediction model, future prediction data for future time intervals is generated. The prediction model is trained and learns the posture changes of the target part through a machine learning algorithm. The future prediction data includes the predicted rotation angle of the target part in the future time interval predicted in the current time interval. The previous prediction data is the prediction data predicted by the prediction model corresponding to the current time interval. as well as The acoustic features of the sound signal are adjusted to correspond to the predicted rotation angle of the future time interval, wherein the acoustic features are related to at least one of the amplitude and phase of the sound signal.

2. The method for adjusting a sound signal according to claim 1, wherein the error threshold includes an upper error limit and a lower error limit, and the step of determining the data to be evaluated based on the angle error between the current attitude data and the previously predicted data includes: Compare the angle error with the lower limit of the error, wherein, in response to the angle error being less than the lower limit of the error, all the measured rotation angles and the changes in the measured rotation angles of the time intervals are selected and incorporated into the data to be evaluated. Compare the angle error with the upper limit of the error, and compare the angle error with the lower limit of the error, wherein the measured rotation angle and the change of the measured rotation angle during a selected portion of the time interval, which reflects the angle error being between the lower limit of the error and the upper limit of the error, are incorporated into the data to be evaluated; and The angle error is compared with the upper limit of the error, wherein the measured rotation angle of the current time interval is selected to the data to be evaluated in response to the angle error being greater than the upper limit of the error.

3. The method for adjusting the sound signal according to claim 1, wherein the change in the measured rotation angle includes the difference in the measured rotation angle between the first time interval and the second time interval in the time interval and the change in the difference.

4. The method for adjusting sound signals according to claim 1, wherein the machine learning algorithm includes a convolutional neural network and a long short-term memory network.

5. The method for adjusting a sound signal according to claim 1, wherein the future time interval includes a first interval and a second interval, the first interval being earlier than the second interval, and the adjustment method further includes: The new predicted rotation angle corresponding to the first interval is determined as the average of the predicted rotation angle of the current time interval and the predicted rotation angle of the future time interval; and The new predicted rotation angle corresponding to the second interval is determined as the predicted rotation angle for the future time interval.

6. The method for adjusting a sound signal according to claim 1, wherein the sound features include frequency response and signal delay, the frequency response being the amplitude of the sound signal at multiple frequencies, the signal delay being the time difference of the sound signal between two channels, and the step of adjusting the sound features of the sound signal to correspond to the predicted rotation angle of the future time interval includes: The frequency response of the sound signal is adjusted by a first parameter of the equalizer, wherein the first parameter corresponds to the spatial sound effect of the predicted rotation angle; as well as The signal delay of the two channels of the sound signal is adjusted to a corrected delay, wherein the corrected delay corresponds to the spatial sound effect of the predicted rotation angle.

7. A computing device for adjusting sound signals, characterized in that, include: Memory, used to store program code; as well as A processor, coupled to the memory, and configured to load the program code for execution: Measure the current attitude data for the current time interval, wherein the current attitude data includes the measured rotation angle of the target part in the current time interval; The data to be evaluated is determined based on the angle error between the current attitude data and the previously predicted data. The previously predicted data includes the predicted rotation angle of the target part in the current time interval predicted in the previous time interval. The angle error is the error between the measured rotation angle and the predicted rotation angle. The comparison result of the angle error and the error threshold is used to select the attitude data of at least one of multiple time intervals to the data to be evaluated. The closer the comparison result is, the more attitude data of the time intervals are selected. The farther the comparison result is, the fewer attitude data of the time intervals are selected. The attitude data of the time interval includes the measured rotation angle of the target part in the time interval and the change of the measured rotation angle. By inputting the data to be evaluated into the prediction model, future prediction data for future time intervals is generated. The prediction model is trained and learns the posture changes of the target part through a machine learning algorithm. The future prediction data includes the predicted rotation angle of the target part in the future time interval predicted in the current time interval. The previous prediction data is the prediction data predicted by the prediction model corresponding to the current time interval. as well as The acoustic features of the sound signal are adjusted to correspond to the predicted rotation angle of the future time interval, wherein the acoustic features are related to at least one of the amplitude and phase of the sound signal.

8. The computing device for adjusting a sound signal according to claim 7, wherein the error threshold includes an upper error limit and a lower error limit, and the processor is further configured to: Compare the angle error with the lower limit of the error, wherein, in response to the angle error being less than the lower limit of the error, all the measured rotation angles and the changes in the measured rotation angles of the time intervals are selected and incorporated into the data to be evaluated. Compare the angle error with the upper limit of the error, and compare the angle error with the lower limit of the error, wherein the measured rotation angle and the change of the measured rotation angle during a selected portion of the time interval, which reflects the angle error being between the lower limit of the error and the upper limit of the error, are incorporated into the data to be evaluated; and The angle error is compared with the upper limit of the error, wherein the measured rotation angle of the current time interval is selected to the data to be evaluated in response to the angle error being greater than the upper limit of the error.

9. The computing device for adjusting a sound signal according to claim 7, wherein the change in the measured rotation angle includes the difference in the measured rotation angle between a first time interval and a second time interval in the time interval and the change in the difference.

10. The computing device for adjusting sound signals according to claim 7, wherein the machine learning algorithm includes a convolutional neural network and a long short-term memory network.

11. The computing device for adjusting a sound signal according to claim 7, wherein the future time interval includes a first interval and a second interval, the first interval being earlier than the second interval, and the processor is further configured to: The new predicted rotation angle corresponding to the first interval is determined as the average of the predicted rotation angle of the current time interval and the predicted rotation angle of the future time interval; and The new predicted rotation angle corresponding to the second interval is determined as the predicted rotation angle for the future time interval.

12. The computing device for adjusting a sound signal according to claim 7, wherein the sound features include frequency response and signal delay, the frequency response being the amplitude of the sound signal at a plurality of frequencies, the signal delay being the time difference of the sound signal between two channels, and the processor is further configured to: The frequency response of the sound signal is adjusted by a first parameter of the equalizer, wherein the first parameter corresponds to the spatial sound effect of the predicted rotation angle; and The signal delay of the two channels of the sound signal is adjusted to a corrected delay, wherein the corrected delay corresponds to the spatial sound effect of the predicted rotation angle.

Images