A method, system, and vehicle for adaptive adjustment of human voice location in vehicle-mounted video.
By classifying and recognizing in-vehicle video content, and combining lip movement detection and voiceprint features, the audio parameters of the in-vehicle speaker array are dynamically adjusted. This solves the problem of insufficient adjustment of human voice position in in-vehicle audio systems during video playback, achieving precise synchronization between human voice and the position of people in the video, and enhancing the immersive audiovisual experience.
Patent Information
- Application Number
- CN202611104463.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-23
- Publication Date
- 2026-11-17
AI Technical Summary
Existing in-vehicle audio systems cannot dynamically adjust the spatial orientation of the speaker's voice based on the real-time position of the person speaking in the video, resulting in audio-visual separation and insufficient immersion for drivers and passengers.
By classifying video content into scenes, combining image recognition and lip movement detection, the normalized two-dimensional continuous coordinates of the person are obtained. The voiceprint features are used to identify the target speaker, and the voice is rendered to the three-dimensional sound field coordinates of the vehicle speaker array through a preset mapping function. The audio gain and delay parameters of the speaker channel are dynamically calculated to achieve adaptive adjustment of the voice's location.
It achieves real-time and accurate synchronization between the location of human voices and the position of people in the video, solving the problems of sound and image separation and insufficient scene adaptation, and significantly improving the immersive audiovisual experience for drivers and passengers.
Smart Images

Figure CN123123783A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of in-vehicle multimedia audio processing technology, and in particular to an in-vehicle video voice location adaptive adjustment method, system and vehicle. Background Technology
[0002] In-vehicle audio systems typically feature multi-speaker arrays. By combining and arranging speakers of different frequency bands, along with preset sound effect algorithms, various sound effect modes can be achieved, such as panoramic sound for movie viewing and surround sound for music playback. Existing adaptive sound adjustment solutions mainly determine the number and position of occupants based on seat occupancy signals, seatbelt status, or in-vehicle cameras, and then call preset sound effect modes to adjust the overall sound field. However, in in-vehicle video playback scenarios, the above solutions can only achieve static sound field presets and cannot dynamically adjust the spatial orientation of the speaker's voice based on the real-time position of the speaker in the video. When a person in the video is speaking from the left side, their voice still comes from the central channel or a fixed direction, resulting in sound-image separation and insufficient audiovisual immersion for the driver and passengers. Summary of the Invention
[0003] To address the technical problems mentioned above, this invention provides a method, system, and vehicle for adaptive adjustment of human voice location in vehicle-mounted video.
[0004] To achieve the above objectives, a first aspect of the present invention provides a method for adaptive adjustment of the location of human voices in vehicle-mounted video, comprising: The system acquires the currently playing video content, classifies the video content into scenes to obtain the scene type to which the currently playing content belongs, and determines the corresponding sound field processing mode based on the scene type. When the sound field processing mode is human voice location tracking, image recognition is performed on the video frame of the video content to obtain the position coordinates of each person in the frame, and lip movement detection is performed on the lip area of each person in the frame to obtain the speaking state marker of each person. The system acquires the currently playing audio signal, separates the human voice component from the audio signal, determines the target speaker from multiple speakers based on the speaking state marker and the voiceprint features of the human voice component, and obtains the normalized two-dimensional continuous coordinates corresponding to the target speaker. The normalized two-dimensional continuous coordinates of the target speaker are converted into three-dimensional sound field coordinates of the vehicle speaker array through a preset mapping function. The audio gain and delay parameters of each speaker channel are dynamically calculated based on the three-dimensional sound field coordinates. The human voice component is rendered to the corresponding spatial orientation using the audio gain and delay parameters. The playback method of the background sound component is determined according to the scene type.
[0005] Furthermore, the scene types include dialogue scenes, action scenes, music scenes, and narration scenes; determining the corresponding sound field processing mode based on the scene type specifically includes: When the scene type is a dialogue scene, the voice location tracking mode is enabled, and the speaker's voice is rendered to the spatial location corresponding to the screen position. When the scene type is an action scene, pause human voice location tracking and resume panoramic sound mode; When the scene type is a music scene, the voice location tracking mode is turned on to locate the singer's voice to the position in the screen, while the background music is played in immersive sound. When the scene type is a narration scene, maintain the center channel playback mode and keep the narrator's voice in the center of the sound field.
[0006] Furthermore, when a scene type switch is detected, a cross-fade-in / fade-out method is used to smoothly switch the sound field processing strategy within a preset transition time. Within the transition time window, the gain coefficient of each speaker channel in the current sound field processing mode is gradually reduced, while the gain coefficient of each speaker channel in the target sound field processing mode is gradually increased. The sum of the gain weights of the two modes is always 1.
[0007] Furthermore, the normalized two-dimensional continuous coordinates include x-coordinates and y-coordinates. The x-coordinate ranges from [0,1], where 0 represents the leftmost side of the screen and 1 represents the rightmost side, used to control the balanced distribution of the left and right audio channels. The y-coordinate ranges from [0,1], where 0 represents the bottom of the screen and 1 represents the top of the screen, used to control the perception of the distance and height of the sound field. When the value of the y-coordinate is greater than a preset threshold, reverberation and distance attenuation are added in the sound field rendering to simulate the effect of sound coming from a distance. When the value of the y-coordinate is less than the preset threshold, reverberation and distance attenuation are reduced to simulate the effect of sound coming from a nearby location.
[0008] Furthermore, the normalized two-dimensional continuous coordinates are converted into three-dimensional sound field coordinates of the vehicle speaker array through a preset mapping function, including: based on the vector basis amplitude phase algorithm, the x-coordinate is mapped to the gain distribution ratio of the left and right channel speakers, and the y-coordinate is mapped to the distance attenuation coefficient and early reflection parameters in the sound field rendering.
[0009] Furthermore, lip movement detection is performed on the lip region of each character to obtain a speech state marker for each character. Specifically, this includes: extracting the lip region image sequence of each character in the image, and inputting N consecutive frames of lip region images into a temporal convolutional neural network for lip movement analysis; the temporal convolutional neural network adopts a structure combining two-dimensional convolutional layers and one-dimensional temporal convolutional layers. The spatial features of the lip region of each frame are extracted frame by frame through the two-dimensional convolutional layer, and the spatial feature sequence of each frame is input into the one-dimensional temporal convolutional layer to capture the dynamic pattern of lip opening and closing changes along the time dimension. The lip movement binary classification result and confidence score are output through a fully connected layer and a Softmax classifier; when the lip movement classification result is that the character is speaking and the confidence score exceeds a preset threshold, and the temporal pattern of lip movement changes matches the characteristics of human speech, it is determined that the character is speaking.
[0010] Furthermore, based on the speaking state marker and the voiceprint features of the voice components, the target speaker is determined from multiple individuals. Specifically, this includes: when multiple speakers are present in the frame, a frequency-domain blind source separation method based on independent component analysis is used to convert the mixed audio signal to the time-frequency domain for demixing and separation, reconstructing the time-domain signal of each voice component; extracting the voiceprint feature vector of each voice component, and calculating the cosine similarity between the extracted voiceprint feature vector and the established voiceprint model of the individual. The cosine similarity between two n-dimensional voiceprint feature vectors A and B is defined as... When the cosine similarity exceeds a preset threshold, it is determined to be the same speaker; the temporal features of lip movement detection are aligned and verified with the start time of the human voice segment in the audio, and the person in the picture corresponding to the current human voice is confirmed as the target speaker based on the alignment and verification result.
[0011] Furthermore, after obtaining the normalized two-dimensional continuous coordinates corresponding to the target speaker, the method further includes: maintaining a historical buffer of the target speaker's position coordinates in the most recent multiple frames; smoothing the coordinate sequence in the historical buffer using Kalman filtering or exponential moving average filtering to generate smoothed continuous position coordinates; when the target speaker is briefly occluded, causing the current frame position detection to fail, performing position prediction based on a motion model of the historical trajectory to fill in the position information of the missing frame; if the occlusion duration exceeds a preset threshold, pausing position prediction, gradually transitioning the voice to the center channel, and resuming directional tracking after the speaker reappears.
[0012] A second aspect of the present invention provides an in-vehicle video voice orientation adaptive adjustment system, comprising: The scene perception module is used to acquire the currently playing video content, classify the video content into scenes, obtain the scene type to which the currently playing content belongs, and determine the corresponding sound field processing mode based on the scene type. The image recognition module is used to perform image recognition on the video frame of the video content when the sound field processing mode is human voice orientation tracking, to obtain the position coordinates of each person in the frame, and to perform lip movement detection on the lip area of each person in the frame to obtain the speaking state marker of each person. The speaker recognition module is used to acquire the currently playing audio signal, separate the human voice component from the audio signal, determine the target speaker from multiple people based on the speaking state marker and the voiceprint features of the human voice component, and obtain the normalized two-dimensional continuous coordinates corresponding to the target speaker. The sound effect rendering module is used to convert the normalized two-dimensional continuous coordinates of the target speaker into three-dimensional sound field coordinates of the vehicle speaker array through a preset mapping function, dynamically calculate the audio gain and delay parameters of each speaker channel based on the three-dimensional sound field coordinates, render the human voice component to the corresponding spatial orientation using the audio gain and delay parameters, and determine the playback mode of the background sound component according to the scene type.
[0013] A third aspect of the present invention provides a vehicle including the aforementioned in-vehicle video voice orientation adaptive adjustment system.
[0014] A fourth aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the vehicle-mounted video voice orientation adaptive adjustment method as described in the first aspect of the present invention.
[0015] A fifth aspect of the present invention provides a computer program product including software code, wherein the program in the software code executes the vehicle-mounted video voice orientation adaptive adjustment method as described in the first aspect of the present invention.
[0016] Compared with existing technologies, the present invention provides a vehicle-mounted video voice location adaptive adjustment method, system, and vehicle. By classifying video content into scenes and adaptively determining the sound field processing mode according to scene type, when voice location tracking is required, the system performs image recognition on the video screen to obtain the normalized two-dimensional continuous coordinates of each person and simultaneously performs lip movement detection. Then, the voice component is separated from the audio signal and the target speaker is determined by combining voiceprint features. The screen coordinates of the target speaker are then converted into three-dimensional sound field coordinates of the vehicle speaker array, and the gain and delay parameters of each speaker channel are dynamically calculated. The voice component is rendered to the corresponding spatial location, and the playback mode of the background sound component is determined according to the scene type. This achieves real-time and accurate synchronization between the voice location and the position of the video person. Furthermore, the system adaptively switches the sound field processing strategy according to the scene type of the playback content, effectively solving the problems of sound-image separation and insufficient scene adaptation, and significantly improving the audiovisual immersion of the driver and passengers. Attached Figure Description
[0017] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0018] Figure 1 This is a flowchart of the vehicle-mounted video voice orientation adaptive adjustment method provided in Embodiment 1 of the present invention.
[0019] Figure 2 This is an architecture diagram of the vehicle-mounted video voice orientation adaptive adjustment system provided in Embodiment 2 of the present invention. Detailed Implementation
[0020] It should be noted that the following detailed descriptions are exemplary and are intended only to describe specific embodiments and to provide further explanation of the invention, and are not intended to limit the scope of exemplary embodiments of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0021] Terminology Explanation 1. VBAP (Vector Base Amplitude Panning): A sound field rendering algorithm for multi-channel speaker arrays. Based on the principle of vector synthesis, it represents the desired direction vector of the target sound source as a linear combination of the direction vectors of adjacent speakers, and obtains the gain coefficient of each speaker by solving a system of linear equations.
[0022] 2. ICA (Independent Component Analysis): A blind source separation method based on the statistical independence of signals, used to separate mutually independent source signal components from mixed signals.
[0023] 3. STFT (Short-Time Fourier Transform): A time-frequency analysis method that performs Fourier transform on each frame of a time-domain signal after windowing to obtain the signal's distribution representation in the time-frequency domain.
[0024] 4. MFCC (Mel-Frequency Cepstral Coefficients): A type of acoustic feature parameter for speech, which is a feature vector extracted by performing spectral analysis on a speech signal by simulating the characteristics of human hearing. It is widely used in the fields of speaker recognition and speech recognition.
[0025] Example 1 like Figure 1 This embodiment provides a method for adaptive adjustment of the location of human voices in in-vehicle video, applied to an in-vehicle multimedia system. The in-vehicle multimedia system includes an in-vehicle display screen, a host audio source, an audio amplifier, and a multi-speaker array distributed throughout the vehicle. The method specifically includes the following steps: S101. Obtain the currently playing video content, classify the video content into scenes, obtain the scene type to which the currently playing content belongs, and determine the corresponding sound field processing mode according to the scene type.
[0026] Specifically, the scene perception module exists as software within the host machine and is automatically activated when video playback begins, performing real-time scene classification of the video content. This scene perception module is based on a deep learning model, using video frame sequences and audio spectral features (such as Mel spectrograms) as joint input. It extracts visual and audio feature vectors through visual and audio feature extraction networks, respectively, and then concatenates them before inputting them into a scene classifier. The classifier outputs the scene type label of the currently playing content and its corresponding confidence score. The scene types include the following four: (1) Dialogue Scene: There are one or more characters in the scene engaging in dialogue. The detection criteria include close-ups of the characters' faces, frequent lip movements, a high proportion of human voices in the audio, and alternating dialogue characteristics. In this scene, the system activates the full human voice location tracking mode to accurately render the speaker's voice to the spatial location corresponding to their position in the scene.
[0027] (2) Action / Explosion Scene: The scene contains a large number of visual elements such as fast movement, explosion, and collision. The detection criteria include high intensity of motion vector in the scene, high proportion of explosion and impact sounds in the audio, and discontinuous or missing human voices. In this scene, the system pauses human voice location tracking and restores to the preset panoramic sound mode, making full use of the vehicle's multi-speaker array to create an immersive environmental sound effect.
[0028] (3) Music / MV Scene: The scene may contain a singer singing or an instrument playing. The detection criteria include the continuous and high proportion of music in the audio and the presence of human voice in the form of singing. In this scene, the system enables human voice location tracking to locate the singer's voice to its position in the scene, while maintaining the panoramic sound playback of the background music to achieve a balance between human voice location and music atmosphere.
[0029] (4) Narration Scene: There is no clear speaker in the picture or it is a landscape / empty scene, and there is narration or voice-over in the audio. In this scene, the system maintains the center channel playback mode and keeps the narrator's voice in the center of the sound field to avoid abnormal deviation of the voice position due to incorrect character detection.
[0030] The step of determining the corresponding sound field processing mode based on the scene type specifically includes: when the scene type is a dialogue scene, enabling the voice location tracking mode to render the speaker's voice to the spatial location corresponding to their screen position; when the scene type is an action / explosion scene, pausing the voice location tracking and resuming the panoramic sound mode; when the scene type is a music / MV scene, enabling the voice location tracking mode to locate the singer's voice to their screen position while maintaining panoramic sound playback of the background music; when the scene type is a narration scene, maintaining the center channel playback mode to keep the narrator's voice in the center of the sound field.
[0031] The scene perception module continuously monitors changes in video content. When a scene type change is detected (such as switching from a dialogue scene to an action scene), the system uses a cross-fade-in / fade-out method to smoothly switch the sound field processing strategy within a preset transition time (such as 0.5 to 1 second): within the transition time window, the gain coefficient of each speaker channel in the current sound field processing mode is gradually reduced, while the gain coefficient of each speaker channel in the target sound field processing mode is gradually increased. The sum of the gain weights of the two modes is always 1, thereby achieving a natural transition without abruptness in hearing.
[0032] This step analyzes the video content in real time through a scene classification module to determine the scene type of the currently playing content and then determines the appropriate sound field processing mode based on the scene type. By subdividing scene types into four categories—dialogue, action / explosion, music / MV, and narration—and pre-setting differentiated sound field processing strategies for each category, this step allows the sound field processing mode to match the semantic context of the video content, compared to the uniform processing approach used in existing technologies. In dialogue scenes, the location of human voices is accurately tracked to enhance immersion; in action scenes, tracking is paused and panoramic sound is enabled to highlight the impact of the environment; in music scenes, human voices follow the visuals while the background music remains surround sound to balance positioning accuracy and musical immersion; and in narration scenes, the center channel is fixed to avoid meaningless directional shifts that interfere with auditory focus, thus providing the most suitable auditory experience in different viewing scenarios.
[0033] S102. When the sound field processing mode is human voice location tracking, image recognition is performed on the video frame of the video content to obtain the position coordinates of each person in the frame, and lip movement detection is performed on the lip area of each person in the frame to obtain the speaking state marker of each person.
[0034] Specifically, the video information processing unit exists in the host in the form of software. Based on the current screen display status, it obtains the audio playback status information of the whole vehicle: (1) whether music is currently playing, and whether the current playback is video or audio; (2) if it is video, and the scene perception module determines that the current scene needs to start human voice location tracking, the image recognition module is activated synchronously, and the corresponding analysis software starts working.
[0035] The image recognition module analyzes the video frame by frame, using a face detection algorithm to detect the facial region of each person in the frame, and outputs the normalized two-dimensional continuous coordinates (x, y) of the center point of each person's face in the frame. The x-coordinate ranges from [0, 1], where 0 represents the far left of the frame, 1 represents the far right, and 0.5 represents the horizontal center of the frame. The x-coordinate is used to control the balanced distribution of the left and right audio channels. The y-coordinate ranges from [0, 1], where 0 represents the bottom of the frame and 1 represents the top. The y-coordinate is used to control the perception of distance / height in the sound field: when the y-value is large (the person is near the top of the frame), appropriate reverberation and distance attenuation are added in the sound field rendering to simulate the auditory effect of sound coming from a distance; when the y-value is small (the person is near the bottom of the frame / close-up), reverberation and attenuation are reduced to simulate the auditory effect of sound coming from close up. Simultaneously, the image recognition module also analyzes auxiliary features such as the number of people, their gender, and their age.
[0036] The lip movement detection module works in parallel with the image recognition module. For each person detected in the scene, it extracts a sequence of lip region images. Taking N consecutive frames (e.g., N=16 frames) of lip region images as input, it performs lip movement analysis through a temporal convolutional neural network. The temporal convolutional neural network employs a structure combining two-dimensional and one-dimensional convolutional layers: first, the spatial features of the lip region in each frame are extracted frame-by-frame using 2D convolutional layers; then, the spatial feature sequences of each frame are input into a 1D temporal convolutional layer to capture the dynamic pattern of lip opening and closing along the time dimension; finally, a fully connected layer and a Softmax classifier output a binary classification result of lip movement (speaking / not speaking) and the corresponding confidence score. When a person's lip movement classification result is "speaking" and the confidence score exceeds a preset threshold, and the temporal pattern of lip movement changes matches the characteristics of human speech (e.g., lip opening and closing frequency within the range of 2 to 8 Hz), then the person is determined to be speaking. The lip movement detection module outputs a speaking status marker (speaking / not speaking) for each person in each frame.
[0037] This step employs an architecture where the image recognition module and the lip movement detection module work in parallel, enabling the simultaneous capture of the positional information and speaking state of all individuals in the video frame. The introduction of normalized two-dimensional continuous coordinates (x, y) is a key design feature that distinguishes this solution from existing ones. Existing technologies can only recognize coarse spatial semantics such as "someone is on the left side of the frame" or "someone is on the right side of the frame," failing to distinguish the specific degree to which a person is off-left (e.g., 10% or 80%), nor can they perceive changes in a person's position in the depth direction of the frame (e.g., a person moving from foreground to background). This results in sound image localization only achieving abrupt transitions between "left-center-right," with sound image movement exhibiting a stepped, abrupt change rather than a smooth, continuous transition. This step outputs continuous x-coordinates and y-coordinates ranging from 0 to 1. Continuous x-coordinate values allow for smooth movement of the sound image between the left and right channels, while continuous y-coordinate values allow for natural transitions between near and far depths. Combined with subsequent steps where the x-coordinate is mapped to speaker gain ratio using the VBAP algorithm, and the y-coordinate is mapped to distance attenuation coefficients and early reflection parameters, this achieves a fundamental improvement in sound image movement, transforming abrupt transitions into smooth, continuous motion. This step solves the technical problem in existing technologies where only a rough area of a person in the image (e.g., left, center, right) can be identified, and continuous position coordinates cannot be output, leading to abrupt changes in sound image positioning. By normalizing two-dimensional continuous coordinates, a precise description of the person's position is achieved, providing a data foundation for smooth sound field rendering. Simultaneously, lip movement detection determines the current speaker's range, providing visual evidence for subsequent speaker recognition.
[0038] S103. Obtain the currently playing audio signal, separate the human voice component from the audio signal, determine the target speaker from multiple people based on the speaking state marker and the voiceprint characteristics of the human voice component, and obtain the normalized two-dimensional continuous coordinates corresponding to the target speaker.
[0039] Specifically, when the scene perception module confirms the need for voice location tracking and the lip movement detection module detects the presence of a speaker in the scene, the voice processing unit activates the audio processing unit to extract the corresponding voice from the audio and analyze whether there is a voice (including singing, dialogue, etc.) in the current audio.
[0040] When multiple speakers are present in the video feed, the speaker recognition module initiates its operation: First, the audio processing unit employs a frequency-domain blind source separation method based on Independent Component Analysis (ICA). Utilizing the statistical independence of human voice components and background sound components in the time-frequency domain, the mixed audio signal is converted to the time-frequency domain via Short-Time Fourier Transform (STFT). In the time-frequency domain, the separation matrix is estimated using the ICA algorithm, and the mixed signal is demixed. Then, the time-domain signal of each human voice component is reconstructed using Inverse Short-Time Fourier Transform (ISTFT). Next, the voiceprint feature vector for each human voice component is extracted. This voiceprint feature vector includes acoustic features such as Mel-frequency cepstral coefficients (MFCC), fundamental frequency F0, and formants. The extracted voiceprint feature vector is then compared with the established human voiceprint model in the system using cosine similarity calculation. The cosine similarity between two n-dimensional voiceprint feature vectors A and B is defined as: ,in, Representing vectors with vector The dot product, where |A| and |B| represent the vectors respectively. sum vector The similarity value ranges from -1 to 1, with values closer to 1 indicating greater similarity between the two voiceprint features. When the cosine similarity exceeds a preset threshold (e.g., 0.75), the voiceprint is determined to be from the same speaker. The voiceprint model is established and continuously updated each time a speaker is detected speaking for the first time. Finally, combining the speaking state markers and lip movement timing features output by the lip movement detection module, the lip movement time sequence is aligned and verified with the start time of the voice segment to confirm the current voice-corresponding person in the video, which is then identified as the target speaker.
[0041] When multiple speakers are present in the scene, this step triggers the speaker recognition module to start running: First, the temporal signals of each voice component are reconstructed through blind source separation; then, voiceprint features are extracted and matched with the voiceprint model using cosine similarity; finally, alignment verification is completed by combining lip movement timing. In existing technologies, when multiple people are in the same frame, relying solely on face detection cannot determine the actual speaker. Multiple people in the scene may have visible faces, but only some may actually be speaking. If the current speaker is distinguished only by the position of the person, the voice will point to the wrong person in the scene, causing a sound-image misalignment problem. This step effectively solves this problem through a triple verification mechanism of lip movement detection, voiceprint recognition, and timing alignment: Lip movement detection filters out the candidate set of people currently speaking from the visual dimension, narrowing the matching range; voiceprint recognition extracts the identity features of the voice from the audio dimension, accurately confirming the speaker's identity in the candidate set; timing alignment verifies the consistency between the start time of lip movement and the start time of the voice from the audio-video synchronization dimension, further eliminating false matches. The three verification steps are independent and complementary. If the confidence level of the output result in any one step is insufficient, it can be corrected by the other steps, which greatly improves the robustness and accuracy of speaker recognition in complex multi-person scenarios. This step solves the core technical problems of not being able to determine the current speaker and the inability to accurately match the location of the voice with the speaker in multi-person scenarios. The accuracy of speaker recognition is ensured by the triple verification mechanism of lip movement detection, voiceprint recognition and temporal alignment.
[0042] S104. The normalized two-dimensional continuous coordinates of the target speaker are converted into three-dimensional sound field coordinates of the vehicle speaker array through a preset mapping function. The audio gain and delay parameters of each speaker channel are dynamically calculated based on the three-dimensional sound field coordinates. The human voice component is rendered to the corresponding spatial orientation using the audio gain and delay parameters. The playback method of the background sound component is determined according to the scene type.
[0043] Specifically, the sound effect algorithm processing unit receives the normalized two-dimensional continuous coordinates (x, y) of the speaker output by the video information processing unit, and converts the screen coordinates into three-dimensional sound field coordinates of the vehicle speaker array through a preset mapping function. The mapping function considers the following factors: the actual layout parameters of the vehicle speaker array (the three-dimensional spatial position of each speaker), the position of the listening area in the vehicle (such as the center position of the driver's seat headrest), and the coordinate system definition of the sound field rendering algorithm.
[0044] The specific mapping process is as follows: the x-coordinate is mapped to the gain distribution ratio of the left and right channel loudspeakers using the Vector Basis Amplitude Phase (VBAP) algorithm. The VBAP algorithm, based on the principle of vector synthesis, represents the desired direction vector of the target sound source as a linear combination of the direction vectors of two (two-dimensional sound field) or three (three-dimensional sound field) adjacent loudspeakers. Let the target direction vector be p, and the basis vector matrix formed by the direction vectors of the adjacent loudspeakers be... Then the gain coefficient of each loudspeaker By solving the system of linear equations To obtain, that is .
[0045] The gain coefficient, after normalization, is used to control the output amplitude of each speaker. The y-coordinate is mapped to the distance attenuation coefficient and early reflection parameters in the sound field rendering, enabling differentiated rendering of near-field sound (small y-value) and far-field sound (large y-value).
[0046] The audio processing unit separates the human voice and background sound components from the raw audio stream. Based on the three-dimensional sound field coordinates, it dynamically calculates the audio gain and delay parameters of each speaker channel and renders the human voice component to its corresponding spatial orientation. Simultaneously, based on the scene type determined by the scene perception module, it decides how the background sound component is played. In dialogue scenarios, the background sound maintains its original panoramic sound field; in music / MV scenarios, the background music maintains panoramic sound playback, while only the human voice component undergoes orientation tracking.
[0047] Furthermore, this embodiment also includes a trajectory smoothing and real-time update mechanism: the trajectory smoothing and motion prediction unit maintains a historical buffer of the speaker's position coordinates within the most recent N frames (e.g., N=30, corresponding to approximately 1 second of video duration). The coordinate sequence within the buffer is smoothed using Kalman filtering or exponential moving average filtering to eliminate inter-frame position jitter caused by detection errors. When the speaker moves in the frame, the smoothed coordinate sequence ensures a smooth and continuous transition between sound and image, avoiding abrupt jumps. When the speaker is briefly obscured (e.g., by other people, objects, or subtitles), causing the current frame's position detection to fail, position prediction is performed based on a motion model (e.g., a uniform motion model or a uniformly accelerated motion model) based on the historical trajectory, filling in the position information of the missing frames and maintaining the continuity of sound and image localization. If the obscuration duration exceeds a preset threshold (e.g., 2 seconds), position prediction is paused, the voice is gradually transitioned to the center channel, and directional tracking resumes once the speaker reappears.
[0048] The sound effect algorithm processing unit sends the real-time scheme signal to the vehicle speaker system. The vehicle speaker system, having been calibrated and pre-set with corresponding sound effect playback schemes, selects the appropriate audio effect for playback upon receiving the signal from the logic processing unit. The latency in this processing is minimal and adjustments are made in real time; for example, if a character moves laterally, the sound effect processing changes synchronously.
[0049] In this step, the audio effect algorithm processing unit receives the speaker's normalized two-dimensional continuous coordinates output by the video information processing unit and converts the screen coordinates into three-dimensional sound field coordinates of the vehicle speaker array using a preset mapping function. The x-coordinate is mapped to a gain allocation ratio using the VBAP algorithm. This algorithm utilizes the principle of approximating the target direction by linearly combining the direction vectors of adjacent speakers. Compared to simple intensity ratio allocation, it can more accurately match the geometric layout of the speaker array, ensuring the directional accuracy and spatial stability of the target sound source localization. The y-coordinate is mapped to the distance attenuation coefficient and early reflection sound parameters. Specifically, when the person is in the distance (large y-value), the voice is perceived as coming from a greater distance, and the natural attenuation effect of sound waves propagating in space is simulated by increasing reverberation and attenuation. When the person is in the foreground (small y-value), the voice is clear and direct, creating a sense of presence for face-to-face dialogue. Through the joint mapping of x and y coordinates, precise control of sound field localization in both the horizontal and vertical directions is achieved. The audio processing unit separates the human voice and background sound components from the raw audio stream. Based on the three-dimensional sound field coordinates, it dynamically calculates the audio gain and delay parameters of each speaker channel, rendering the human voice component to its corresponding spatial location. Simultaneously, it determines the playback method of the background sound component based on the scene type. This step solves the technical problem of the disconnect between the human voice location and the position of the person in the video. By mapping continuous coordinates to speaker gain allocation, it achieves precise synchronization between the human voice location and the position of the person in the video. Trajectory smoothing and motion prediction mechanisms further ensure the continuity of sound image localization when the person moves quickly or is briefly obscured: when the person moves quickly horizontally in the frame, Kalman filtering can predict the position of the next frame based on historical motion trends, keeping the sound image movement speed synchronized with the image movement speed, avoiding sound image lag or jitter caused by image recognition delays or errors; when the person is briefly obscured by other objects, causing face detection failure, motion prediction fills in the missing frame's position information, preventing the sound image from suddenly jumping back to the center channel or becoming silent at the moment of obstruction, ensuring the stability and continuity of the auditory experience.
[0050] In a preferred embodiment, the scene perception module, image recognition module, lip movement detection module, and speaker recognition module all run on the vehicle's main processor, employing lightweight deep learning models (such as MobileNet or EfficientNet series) to ensure real-time processing within the vehicle's computing power. The inference latency of each module is controlled within 50 milliseconds, and the end-to-end latency of the entire system from video frame input to sound field parameter update is controlled within 150 milliseconds, ensuring audio-visual synchronization remains within the human perception threshold.
[0051] Example 2 like Figure 2This embodiment provides an in-vehicle video and voice orientation adaptive adjustment system. The system is applied to an in-vehicle multimedia system, which includes an in-vehicle display screen, a host audio source, an audio amplifier, and a multi-speaker array distributed throughout the vehicle. The in-vehicle equipment includes a front-row display screen, a host audio source, and an audio processing amplifier.
[0052] The system includes the following modules: The 201 scene perception module is used to acquire the currently playing video content, classify the video content into scenes, obtain the scene type to which the currently playing content belongs, and determine the corresponding sound field processing mode based on the scene type.
[0053] The scene perception module exists as software within the host machine and is automatically activated when video playback begins, performing real-time scene classification of the video content. Based on a deep learning model, the module uses video frame sequences and audio spectral features (such as Mel spectrograms) as joint input. It extracts visual and audio feature vectors through visual and audio feature extraction networks, respectively, concatenates them, and inputs the result into a scene classifier. The classifier outputs the scene type label of the currently playing content and its corresponding confidence score. Scene types include the following four: (1) Dialogue Scene: There are one or more characters in the scene engaging in dialogue. The detection criteria include close-ups of the characters' faces, frequent lip movements, a high proportion of human voices in the audio, and alternating dialogue characteristics. In this scene, the system activates the full human voice location tracking mode to accurately render the speaker's voice to the spatial location corresponding to their position in the scene.
[0054] (2) Action / Explosion Scene: The scene contains a large number of visual elements such as fast movement, explosion, and collision. The detection criteria include high intensity of motion vector in the scene, high proportion of explosion and impact sounds in the audio, and discontinuous or missing human voices. In this scene, the system pauses human voice location tracking and restores to the preset panoramic sound mode, making full use of the vehicle's multi-speaker array to create an immersive environmental sound effect.
[0055] (3) Music / MV Scene: The scene may contain a singer singing or an instrument playing. The detection criteria include the continuous and high proportion of music in the audio and the presence of human voice in the form of singing. In this scene, the system enables human voice location tracking to locate the singer's voice to its position in the scene, while maintaining the panoramic sound playback of the background music to achieve a balance between human voice location and music atmosphere.
[0056] (4) Narration Scene: There is no clear speaker in the picture or it is a landscape / empty scene, and there is narration or voice-over in the audio. In this scene, the system maintains the center channel playback mode and keeps the narrator's voice in the center of the sound field to avoid abnormal deviation of the voice position due to incorrect character detection.
[0057] The scene perception module determines the corresponding sound field processing mode according to the scene type, specifically including: when the scene type is a dialogue scene, the voice location tracking mode is activated, and the speaker's voice is rendered to the spatial location corresponding to its screen position; when the scene type is an action / explosion scene, the voice location tracking is paused and the panoramic sound mode is restored; when the scene type is a music / MV scene, the voice location tracking mode is activated, and the singer's voice is located to its screen position, while maintaining the panoramic sound playback of the background music; when the scene type is a narration scene, the center channel playback mode is maintained, and the narrator's voice is kept in the center of the sound field.
[0058] The scene perception module continuously monitors changes in video content. When a scene type change is detected (such as switching from a dialogue scene to an action scene), the system uses a cross-fade-in / fade-out method to smoothly switch the sound field processing strategy within a preset transition time (such as 0.5 to 1 second): within the transition time window, the gain coefficient of each speaker channel in the current sound field processing mode is gradually reduced, while the gain coefficient of each speaker channel in the target sound field processing mode is gradually increased. The sum of the gain weights of the two modes is always 1, thereby achieving a natural transition without abruptness in hearing.
[0059] The 202 image recognition module is used to perform image recognition on the video frame of the video content when the sound field processing mode is human voice location tracking, to obtain the position coordinates of each person in the frame, and to perform lip movement detection on the lip area of each person in the frame to obtain the speaking state marker of each person.
[0060] The image recognition module specifically includes an image recognition submodule and a lip movement detection submodule.
[0061] The image recognition submodule exists in the host as software. It obtains the audio playback status information of the whole vehicle based on the current screen display status: (1) whether music is currently playing, and whether the current playback is video or audio; (2) if it is video, and the scene perception module determines that the current scene needs to start human voice location tracking, the image recognition module is activated synchronously, and the corresponding analysis software starts working. The image recognition submodule performs frame-by-frame analysis of the video screen, uses a face detection algorithm to detect the facial area of each person in the screen, and outputs the normalized two-dimensional continuous coordinates (x, y) of the center point of each person's face in the screen. The x-coordinate ranges from [0, 1], where 0 represents the far left of the image, 1 represents the far right, and 0.5 represents the horizontal center. The x-coordinate controls the balanced distribution of the left and right audio channels. The y-coordinate ranges from [0, 1], where 0 represents the bottom of the image and 1 represents the top. The y-coordinate controls the perception of distance / height in the sound field. When the y-value is large (the person is near the top of the image), appropriate reverberation and distance attenuation are added to the sound field rendering to simulate the auditory effect of distant sound. When the y-value is small (the person is near the bottom of the image / close-up), reverberation and attenuation are reduced to simulate the auditory effect of close-up sound. Simultaneously, the image recognition module also analyzes auxiliary features such as the number of people, their gender, and their age.
[0062] The lip movement detection submodule works in parallel with the image recognition submodule. For each person detected in the scene, it extracts a sequence of lip region images. Taking N consecutive frames (e.g., N=16 frames) of lip region images as input, it performs lip movement analysis through a temporal convolutional neural network. The temporal convolutional neural network employs a structure combining two-dimensional and one-dimensional convolutional layers: first, the spatial features of the lip region in each frame are extracted frame-by-frame using 2D convolutional layers; then, the spatial feature sequences of each frame are input into a 1D temporal convolutional layer to capture the dynamic pattern of lip opening and closing along the time dimension; finally, a fully connected layer and a Softmax classifier output a binary classification result of lip movement (speaking / not speaking) and the corresponding confidence score. When a person's lip movement classification result is "speaking" and the confidence score exceeds a preset threshold, and the temporal pattern of lip movement changes matches the characteristics of human speech (e.g., lip opening and closing frequency within the range of 2 to 8 Hz), then the person is determined to be speaking. The lip movement detection module outputs a speaking status marker (speaking / not speaking) for each person in each frame.
[0063] The 203 speaker recognition module is used to acquire the currently playing audio signal, separate the human voice component from the audio signal, determine the target speaker from multiple people based on the speaking state marker and the voiceprint features of the human voice component, and obtain the normalized two-dimensional continuous coordinates corresponding to the target speaker.
[0064] Specifically, when the scene perception module confirms the need for voice location tracking and the lip movement detection module detects the presence of a speaker in the scene, the voice processing unit activates the audio processing unit to extract the corresponding voice from the audio and analyze whether there is a voice (including singing, dialogue, etc.) in the current audio.
[0065] When multiple speakers are present in the video feed, the speaker recognition module initiates its operation: First, the audio processing unit employs a frequency-domain blind source separation method based on Independent Component Analysis (ICA). Utilizing the statistical independence of human voice components and background sound components in the time-frequency domain, the mixed audio signal is converted to the time-frequency domain via Short-Time Fourier Transform (STFT). In the time-frequency domain, the separation matrix is estimated using the ICA algorithm, and the mixed signal is demixed. Then, the time-domain signal of each human voice component is reconstructed using Inverse Short-Time Fourier Transform (ISTFT). Next, the voiceprint feature vector for each human voice component is extracted. This voiceprint feature vector includes acoustic features such as Mel-frequency cepstral coefficients (MFCC), fundamental frequency F0, and formants. The extracted voiceprint feature vector is then compared with the established human voiceprint model in the system using cosine similarity calculation. The cosine similarity between two n-dimensional voiceprint feature vectors A and B is defined as: Where A·B represents the dot product of vectors A and B, and |A| and |B| represent the magnitudes of vectors A and B, respectively. The similarity value ranges from [-1, 1], with values closer to 1 indicating greater similarity between the two voiceprint features. When the cosine similarity exceeds a preset threshold, such as 0.75, the voiceprint is determined to be from the same speaker. The voiceprint model is established and continuously updated each time a speaker is first detected speaking. Finally, combining the speaking state markers and lip movement temporal features output by the lip movement detection module, the lip movement time sequence is aligned and verified with the start time of the voice segment to confirm the current voice-corresponding person in the video, which is then identified as the target speaker.
[0066] The 204 sound effect rendering module is used to convert the normalized two-dimensional continuous coordinates of the target speaker into three-dimensional sound field coordinates of the vehicle speaker array through a preset mapping function, dynamically calculate the audio gain and delay parameters of each speaker channel based on the three-dimensional sound field coordinates, render the human voice component to the corresponding spatial orientation using the audio gain and delay parameters, and determine the playback mode of the background sound component according to the scene type.
[0067] Specifically, the sound effect algorithm processing unit receives the normalized two-dimensional continuous coordinates (x, y) of the speaker output by the video information processing unit, and converts the screen coordinates into three-dimensional sound field coordinates of the vehicle speaker array through a preset mapping function. The mapping function considers the following factors: the actual layout parameters of the vehicle speaker array (the three-dimensional spatial position of each speaker), the position of the listening area in the vehicle (such as the center position of the driver's seat headrest), and the coordinate system definition of the sound field rendering algorithm.
[0068] The specific mapping process is as follows: the x-coordinate is mapped to the gain distribution ratio of the left and right channel loudspeakers using the Vector Basis Amplitude Phase (VBAP) algorithm. The VBAP algorithm, based on the principle of vector synthesis, represents the desired direction vector of the target sound source as a linear combination of the direction vectors of two (two-dimensional sound field) or three (three-dimensional sound field) adjacent loudspeakers. Let the target direction vector be p, and the basis vector matrix formed by the direction vectors of the adjacent loudspeakers be... Then the gain coefficient of each loudspeaker By solving the system of linear equations To obtain, that is .
[0069] The gain coefficient, after normalization, is used to control the output amplitude of each speaker. The y-coordinate is mapped to the distance attenuation coefficient and early reflection parameters in the sound field rendering, enabling differentiated rendering of near-field sound (small y-value) and far-field sound (large y-value).
[0070] The audio processing unit separates the human voice and background sound components from the raw audio stream. Based on the three-dimensional sound field coordinates, it dynamically calculates the audio gain and delay parameters of each speaker channel and renders the human voice component to its corresponding spatial orientation. Simultaneously, based on the scene type determined by the scene perception module, it decides how the background sound component is played: in dialogue scenes, the background sound maintains its original panoramic sound field; in music / MV scenes, the background music maintains panoramic sound playback, while only the human voice component undergoes orientation tracking.
[0071] The sound effect algorithm processing unit sends the real-time scheme signal to the vehicle speaker system. The vehicle speaker system has been calibrated and preset with the corresponding sound effect playback scheme. After receiving the signal provided by the logic processing unit, it selects the corresponding audio sound effect for playback.
[0072] The system in this embodiment also includes a trajectory smoothing and motion prediction unit: This unit maintains a historical buffer of the speaker's position coordinates within the most recent N frames (e.g., N=30, corresponding to approximately 1 second of video duration). The coordinate sequence within the buffer is smoothed using Kalman filtering or exponential moving average filtering to eliminate inter-frame position jitter caused by detection errors. When the speaker moves in the frame, the smoothed coordinate sequence ensures a smooth and continuous transition between sound and image, avoiding abrupt jumps. When the speaker is briefly obscured (e.g., by other people, objects, or subtitles), causing position detection to fail in the current frame, position prediction is performed based on a motion model (e.g., a uniform motion model or a uniformly accelerated motion model) based on the historical trajectory, filling in the position information of the missing frames and maintaining the continuity of sound and image localization. If the obscuration duration exceeds a preset threshold (e.g., 2 seconds), position prediction is paused, and the voice is gradually transitioned to the center channel until the speaker reappears, at which point position tracking resumes. The trajectory smoothing and motion prediction unit is signal-connected to the image recognition module and the sound effect rendering module.
[0073] In a preferred embodiment, the scene perception module, image recognition module, lip movement detection module, and speaker recognition module all run on the vehicle's main processor, employing lightweight deep learning models (such as MobileNet or EfficientNet series) to ensure real-time processing within the vehicle's computing power. The inference latency of each module is controlled within 50 milliseconds, and the end-to-end latency of the entire system from video frame input to sound field parameter update is controlled within 150 milliseconds, ensuring audio-visual synchronization remains within the human perception threshold.
[0074] In this embodiment, the signal connection relationships of each module are as follows: the scene perception module is signal-connected to the image recognition module, and is used to send control commands to the image recognition module regarding the scene type and whether to enable voice location tracking; the image recognition module is signal-connected to both the speaker recognition module and the sound effect rendering module, and is used to send the position coordinates and speaking status markers of each person to the speaker recognition module, and the position coordinates of the target speaker to the sound effect rendering module; the speaker recognition module is signal-connected to the sound effect rendering module, and is used to send the confirmed target speaker and its position coordinates to the sound effect rendering module; the scene perception module is also signal-connected to the sound effect rendering module, and is used to send the scene type to the sound effect rendering module to control the background sound playback mode; the trajectory smoothing and motion prediction unit is signal-connected to both the image recognition module and the sound effect rendering module, and is used to receive the position coordinate sequence and output the smoothed position coordinates to the sound effect rendering module.
[0075] Example 3 This embodiment provides a vehicle, which includes an in-vehicle speaker array disposed within the vehicle body; a display screen disposed within the vehicle body; and an in-vehicle video voice orientation adaptive adjustment system as described in Embodiment 2.
[0076] The vehicle's speaker array is calibrated and pre-programmed with corresponding sound effect playback schemes. Upon receiving a signal from the sound effect rendering module, it selects the appropriate audio effect for playback. When playing video content, the vehicle can dynamically adjust the spatial orientation of the speaker's voice based on the real-time position of the speaker in the video frame, and adaptively switch sound field processing strategies according to the scene type of the content being played. This achieves precise synchronization between the speaker's voice location and the position of the person in the video, significantly enhancing the immersive audiovisual experience for passengers.
[0077] Example 4 Embodiment 4 of the present invention provides a computer-readable storage medium.
[0078] A computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the vehicle-mounted video voice orientation adaptive adjustment method as described in Embodiment 1 of the present invention.
[0079] The detailed steps are the same as the vehicle-mounted video voice orientation adaptive adjustment method provided in Example 1, and will not be repeated here.
[0080] Example 5 Embodiment 5 of the present invention provides a computer program product.
[0081] A computer program product includes software code, wherein the program in the software code executes the vehicle-mounted video voice orientation adaptive adjustment method as described in Embodiment 1 of the present invention.
[0082] The detailed steps are the same as the vehicle-mounted video voice orientation adaptive adjustment method provided in Example 1, and will not be repeated here.
[0083] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages. For example, in one implementation, the methods and systems can be developed based on deep learning frameworks (such as TensorFlow, PyTorch, etc.) and using the Python language. Those skilled in the art will understand that other suitable programming languages or tools can also be used for implementation without departing from the core ideas of the present invention.
[0084] The above description is only a preferred embodiment of the present invention. Although the specific implementation of the present invention has been described in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the present invention.
Claims
1. A method for adaptive adjustment of the location of human voice in vehicle-mounted video, characterized in that, include: The system acquires the currently playing video content, classifies the video content into scenes to obtain the scene type to which the currently playing content belongs, and determines the corresponding sound field processing mode based on the scene type. When the sound field processing mode is human voice location tracking, image recognition is performed on the video frame of the video content to obtain the position coordinates of each person in the frame, and lip movement detection is performed on the lip area of each person in the frame to obtain the speaking state marker of each person. The system acquires the currently playing audio signal, separates the human voice component from the audio signal, determines the target speaker from multiple speakers based on the speaking state marker and the voiceprint features of the human voice component, and obtains the normalized two-dimensional continuous coordinates corresponding to the target speaker. The normalized two-dimensional continuous coordinates of the target speaker are converted into three-dimensional sound field coordinates of the vehicle speaker array through a preset mapping function. The audio gain and delay parameters of each speaker channel are dynamically calculated based on the three-dimensional sound field coordinates. The human voice component is rendered to the corresponding spatial orientation using the audio gain and delay parameters. The playback method of the background sound component is determined according to the scene type.
2. The vehicle-mounted video voice location adaptive adjustment method according to claim 1, characterized in that, The scene types include dialogue scenes, action scenes, music scenes, and narration scenes; determining the corresponding sound field processing mode based on the scene type specifically includes: When the scene type is a dialogue scene, the voice location tracking mode is enabled, and the speaker's voice is rendered to the spatial location corresponding to the screen position. When the scene type is an action scene, pause human voice location tracking and resume panoramic sound mode; When the scene type is a music scene, the voice location tracking mode is turned on to locate the singer's voice to the position in the screen, while the background music is played in immersive sound. When the scene type is a narration scene, maintain the center channel playback mode and keep the narrator's voice in the center of the sound field.
3. The vehicle-mounted video voice location adaptive adjustment method according to claim 1, characterized in that, When a scene type switch is detected, the sound field processing strategy is smoothly switched using a cross-fade-in / fade-out method within a preset transition time. Within the transition time window, the gain coefficient of each speaker channel in the current sound field processing mode is gradually reduced, while the gain coefficient of each speaker channel in the target sound field processing mode is gradually increased. The sum of the gain weights of the two modes is always 1.
4. The vehicle-mounted video voice location adaptive adjustment method according to claim 1, characterized in that, The normalized two-dimensional continuous coordinates include x-coordinates and y-coordinates. The x-coordinate ranges from [0, 1], where 0 represents the leftmost side of the screen and 1 represents the rightmost side, used to control the balanced distribution of the left and right audio channels. The y-coordinate ranges from [0, 1], where 0 represents the bottom of the screen and 1 represents the top of the screen, used to control the perceived distance and height of the sound field. When the y-coordinate value is greater than a preset threshold, reverberation and distance attenuation are added in the sound field rendering to simulate the effect of sound coming from a distance. When the y-coordinate value is less than the preset threshold, reverberation and distance attenuation are reduced to simulate the effect of sound coming from a nearby location.
5. The vehicle-mounted video voice location adaptive adjustment method according to claim 2, characterized in that, The normalized two-dimensional continuous coordinates are converted into three-dimensional sound field coordinates of the vehicle speaker array through a preset mapping function, including: based on the vector basis amplitude phase algorithm, the x-coordinate is mapped to the gain distribution ratio of the left and right channel speakers, and the y-coordinate is mapped to the distance attenuation coefficient and early reflection parameters in the sound field rendering.
6. The vehicle-mounted video voice location adaptive adjustment method according to claim 1, characterized in that, Lip movement detection is performed on the lip region of each character to obtain a speech state marker for each character. Specifically, this includes: extracting the lip region image sequence of each character in the image, and inputting N consecutive frames of lip region images into a temporal convolutional neural network for lip movement analysis; the temporal convolutional neural network adopts a structure combining two-dimensional convolutional layers and one-dimensional temporal convolutional layers. The spatial features of the lip region of each frame are extracted frame by frame through the two-dimensional convolutional layer, and the spatial feature sequence of each frame is input into the one-dimensional temporal convolutional layer to capture the dynamic pattern of lip opening and closing changes along the time dimension. The lip movement binary classification result and confidence score are output through a fully connected layer and a Softmax classifier; when the lip movement classification result is that the character is speaking and the confidence score exceeds a preset threshold, and the temporal pattern of lip movement changes matches the characteristics of human speech, it is determined that the character is speaking.
7. The vehicle-mounted video voice location adaptive adjustment method according to claim 1, characterized in that, Based on the speaking state marker and the voiceprint features of the voice components, the target speaker is determined from multiple speakers. Specifically, this includes: when multiple speakers are present in the frame, a frequency-domain blind source separation method based on independent component analysis is used to convert the mixed audio signal to the time-frequency domain for demixing and separation, reconstructing the time-domain signal of each voice component; extracting the voiceprint feature vector of each voice component; and calculating the cosine similarity between the extracted voiceprint feature vector and the established voiceprint model of the speaker. The cosine similarity between two n-dimensional voiceprint feature vectors A and B is defined as... When the cosine similarity exceeds a preset threshold, it is determined to be the same speaker; the temporal features of lip movement detection are aligned and verified with the start time of the human voice segment in the audio, and the person in the picture corresponding to the current human voice is confirmed as the target speaker based on the alignment and verification result.
8. The vehicle-mounted video voice location adaptive adjustment method according to claim 1, characterized in that, After obtaining the normalized two-dimensional continuous coordinates corresponding to the target speaker, the method further includes: maintaining a historical buffer of the target speaker's position coordinates in the most recent multiple frames; smoothing the coordinate sequence in the historical buffer using Kalman filtering or exponential moving average filtering to generate smoothed continuous position coordinates; when the target speaker is briefly occluded, causing the position detection of the current frame to fail, performing position prediction based on the motion model of the historical trajectory to fill in the position information of the missing frame; if the occlusion duration exceeds a preset threshold, pausing position prediction, gradually transitioning the voice to the center channel, and resuming directional tracking after the speaker reappears.
9. A vehicle-mounted video voice orientation adaptive adjustment system, characterized in that, include: The scene perception module is used to acquire the currently playing video content, classify the video content into scenes, obtain the scene type to which the currently playing content belongs, and determine the corresponding sound field processing mode based on the scene type. The image recognition module is used to perform image recognition on the video frame of the video content when the sound field processing mode is human voice orientation tracking, to obtain the position coordinates of each person in the frame, and to perform lip movement detection on the lip area of each person in the frame to obtain the speaking state marker of each person. The speaker recognition module is used to acquire the currently playing audio signal, separate the human voice component from the audio signal, determine the target speaker from multiple people based on the speaking state marker and the voiceprint features of the human voice component, and obtain the normalized two-dimensional continuous coordinates corresponding to the target speaker. The sound effect rendering module is used to convert the normalized two-dimensional continuous coordinates of the target speaker into three-dimensional sound field coordinates of the vehicle speaker array through a preset mapping function, dynamically calculate the audio gain and delay parameters of each speaker channel based on the three-dimensional sound field coordinates, render the human voice component to the corresponding spatial orientation using the audio gain and delay parameters, and determine the playback mode of the background sound component according to the scene type.
10. A vehicle, characterized in that, Includes the vehicle-mounted video voice orientation adaptive adjustment system as described in claim 9.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the vehicle-mounted video voice orientation adaptive adjustment method according to any one of claims 1 to 8.
12. A computer program product, comprising software code, characterized in that, When the program in the software code is executed, it implements the vehicle-mounted video voice orientation adaptive adjustment method as described in any one of claims 1 to 8.
