Vehicle-mounted microphone-free KTV (Karaoke Television) partition pickup and sound mixing method for multiple passenger seat areas

Abstract

The invention relates to the technical field of voice data processing, in particular to a multi-passenger seat area-oriented vehicle-mounted microphone-free KTV partition pickup and sound mixing method, which comprises the following steps of: 1, controlling a trigger microphone array to enter a partition pickup mode, and simultaneously sending a partition pickup stream and an accompaniment signal to an audio channel manager according to an uplink channel; step 2, after the audio channel manager receives the subarea pickup stream and the accompaniment signal, executing a seat area multi-shadow separation and resonance gating process, aligning each seat area reconstruction result with a beat skeleton by accompaniment phase anchoring to obtain a seat area aligned dry sound stream, and performing a resonance gating process; and after space reverberation and human voice optimization are carried out on the seat area alignment trunk sound stream and the accompaniment signal of each seat area, a processed audio signal is obtained. According to the method, the problems of unclear partition, tone quality distortion, difficulty in synchronization and the like of the existing microphone-free KTV under the condition of multiple sound sources are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of voice data processing, and in particular relates to a method for picking up and mixing voices in a car without microphones in a multi-passenger seating area. Background Art

[0002] With the development of intelligent, connected vehicles, in-car entertainment systems have gradually become a key value-added feature. Traditional in-car entertainment primarily relies on radio, CD player, and Bluetooth music player. However, with the recent rise in the popularity of mobile internet and intelligent voice technology, in-car users are increasingly demanding diverse entertainment options. Against this backdrop, in-car KTV functionality has gradually entered the industry's radar, becoming a key area of ​​exploration for automakers and audio technology manufacturers. Existing in-car KTV systems mostly rely on microphone pickup and sound system playback. Users sing through wireless or wired microphones, and the system then generates reverberation and accompaniment, creating a KTV-like experience.

[0003] However, these traditional solutions based on physical microphones have many practical limitations. First, the microphones need to be handheld or mounted, which is inconvenient, takes up space in the vehicle, and incurs additional hardware costs. Second, microphones are prone to introducing echo and howling in the confined space of the vehicle. This is especially true when the speaker and microphone are close together, significantly increasing the difficulty of echo cancellation and reducing the singing experience. Third, in multi-passenger scenarios, providing only one or two microphones results in insufficient passenger interaction, making it impossible to meet the demands of simultaneous chorus, role-based singing, or multi-person interaction.

[0004] To address the microphone dependency issue, some manufacturers have proposed "microphone-free KTV" solutions. These solutions utilize in-vehicle microphone arrays (typically located on the ceiling, doors, or center console) to capture vocals, mix them with accompanying music, and then play them back. This approach eliminates the need for users to hold a microphone while also leveraging existing in-vehicle hardware to reduce system costs. However, existing "microphone-free KTV" solutions still present several challenges. Summary of the Invention

[0005] The main purpose of the present invention is to provide a zoned sound pickup and mixing method for in-vehicle microphone-free KTVs with multiple seating areas. By constructing a fixed acoustic path within the vehicle and generating a topological map of the seating area sound field, the method combines a virtual microphone set to independently capture and process the human voice in each seating area. Furthermore, the method uses a lead vocal template and a leakage template to interleave and annotate the zoned sound pickup streams, effectively separating the lead vocal shadow, echo shadow, and leakage shadow, and suppressing non-target segments according to the path order. When multiple seating areas sound simultaneously, resonant gating is used to achieve intermittent reconstruction of occluded syllables, and the accompaniment phase is used to anchor the beat to ensure the synchronization of the dry sound streams output by each seating area. Finally, the method combines spatial reverberation with vocal optimization processing to output a natural, clear, and layered audio signal. This invention not only overcomes the problems of unclear zone division, echo interference, and rhythm misalignment in existing microphone-free KTV technologies under multi-sound source conditions, but also enhances the interactivity and immersion of simultaneous singing by multiple passengers in the vehicle, significantly improving the overall experience and application value of the in-vehicle entertainment system.

[0006] In order to solve the above problems, the technical solution of the present invention is achieved as follows: refer to Figure 1 A method for picking up and mixing microphones in a car without microphones in a multi-passenger seating area in a KTV, comprising the following steps: Step 1: Control and trigger the microphone array to enter the zone pickup mode, perform echo cancellation, noise reduction and voice optimization on each channel of sound picked up by the microphone array, obtain the zone pickup stream, and send the zone pickup stream and accompaniment signal to the audio channel manager through the upstream path; Step 2: After the audio path manager receives the partition pickup stream and accompaniment signal, it executes the seat multiple shadow separation and resonance gating process, which includes: generating a seat area sound field topology map based on the fixed sound path in the car, and forming a virtual microphone set based on the sound field topology map; using the virtual microphone set to generate a lead singer template and a leakage template, interlaced matching the partition pickup stream, marking the lead singer shadow, echo shadow and leakage shadow, and suppressing non-target segments according to the path order; when multiple seats sound at the same time, the blocked syllables are discontinuously reconstructed based on the resonance gating; the reconstruction results of each seat area are aligned with the beat skeleton using the accompaniment phase anchor to obtain the seat area aligned dry sound stream, and the seat area aligned dry sound stream of each seat area is spatially reverberated and vocally optimized with the accompaniment signal to obtain the processed audio signal.

[0007] Furthermore, after receiving the control instruction for triggering the partitioned sound pickup mode in step 1, the geometric parameters and seating area pointing parameter set of the microphone array are loaded, the sampling rate is set to 48,000 Hz, the quantization bit depth is set to 24 bits, the frame length is set to 20 milliseconds, and the frame shift is set to 10 milliseconds. A unified time scale is established based on the microphone array master clock; the sampling rate fine-tuning resampling method is used to control the sampling frequency deviation between the accompaniment signal and the microphone array acquisition signal to within ±50 millionths, so that the two can be compared under the unified time scale; in order to reduce the overall delay, the target end-to-end delay budget of the uplink path of the partitioned sound pickup mode is set to no more than 80 milliseconds.

[0008] Furthermore, the seat area reference coordinates and the fixed sound path in the car are constructed through the following process: the seat area reference coordinates are constructed with the vehicle's geometric center as the origin, the front as the positive direction of the x-axis, the left as the positive direction of the y-axis, and the top as the positive direction of the z-axis; the center of the headrest of each seat area is used as the reference, and the mouth reference point coordinates are obtained by moving 70 mm forward along the x-axis, moving 120 mm downward along the z-axis, and placing the y-axis coordinate on the center line of the seat; the driver's seat, front passenger seat, rear left seat, rear middle seat, and rear right seat are marked as seat areas one to five in sequence; the three-dimensional coordinates of each array pickup unit in the microphone array are recorded; the direct propagation path from the mouth reference point of each seat area to each array pickup unit and the reflection path generated on the reflection surface are defined as the fixed sound path in the car.

[0009] Furthermore, in step 2, the seat area reference coordinates, the reference point coordinates of each seat area mouth, the three-dimensional coordinates of each array pickup unit, and the spatial pointing direction are read, and the front windshield, rear windshield, left window, right window, instrument panel, door interior panel, roof lining, and floor are determined to be reflective surfaces, and the seat backs, headrests, instrument panel body, center console, and trunk partition are determined to be impenetrable bodies. For each seat area and each array pickup unit, the direct propagation path, primary reflection path, and secondary reflection path are enumerated, and samples are taken at intervals of 10 mm along the path. If any sampling point falls within the outer boundary of the impenetrable body, the path is discarded. The retained paths are assigned an arrival order index from short to long according to their geometric length, and the path type and reflective surface sequence are recorded. At the same time, the alignment delay is converted to an integer sampling point using a sound speed of 343 meters per second and a sampling rate of 48,000 Hz.

[0010] Further, in step 2, the process of generating the seat sound field topology map based on the fixed sound path in the vehicle includes: performing seat crossing detection on the reserved path, using a 50mm expansion in each direction for the non-native seat area surrounding boundary, and marking the leakage risk level as high if at least one crossing occurs; calculating the minimum distance of the path to the non-native seat area mouth reference point, and marking it as high if it is less than or equal to 0.25 meters, and marking it as medium if it is between 0.25 meters and 0.40 meters; calculating the angle between the path incident direction and the spatial pointing direction of the array pickup unit, and marking the leakage risk level as high if the non-native seat area path is less than or equal to 25 degrees, and marking the leakage risk level as medium if it is between 25 degrees and 50 degrees; additionally marking the secondary reflection path as medium in the leakage risk level; taking the mouth reference point and the array pickup unit as nodes, and taking the reserved path as edges, and adding the attributes of arrival index, path type, reflection surface sequence, delay sampling point and leakage risk level; organizing in layers within each seat area subgraph, first sorting by arrival index, then sorting by path type as direct propagation priority over one reflection, one reflection priority over two reflections, and sorting by leakage risk level from low to high within the same type, to obtain the seat sound field topology map.

[0011] Further, in step 2, the process of forming a virtual microphone set based on the sound field topology map includes: selecting the array pickup unit with the smallest arrival index and the direct propagation path as the main channel under frame length 20 milliseconds, frame shift 10 milliseconds, if the short-time energy of the main channel is less than the quiet baseline by more than 6 decibel threshold, then sequentially select a direct propagation path with smaller index, if none of the direct propagation paths meet the condition, then select a one-reflection path; aligning the selected channels with the corresponding delay sampling points, and using a 5 millisecond linear cross transition for channel switching; when the evaluated speech intelligibility meets the preset rules, select at most 2 one-reflection paths with arrival index not greater than 3 and leakage risk level low or medium as support channels, and mix them in with a gain of 6 decibels lower than the main channel; applying a 12 decibel fixed attenuation to the non-native seat area incident direction with a high leakage risk level; suspending the support channels when the short-time energy is less than the quiet baseline by more than 3 decibel threshold for 300 milliseconds; when the similarity to the adjacent seat area virtual microphone reaches 0.8 and lasts for more than 100 milliseconds, add its direction to the side suppression direction set; output at least one main virtual microphone for each seat area, if there are two symmetric direct propagation paths with arrival index difference not greater than 2 and leakage risk level low, then additionally output one auxiliary virtual microphone.

[0012] Further, in step 2, select continuous voiced candidate segments on each seat area main virtual microphone with frame length 20 milliseconds, frame shift 10 milliseconds and sampling rate 48000 hertz, extract beat key points, energy envelope, frequency band energy ratio and rise-fall transient, and form the main singer template; collect leakage samples on adjacent seat areas based on the seat sound field topology map and arrival index, and align them by delay sampling points, record the lag, frequency band energy ratio and direction consistency, and form the leakage template.

[0013] Furthermore, in step 2, a sliding window is used on the partitioned sound pickup stream to calculate lead vocal similarity and leakage similarity, respectively, using the lead vocal template and the leakage template. The echo similarity is then calculated by combining the accompaniment beat repetition and the number of delayed sampling points of the retained path. Each window is initially judged as a lead vocal shadow, echo shadow, or leakage shadow based on a preset threshold. Conflicts are resolved by prioritizing scores and inheriting from the previous window, and a confirmation criterion is reached when three consecutive windows are determined. If a lead vocal shadow is determined, a graded attenuation with a strength positively correlated with the arrival order index is applied to path components with arrival order indices starting from 2, and additional attenuation is applied to paths with a high leakage risk level. If an echo shadow is determined, a uniform attenuation is applied within a range of ±15 milliseconds from the beat repetition, and additional attenuation is applied to paths with arrival order indices of 3 or greater. If a leakage shadow is determined, the source path and its neighboring paths are graded and suppressed using a fixed attenuation amount. A rise and fall transition and a maximum total attenuation cap are set for all suppression operations. A timestamped shadow category label sequence and a suppression instruction sequence are generated, and the suppressed partitioned sound pickup stream is output.

[0014] Furthermore, in step 2, frame-level detection is performed on the partitioned sound pickup streams and virtual microphones of all seating areas. When two or more seating areas are confirmed to be overlapped by the lead singer shadow in the same time window, the envelope reference and beat key points of the lead singer template and the leakage template are combined to determine the occluded syllable segment of the target seating area; resonance peaks are extracted in the front and back neighborhoods of the occluded syllable segment, and a resonance gated map of open, semi-open and closed states is formed according to the frequency band and time, and a side inhibition mark is set at the corresponding lag position according to the leakage template of the adjacent seating area to trigger subsequent leakage compensation; the occluded syllable segment is baseline restored with the lead singer template as the envelope reference; in the time slice marked as open in the resonance gated map, the unoccluded detail segment of the same seating area is re-injected according to the level and boundary transition rules; when the side inhibition mark is triggered, the leakage template matching it is introduced for compensation and superposition to obtain the reconstruction result.

[0015] The multi-occupant seat area-oriented vehicle-mounted KTV partition sound pickup and mixing method of the present application has the following beneficial effects: high-quality voice partition collection and intelligent mixing can be realized in a complex acoustic environment in the vehicle, thereby effectively improving the KTV experience in the vehicle. By establishing a fixed sound path in the vehicle and generating a seat area sound field topology map, the present application forms a virtual microphone set that can accurately correspond to the mouth reference point of each seat area, ensuring that the voice of each occupant can be independently captured and recognized. This approach not only avoids the space occupation and operational inconvenience caused by traditional handheld microphones, but also eliminates the sound pickup distortion caused by multi-path echoes and mutual interference between seat areas. Further, by generating and interleaving the main singer template and the leakage template, the system can label and distinguish the main singer shadow, echo shadow and leakage shadow that exist simultaneously in multiple seat areas, and through the suppression of non-target segments in the path order, the clean separation of the voice signal is realized. Especially when multiple seat areas sing at the same time, the present application introduces a resonance gating mechanism that can intermittently reconstruct the obstructed syllables, avoiding the sense of broken chorus caused by missing voice segments. At the same time, taking the accompaniment phase anchoring as a unified reference, the reconstruction results of each seat area are strictly aligned with the beat skeleton, so that the final output of the seat area-aligned dry sound stream remains synchronized with the accompaniment, ensuring the rhythm consistency of the singing process. Finally, the present application generates processed audio signals with spatial hierarchy and natural voice quality for each seat area through spatial reverberation and voice optimization processing, thereby realizing an immersive KTV experience in the vehicle with multiple users singing simultaneously and without mutual interference. The implementation of the present application not only solves the problems of unclear partition, distorted sound quality and synchronization difficulty of existing KTV without microphones in the case of multiple sound sources, but also provides a solution with high robustness and high scalability, significantly improving the overall value and user experience of interactive entertainment for multiple occupants in the vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The method flowchart of the multi-occupant seat area-oriented vehicle-mounted KTV partition sound pickup and mixing method provided for the embodiments of the present application is shown in the figure. Figure 2 The schematic diagram of the in-vehicle fixed sound path propagation path in the embodiments of the present application is shown in the figure. Figure 3 The in-vehicle seat area arrangement and microphone array schematic diagram in the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0018] refer to Figure 1 A method for picking up and mixing microphones in a car without microphones in a multi-passenger seating area in a KTV, comprising the following steps: Step 1: Control and trigger the microphone array to enter the zone pickup mode, perform echo cancellation, noise reduction and voice optimization on each channel of sound picked up by the microphone array, obtain the zone pickup stream, and send the zone pickup stream and accompaniment signal to the audio channel manager through the upstream path; After receiving control commands, a seating list is created based on the correspondence between vehicle seats and microphones. Each channel of the microphone array is enabled sequentially, completing time base alignment and channel connectivity verification. A silent segment is captured while the vehicle maintains a stable environment to extract baseline characteristics of the ambient noise, which serve as a reference for subsequent noise reduction and vocal optimization. The individual pickups from the microphone array are mapped into several seating channel sets according to the seating list. A time-slice-based processing flow is established to ensure that each seating channel set is processed synchronously on the same time base. For each time slice, the corresponding pickups are read to form the original seating channel segment for that time slice. For each seating channel set, the currently playing accompaniment signal is used as an echo reference, and echo path estimation and reference alignment are performed on the original seating channel segment. Components associated with the accompaniment signal are first offset, and the remaining distorted echo components are then subjected to nonlinear suppression and boundary smoothing to produce the echo-cancelled seating channel segment. Throughout the entire processing process, the convergence rate of the echo cancellation is dynamically adjusted based on the results of voice activity detection to avoid over-processing in silent areas.

[0019] For the seat pickup clips that have completed the echo cancellation process, the baseline characteristics of the ambient noise obtained in the silent clips are used to suppress steady-state noise and perform transient suppression on impulsive interference; the noise reduction intensity is controlled in the segments where voice activity is detected as sound to maintain the naturalness of the human voice, and the noise reduction intensity is increased in the segments where voice activity is detected as silent to reduce the background noise; distortion compensation and edge continuity repair are performed on the processed segments to maintain the coherence between the segments and adjacent time slices. The seat pickup clips that have completed the noise reduction process are then subjected to vocal optimization, with loudness equalization, dynamic range control, sibilance suppression, and formant modification implemented in sequence. The spatial directionality of the human voice is fixed according to the seat list to obtain a stable lead singer perception; the rise and fall characteristics and beat boundaries of the original voice are retained during the vocal optimization process to avoid affecting the time relationship with the accompaniment signal in subsequent steps. After echo cancellation, noise reduction, and vocal optimization, the audio pickup segments of each seating area are spliced ​​into corresponding partitioned audio pickup streams in chronological order, and a seating area identifier and a timestamp are added to each partitioned audio pickup stream to ensure synchronization among multiple seating areas. The accompaniment signal of the corresponding time period of the partitioned audio pickup stream is extracted on the same time base to generate an accompaniment signal segment with a timestamp identifier. The partitioned audio pickup stream and the accompaniment signal are encapsulated into a transmittable data unit according to the uplink path and sent to the audio path manager in the order of the seating area list for use in subsequent steps.

[0020] Step 2: After the audio path manager receives the zone pickup stream and the accompaniment signal, it executes the seat zone multiple image separation and resonance gating process, which includes: Specifically, a seat area sound field topology map is generated based on the fixed acoustic paths within the vehicle, and a virtual microphone set is formed based on the sound field topology map. This includes: according to the definitions of seat area reference coordinates and mouth reference points, the coordinates of the mouth reference point of each seat area and the three-dimensional coordinates of the array pickup unit are read, and an index table is established for seat areas one to five; the front windshield, rear windshield, left window, right window, instrument panel, door interior panel, roof lining, and floor are used as the main reflection surfaces, and the geometric boundaries and normal orientation of each reflection surface are recorded; this information is written into the fixed acoustic path candidate library within the vehicle as the sole input for subsequent path enumeration and screening. For each seat area and each array pickup unit, a path set is generated in the following order: direct propagation path, primary reflection path, and secondary reflection path. The direct propagation path is obtained by connecting the mouth reference point with the array pickup unit; the primary reflection path is obtained by geometrically mirroring the mouth reference point on a single reflection surface and then connecting the mirror point with the array pickup unit; the secondary reflection path is obtained by geometrically mirroring two different reflection surfaces in sequence and then connecting the final mirror point with the array pickup unit. The reachability of each candidate path is determined. If a straight line segment or a segmented polyline segment is blocked by the geometric boundary of an obstacle inside the vehicle, the path is eliminated. For the paths that are not eliminated, the path type, the sequence of reflective surfaces involved, and the relative level of geometric length are recorded.

[0021] Each retained path is assigned an arrival order index based on geometric length from shortest to longest, numbered consecutively starting with index 1. Path types are labeled as direct, single-reflection, or double-reflection. The sequence of reflective surfaces involved is written into the path label in a fixed order. Each path is assigned a leakage risk level of low, medium, or high based on the spatial orientation of the seating area traversed by the path and the array pickup unit. Each path is assigned a reflection stability level of high, medium, or low based on the reflective surface material and the range of incidence angles. This results in a seat-based path attribute table. A graph structure is constructed with the mouth reference point and the array pickup unit as nodes and the determined retained fixed acoustic paths within the vehicle as edges. Each edge is assigned the arrival order index, path type, reflective surface sequence, geometric length level, leakage risk level, and reflection stability level. The entire graph is divided into five subgraphs by seating area, and each subgraph is further layered by arrival order index, giving priority to layers with smaller indexes, forming a hierarchical structure from near to far and from direct to multiple reflections. The seat area sound field topology map is now generated.

[0022] For each seating area, all array pickup units are directional sampled in the corresponding subgraph. The direction from the array pickup unit to the mouth reference point is defined as the main sampling direction, and the direction connected to the adjacent seating area with a medium or high leakage risk level is defined as the side suppression direction. Within each layer, each edge is assigned a sampling priority according to the path type and reflection stability level. The direct path has a higher priority than the first-reflection path, and the first-reflection path has a higher priority than the second-reflection path. A weight plan table is generated for each edge, which includes the sampling priority, boundary smoothing duration, switching order with adjacent layers, suppression strength level for leakage risk level, and silence threshold level when voice activity is detected as silent. According to the weighted schedule in step 5, under the time organization of 20 milliseconds frame length and 10 milliseconds frame shift set in step 1, the following rules are applied frame by frame: the array pickup unit signal corresponding to the edge with the smallest arrival order index and a direct path type is preferentially selected as the main channel; if the main channel has insufficient short-term energy or voice activity detection indicates silence, the array pickup unit signals corresponding to the primary and secondary reflection paths are supplemented in sequence according to the acceptance priority, and boundary smoothing is applied within the frame boundary during supplementation; the array pickup unit signals corresponding to the edges marked as side suppression directions and with a medium or high leakage risk level are attenuated according to the suppression strength level, and the arrival order index is maintained in the same order as the main channel within the frame; a fixed order is implemented for cross-layer switching, and multiple switching within the same frame is prohibited to ensure stability. The signal synthesized frame by frame is defined as the virtual microphone of the seating area, and at least one virtual microphone is formed as the main virtual microphone. When the sound field topology diagram of the seating area shows that there are two symmetrical and effective sets of main sampling directions and their leakage risk level is low, it is allowed to form an additional virtual microphone in the same seating area as a secondary virtual microphone to enhance the audibility of the side seats.

[0023] Over a continuous time period, the short-term spectral shapes and tempo variations of the primary and secondary virtual microphones are compared for similarity. If the correlation with the corresponding seat's mouth reference point decreases and the correlation with the adjacent seat increases, the leakage risk level of the adjacent seat's related edge is increased by one level, and the priority of the corresponding edge within the current seat is decreased by one level. If silence persists for several frames, the weighting schedule for the current layer is restored to the default settings. This process outputs a virtual microphone set containing the virtual microphones for each seat, the corresponding array pickup unit list, the weighting schedule, the arrival order index table, and the leakage suppression table for use in subsequent steps.

[0024] The clarity is evaluated frame by frame on the main virtual microphone signal in each seating area. If any two of the following conditions are met, the clarity is considered insufficient: Voice on-off boundary deviation: within a continuous voiced period, the voice on-set and off-set boundaries of the main virtual microphone in the seat zone are detected with reference to the beat boundaries of the accompaniment beat skeleton; when the time deviation of the on-set or off-set boundary exceeds 30 milliseconds and occurs 3 times or more within a 500-millisecond window, it is recorded as a boundary deviation unqualified. High-frequency detail loss: in a transient segment containing a fast rising edge, the high-frequency energy proportion is calculated; when the high-frequency energy proportion is less than 15% of the total energy and the situation accumulates for 120 milliseconds or more within a 300-millisecond window, it is recorded as a high-frequency detail loss unqualified. The transient segment is determined by a rising edge threshold, which is set to an adjacent frame energy increase of 6 decibels and a duration of 5 to 15 milliseconds. Discontinuous distortion insertion: a voice insertion event occurs in a segment determined to be voiced, with a single insertion duration of 20 milliseconds or more; when 3 or more occur within 1 second, it is recorded as a discontinuous distortion unqualified. Voice insertion is determined by a threshold of 3 decibels above the silence baseline. Residual noise masking: in a pause segment that should be blank, the average energy is continuously higher than 3 decibels above the silence baseline for more than 200 milliseconds; or the same texture is detected to occur more than 2 times in the pause segment, which is recorded as a residual noise masking unqualified. Direction consistency decrease: with reference to the main sampling direction, the incident direction of the main virtual microphone is estimated; when the deviation angle of the incident direction relative to the main sampling direction is greater than 25 degrees and the duration is more than 100 milliseconds, it is recorded as a direction consistency unqualified. The incident direction is estimated according to the arrival index of the selected path in the seat zone sound field topology and the spatial pointing direction of the array pickup unit. Echo residual synchronization: at the time point near the accompaniment beat kick, the main virtual microphone appears a peak value synchronized with the beat kick and the amplitude is higher than 6 decibels above the silence baseline; when 3 or more occur continuously, it is recorded as an echo residual unqualified. Pronunciation intensity imbalance: within a continuous three-syllable range, the difference between the peak value of any syllable and its adjacent valley value is less than 6 decibels, and the imbalance phenomenon occurs in two or more syllables at the same time, which is recorded as a pronunciation intensity imbalance unqualified. Syllable boundaries are determined by energy rising and falling inflection points.

[0025] The execution criteria are as follows: in each frame processing period, the above 7 indicators are calculated in turn; when any 2 are determined to be unqualified, the "insufficient intelligibility" flag is triggered; once marked, immediately enable the support channel mixing according to the previously agreed process, and keep the 5-millisecond linear cross transition of channel switching and the corresponding delay alignment; if "insufficient intelligibility" is still continuously triggered within 200 milliseconds, add 1 path with arrival index not greater than 3 and leakage risk level low to the support direction set to participate in mixing; if "insufficient intelligibility" is not triggered again after the addition for 300 milliseconds, restore the use of only the main channel; all determination results and processing actions are recorded with frame timestamps and written into the process log of the seat zone virtual microphone for subsequent steps of interleaving matching, labeling and alignment.

[0026] Specifically, a virtual microphone set is used to generate a lead vocal template and a leakage template. Interleaved matching is performed on the partitioned sound pickup streams, annotating the lead vocal shadow, echo shadow, and leakage shadow. Non-target segments are suppressed according to the path order. Inputs include the virtual microphone set for each seating area, the partitioned sound pickup stream, the accompaniment signal, the seating area sound field topology, the arrival order index, and the number of delayed sampling points for each retained path. Processing is performed with a 20 millisecond frame length and a 10 millisecond frame shift. All signals are sampled at a 48,000 Hz rate, a 24-bit quantization bit depth, and a unified time scale.

[0027] On each zone's main virtual microphone, search for consecutive segments that meet the following three criteria as candidate segments: voice activity detected as active and lasting at least 400 milliseconds; the average energy of the segment is at least 9 dB above the silence baseline; and it is aligned with the beat skeleton of the accompaniment signal, covering four or more consecutive beats. Normalization and Alignment: Candidate segments are normalized to the reference level of the zone's main virtual microphone. If the candidate segment is composed of multiple array pickup units, each segment is aligned according to the number of delayed sampling points of the corresponding path before being spliced ​​together.

[0028] For each candidate segment, the following four elements are extracted to form the main vocal template entry for that segment: Beat Keypoint Sequence: records the relative timing of each beat's strong and weak beats, with an allowable deviation of no more than ±10 milliseconds; Energy Envelope Curve: records the normalized energy envelope at 10-millisecond intervals; Band Energy Ratio: divides the audible frequency into low, mid, and high bands, and records the energy ratio (expressed as a percentage) of these three bands; Rise and Fall Transient Morphology: records the duration of the rising and falling edges, expressed in milliseconds. Representative Template Determination: Based on the singing styles covered by the segment, up to three main vocal templates are selected as representative templates for that segment, corresponding to strong, weak, and continuous singing segments. If fewer than three candidates are available, all available templates are selected in descending order of frequency.

[0029] In the seat area sound field topology diagram of the seat area, find the edges with medium or high leakage risk level and coming from the adjacent seat area, and record the arrival order index and delayed sampling points of these edges to form a leakage source list. Leakage sample collection: On the main virtual microphone of the adjacent seat area, select the segments that meet the following two conditions as leakage samples: the average energy is 6 decibels to 12 decibels lower than the average energy of the lead singer template; it is aligned with the beat skeleton of the adjacent seat area, covering 2 or more consecutive beats. Path consistency processing: align the leakage samples according to the delayed sampling points of the corresponding edges in the leakage source list; if the edge is a single reflection path, reduce the high-frequency band energy of the leakage sample by 3 decibels; if the edge is a double reflection path, reduce it by an additional 3 decibels on top of the 3 decibel reduction.

[0030] Leak template elements: record the following elements for the aligned leak sample to form a leak template entry: arrival index and corresponding delay sample point number; energy envelope flatness, expressed as the decibel value of envelope peak-valley difference; time lag with the seat area beat skeleton, expressed in milliseconds; frequency band energy proportion and direction consistency, direction consistency is grouped by the angle between the incident direction and the spatial pointing direction of the array pickup unit, less than or equal to 25 degrees is strong consistency, between 25 degrees and 50 degrees is moderate consistency, and greater than 50 degrees is weak consistency. According to the occurrence frequency and direction consistency of the leak source edge, at most 2 leak templates are retained for each adjacent seat area.

[0031] On the partitioned pickup stream of each seat area, a sliding window with the same length as the template is traversed with a step of 10 milliseconds. Three scores are calculated between each window and each lead vocal template, and the lead vocal similarity (range 0 to 1) is obtained by weighting: beat key point alignment score, accounting for 50%. If the average deviation of the key points in the window and the template key points is not more than 20 milliseconds, it is recorded as 1, the deviation is between 20 milliseconds and 40 milliseconds, and it is reduced from 1 to 0 in a linear manner, and more than 40 milliseconds is recorded as 0; energy envelope similarity score, accounting for 30%. The normalized energy difference between the window and the template is linearly mapped from 0 to the maximum difference to 1 to 0; frequency band proportion similarity score, accounting for 20%. The absolute difference of the three proportion ratios is less than or equal to 10% for 1, any proportion absolute difference reaches 30% for 0, and the rest is interpolated in a linear manner. Take the highest lead vocal similarity of all lead vocal templates in this window as the lead vocal similarity of the window.

[0032] Echo Similarity Calculation: Within each window, the number and amplitude percentage of peaks synchronized with the accompaniment beat are counted, and combined with the delay constraints corresponding to the arrival order indexes, the echo similarity (ranging from 0 to 1) is calculated: the synchronization rate score, which accounts for 70%. A peak occurring within ±10 milliseconds of each hit point is considered a hit, with a hit rate of 1; otherwise, it is counted as 0; the delay compliance score, which accounts for 30%. If the time difference between the window peak and the hit point is equal to the time corresponding to the number of delayed sampling points on any retained path and the deviation does not exceed ±10 milliseconds, it is scored as 1. Deviations between ±10 milliseconds and ±30 milliseconds are linearly reduced from 1 to 0, and those exceeding ±30 milliseconds are scored as 0. Leakage Similarity Calculation: Three scores are calculated between each window and each leakage template and weighted to obtain the leakage similarity (ranging from 0 to 1): the delay compliance score, which accounts for 40%. If the lag of the window's peak moment relative to the tempo skeleton of the current seat is consistent with the lag of the leakage template and the deviation does not exceed ±10 milliseconds, it is scored as 1. If the deviation is between ±10 milliseconds and ±30 milliseconds, the score is linearly reduced from 1 to 0. The directional consistency score accounts for 30%. If the directional consistency of the leakage template is strong, it is scored as 1, moderate, as 0.5, and weak, as 0. The frequency band ratio matching score accounts for 30%. If the absolute difference of the three ratios is less than or equal to 15%, it is scored as 1. If the absolute difference of any ratio reaches 35%, it is scored as 0. The remaining scores are linearly interpolated. The highest leakage similarity of the window with all leakage templates is taken as the leakage similarity of the window.

[0033] Shadow category labeling: For each window, a preliminary classification is performed using the following thresholds: Vocalist shadow: Vocalist similarity greater than or equal to 0.70, leakage similarity less than or equal to 0.40, and echo similarity less than or equal to 0.50; Echo shadow: Echo similarity greater than or equal to 0.70, and Vocalist similarity less than or equal to 0.50; Leakage shadow: Leakage similarity greater than or equal to 0.60, and Vocalist similarity less than or equal to 0.60. If the initial classification criteria for two or more categories are met simultaneously, the category with the highest score is selected. If the difference between the highest scores is less than 0.05, the confirmed category from the previous window is retained. If no category was confirmed in the previous window, it is temporarily marked as pending and merged for classification in the next window. If the same category appears in three consecutive windows, it is confirmed. If the same category appears twice within three windows, and the other appears as pending, it is also confirmed.

[0034] Lead Vocal Shadow Window Suppression: When a window is identified as a Lead Vocal Shadow, graded suppression is applied to all components of paths within that area with an arrival order index greater than or equal to 2: 6dB attenuation for arrival order indexes 2, 12dB attenuation for arrival order indexes 3 or 4, and 18dB attenuation for arrival order indexes 5 or greater. If the path's leakage risk is high, an additional 6dB of attenuation is applied. Suppression is applied with a 5ms linear transition between the start and end of the suppression. Echo Shadow Window Suppression: When a window is identified as an echo shadow, a uniform 12dB attenuation is applied to all path components within ±15ms of the beat re-attack. Outside this range, the attenuation remains unchanged. If the echo shadow persists for three consecutive windows, an additional 6dB of attenuation is applied to paths with an arrival order index greater than or equal to 3. Leakage Shadow Window Suppression: When a window is identified as a leakage shadow, the adjacent source path corresponding to the leakage template is identified and the corresponding path component is attenuated by 18dB. Path components from the same adjacent source path with an arrival order index difference of no more than 1 are attenuated by 12dB. Path components from other non-local sources with medium or high leakage risk levels are attenuated by 6dB. Continuity and Upper Limit Constraints: The total attenuation after any single suppression does not exceed 24dB. If the category changes more than twice within 200 milliseconds, the transition time of all suppressed edges is increased to 10 milliseconds to prevent audible abrupt changes. For each window, the lead vocal similarity, echo similarity, and leakage similarity, shadow category, list of suppressed paths, corresponding arrival order index, and attenuation are recorded and timestamped. A shadow category label sequence and suppression instruction sequence are generated as input for discontinuous reconstruction and phase anchoring in subsequent steps. The suppressed partition pickup stream is retained as an updated version for subsequent processing of the corresponding seat.

[0035] Specifically, when multiple seats sound at the same time, the blocked syllables are discontinuously reconstructed based on the resonance gating; the reconstruction results of each seat are aligned with the beat skeleton using the accompaniment phase anchor to obtain the seat-aligned dry sound stream, and the seat-aligned dry sound stream of each seat is spatially reverberated and vocally optimized with the accompaniment signal to obtain a processed audio signal, including: under a unified time scale, a sampling rate of 48,000 Hz, a quantization bit depth of 24 bits, a frame length of 20 milliseconds, and a frame shift of 10 milliseconds, the partitioned pickup streams of all seats and the virtual microphones are checked frame by frame to see whether there are two or more seats marked as the main singer shadow in the same time window, and if the continuous overlapping time reaches or exceeds 50 milliseconds, it is determined that multiple seats are sounding at the same time. In each determined time window, the main virtual microphone of the target seat area is divided according to the syllable boundary; if the syllable has an energy trough in the middle section and is more than 6 decibels lower than the envelope reference value of the lead singer template, and the leakage similarity of the adjacent seat area leakage template reaches or exceeds 0.60, the trough segment of the syllable is marked as an occluded syllable segment; the boundary of the occluded syllable segment is determined by the intersection of the energy turning point and the envelope reference value, and the minimum reconstruction unit is not less than 30 milliseconds.

[0036] The target seat zone main virtual microphone is within the range of 100 milliseconds before and after the occluded syllable segment, and the duration and relative energy of the stable peak are counted in the low frequency band, the medium frequency band and the high frequency band respectively. The peak value with duration reaching or exceeding 40 milliseconds and relative energy higher than the silence baseline by more than 9 decibels is identified as a formant. The formant gating map is generated with the formant as the center: the segment within the frequency band where the formant is located and the range of 10 milliseconds before and after it is marked as "open", the segment without formant and with energy lower than the silence baseline by more than 3 decibels is marked as "closed", and the remaining segment is marked as "half open". The minimum holding time of "open" state is 30 milliseconds, the minimum holding time of "closed" state is 20 milliseconds, and "half open" state is used for state transition and keeps for no less than 10 milliseconds. When the arrival index corresponding to the adjacent seat zone leakage template is less than or equal to 3 and the direction consistency is strong consistency, the time point matching the lag of the leakage template is added with a "side suppression mark" on the formant gating map of the target seat zone as a trigger condition for subsequent leakage compensation.

[0037] The discontinuous reconstruction (performed for each occluded syllable segment) process includes: baseline recovery: using the main singer template entry most similar to the syllable as the envelope reference to generate the target envelope within the occluded syllable segment; the residual signal within the occluded syllable segment is amplitude recovered according to the target envelope, so that the peak-to-valley difference is not less than 70% of the template peak-to-valley difference. When recovering, a 5-millisecond linear cross transition is set before and after the segment to avoid boundary mutation. Detail restoration: search for an unoccluded segment with the same vowel category or a beat key point sequence similarity reaching or exceeding 0.70 within the same seat zone in the last 1 second, and cut off a detail segment of no more than 120 milliseconds as a formant re-injection source; within the time slice where the formant gating map is "open", superimpose the detail segment to the target envelope with a mixing level not higher than 6 decibels of the current frame main virtual microphone level, and use 10-millisecond linear cross transition for mixing and removal. Leakage compensation: when the "side suppression mark" is triggered, select the entry with the most matching lag from the leakage template, align the number of delay sampling points according to the entry, and perform compensation superposition on the corresponding time slice within the occluded syllable segment, with the compensation level set to 80% of the leakage component estimated level of the time slice and linearly started and stopped within 5 milliseconds. If the feature peak value with consistent direction with the entry is still detected after compensation, add a 20% compensation. Consistency check: after reconstruction, check whether the beat key point sequence deviation of the syllable is less than or equal to ±20 milliseconds, and whether the energy envelope peak-to-valley difference reaches or exceeds 70% of the corresponding main singer template peak-to-valley difference. If either is not met, perform a one-time smooth stretching or compression on the entire syllable, with the absolute time length of stretching or compression not exceeding 20 milliseconds, and maintain a 5-millisecond cross transition at both ends. Output the reconstruction result of the syllable after completion.

[0038] The process of accompaniment phase anchoring and seat-aligned dry stream generation involves extracting a beat skeleton from the accompaniment signal and determining the timing of each strong and weak beat. The target seat is then time-shifted relative to adjacent anchor points at three key points in the reconstructed syllable: the "starting point," "core point," and "ending point." When the absolute deviation at any key point is less than or equal to 15 milliseconds, the segment containing that key point is directly fine-tuned. When the absolute deviation is between 15 and 40 milliseconds, the correction is performed by shortening or lengthening the adjacent pause segment without changing the internal structure of the syllable, with a single adjustment not exceeding 20 milliseconds. When the absolute deviation is greater than 40 milliseconds, the correction is amortized to the next three anchor points, with a single adjustment not exceeding 20 milliseconds, and a cumulative adjustment not exceeding 50 milliseconds within any one-second period. Each corrected syllable is concatenated in chronological order to form the seat-aligned dry stream. A uniform 5-millisecond linear crossover transition is applied to the syllable joints, and timestamps are added to ensure synchronization during subsequent mixing.

[0039] The process of optimizing spatial reverberation, vocals, and mixing the sound with the accompaniment output includes the following: Spatial reverberation: Based on the seating area's sound field topology, the direct propagation path and the three first-order reflection paths with the smallest arrival order indexes are selected as early reflection references. The relative levels of the early reflections are set to the main direct level minus 6dB, minus 9dB, and minus 12dB, respectively. The early reflections are started and stopped with a 5-millisecond linear transition. A decaying reverberation tail is then superimposed, with the tail length preset according to the vehicle volume category and decreasing in 100-millisecond increments until the overall sound decays to within 3dB of the silent baseline. Vocal optimization: Loudness equalization is first performed on the seating area's aligned dry sound stream, followed by two stages of dynamic range control. The first stage controls fast peaks with an attack time of 10ms and a recovery time of 80ms; the second stage controls slow fluctuations with an attack time of 60ms and a recovery time of 300ms. Sibilance suppression and formant modification are then applied to bring the frequency band energy ratio close to the target ratio of the lead vocal template, with a deviation of no more than 10%. Mixing with Accompaniment: The average energy of the seat-aligned dry sound stream and the accompaniment signal is compared within a 200ms sliding window. If the accompaniment is 6dB or higher than the vocals, the accompaniment is reduced by 3dB; if the vocals are 6dB or higher than the accompaniment, the vocals are reduced by 1dB to avoid overemphasis. All gain changes are implemented with a 10ms linear transition. After mixing, a peak limiter is used to maintain a fixed instantaneous peak margin of 6dB. Output: The processed audio signal, with spatial reverberation and vocal optimization, is output for each seat, retaining time alignment markers with the beat skeleton for full-vehicle broadcast and subsequent scoring.

[0040] A complete embodiment of the present invention is given below: 1. Vehicle and sampling parameter settings: Sampling rate is set to hertz( is the audio sampling rate), the quantization bit depth is 24 bits. The frame length and frame shift are Second( is the duration of each frame) and Second( is the time step of adjacent frames). The corresponding number of sampling points is ;in is the number of sampling points per frame, is the number of frame shift sampling points. The origin of the vehicle coordinate system is at the geometric center of the passenger compartment, and the forward direction is Axis positive, left side Axis positive direction, upward Axis positive direction. Seat mouth reference point (mouth reference point is the singer's mouth space position) value: Seat mouth reference point Meter; reference point of the second mouth of the seat area The microphone array pickup unit (the array pickup unit is a fixed microphone position in space) takes 6: rice, rice, rice, rice, rice, Meters. Select the tempo of the song ( is the number of beats per minute), beat cycle ;in The time between adjacent strong beats.

[0041] 2. Example of fixed sound path and arrival order index (direct path from seat 1 to several microphones): Path geometric length ( and The Euclidean distance of ;in is the path length. Take the speed of sound in air Meters per second ( is the speed of sound), arrival time Corresponding alignment delay sampling points ;in is the propagation time, Delay points.

[0042] Calculation Example: Direct Path from Seat 1 to Microphone 4 ; .

[0043] Calculation example: Direct path from seat 1 to microphones 1 and 2 ; .

[0044] Therefore, the arrival order index of seat zone one is one, two, and three according to microphone four, microphone one, and microphone two.

[0045] Frame-level synthesis of a virtual microphone in the seating area (main channel and supporting channel): Assume the short-time RMS envelope within the frame ;in For the Frame energy envelope, is the waveform sampling sequence. If the main channel threshold is the relative silence baseline decibel (in decibel as logarithmic dimension), when the frame energy of microphone 4 meets When the threshold is exceeded, select microphone 4 as the main channel and delay Point alignment; when When the clarity is below the threshold, microphone 1 and microphone 2 are examined in turn. When the clarity is insufficient (according to the previously defined objective rules, any two of the seven indicators are met), 1 to 2 supporting channels are selected from the primary reflection or the nearest direct path to be mixed in, and the mixed level is lower than the main channel. Decibel. Level converted to linear gain ;in is the linear gain, decibel hour .

[0046] Examples of lead vocal template and leaked template: The beat skeleton time point is ( For the A powerful moment, to align the song start point). seconds, select the lead singer candidate segments covering 4 strong beats Seconds. The beat key point sequence and the segment alignment deviation are milliseconds is considered perfectly aligned; the energy envelope is recorded in each Millisecond sampling is normalized to a maximum value of 1; the energy ratio of the three frequency bands (low, medium, and high) in this segment is set to the average value Seat 2 sings during the same time period. The direct path length from seat 2 to microphone 4 is Corresponding delay The delay difference between seat 1 and microphone 4 ; Leak template recording lag The energy ratio of point and frequency band is about , the directional consistency is moderate (angle is about 35 degrees).

[0047] Interleaved matching and shadow annotation window calculation: Let the sliding window length be equal to the template length, step Seconds. In the window Calculate the three scores of the lead singer template within seconds. The lead singer similarity is composed of three weighted parts ;in Similarity to the lead singer, is the beat alignment score, is the envelope similarity score, is the frequency band ratio score. Let the average deviation of the strong beat of the window be Milliseconds are mapped to , envelope mean absolute difference Mapped to The three-segment proportional differences are Mapped to ,but The echo similarity is calculated by combining the strong beat synchronization and the delay (the score is ). Leakage similarity combined with hysteresis , medium directional consistency and matching of frequency band ratio (set score ). According to the threshold rule (lead singer shadow threshold ), the window is marked as the lead singer's shadow.

[0048] A segment of discontinuous reconstruction of simultaneous voicing and occluded syllables: The energy of the main virtual microphone in the seating area is lower than that of the template within seconds. decibels and the leakage similarity peak reached , which is determined to be an occluded syllable segment. The template peak-to-valley difference is decibels, aiming for at least Recovery, that is Set the gain of this section to ; and apply at both ends of the section Millisecond linear crossover transition. Retrieve similar syllable segments within seconds Seconds, intercept milliseconds and aligned to mix level Decibel superposition, that is , the injection interval is limited to the time slice of the resonance gate "on" state. Alignment leakage template hysteresis Point, the compensation component is superimposed in the occlusion section, and the compensation level is set to the leakage estimation level. If the peak that is consistent with the leakage template direction is still visible, add it again Compensation. Deviation of strong shot key points after reconstruction milliseconds, less than Millisecond threshold; the envelope peak-to-valley difference reaches decibel, satisfied Request, reconstruction completed.

[0049] Accompaniment phase anchoring and alignment: The accompaniment strong beat sequence is Seconds. Calculate the deviations at the starting point, core point, and end point of the reconstructed syllable: For example, the starting point deviation milliseconds. Shorten the adjacent pause segments milliseconds and slightly shift the entire syllable milliseconds, so that the absolute deviation is no greater than milliseconds. All nudges are limited to a single within milliseconds and using Millisecond linear crossover transition. All aligned syllables are spliced ​​in time sequence to obtain the seat-aligned dry sound stream.

[0050] Spatial reverberation, vocal optimization and accompaniment mixed output: Early reflection: Select the three shortest reflection paths related to seat area 1, and set the difference between the early reflection and the main direct level to be 、 、 decibels, start-stop transition milliseconds. Reverb tail envelope ;in For the tail envelope, is the tail time constant. seconds, so that the tail sound is about seconds below the silence baseline Decibel. First loudness equalization, then two-stage dynamic range control. First stage start time Milliseconds, recovery time milliseconds, second stage starting time Milliseconds, recovery time milliseconds; followed by sibilance suppression and formant modification, so that the frequency band ratio converges to the lead vocal template The deviation does not exceed .exist Compare the average energy of vocals and accompaniment in the millisecond sliding window. decibels and above, accompaniment attenuation decibel; when the human voice is higher than the accompaniment decibels and above, human voice attenuation All gain changes are in decibels. Millisecond linear transition is achieved. Peak limiter is retained decibel margin, linearity coefficient ; The processed audio signals of seat area 1 and seat area 2 are obtained respectively, and each signal carries a time alignment mark with the beat skeleton for partition broadcasting and scoring.

[0051] like Figure 2As shown, the sound path analysis system of the present application details the sound propagation paths and their geometric characteristics from the mouth reference point of the seating zone to each array pickup unit. In the figure, the mouth reference point of the first seating zone is taken as the sound source starting point, and a certain array pickup unit is taken as the end point, showing three different types of sound propagation paths: direct propagation path, once-reflected path, and twice-reflected path. The direct propagation path is represented by a solid line, which is the shortest geometric path from the mouth reference point of the first seating zone to the array pickup unit. This path does not pass through any reflecting surface, and the sound signal propagates in a straight line, with the shortest propagation time and the smallest energy loss. The direct propagation path has the highest priority and the lowest risk level of leakage in the sound field topology map. The once-reflected path is represented by a dashed line and includes two typical cases. The first is the reflection path through the roof lining, where the sound travels from the mouth reference point of the first seating zone, reflects off the surface of the roof lining, and then propagates to the array pickup unit. The second is the reflection path through the floor, where the sound reflects off the surface of the floor before reaching the target location. The once-reflected path has a longer propagation distance than the direct propagation path, and the corresponding arrival time is also delayed accordingly, giving it a higher arrival index value in the system. The twice-reflected path is represented by a dotted and dashed line and involves two consecutive reflection processes. The twice-reflected path shown in the figure first reflects off the left window surface for the first time, then reflects off the roof lining surface for the second time, and finally reaches the array pickup unit. The twice-reflected path has the longest propagation distance and the largest propagation delay, and its signal strength is relatively weak due to the energy attenuation caused by multiple reflections. The interior reflecting surfaces include the front windshield, rear windshield, left window, right window, instrument panel, door trim panel, roof lining, and floor surfaces. These reflecting surfaces have different acoustic characteristics and reflection coefficients, affecting the spectral characteristics and energy distribution of the reflected sound. In contrast, the non-traversable bodies include the seat backrest, headrest, instrument panel body, center console channel, and trunk partition, which block the sound propagation path. The system analyzes all possible paths between each seating zone and each array pickup unit, sampling and detecting at 10mm intervals along the path. If any sampling point falls within the outer boundary of the non-traversable body, the path is automatically excluded. The remaining paths are sorted by geometric length from short to long and assigned a corresponding arrival index, while recording path type, reflection surface sequence, and other attribute information, providing basic data for subsequent sound field topology map construction and virtual microphone formation.

[0052] As Figure 3As shown, the vehicle-mounted microphone-free KTV partition sound pickup system of the present application adopts a specific seat zone division scheme and microphone array arrangement structure in the vehicle interior. The vehicle interior space is divided into five independent seat zones, which are sequentially identified as seat zone one to seat zone five from front to back and from left to right. Specifically, seat zone one is located at the driver's seat, seat zone two is located at the front passenger seat, and seat zone three, seat zone four, and seat zone five are respectively located at the left, middle, and right rear seats. Each seat zone is identified as a rectangular area, and its boundary range is determined based on the normal sitting posture and sound production position of the occupant, ensuring that the main activity range of the occupant in each seat zone is covered. The microphone array system includes eight array pickup units, which are labeled as Mic 1 to Mic 8, and adopts a distributed arrangement strategy to achieve effective coverage of the entire vehicle space. In the front area of the vehicle, Mic 1 to Mic 5 are distributed in a horizontal line, with Mic 1 located on the left side of the front of the vehicle, Mic 2, Mic 3, and Mic 4 arranged in sequence to the right in the central area of the front of the vehicle, and Mic 5 located on the right side of the front of the vehicle. This front array arrangement can effectively pick up the sound signals of the two front seat zones, while providing appropriate directional support for the rear seat zones. In the side area of the vehicle, Mic 6 is arranged in the middle left position, and Mic 7 is arranged in the middle right position, forming a symmetrical lateral pickup configuration, mainly responsible for picking up sound signals from the sides and providing spatial positioning reference. In the rear area of the vehicle, Mic 8 is located in the central position of the rear, and is specially responsible for sound pickup of the rear seat zones. The spatial geometric configuration of the entire microphone array system fully considers the complexity of the vehicle interior acoustic environment, and realizes effective separation and accurate positioning of sound sources in different seat zones through multi-point distributed arrangement. The array pickup units maintain appropriate spatial distance, which not only ensures the continuity of sound field coverage, but also avoids signal interference between adjacent units. This arrangement scheme lays the foundation for subsequent partition sound pickup, sound field topology construction, and virtual microphone formation.

[0053] The above-described embodiments are merely used to illustrate the technical solutions of the present application, but not limit it; although the foregoing embodiments of the present application have been described in detail, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for picking up and mixing microphones in a car without microphones in a multi-passenger seating area, characterized in that: The method comprises the following steps: Step 1: Control and trigger the microphone array to enter the zone pickup mode, perform echo cancellation, noise reduction and voice optimization on each channel of sound picked up by the microphone array, obtain the zone pickup stream, and send the zone pickup stream and accompaniment signal to the audio channel manager through the upstream path; Step 2: After the audio path manager receives the partition pickup stream and accompaniment signal, it executes the seat area multiple image separation and resonance gating process, which includes: generating a seat area sound field topology map based on the fixed sound path in the car, and forming a virtual microphone set based on the sound field topology map; A lead singer template and a leakage template are generated using a virtual microphone set. The partitioned sound pickup streams are interlaced and matched, the lead singer shadow, echo shadow and leakage shadow are marked, and non-target segments are suppressed in the order of the paths. When multiple seats sound at the same time, the blocked syllables are discontinuously reconstructed based on the resonance gating. The reconstruction results of each seat are aligned with the beat skeleton using the accompaniment phase anchor to obtain the seat-aligned dry sound stream. After spatial reverberation and vocal optimization of the seat-aligned dry sound stream of each seat and the accompaniment signal, the processed audio signal is obtained.

2. The method for picking up and mixing zoning sounds of a car-mounted KTV without microphones for multiple passenger seating areas as claimed in claim 1 is characterized in that: After receiving the control command for triggering the zoned pickup mode in step 1, the geometric parameters and seating area directional parameter set of the microphone array are loaded, the sampling rate is set to 48,000 Hz, the quantization bit depth is set to 24 bits, the frame length is set to 20 milliseconds, and the frame shift is set to 10 milliseconds. A unified time scale is established based on the microphone array master clock. The sampling frequency deviation between the accompaniment signal and the microphone array acquisition signal is controlled within ±50 parts per million using sampling rate fine-tuning resampling, making the two comparable under the unified time scale. To reduce overall latency, the target end-to-end latency budget for the uplink path of the zoned pickup mode is set to no more than 80 milliseconds.

3. The method for picking up and mixing zoning sounds of a car-mounted KTV without microphones for multiple passenger seating areas as claimed in claim 2 is characterized in that: The seat reference coordinates and the fixed sound path in the car are constructed through the following process: the seat reference coordinates are constructed with the vehicle's geometric center as the origin, the front as the positive x-axis, the left as the positive y-axis, and the top as the positive z-axis; the mouth reference point coordinates are obtained by taking the center of each seat headrest as the reference, moving 70 mm forward along the x-axis, 120 mm downward along the z-axis, and placing the y-axis coordinate on the center line of the seat; the driver's seat, front passenger seat, rear left seat, rear middle seat, and rear right seat are labeled as seats one to five in sequence; the three-dimensional coordinates of each array pickup unit in the microphone array are recorded; the direct propagation path from the mouth reference point of each seat to each array pickup unit and the reflection path generated on the reflection surface are defined as the fixed sound path in the car.

4. The method for picking up and mixing zoning sounds of a car-mounted KTV without microphones for multiple passenger seating areas as claimed in claim 3 is characterized in that: In step 2, the seat reference coordinates, the reference point coordinates of each seat mouth, and the three-dimensional coordinates and spatial pointing directions of each array pickup unit are read. The front windshield, rear windshield, left window, right window, instrument panel, door interior panel, roof lining, and floor are determined to be reflective surfaces, and the seat backs, headrests, instrument panel body, center console, and trunk partition are determined to be impenetrable bodies. For each seat and each array pickup unit, the direct propagation path, primary reflection path, and secondary reflection path are enumerated, and samples are taken at 10 mm intervals along the path. If any sampling point falls within the enclosing boundary of the impenetrable body, the path is discarded. The retained paths are assigned an arrival order index from short to long according to their geometric length, and the path type and reflective surface sequence are recorded. At the same time, the alignment delay is converted to an integer sampling point using a sound speed of 343 meters per second and a sampling rate of 48,000 Hz.

5. The method for picking up and mixing zoning sounds of a car-mounted KTV without microphones for multiple passenger seating areas as claimed in claim 4 is characterized in that: In step 2, the process of generating the seat area sound field topology map based on the fixed sound path in the car includes: performing seat area crossing detection on the reserved path, and the outer boundary of the non-seat area is extended by 50 mm in each direction. If at least one crossing occurs, the leakage risk level is marked as high; calculating the minimum distance of the path to the reference point of the mouth of the non-seat area, if it is less than or equal to 0.25 meters, it is marked as high; if it is between 0.25 meters and 0.40 meters, it is marked as medium; calculating the angle between the incident direction of the path and the spatial pointing direction of the array pickup unit, if the non-seat area path is less than or equal to 25 degrees, the leakage risk level is marked as high, If it is located between 25 degrees and 50 degrees, the leakage risk level is marked as medium; the secondary reflection path is additionally marked with a medium leakage risk level; the mouth reference point and the array pickup unit are used as nodes, the retained path is used as the edge, and the arrival order index, path type, reflection surface sequence, delayed sampling point and leakage risk level attributes are attached; in each seat area sub-graph, it is organized in layers, first sorted by the arrival order index, and then sorted by the path type, with direct propagation taking precedence over primary reflection, and primary reflection taking precedence over secondary reflection. Within the same type, it is sorted from low to high by the leakage risk level to obtain the seat area sound field topology map.

6. The method for picking up and mixing zoning sounds of a car-mounted KTV without microphones for multiple passenger seating areas as claimed in claim 5 is characterized in that: In step 2, the process of forming a virtual microphone set based on the sound field topology diagram includes: selecting the array pickup unit with the smallest arrival order index and the direct propagation path as the main channel frame by frame under the frame length of 20 milliseconds and the frame shift of 10 milliseconds; if the short-term energy of the main channel is lower than the silent baseline by more than 6 decibel threshold, the direct propagation path with the smaller index is selected in sequence; if the direct propagation path does not meet the conditions, the reflection path is selected again; the corresponding delayed sampling point alignment is applied to the selected channel, and the channel switching adopts a 5 millisecond linear crossover transition; when the evaluated speech clarity meets the preset rules, the arrival order index is not greater than 3 and the leakage risk level is low or At most two of the primary reflection paths are selected as support channels and mixed in with a gain 6 dB lower than the main channel; a fixed attenuation of 12 dB is applied to the incident direction of non-seat areas with a high leakage risk level; the support channel is suspended when the short-term energy is lower than the silent baseline by 3 dB for 300 consecutive milliseconds, and its direction is added to the side suppression direction set when the similarity with the virtual microphone in the adjacent seat reaches 0.8 and lasts for more than 100 milliseconds; each seat outputs at least one main virtual microphone. If there are two symmetrical direct propagation paths with an arrival index difference of no more than 2 and a low leakage risk level, an additional auxiliary virtual microphone is output.

7. The method for picking up and mixing zoning sounds of a car-mounted KTV without microphones for multiple passenger seating areas as claimed in claim 6, characterized in that: In step 2, continuous vocal candidate segments are selected on the main virtual microphone of each seat according to the frame length of 20 milliseconds, frame shift of 10 milliseconds and sampling rate of 48,000 Hz, and the beat key points, energy envelope, frequency band energy ratio and rise and fall transients are extracted to form a lead singer template; leakage samples are collected in adjacent seats based on the seat sound field topology map and arrival order index, and aligned according to the number of delayed sampling points, and the lag, frequency band energy ratio and direction consistency are recorded to form a leakage template.

8. The method for picking up and mixing zoning sounds of a car-mounted KTV without microphones for multiple passenger seating areas as claimed in claim 7 is characterized in that: In step 2, the lead vocal similarity and leakage similarity are calculated using a sliding window on the partitioned sound pickup stream, along with the lead vocal template and the leakage template, respectively. The echo similarity is then calculated by combining the accompaniment beat repetition and the number of sampling points of the retained path delay. Each window is initially judged as a lead vocal shadow, echo shadow, or leakage shadow based on a preset threshold. Conflicts are resolved by prioritizing scores and inheriting from the previous window, with the confirmation criterion being three consecutive windows. If a lead vocal shadow is determined, a graded attenuation is applied to path components with arrival order indices starting from 2, which are positively correlated with the arrival order index. Additional attenuation is applied to paths with a high leakage risk level. If an echo shadow is determined, uniform attenuation is applied within ±15 milliseconds of the beat repetition, and additional attenuation is applied to paths with arrival order indices of 3 or greater. If a leakage shadow is determined, the source path and its neighboring paths are graded and suppressed with a fixed attenuation amount. A rise and fall edge transition and a maximum total attenuation cap are set for all suppression operations. Generates a timestamp-equipped shadow category label sequence and a suppression instruction sequence, and outputs a suppressed partition pickup stream.

9. The method for picking up and mixing zoning sounds of a car-mounted KTV without microphones for multiple passenger seating areas as claimed in claim 8, characterized in that: In step 2, frame-level detection is performed on the partitioned sound pickup streams and virtual microphones of all seating areas. When two or more seating areas are confirmed to overlap with the lead singer's shadow within the same time window, the occluded syllable segment of the target seating area is determined by combining the envelope reference and beat key points of the lead singer template and the leakage template. Resonant peaks are extracted in the front and back neighborhoods of the occluded syllable segment, and a resonance gating map of open, half-open, and closed states is formed according to the frequency band and time. A side inhibition mark is set at the corresponding lag position based on the leakage template of the adjacent seating area to trigger subsequent leakage compensation. Baseline restoration of the occluded syllable segment was performed using the lead vocal template as the envelope reference; In the time slice marked as open in the resonance gated spectrum, the unobstructed detail segments of the co-seat area are re-injected according to the level and boundary transition rules; when the lateral inhibition mark is triggered, the matching leakage template is introduced for compensation and superposition to obtain the reconstruction result.

Images