Children inquiry method and device based on large language model, medium and program product

Abstract

The invention discloses a children inquiry method and device based on a large language model, a medium and a program product, relates to the field of intelligent auxiliary medical treatment, and aims to solve the problems of voice distortion and inaccurate positioning caused by the fact that a single-channel noise reduction method cannot give consideration to voice quality and spatial information in an intelligent inquiry process. The method comprises the steps of performing short-time Fourier transform on inquiry voice data to obtain spatial information of the inquiry voice data to obtain a multi-channel time-frequency mask, obtaining a voice feature matrix based on the multi-channel time-frequency mask and a short-time Fourier transform coefficient, generating a generalized mutual information entropy value through kernel density estimation, quantifying direction consistency between channels, and obtaining a multi-channel mutual information entropy value. Noise reduction can be ensured, spatial information is effectively reserved, and the efficiency of intelligent inquiry is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent assisted medical care, and more specifically, to a method, device, medium, and program product for pediatric medical consultation based on a large language model. Background Technology

[0002] With the deepening application of artificial intelligence technology in the medical field, intelligent consultation systems based on large language models have shown great potential. However, in the diagnosis and treatment of children, a special group, the effectiveness of such systems faces severe challenges. First, because children's language abilities are not yet mature and they are easily affected by emotions, their speech is often ambiguous and incoherent. Second, the consultation room environment is usually noisy, resulting in a low signal-to-noise ratio of the collected children's speech signals, accompanied by severe environmental reverberation and background noise, which brings core difficulties to the speech recognition module.

[0003] While existing technologies (taking CN120727013A as an example) incorporate multimodal and knowledge graph approaches, their underlying core algorithms still suffer from fundamental flaws: single-channel noise reduction sacrifices speech quality and spatial information; sound source localization methods experience drastic performance degradation under noise and reverberation, and rely on prior knowledge of the number of sound sources; multi-task learning mechanisms become unstable due to feature conflicts; and simple weighted and multi-objective optimization ignores the distribution consistency between tasks, resulting in insufficient generalization ability. These flaws collectively limit the effective application of this technology in high-noise, high-interference real-world scenarios. Summary of the Invention

[0004] This invention is an auxiliary diagnostic method based on multi-source information fusion. First, the diagnostic speech data is frequency-domain transformed by short-time Fourier transform and combined with multi-channel time-frequency masking to obtain processed speech data; then, modeling is performed based on the processed speech data, and the data is input into a large language model to obtain a diagnostic file.

[0005] This invention discloses a speech signal processing method, comprising:

[0006] S1: Obtain consultation voice data from N channels, where N is a natural integer;

[0007] S2: Calculate the short-time Fourier transform amplitude spectrum and phase spectrum of the consultation voice data of the N channels to obtain the Fourier transform coefficients of each channel, and calculate the coherence of the short-time Fourier transform amplitude spectrum among the N channels to obtain weighting coefficients. Combine the Fourier transform coefficients and the weighting coefficients with the cosine function of the phase difference between the channels to obtain a multi-channel time-frequency mask;

[0008] S3: Multiply the Fourier transform coefficients of the multi-channel time-frequency mask to obtain the speech feature matrix, and calculate the generalized mutual information entropy value of the speech feature matrix through kernel density estimation;

[0009] Further, the calculation of the generalized mutual information entropy value of the speech feature matrix by kernel density estimation is specifically: based on the speech feature matrix, the joint probability distribution and the marginal probability distribution are obtained by kernel density estimation calculation of the probability density of each sample point through the Gaussian kernel function, and the phase weight coefficient is combined to generate the generalized mutual information entropy value representing the direction correlation of the channel pair.

[0010] S4: generating a sound source direction feature vector by aggregating the generalized mutual information entropy values.

[0011] Further, the S4 is specifically: aggregating the N channel pair generalized mutual information entropy values and weighting the azimuth angle prior vector to generate a vector representing the sound source direction probability distribution, and then normalizing the vector through an activation function to generate a sound source direction feature vector, thereby obtaining processed speech data; the azimuth angle prior vector is calculated based on the microphone array geometry.

[0012] Optionally, the activation function is selected from one or more of the following activation functions: Softmax function, tanh function, ReLU activation function, Sigmoid function, SeLU function, ELU function, Swish function, Maxout activation function, Softplus function.

[0013] Further, the speech signal processing method further comprises:

[0014] performing multi-head attention calculation on the sound source direction feature vector to obtain output features of M attention heads, M being a natural number integer;

[0015] stitching the output features of the M attention heads to obtain a sound source azimuth probability distribution, and generating a sound source feature through a sound source azimuth discriminator to obtain processed speech data.

[0016] Further, the multi-head attention calculation on the sound source direction feature vector to obtain output features of M attention heads further comprises: performing feature extraction on the speech feature matrix through deep convolution to obtain deep separable convolution output features; and stitching the deep separable convolution output features and the sound source direction feature vector and performing multi-head attention calculation to obtain output features of M attention heads, M being a natural number integer.

[0017] Further, the stitching of the deep separable convolution output features and the sound source direction feature vector and the multi-head attention calculation to obtain output features of M attention heads are specifically:

[0018] S31: splice the deep separable convolution output feature and the sound source direction feature vector into a unified representation, input the unified representation into a gated recurrent unit, fuse the time-dependent information through an update gate and a reset gate mechanism, and output a shared encoding sequence;

[0019] S32: generate a task-specific query matrix, a key matrix and a value matrix based on the shared encoding sequence, fuse the sound source direction feature bias through scaled dot-product attention, and calculate the output feature of M attention heads.

[0020] Further, the sound source feature generated by the sound source orientation discriminator and the processed speech data further comprise:

[0021] The processed speech data are respectively subjected to full connection layer, linear transformation and activation function to obtain noise reduction speech time-frequency representation, sound source orientation probability distribution and text sequence probability distribution.

[0022] The noise reduction speech time-frequency representation, the sound source orientation probability distribution and the text sequence probability distribution are respectively generated by the noise reduction speech discriminator, the sound source orientation discriminator and the text sequence discriminator to obtain noise reduction features, sound source features and text sequence features to obtain processed speech data.

[0023] The application also discloses a construction method of a speech signal processing model, comprising:

[0024] S61: obtaining N-channel inquiry speech data and real speech labels, N being a natural integer;

[0025] S62: inputting the N-channel inquiry speech data into the speech signal processing method to obtain processed speech data;

[0026] S63: inputting the processed speech data and the real speech labels into a generator and a discriminator respectively to calculate an adversarial loss, optimizing the adversarial loss, and obtaining the speech signal processing model.

[0027] The application also discloses an auxiliary inquiry method based on a large language model, comprising:

[0028] S71: obtaining N-channel inquiry speech data, patient test report data and case document images;

[0029] S72: inputting the inquiry speech data into the speech signal processing model to obtain processed speech data;

[0030] S73: inputting the processed speech data, the patient test report data and the case document images into a large language model to read context and generate a diagnosis file.

[0031] The application further discloses an auxiliary diagnosis device based on multi-source information fusion, which comprises a memory and a processor.

[0032] The memory is used for storing program instructions.

[0033] The processor is used for calling the program instructions, so as to realize the steps of the voice signal processing method, or realize the steps of the construction method of the voice signal processing model, or realize the steps of the auxiliary diagnosis method based on the large language model.

[0034] The application further discloses a computer program product comprising a computer program, which is used for realizing the steps of the voice signal processing method, or realizing the steps of the construction method of the voice signal processing model, or realizing the steps of the auxiliary diagnosis method based on the large language model.

[0035] The application further discloses a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is used for realizing the steps of the voice signal processing method, or realizing the steps of the construction method of the voice signal processing model, or realizing the steps of the auxiliary diagnosis method based on the large language model.

[0036] The application has the following advantages:

[0037] 1. In order to solve the problems that the single-channel noise reduction method cannot consider the voice quality and spatial information in the intelligent diagnosis process, leading to voice distortion and inaccurate positioning, the application obtains the amplitude spectrum and phase spectrum by performing short-time Fourier transform on the diagnosis voice data, obtains the multi-channel time-frequency mask based on the phase spectrum, obtains the voice feature matrix by combining the multi-channel time-frequency mask with the amplitude spectrum and phase spectrum, generates the generalized mutual information entropy value by performing kernel density estimation on the voice feature matrix, and quantifies the directional consistency between channels, so that the noise reduction can be ensured, the spatial information can be effectively preserved, the limitations of the traditional positioning method in the noise and reverberation environment can be successfully overcome, the sound source direction feature can be dynamically generated, and accurate positioning can be realized.

[0038] 2. In view of the problems that the multi-task learning mechanism in the prior art is unstable due to feature conflicts, and simple weighted sum and multi-objective optimization ignore the distribution consistency between tasks, resulting in insufficient generalization ability, the present application can optimize the shared encoding representation of speech and direction features, reduce model complexity and improve the effect of multi-task learning, further improve the precision of the multi-task decoder, and solve the gradient conflict and overfitting problems in the traditional method in multi-task learning, by performing deep separable convolution on the speech feature matrix and splicing it with the sound source direction feature vector, calculating multi-head attention output features, generating noise-reduced speech time-frequency representation, sound source probability distribution and text sequence probability distribution based on the multi-head attention output features, and inputting them into a generative adversarial network for optimization. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0040] Figure 1 is a schematic flow chart of the speech signal processing method provided by the embodiments of the present application;

[0041] Figure 2 is a performance comparison schematic diagram of different noise reduction methods provided by the embodiments of the present application;

[0042] Figure 3 is a positioning accuracy comparison schematic diagram in a multi-source scene provided by the embodiments of the present application;

[0043] Figure 4 is a schematic flow chart of the construction method of the speech signal processing model provided by the embodiments of the present application;

[0044] Figure 5 is a precision comparison schematic diagram of different sound source positioning methods in different azimuth angles provided by the embodiments of the present application;

[0045] Figure 6 is a generalization ability comparison schematic diagram of different methods in a diversity test scene provided by the embodiments of the present application;

[0046] Figure 7 is a schematic flow chart of the auxiliary diagnosis method based on the large language model provided by the embodiments of the present application;

[0047] Figure 8 is a schematic diagram of a computing device based on intelligent retrieval of industrial intelligence data provided by the embodiments of the present application. DETAILED DESCRIPTION

[0048] In order to make the person skilled in the art better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0049] In some of the processes described in this specification and in the claims and in the above description, the processes can occur in different order than described. It is to be understood that the ordering only implies no certain order. Also, some of the processes can occur concurrently, be divided into sub-processes, be performed in an iterative fashion, or be performed at different times, as can be advantageous. The processes described herein are also intended to cover any process that can be modified or changed in light of the above description.

[0050] The technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0051] Figure 1 is a schematic flowchart of the voice signal processing method provided by the embodiments of the present application, and specifically, comprises the following steps:

[0052] S1: acquiring N-channel inquiry voice data, N being a natural integer;

[0053] In an embodiment, the acquisition of the inquiry voice data comprises:

[0054] A multi-channel voice training data set is constructed for a child inquiry scene. The data collection process is carried out in a simulated medical environment. A six-channel microphone array is used to record the dialogue voice of a doctor and a sick child. The sampling rate is set to 16 kHz to cover the main frequency range of human voice.

[0055] The recording environment contains typical medical scene noise, including equipment operation sound, personnel walking sound and background conversation sound, to simulate real inquiry conditions.

[0056] S2: calculating the short-time Fourier transform amplitude spectrum and phase spectrum of the N-channel inquiry voice data, obtaining the Fourier transform coefficient of each channel, and calculating the short-time Fourier transform amplitude spectrum coherence between the N channels to obtain a weighting coefficient, and combining the Fourier transform coefficient, the weighting coefficient and the cosine function of the phase difference between the channels to obtain a multi-channel time-frequency mask;

[0057] In a specific embodiment, the specific formula of S2 is:

[0058] ,

[0059] in, Indicates the first The first channel in the Frequency and the The time-frequency mask value of the time frame is used to enhance the target speech and preserve spatial information; This is the channel index, and its value range is... arrive Channel identifiers that represent multi-channel data are used to distinguish different microphones; This indicates the total number of microphone channels, representing the number of microphones in the array; the default setting. ; This is a frequency index, with a value range of [value range missing]. arrive The frequency points that characterize the short-time Fourier transform determine the frequency resolution. Indicates the number of frequency points, representing the frequency resolution; default setting. ; This is the time frame index, and its value range is... arrive The time frame identifier, which characterizes the short-time Fourier transform, determines the time resolution; Indicates the total number of time frames, representing the signal duration; Indicates the first The short-time Fourier transform coefficients of the i-th channel are complex numbers containing amplitude and phase information. This is achieved by... The speech signal from each channel is obtained by performing a short-time Fourier transform. Indicates the first The magnitudes of the short-time Fourier transform coefficients of each channel characterize the signal in the th... Frequency and the Energy of a time frame; Indicates the first The first channel and the first The weighting coefficients between channels are calculated based on amplitude spectrum correlation and are used to measure the similarity between channels. The larger the value, the higher the similarity. To distinguish from The channel index, with a value range of 100. arrive ; Indicates the first The first channel and the first The phase difference between each channel is calculated as follows: It represents the time difference of sound waves arriving at different channels and is used to capture spatial information; This indicates a phase extraction operation, which extracts the phase angle of a complex number, in radians. Indicates the first The short-time Fourier transform coefficients of each channel; Represents the cosine function; This indicates a preferred setting to prevent division by zero by a small constant and ensure numerical stability. ; Indicates the first The short-time Fourier transform coefficients of each channel; Indicates the first The magnitude of the short-time Fourier transform coefficients of each channel; To distinguish from and The channel index, with a value range of 100. arrive .

[0060] In one specific embodiment, the short-time Fourier transform coefficients of each channel are obtained through short-time Fourier transform, with the first... Taking one channel as an example, the voice signal is first divided into frames, preferably with a frame length of [value missing]. Frame shift is Then, a Fourier transform is applied to each frame of the signal to obtain complex time-frequency coefficients. For example, for a sampling rate of... The voice, frame length Points, Fourier transform points ,but Indicates the first The first frame frequency The complex value of , where frequency It refers to the frequency unit corresponding to the frequency index, that is, the frequency range represented by each frequency point.

[0061] S3: Multiply the Fourier transform coefficients of the multi-channel time-frequency mask to obtain the speech feature matrix, and calculate the generalized mutual information entropy value of the speech feature matrix through kernel density estimation;

[0062] In one embodiment, multiplying the multi-channel time-frequency mask with the amplitude spectrum and the phase spectrum to obtain the speech feature matrix specifically involves multiplying the multi-channel time-frequency mask with the short-time Fourier transform coefficients of the consultation speech data to obtain the speech feature matrix. The short-time Fourier transform coefficients are obtained by performing short-time Fourier transform on N channels of the consultation speech data.

[0063] In a specific embodiment, the formula for multiplying the multi-channel time-frequency mask with the amplitude spectrum and the phase spectrum to obtain the speech feature matrix is ​​as follows:

[0064] ,

[0065] in, Indicates the first enhanced speech features of the first channel enhanced signal energy and phase of the first channel at the first frequency and the first time frame, are complex time-frequency coefficients that preserve the phase information of the original speech while the amplitude spectrum is adjusted by the time-frequency mask to suppress noise and enhance the target speech.

[0066] Figure 2 is a performance comparison diagram of different noise reduction methods provided by the embodiments of the present application, which is a comparison of time-frequency domain noise reduction effects of the present application and directly shows the noise reduction performance of the multi-channel time-frequency mask joint optimization method proposed by the present application in a medical environment. The experiment compares four cases: pure speech (ideal state without noise), spectral subtraction (traditional single-channel noise reduction technology), beamforming (conventional multi-channel noise reduction technology), and the method of the present application. The experimental configuration uses six-channel microphone array to collect children's speech data with a sampling rate of 16000 Hz, simulates typical device noise (500 Hz) and background human voice interference (1200 Hz) in a medical environment. In the frequency spectrum, the horizontal axis represents frequency (Hz), and the vertical axis represents power (dB). The pure speech spectrum clearly shows the 250 Hz fundamental frequency and its harmonic components (500 Hz, 750 Hz, etc.). Although the spectral subtraction suppresses part of the noise, there are still obvious noise peaks near 500 Hz, and the speech harmonic structure is broken. The beamforming has significant noise residues at 1200 Hz, and the speech harmonic amplitude is reduced as a whole. In contrast, the frequency spectrum of the method of the present application is closest to the pure speech: the noise peaks at 500 Hz and 1200 Hz are completely suppressed, while the 250 Hz fundamental frequency and its harmonic structure are completely preserved, verifying that the time-frequency mask joint optimization of the present application can reduce noise while protecting the phase information of the speech, avoiding the speech distortion problem caused by traditional methods.

[0067] In one embodiment, the calculation of the generalized mutual information entropy value of the speech feature matrix through kernel density estimation is specifically: based on the speech feature matrix, the joint probability distribution and the marginal probability distribution of each sample point are obtained by kernel density estimation through a Gaussian kernel function, and the generalized mutual information entropy value representing the direction correlation of the channel pair is generated in combination with the phase weight coefficient.

[0068] In a specific embodiment, the formula for calculating the generalized mutual information entropy value of the speech feature matrix through kernel density estimation is specifically:

[0069] ,

[0070] wherein, represents the enhanced signal energy and phase of the first channel and the second channel at the first frequency and the first time frame, are complex time-frequency coefficients that preserve the phase information of the original speech while the amplitude spectrum is adjusted by the time-frequency mask to suppress noise and enhance the target speech. The generalized mutual information entropy value of the time frame is used to measure the consistency of the sound source direction, and the greater the value is, the higher the direction consistency is; represents the empirical probability distribution based on the enhanced speech feature, which is used to estimate the probability density of the feature, and is calculated by kernel density estimation; represents the enhanced speech feature of the th channel; represents the phase weight coefficient, which is used to balance the contribution of the mutual information term and the phase difference term, and is preferably set to ; represents the logarithmic function, and the default base is the natural constant, represents the phase difference between the th channel and the th channel, represents the phase difference between the th channel and the th channel.

[0071] In a specific embodiment, the empirical probability distribution based on the enhanced speech feature uses a Gaussian kernel function to estimate the smoothed probability density of the time-frequency point of the enhanced speech feature. Specifically, first, the amplitude and phase values of and are regarded as sample points, and then the probability density of each sample point is calculated by a Gaussian kernel function, so as to obtain the joint probability distribution and the marginal probability distribution , The kernel bandwidth is adaptively selected by the criterion to ensure the stability of the estimation.

[0072] S4: generating a sound source direction feature vector by aggregating the generalized mutual information entropy values.

[0073] In an embodiment, the S4 is specifically: generating a vector representing the probability distribution of the sound source direction by aggregating the generalized mutual information entropy values of the N channels and weighting the azimuth angle prior vector, and then generating the sound source direction feature vector by normalizing the vector through an activation function, to obtain the processed speech data; the azimuth angle prior vector is calculated based on the microphone array geometry.

[0074] In a specific embodiment, the specific formula of the S4 is:

[0075] ,

[0076] wherein, represents the sound source direction feature vector of the th time frame, and the dimension is , representing the The probability distribution of the time frame sound source from different azimuth partitions is generated based on generalized mutual information entropy and azimuth prior, and can dynamically capture the consistency of the sound source direction without pre-assumption of the number of sound sources. represents the number of azimuth partitions, representing the direction resolution accuracy, preferably set to ; represents function; represents the azimuth prior vector corresponding to the th channel and the th channel, with a dimension of , calculated based on the geometry of the microphone array, representing the direction sensitivity of the channel pair.

[0077] In a specific embodiment, the azimuth prior vector corresponding to the th channel and the th channel is pre-calculated based on the geometric layout of the microphone array, without real-time training, and is a fixed vector for the th channel pair, used to represent the direction sensitivity of the channel pair in different azimuth partitions. Specifically, by measuring or simulating the sound wave response of the microphone array at different azimuths, such as phase difference or time difference expectation, a weight vector is assigned to each channel pair, and the higher the weight value, the more sensitive the channel pair is to the corresponding azimuth of the sound source. In a linear array, the distance and angle relationship between the channel pairs can be used to generate the prior vector, and normalization processing is used to ensure numerical stability.

[0078] Figure 3 is a positioning accuracy comparison diagram provided by the embodiments of the present application in a multi-source scene. The present application compares the positioning accuracy of multiple sound sources, and evaluates the performance of the sound source positioning method based on generalized mutual information entropy in complex medical environments. The experiment simulates three real sound sources in a clinic: a doctor (60-degree direction), a sick child (150-degree direction), and an interference sound source (240-degree direction). The comparison methods include time difference positioning (traditional sound time difference method), fixed beamforming (conventional spatial filtering technology), and the method of the present application. All methods process the same six-channel noisy speech data, including device noise and personnel walking sound. In the polar coordinate graph, the golden five-point star represents the true sound source azimuth. The time difference positioning result (blue square) has a significant deviation: the doctor's position is shifted by about 20 degrees, and the child's position is shifted by nearly 30 degrees. Fixed beamforming (orange triangle) is more accurate in positioning the doctor, but the child's position is shifted by about 15 degrees, and the interference source is shifted by about 20 degrees. The method of the present application (green dot) is closest to the true position: the doctor's azimuth error is about 5 degrees, the child's is about 8 degrees, and the interference source is about 10 degrees. The connection line directly shows that the positioning error of the present application is the smallest, proving that the generalized mutual information entropy method can effectively overcome noise and reverberation interference, and can achieve accurate positioning without prior knowledge of the number of sound sources.

[0079] Optionally, the activation function is selected from one or more of the following activation functions: Softmax function, tanh function, ReLU activation function, Sigmoid function, SeLU function, ELU function, Swish function, Maxout activation function, Softplus function.

[0080] In one embodiment, the speech signal processing method further comprises:

[0081] The sound source direction feature vector is subjected to multi-head attention calculation to obtain output features of M attention heads, M being a natural number integer.

[0082] The output features of the M attention heads are spliced to obtain a sound source orientation probability distribution, and a sound source feature is generated by a sound source orientation discriminator to obtain processed speech data.

[0083] In one embodiment, the multi-head attention calculation of the sound source direction feature vector to obtain output features of M attention heads further comprises: feature extraction of the speech feature matrix through deep convolution to obtain deep separable convolution output features; splicing and multi-head attention calculation of the deep separable convolution output features and the sound source direction feature vector to obtain output features of M attention heads, M being a natural number integer.

[0084] In one embodiment, the feature extraction of the speech feature matrix through deep convolution to obtain deep separable convolution output features specifically comprises: feature extraction of the speech feature matrix through deep convolution to obtain spatial features, fusion of spatial features through point convolution to fuse channel information, and obtaining of deep separable convolution output features fused with spatial and channel information through an activation function.

[0085] In a specific embodiment, the specific formula for the feature extraction of the speech feature matrix through deep convolution to obtain deep separable convolution output features is:

[0086] ,

[0087] wherein, represents the deep separable convolution output feature of the i-th channel, and represents the spatial and channel fusion feature of the i-th channel speech signal; represents the deep separable convolution output feature of the i-th channel, and represents the spatial and channel fusion feature of the i-th channel speech signal; represents the deep convolution kernel weight, which is a trainable parameter, for extracting spatial features; represents the convolution operation; represents the deep convolution bias, which is a trainable parameter, for adjusting the baseline of the convolution output; represents the point convolution kernel weight, which is a trainable parameter, for channel feature fusion;​ This represents the modified linear unit activation function.

[0088] In one embodiment, the step of concatenating the depthwise separable convolutional output features and the sound source direction feature vector and performing multi-head attention calculation to obtain the output features of M attention heads specifically involves:

[0089] S31: The depthwise separable convolution output features are concatenated with the sound source direction feature vector to form a unified representation. The unified representation is input into a gated recurrent unit. Temporal dependency information is fused through update gate and reset gate mechanisms to output a shared coding sequence.

[0090] In a specific embodiment, the specific formula for S31 is:

[0091] ,

[0092] in, This represents the shared coded sequence of the t-th time frame, with dimension . , which serves as the input for the subsequent decoder; This represents the concatenation of depthwise separable convolution output features across all channels, consisting of features from the first convolution. Depth-separable convolution output characteristics of each channel The output features of convolution are concatenated along the channel dimension; Represents the encoding dimension, characterizing the capacity of the feature representation, and is preferably set. ; Gated loop units capture long-term dependencies through gating mechanisms; This indicates a feature concatenation operation that connects features from different modalities into a unified representation. Indicates the first A shared encoded sequence of time frames, used to transmit historical information. hour Dimensions .

[0093] In one specific embodiment, the gated recurrent unit is a recurrent neural network that includes update gates and reset gates. The update gate controls the degree to which historical information is retained, and the reset gate controls the degree to which the current input is combined with historical information. Through the gating mechanism, the gated recurrent unit can adaptively capture temporal dependencies, avoid the gradient vanishing problem, and reduce the number of parameters.

[0094] S32: Generate a task-specific query matrix, key matrix, and value matrix based on the shared encoding sequence, and calculate the output features of M attention heads by scaling dot product attention fusion of sound source direction feature bias.

[0095] In a specific embodiment, the specific formula for S32 is:

[0096] ,

[0097] in, Indicates the first The output of each attention head is used as a shared feature of the multi-task output; This represents the directional bias weighting coefficient, used to balance content attention and directional attention; a preferred setting is... ; Indicates the first The directional projection matrix of each attention head is a trainable parameter with dimension . This is used to project the transpose of the sound source direction feature matrix onto the attention space and fuse it with the query-key interaction; The sound source direction feature matrix is ​​represented by the matrix containing the first... Sound source direction feature vector of time frame It is composed of the sound source direction feature vectors of all time frames, including the one with dimension of . ; This represents the transpose of the sound source direction feature matrix; This is the attention head index, with a value range of [value range missing]. arrive Identifiers representing different attention mechanisms; This represents the total number of attention heads, characterizes model complexity, and is a preferred setting. ; This represents the query matrix, calculated as follows: ; The key matrix is ​​represented as follows: ; Represents the transpose of the key matrix; The value matrix is ​​represented as follows: ; Indicates the first The query transformation weight matrix of each attention head is a trainable parameter used to linearly transform the shared encoding sequence into a query matrix. Indicates the first The key transformation weight matrix of each attention head is a trainable parameter used to linearly transform the shared coding sequence into a key matrix; Indicates the first The value transformation weight matrix of each attention head is a trainable parameter used to linearly transform a shared coding sequence into a value matrix; Indicates a shared encoded sequence, consisting of the first... It is composed of the shared encoded sequences of all time frames, including the shared encoded sequence of the time frame, with a dimension of [missing information]. ; Key dimension, feature scaling factor, preferred settings . The term serves as a directional bias, used to incorporate sound source directional features into the attention weights, helping the model prioritize directional features during decoding.

[0098] In one embodiment, the step of generating sound source features through a sound source location discriminator to obtain processed speech data further includes:

[0099] The processed speech data is passed through a fully connected layer, linear transformation and activation function to obtain the time-frequency representation of the denoised speech, the probability distribution of the sound source location and the probability distribution of the text sequence.

[0100] The specific formulas for obtaining the time-frequency representation of the denoised speech, the probability distribution of the sound source location, and the probability distribution of the text sequence by passing the processed speech data through a fully connected layer, linear transformation, and activation function are as follows:

[0101] ,

[0102] ,

[0103] ,

[0104] in, The time-frequency representation of the denoised speech is the enhanced speech signal, with dimensions of [missing information]. ; The feature reshaping operation rearranges the output of the denoising task into dimensions. To match the time-frequency representation of the original speech signal; This represents the probability distribution of sound source location, which is a probability distribution of sound source location with dimension . ; This represents the probability distribution of a text sequence, and its dimension is . ; Indicates the size of the vocabulary; The output weight matrix represents the noise reduction task and consists of trainable parameters used to linearly transform the spliced ​​attention head output into a noise-reduced speech signal. The output weight matrix represents the sound source localization task. It is a trainable parameter used to linearly transform the spliced ​​attention head output into the sound source location probability. The output weight matrix represents the speech recognition task and consists of trainable parameters used to linearly transform the concatenated attention head output into text sequence probabilities. This represents the output of the first attention head. Indicates the first The output of each attention head; Indicates a splicing operation; This represents the output bias of the noise reduction task; it is a trainable parameter used to adjust the baseline of the output layer. an output bias for the speech recognition task, which is a trainable parameter to adjust the baseline of the output layer; an output bias for the speech recognition task, which is a trainable parameter to adjust the baseline of the output layer; an output bias for the speech recognition task, which is a trainable parameter to adjust the baseline of the output layer; an activation function.

[0105] In one specific embodiment, the vocabulary is constructed from the training data, containing all possible characters or subword units. For example, in the pediatric consultation scenario, the vocabulary can cover medical terms and daily language, which is generated by counting the frequency of characters in the training text. The size of the vocabulary is usually several hundred to several thousand.

[0106] Figure 5 is a precision comparison diagram of different sound source positioning methods provided by the embodiments of the present application under different azimuth angles. The present application evaluates the azimuth recognition accuracy of different sound source positioning methods in the pediatric consultation scenario. The experiment compares four positioning technologies: time difference positioning is a traditional method of estimating the azimuth based on the time difference of sound arriving at different microphones; beamforming finds the direction with the maximum energy through spatial scanning; the traditional mutual information method uses the statistical correlation between channels; and the method proposed in the present application uses generalized mutual information entropy to dynamically generate sound source direction features by quantifying the direction consistency between channels. The experimental results show that the horizontal coordinate represents the actual azimuth of the sound source, with a unit of degrees, covering 36 test points in all directions; the vertical coordinate is the positioning error, also with a unit of degrees, and the smaller the value, the more accurate the positioning. From the column chart, it can be seen that the positioning error of the method of the present application under different azimuth angles is significantly smaller than that of other methods, showing better stability and accuracy. The traditional time difference positioning method is sensitive to noise, with large error fluctuations; beamforming performs well at some angles, but its performance decreases when the sound source direction coincides with the array axis; the traditional mutual information method has improved, but it is still inferior to the method of the present application. The method of the present application can better maintain positioning accuracy in noisy and reverberation environments by using generalized mutual information entropy combined with phase difference weighting, which has important practical significance for accurately distinguishing between doctor-patient conversations.

[0107] The noise-reduced speech time-frequency representation, the sound source azimuth probability distribution, and the text sequence probability distribution are respectively processed by a noise-reduced speech discriminator, a sound source azimuth discriminator, and a text sequence discriminator to generate noise-reduced features, sound source features, and text sequence features to obtain processed speech data.

[0108] In one specific embodiment, the specific formula for generating noise-reduced features, sound source features, and text sequence features from the noise-reduced speech time-frequency representation, the sound source azimuth probability distribution, and the text sequence probability distribution by the noise-reduced speech discriminator, the sound source azimuth discriminator, and the text sequence discriminator is as follows:

[0109] ​

[0110]

[0111]

[0112]

[0113] wherein, represents a noise reduction feature, representing an input noise reduction speech time-frequency, represents a probability that is a real noise reduction speech, the value is closer to the more real; represents the sound source orientation discriminator output, representing the input sound source orientation probability distribution is the probability of a real sound source orientation probability distribution; represents a text sequence feature, representing an input text sequence probability distribution is the probability of a real text sequence probability distribution; represents a sound source feature, representing an input shared encoding sequence is the probability of a real data encoding sequence; represents the convolution kernel weight of the noise reduction discriminator, which is a trainable parameter, and the dimension is , used to extract time-frequency features; represents a convolution operation; represents the bias of the noise reduction discriminator, which is a trainable parameter; represents the fully connected weight of the sound source positioning discriminator, which is a trainable parameter, and the dimension is ; represents the bias of the sound source positioning discriminator, which is a trainable parameter; represents the input weight of the speech recognition discriminator, which is a trainable parameter, and the dimension is ; represents the bias of the speech recognition discriminator, which is a trainable parameter; represents the fully connected weight of the shared encoding discriminator, which is a trainable parameter, and the dimension is ; represents the bias of the shared encoding discriminator, which is a trainable parameter; represents the noise reduction discriminator convolution kernel size, preferably set to ; represents the sound source positioning discriminator hidden layer dimension, preferably set to ; represents the speech recognition discriminator hidden layer dimension, preferably set to ; represents the shared encoding discriminator hidden layer dimension, preferably set to .

[0114] In a specific embodiment, the noise reduction discriminator uses a three-layer convolutional network to process the time-frequency representation, the sound source positioning discriminator uses a two-layer fully connected network to process the probability vector, the speech recognition discriminator uses a single-layer gated recurrent unit to process the sequence probability, the shared encoding discriminator uses a two-layer fully connected network to process the encoding sequence, all discriminators finally output probability values through a Sigmoid function, and the gradient direction is reversed during back propagation through a gradient reversal layer.

[0115] Figure 4 is a schematic flowchart of a method for constructing a speech signal processing model according to an embodiment of the present application, which specifically comprises:

[0116] S61: Obtain N-channel inquiry speech data and real speech labels, N being a natural integer;

[0117] In an embodiment, the acquisition of the real speech labels comprises:

[0118] Based on the pre-acquired orientation information of each sound source, the relative position of the microphone array and the sound source is accurately measured to generate a sound source orientation label, which is used for subsequent sound source positioning tasks.

[0119] In an embodiment, the acquisition of the labels further comprises a data annotation process, which comprises three main categories:

[0120] Speech transcription text annotation involves professional annotators transcribing the content of the recording sentence by sentence, covering medical terminology and everyday language, to form a text sequence label;

[0121] Sound source orientation annotation divides the sound source direction into 36 azimuth partitions based on the geometric layout of the microphone array, and assigns a corresponding orientation probability distribution to each time frame;

[0122] Noise reduction target annotation generates the time-frequency representation of the target noise reduction speech by synchronously collecting clean speech and noisy speech.

[0123] S62: Input the N-channel inquiry speech data into the speech signal processing method to obtain processed speech data;

[0124] S63: Input the processed speech data and the real speech labels into the generator and the discriminator respectively to calculate the adversarial loss, optimize the adversarial loss, and obtain the speech signal processing model.

[0125] In a specific embodiment, the calculation formula of the adversarial loss is specifically:

[0126]

[0127] ,

[0128] ,

[0129] wherein, represents the generator adversarial loss for training the main model to generate outputs consistent with the real data distribution; represents the discriminator adversarial loss for training the discriminator to distinguish real from generated data; represents the gradient penalty term for enforcing the discriminator to satisfy continuity; represents the task index, representing different tasks;

[0130] represents the set of task indices, i.e. , represents the denoising task, represents the sound source localization task, represents the speech recognition task; represents the real data of task , represents the real denoised speech time-frequency representation, represents the real sound source orientation probability distribution, represents the real text sequence probability distribution; represents the shared encoding sequence of real data, which is obtained by inputting the real data containing clean speech, real sound source orientation and text into the multi-task convolutional recurrent encoder;

[0131] represents the discriminator output of task , represents the denoised speech discriminator, represents the sound source orientation discriminator, represents the text sequence discriminator; represents the random interpolation sample of task output, calculated as ; represents the model output of task , is the denoised speech time-frequency representation, is the sound source orientation probability distribution, is the text sequence probability distribution; represents a uniformly distributed random number in the interval [0, 1]; represents the random interpolation sample of shared encoding, calculated as ; represents the gradient of the discriminator output of task to the random interpolation sample ; represents the gradient of the shared encoding discriminator to the random interpolation sample gradient of the loss function; denotes the L2 norm.

[0132] In one specific embodiment, the gradient reversal layer is an identity transformation during forward propagation and multiplies the gradient by a negative constant during backward propagation, the negative constant is preferably set to realize the adversarial training, the gradient penalty term is calculated by randomly interpolated samples to avoid the discriminator gradient explosion or disappearance.

[0133] In one specific embodiment, the adversarial loss further includes a weighted sum of the task-specific loss, the adversarial regularization loss and the gradient penalty, and the specific formula is as follows:

[0134] ,

[0135] ,

[0136] wherein, denotes the weighted sum of the task-specific loss, which is used to measure the independent performance of the model in the denoising, positioning and identification tasks; denotes the total loss, which combines the task loss, the adversarial regularization loss and the gradient penalty, and is used for overall model optimization; denotes the denoising task loss, and the calculation method is the mean square error loss, and the specific formula is as follows: ,

[0137] denotes the time-frequency representation of the denoised speech at the specific value of the i-th frequency, the j-th time frame and the k-th channel, which is a tensor element of the time-frequency representation of the denoised speech denotes the time-frequency representation of the real denoised speech at the specific value of the i-th frequency, the j-th time frame and the k-th channel, which is a tensor element of the time-frequency representation of the real denoised speech denotes the sound source positioning task loss, and the calculation method is the cross-entropy loss, and the calculation formula is as follows: ,

[0138] denotes the value of the real sound source azimuth probability distribution at the i-th azimuth angle partition and the j-th time frame, which is the measured azimuth angle by the microphone array; denotes the sound source azimuth probability distribution at the specific value of the i-th azimuth angle partition and the j-th time frame, which is a tensor element of the sound source azimuth probability distribution ​​​​​​​​​​azimuth angle partition and the feature value at the time frame; denotes the speech recognition task loss, and the calculation method is the connectionist temporal classification loss . denotes the real text sequence, which is the artificially annotated dialogue content; denotes the noise reduction loss weight, which is used to balance the contribution of the task and is calculated by dynamic adjustment . denotes the moving average of the noise reduction task loss , which is obtained by exponential smoothing calculation; denotes the natural exponential function; denotes the sound source positioning loss weight, which is used to dynamically adjust the contribution of the task in the total loss, and the calculation method is . denotes the moving average of the sound source positioning task loss , which is obtained by exponential smoothing calculation; denotes the speech recognition loss weight, which is used to balance the importance of the speech recognition task, and the calculation method is . denotes the moving average of the speech recognition task loss , which is obtained by exponential smoothing calculation; denotes the adversarial loss weight, which is used to control the regularization strength, and is preferably set to . denotes the gradient penalty weight, which is used to enhance the constraint of the discriminator, and is preferably set to . denotes the connectionist temporal classification function, which is used to process sequence alignment problems.

[0139] In a specific embodiment, the connectionist temporal classification is an algorithm for sequence-to-sequence learning, which is used in the speech recognition task, where the lengths of the input sequence and the output sequence may not be consistent. The connectionist temporal classification avoids explicit alignment data by using blank labels. In a specific implementation, the model output layer uses function to generate character and blank probability distribution at each time step, and then converts multiple possible paths into a final sequence through path mapping rules. The loss calculation aggregates the probabilities of all valid paths through dynamic programming to optimize the model parameters. In inference, greedy decoding or beam search is used to generate output text, so that end-to-end speech recognition training can be achieved without manual alignment.

[0140] In an embodiment, the deep learning model is trained end-to-end based on the constructed training data set. The training process uses the mini-batch gradient descent method, and the optimizer is used to update the model parameters. The initial learning rate is set to and the learning rate is dynamically adjusted with an exponential decay strategy; during the parameter update of the deep learning model, multiple batches of training data are first input into the model, including multi-channel speech signals, sound source direction features, and corresponding true labels, the model output is calculated through forward propagation, including noise-reduced speech time-frequency representation, sound source orientation probability distribution, and text sequence probability distribution; then the adversarial loss function is calculated, the adversarial loss function combines task-specific loss, adversarial regularization loss, and gradient penalty term, the gradient is calculated through the back propagation algorithm, and the gradient reversal layer is applied to reverse the gradient direction of the discriminator in the adversarial training; during the parameter update process, the loss changes on the training set and the validation set are monitored, the early stopping mechanism is used to prevent overfitting: when the total loss of the validation set does not decrease for ten consecutive training periods, the training is terminated and the optimal model parameters are saved; at the same time, the task loss weights are dynamically adjusted during the training process, the contribution of multiple tasks is adaptively balanced based on the moving average of each task loss, and the model is ensured to be optimized on the denoising, positioning, and recognition tasks.

[0141] Figure 6is a generalization capability comparison diagram of different methods provided by the embodiment of the present application in diversified test scenarios, and evaluates the generalization capability of the multi-objective adversarial regularization training method proposed by the present application in diversified test scenarios. The experiment compares four different training strategies: traditional spectral subtraction, deep learning noise reduction, multi-task learning without adversarial training, and adversarial regularization training of the present application. The traditional spectral subtraction is used as the baseline method; the deep learning noise reduction represents the current mainstream technology; the multi-task learning without adversarial training uses the network structure of the present application but does not introduce adversarial regularization; the method of the present application adds adversarial regularization loss on the basis of multi-task learning, and forces the model output to be consistent with the real data in distribution. Five typical medical environment test scenarios are designed in the experiment configuration: quiet environment as the benchmark scene, device running sound simulating medical instrument interference, personnel walking sound simulating clinic personnel activity, multiple reverberation simulating different clinic acoustic characteristics, and child crying simulating the most challenging pediatric scene. The evaluation index is the performance retention rate, that is, the performance retention degree of each method in the challenging scene relative to the performance in the quiet environment, ranging from 0 to 1, and the higher the value, the stronger the generalization capability. The left subgraph of the experimental graph shows the performance retention rate of each method in different test scenarios, and the right subgraph quantitatively analyzes the performance decline of each method in the challenging environment. The experimental results show that the method of the present application maintains the highest performance retention rate in all test scenarios, and the advantage is more obvious in extreme scenarios such as child crying. The performance of the traditional spectral subtraction method decreases sharply in complex scenarios, and the deep learning noise reduction method is improved but still cannot effectively cope with the pediatric environment. Although the multi-task learning without adversarial training has an advanced network structure, it is prone to performance degradation on unknown data due to the lack of regularization constraints. The present application constructs a task-specific discriminator and a shared feature discriminator through a multi-objective adversarial regularization loss, forces the model output to align with the real data distribution, and significantly improves the generalization capability of the system in diversified medical environments. The experimental results show that the performance of the method of the present application decreases the least in the challenging environment, which reflects the key role of adversarial regularization in enhancing the robustness of the model.

[0142] Figure 7 is a schematic flowchart of the auxiliary diagnosis method based on a large language model provided by the embodiment of the present application, which specifically comprises:

[0143] S71: Obtain N-channel diagnosis voice data, patient test report data, and case document images;

[0144] In one embodiment, the medical records and test report images of the patient are obtained in advance, and the text information is extracted through optical character recognition technology as part of the multi-modal data.

[0145] S72: input the diagnosis voice data into the voice signal processing model for processing to obtain processed voice data;

[0146] In one embodiment, after the model training is completed, end-to-end processing is performed on new consultation voice data. First, the input multi-channel voice signal is enhanced through a noise reduction module based on time-frequency mask joint optimization, the target voice is enhanced and the environmental noise is suppressed, while the sound source spatial information is preserved; then, the sound source direction feature is extracted using the generalized mutual information entropy method, and the sound source direction probability distribution is dynamically generated; then, the enhanced voice feature and the sound source direction feature are input into a multi-task convolutional recurrent encoder, and the noise-reduced voice, the sound source direction and the text sequence are jointly output through a multi-head attention decoder; in order to distinguish the speaking content of the patient and the doctor, the sound source positioning result and the voice activity detection are combined: the active sound source direction in different time periods is identified according to the sound source direction probability distribution, if the sound source direction corresponds to the preset patient area, the voice in this time period is classified as patient voice, and similarly, the voice in the doctor area is classified as doctor voice; finally, the separated patient voice segment is converted through speech recognition to generate processed voice data of the patient, while the timestamp information of the voice segment is preserved to ensure the time sequence integrity of the consultation dialogue.

[0147] S73: inputting the processed voice data, the test report data of the patient and the case document image into a large language model to generate a diagnosis file through context reading.

[0148] In one specific embodiment, the processed voice data is combined with other input sources: the patient's test report and medical record document image are analyzed through optical character recognition technology to extract key text information such as test indicators, diagnosis records, etc., while the patient's directly input text description of symptoms is received; all data are combined in time sequence and context association to form a unified input sequence, including voice converted text, optical character recognition text and patient input text, which presents the patient's chief complaint, medical history and test results, providing rich consultation context data for natural language processing large language model.

[0149] Optionally, during data fusion, the texts from different sources are standardized to eliminate format differences and preserve the consistency of key medical terms, ensuring the accuracy and integrity of the input information.

[0150] In one specific embodiment, the open source large language model ChatGLM-6B is used as the core reasoning engine of the consultation system, which has been pre-trained and can be directly deployed without fine-tuning. The system deploys the ChatGLM-6B model on a local server equipped with GPU acceleration devices to support real-time reasoning. When the multi-modal fusion text is input, the large language model generates a diagnosis file based on its large medical knowledge base and language understanding ability. The output process includes: analyzing the symptom description, test results and medical history information in the input text, combining the knowledge of common childhood diseases to generate diagnosis suggestions, health guidance or further examination recommendations.

[0151] In one embodiment, when the input text contains "the child has a fever for three days, cough with sore throat, and blood routine shows that the white blood cell count is elevated", the large language model may output "considering the possibility of upper respiratory tract infection, it is recommended to rest and drink more water, and antipyretic drugs can be used as appropriate, and if the symptoms persist, reconsultation is needed". The system provides professional medical advice through concise and clear output content, and clearly marks the disclaimer that needs to be finally confirmed by the doctor.

[0152] The application further discloses an auxiliary inquiry device based on multi-source information fusion, which comprises a memory and a processor. Figure 8 As shown in the figure:

[0153] The memory is used for storing program instructions;

[0154] The processor is used for calling program instructions, and when the program instructions are executed, the voice signal processing method steps are realized, or the voice signal processing model construction method steps are realized, or the auxiliary inquiry method steps based on the large language model are realized.

[0155] The application provides a computer program product, which comprises a computer program, and the computer program is used for realizing the voice signal processing method steps, or realizing the voice signal processing model construction method steps, or realizing the auxiliary inquiry method steps based on the large language model.

[0156] The application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the voice signal processing method steps, or realize the voice signal processing model construction method steps, or realize the auxiliary inquiry method steps based on the large language model.

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and module can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0158] In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other manners. For example, the above-described device embodiments are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual ones can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0159] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, i.e., may be located in one place or distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0160] In addition, each functional module in each embodiment of the present application can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0161] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer readable storage medium, which can include read only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0162] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer readable storage medium, which can include read only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0163] The computer device provided by the present application is described in detail above. For those skilled in the art, according to the idea of the embodiment of the present application, the specific implementation and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A speech signal processing method, characterized in that, include: S1: Obtain consultation voice data from N channels, where N is a natural integer; S2: Calculate the short-time Fourier transform amplitude spectrum and phase spectrum of the consultation voice data of the N channels to obtain the Fourier transform coefficients of each channel, and calculate the coherence of the short-time Fourier transform amplitude spectrum among the N channels to obtain weighting coefficients. Combine the Fourier transform coefficients and the weighting coefficients with the cosine function of the phase difference between the channels to obtain a multi-channel time-frequency mask; S3: Multiply the Fourier transform coefficients of the multi-channel time-frequency mask to obtain the speech feature matrix, and calculate the generalized mutual information entropy value of the speech feature matrix through kernel density estimation; S4: Generate a sound source direction feature vector by aggregating the generalized mutual information entropy values.

2. The speech signal processing method according to claim 1, characterized in that, The step of calculating the generalized mutual information entropy value by kernel density estimation of the speech feature matrix is ​​as follows: based on the speech feature matrix, the probability density of each sample point is calculated by kernel density estimation using a Gaussian kernel function to obtain the joint probability distribution and marginal probability distribution, and the generalized mutual information entropy value representing the directional correlation of the channel pairs is generated by combining the phase weight coefficient.

3. The speech signal processing method according to claim 1, characterized in that, Specifically, S4 involves aggregating the generalized mutual information entropy values ​​of the N channels and weighting the azimuth prior vector to generate a vector representing the probability distribution of the sound source direction, which is then normalized by an activation function to generate a sound source direction feature vector; the azimuth prior vector is obtained based on the geometry of the microphone array.

4. A speech signal processing method according to claims 1-3, characterized in that, Also includes: The sound source direction feature vector is subjected to multi-head attention calculation to obtain the output features of M attention heads, where M is a natural integer. The output features of the M attention heads are concatenated to obtain the probability distribution of the sound source location, and the sound source features are generated by the sound source location discriminator to obtain the processed speech data.

5. The speech signal processing method according to claim 4, characterized in that, The step of performing multi-head attention calculation on the sound source direction feature vector to obtain the output features of M attention heads further includes: extracting features from the speech feature matrix through depthwise convolution to obtain depthwise separable convolution output features; concatenating the depthwise separable convolution output features and the sound source direction feature vector and performing multi-head attention calculation to obtain the output features of M attention heads, where M is a natural integer; And / or the process of generating sound source features through a sound source location discriminator to obtain processed speech data further includes: The processed speech data is passed through a fully connected layer, linear transformation and activation function to obtain the time-frequency representation of the denoised speech, the probability distribution of the sound source location and the probability distribution of the text sequence. The denoised speech time-frequency representation, sound source location probability distribution, and text sequence probability distribution are respectively processed by a denoised speech discriminator, a sound source location discriminator, and a text sequence discriminator to generate denoised features, sound source features, and text sequence features to obtain the processed speech data.

6. A method for constructing a speech signal processing model, characterized in that, include: S61: Obtain consultation voice data and real voice labels from N channels, where N is a natural integer; S62: Input the consultation voice data from the N channels into the voice signal processing method according to any one of claims 1-5 to obtain processed voice data; S63: Input the processed speech data and the real speech labels into the generator and discriminator respectively to calculate the adversarial loss, optimize the adversarial loss, and obtain the speech signal processing model.

7. A diagnostic assistance method based on a large language model, characterized in that, include: S71: Acquire consultation voice data, patient test report data, and medical record document images from N channels; S72: Input the consultation voice data into the voice signal processing model of claim 6 for processing to obtain processed voice data; S73: Input the processed speech data, along with the patient's test report data and the case document image, into a large language model for context reading to generate a diagnostic file.

8. A computer device, characterized in that, include: Memory and processor; The memory is used to store program instructions; The processor is used to call program instructions, which, when executed, implement the steps of the speech signal processing method according to any one of claims 1-5, or the steps of the speech signal processing model construction method according to claim 6, or the steps of the multimodal-based assisted diagnosis method according to claim 7.

9. A computer program product, comprising a computer program, characterized in that, The computer program is implemented by a processor to perform the steps of the speech signal processing method according to any one of claims 1-5, or the steps of the method for constructing a speech signal processing model according to claim 6, or the steps of the multimodal-based assisted diagnosis method according to claim 7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the speech signal processing method according to any one of claims 1-5, or the steps of the speech signal processing model construction method according to claim 6, or the steps of the multimodal-based assisted diagnosis method according to claim 7.

Images