Exhibition hall intelligent guide voice question and answer optimization system based on multi-modal data

Abstract

The invention discloses an exhibition hall intelligent guide voice question and answer optimization system based on multi-modal data, and relates to the technical field of intelligent voice question and answer data processing. The exhibition hall intelligent guide voice question-answer optimization system based on the multi-modal data comprises a voice signal noise interference monitoring module; a key semantic recognition monitoring module; and a voice question-answer response efficiency monitoring module. According to the method, voice signal noise interference evaluation is carried out to determine whether to take an adaptive noise reduction measure, then, based on a voice problem key semantic recognition analysis result, whether to take key semantic recognition correction is determined, and finally, based on a voice problem answer response judgment result, whether to take a triple retrieval strategy is determined. The effect of improving the response efficiency of complex voice question answering is achieved, and the problem that in the prior art, multi-modal data-based exhibition hall intelligent guide voice question answering optimization data is low in accuracy is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent voice question and answer data processing, in particular to an exhibition intelligent tour voice question and answer optimization system based on multi-modal data. BACKGROUND

[0002] In the semantic processing of the exhibition intelligent tour voice question and answer based on multi-modal data, first, through the microphone array and the audio processing chip built-in in the exhibition intelligent tour terminal, the multi-modal data (such as voice signal, object position coordinates, user input text, etc.) collected is suppressed based on the microphone array noise reduction technology, and the starting and ending positions of the voice question initiated through the microphone of the exhibition intelligent tour terminal are accurately identified with the help of the voice endpoint detection technology, and the meaningless blank audio segments are filtered out; then, the processed effective voice signal is transmitted into the automatic speech recognition module, the voice signal is converted into text sequence, and at the same time, the text is preprocessed by Chinese word segmentation tool, such as jieba word segmentation (Jieba Chinese Word Segmentation, Jieba word segmentation), and part-of-speech tagging tool, to remove mood words and redundant characters, to obtain clear question text; subsequently, the exhibition intelligent tour voice system calls the multi-modal data retrieval module, first converts the question text into a semantic vector through a pre-trained model, and then performs cosine similarity calculation with the feature vectors (such as feature vector data of product text introduction, explanation video, etc.) in the exhibition multi-modal database, to quickly match the multi-modal data highly related to the question; then, the matched multi-modal data association information (such as product number, core knowledge points) is input into the question and answer generation model based on domain fine-tuning together with the question text; after that, the text answer is transmitted into the speech synthesis module, the text is converted into natural and fluent human voice, and the user is output through the speaker of the tour terminal.

[0003] In the process of semantic processing of voice question and answer of intelligent guide of exhibition hall based on multi-modal data, the prior art first receives the voice signal containing the wake-up word issued by the user through a non-contact voice collection device (such as a microphone array), pre-processes the signal to extract wake-up features, and matches the pre-set wake-up model to complete the wake-up of the intelligent guide voice system of the exhibition hall. Then, after the intelligent guide voice system of the exhibition hall is woken up, the user's question and answer voice signal is continuously collected, and the environmental interference noise is filtered by using a noise suppression algorithm (such as beamforming technology) to obtain a clear effective voice segment. Then, the processed effective voice segment is input into a voice recognition model to be converted into corresponding text information, and the user's question intention and key information are analyzed in combination with a context semantic analysis model. Then, according to the analysis result, the matching answer content is retrieved from the pre-set knowledge base, and if there is a multi-round question and answer requirement, the historical interaction information is also associated for context coherent processing. Finally, the retrieved answer content is converted into a voice signal through a text-to-speech model, and then fed back to the user through a non-contact audio output device to complete a non-contact voice question and answer interaction. If the user has subsequent questions, the above collection, processing, analysis, retrieval and feedback steps are repeated.

[0004] For example, the adaptive learning question and answer system and method based on multi-modal interaction disclosed in Chinese patent application No. CN120256575A, comprising: first, collecting multi-modal learning data such as text and voice, cleaning, denoising and multi-modal feature extraction of the data, synchronously constructing a domain question and answer knowledge base adapted to a specific learning scene, and initializing a multi-modal interaction interface and a user portrait module (such as recording user knowledge mastery level and learning preference information); then, the user submits a learning question through a multi-modal interaction interface (such as voice input and text editing, etc.), the voice system receives the multi-modal input, calls the corresponding analysis module (voice to text, image content recognition, etc.) to convert it into a unified feature representation, and combines the historical learning records in the user portrait to complete feature fusion; then, the voice system retrieves the question and answer knowledge base based on the fused features, and starts the adaptive learning engine, dynamically adjusts the question and answer matching strategy and answer generation logic according to the problem difficulty, knowledge point category and user historical feedback, to ensure that the answer fits the user's cognitive level; finally, the voice system feeds back the answer in the multi-modal form (text explanation and voice broadcast, etc.) preferred by the user, and collects the user's satisfaction evaluation and subsequent learning needs for the answer.

[0005] For example, Chinese invention patent CN119739840A discloses a multimodal intelligent question-answering and recommendation system that supports emotional voice output. The system includes: a voice system first collects multimodal data such as text and speech, cleans and extracts features to construct a multimodal intelligent question-answering and recommendation knowledge base, and simultaneously trains and establishes voice parameter models containing different emotional styles such as gentle, soothing, and lively. A multimodal input parsing module and a user preference recording module are initialized concurrently. Next, the user submits a request through a multimodal interaction interface. The voice system calls the input parsing module to process the multimodal request uniformly, such as converting speech to text and recognizing key information in images, and then analyzes the core of the request in conjunction with the user preference records. Subsequently, the voice system processes the request type: if it is a question-answering request, it searches the knowledge base to match an accurate answer; if it is a recommendation request, it filters recommended content based on user preferences and the scenario. Simultaneously, based on the atmosphere of the request scenario and the user's potential emotions, it calls the emotional voice model to adjust the voice output parameters, generating appropriate emotional voice content, which can also be combined with text or images to form multimodal output.

[0006] The above-mentioned technology has at least the following technical problems: In the semantic processing of intelligent voice question-and-answer systems for exhibition halls based on multimodal data, complex background noise (such as conversations among people and equipment operation) within the exhibition hall can distort the collected visitor voice signals. Consequently, when processing complex voice questions involving multiple semantic layers based on these distorted voice signals, it becomes impossible to accurately extract key semantic information from the voice, potentially leading to breaks in dialogue logic. Consequently, semantic clarification mechanisms aided by multimodal data, such as visual gesture recognition and spatial location positioning, also fail due to the broken connections caused by voice distortion. Because the intelligent voice guide system for exhibition halls cannot effectively correlate the multiple semantic layers of complex voice questions, deviations occur in the identification of key requirements for complex voice questions. Therefore, when searching the knowledge base based on the keywords of the identified requirements, multiple range adjustments and matching verifications are required. Ultimately, this reduces the response efficiency of the intelligent voice guide system for complex voice questions, resulting in low data accuracy in optimizing intelligent voice question-and-answer systems for exhibition halls based on multimodal data. Summary of the Invention

[0007] To address the issue of low data accuracy in existing technologies for optimizing voice-based question-and-answer responses in intelligent exhibition guides based on multimodal data, this invention provides an optimization system for voice-based question-and-answer responses in intelligent exhibition guides based on multimodal data. The system includes: a voice signal noise interference monitoring module, a key semantic recognition monitoring module, and a voice-based question-and-answer response efficiency monitoring module. The voice signal noise interference monitoring module assesses voice signal noise interference during the semantic processing of voice-based question-and-answer responses in intelligent exhibition guides based on multimodal data. Based on the acquired voice signal noise interference assessment results, it determines whether adaptive noise reduction measures are needed. These adaptive noise reduction measures are used to reduce the impact of complex background noise on the voice-based question-and-answer responses. The system monitors and controls interference in the collected tourist speech signals to improve clarity and intelligibility. A key semantic recognition monitoring module analyzes key semantics of speech issues after the speech signal noise interference assessment is satisfactory. Based on the results of this analysis, it determines whether key semantic recognition correction is necessary. This correction improves the accuracy of identifying key requirements for complex speech problems. A speech question-and-answer response efficiency monitoring module assesses the response to speech questions after the key semantic recognition analysis is satisfactory, determining whether a triplet retrieval strategy is needed. This strategy improves the response efficiency for complex speech problems.

[0008] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: 1. By conducting a speech signal noise interference assessment, the results are used to determine whether adaptive noise reduction measures are needed. This helps to solve the problem of speech signal distortion caused by complex background noise in the exhibition hall, improves the accuracy of the speech signal, and provides basic data support for the effective extraction of multi-layer semantics. After the speech signal noise interference assessment is qualified, key semantic recognition analysis of speech problems is performed. Based on the results of the key semantic recognition analysis, the results are used to determine whether key semantic recognition correction is needed. This helps to improve the effective association ability of multi-layer semantics of complex speech problems and the accuracy of key requirement identification. Key semantic recognition correction is used to correct omissions or errors in keyword extraction, complete missing logical nodes in the semantic chain, and improve the accuracy of key requirement identification of complex speech problems. After the key semantic recognition analysis of speech problems is qualified, the speech problem response judgment is performed to determine whether a triplet retrieval strategy is needed. This helps to reduce invalid question retrieval steps and improve the response efficiency of complex speech problems. The triplet retrieval strategy is used to improve the response efficiency of complex speech problems.

[0009] 2. By selectively choosing the speech signal-to-noise ratio (SNR) verification value and signal distortion verification value as speech signal noise interference assessment data items, the degree of noise interference on the speech signal can be comprehensively captured. The SNR verification value reflects the severity of the effective components being masked by background noise, while the signal distortion verification value reflects the degree of damage to the semantic integrity of the speech signal caused by background noise. This avoids the limitation of a single parameter only being able to assess the impact of noise interference in one dimension. The speech signal noise interference assessment data items are harmonicly averaged to obtain the speech signal noise interference assessment index, which can comprehensively consider the combined impact of noise masking and waveform distortion on the quality of the speech signal. This avoids the bias caused by the assessment of a single data item, making the speech signal noise interference assessment index more objectively reflect the overall severity of the speech signal being interfered with by background noise. Based on the speech signal noise interference assessment index, it is possible to determine whether to take adaptive noise reduction measures, which helps to specifically solve problems such as insufficient speech signal clarity and semantic distortion caused by complex background noise. This ensures that the processed speech signal can provide a high-quality data foundation for subsequent key semantic recognition analysis of speech problems.

[0010] 3. Based on speech-to-text and natural language processing algorithms, key semantic extraction provides a foundation for converting speech questions in intelligent tour guide scenarios into analyzable text keywords. This reduces the potential loss of semantic information when directly processing speech signals. Semantic evaluation data items, composed of key semantic matching rate and semantic completeness, ensure that extracted keywords conform to the terminology norms of the tour guide scenario and avoid requirement identification deviations caused by the omission of core semantic units. A semantic recognition accuracy index is generated through weighted coupling of semantic evaluation data items and their corresponding weight parameters. The weights can be flexibly adjusted according to the actual needs of the tour guide, making the semantic recognition accuracy index more scenario-adaptable. A decision-making mechanism based on the accuracy index to determine whether to implement key semantic recognition correction can specifically screen out and correct cases where semantic recognition is substandard, effectively reducing subsequent question-and-answer matching deviations caused by keyword extraction errors and missing core information. This lays a reliable semantic foundation for the intelligent tour guide voice question-and-answer system to accurately understand the complex needs of visitors and efficiently generate response content, thus improving the accuracy of the intelligent tour guide voice question-and-answer system. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1This is a schematic diagram of the structure of the intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data provided in this embodiment of the invention; Figure 2 This is a flowchart summarizing the overall process of the intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data provided in this embodiment of the invention. Figure 3 This is a logic diagram of the adaptive noise reduction measures of the intelligent exhibition hall guide voice question-and-answer optimization system based on multimodal data provided in this embodiment of the invention; Figure 4 This is a schematic diagram of the question-and-answer generation model of the intelligent exhibition hall guide voice question-and-answer optimization system based on multimodal data provided in this embodiment of the invention. Detailed Implementation

[0013] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0014] like Figure 1 The diagram shown is a structural schematic of the intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data provided in an embodiment of the present invention. The system includes: a voice signal noise interference monitoring module, a key semantic recognition monitoring module, and a voice question-and-answer response efficiency monitoring module. The voice signal noise interference monitoring module is used to assess voice signal noise interference during the semantic processing of intelligent voice question-and-answer tours in exhibition halls based on multimodal data. Based on the acquired voice signal noise interference assessment results, it determines whether adaptive noise reduction measures are needed. Adaptive noise reduction measures are used to reduce the interference of complex background noise on the collected visitor voice signals, thereby improving the clarity and intelligibility of the voice signals. By monitoring the voice signal noise interference assessment results, it helps to identify the degree of interference of complex background noise in the exhibition hall on the voice signals in a timely manner, and to trigger adaptive noise reduction measures in a targeted manner to avoid voice signal distortion and reduce the chain of semantic processing errors caused by voice distortion.

[0015] The key semantic recognition monitoring module is used to perform key semantic recognition analysis of speech problems after the speech signal noise interference assessment is qualified. Based on the obtained key semantic recognition analysis results, it determines whether key semantic recognition correction is needed. Key semantic recognition correction is used to correct omissions or errors in keyword extraction, fill in missing logical nodes in the semantic chain, and improve the accuracy of key requirement identification for complex speech problems. By monitoring the key semantic recognition analysis results of speech problems, it helps to promptly discover problems such as omissions in key semantic extraction, keyword errors, or semantic chain breaks. By triggering correction, it ensures the effective association of multi-layer semantics for complex speech problems, reduces the failure of multimodal auxiliary clarification semantic mechanisms due to semantic recognition deviations, and improves the accuracy of identifying key needs of tourists.

[0016] The voice question-and-answer response efficiency monitoring module is used to judge the voice question answer response after the key semantic recognition and analysis of the voice question is qualified, so as to determine whether the triplet retrieval strategy needs to be adopted. The triplet retrieval strategy is used to improve the response efficiency of complex voice questions. By monitoring the voice question answer response judgment, it helps to reduce process redundancy caused by blind retrieval, optimize the processing speed of the entire voice question-and-answer process, and ensure the smooth interaction between tourists and the intelligent guide voice system in the intelligent guide scenario of the exhibition hall.

[0017] It should be noted that the intelligent tour guide voice question-and-answer optimization system for exhibition halls based on multimodal data provided in this application was designed with a database that stores various preset data. The database includes, but is not limited to, preset voice signal-to-noise ratio threshold, preset signal distortion threshold, preset voice signal noise interference threshold, preset high-frequency noise threshold, preset low-frequency noise threshold, preset total voice signal energy, preset dedicated vocabulary, and preset semantic association database, etc. The preset values ​​are directly set by technical personnel.

[0018] It should also be noted that the intelligent tour guide voice question-and-answer optimization system for exhibition halls based on multimodal data provided in this application has been designed in advance by constructing a mapping set that associates multi-dimensional exhibition hall scene parameters with corresponding output results. This mapping set is configured in advance by preset personnel and stored in the database. Its core function is to reflect the precise correspondence between the input parameters related to the definition of voice signal denoising parameters and the optimization of frame processing ratio in different exhibition hall scenes, and the coefficients and ratios required by each processing link. This provides a decision support basis that can be directly queried and called for the dynamic adjustment of parameters in the voice signal denoising process and subsequent semantic analysis.

[0019] Specifically, the mapping sets constructed in this embodiment include four types: wavelet threshold parameter mapping set, Wiener filter parameter mapping set, least mean square adaptive filter parameter mapping set, and frame overlap ratio parameter mapping set. The formation of each mapping set is not directly predetermined, but rather gradually completed based on statistical analysis and parameter verification of the intelligent guide voice interaction scenario in historical exhibition halls. First, a large amount of specific combination data of input parameters and output results corresponding to each mapping set in actual interaction scenarios is extracted (e.g., the combination of the speech signal noise interference evaluation index, high-frequency noise energy ratio, and corresponding wavelet coefficients required for the wavelet threshold parameter mapping set; the combination of the speech signal noise interference evaluation index, mid-frequency noise energy ratio, noise reference signal, and corresponding Wiener filter coefficients required for the Wiener filter parameter mapping set; and the least mean square adaptive filter parameter mapping set...). The required combination of speech signal noise interference assessment index, low-frequency noise energy ratio and adaptive filtering coefficients, and the combination of frequency band boundary frequency difference, noise energy ratio of each frequency band and corresponding optimal frame overlap ratio required for the frame overlap ratio parameter mapping set are used. Each input parameter is assigned a weight value based on its influence on the output result. The actual effective parameter matching results in each scenario are recorded simultaneously. Then, with the help of correlation analysis (such as Pearson correlation coefficient analysis method), abnormal correlation information caused by temporary equipment failure (such as poor microphone contact) and sudden environmental interference (such as instantaneous broadcast noise) is filtered out. The correspondence between the input parameter combination and the output result with statistical significance is retained. Finally, all effective information is integrated to form various mapping sets that can be directly queried and the corresponding output results can be obtained through input parameters.

[0020] In this embodiment, the voice signal noise interference monitoring module, the key semantic recognition monitoring module, and the voice question-and-answer response efficiency monitoring module form a progressive closed-loop control process through data transmission and process connection: the voice signal noise interference monitoring module prioritizes solving the noise interference problem at the source of the voice data, providing a high-quality, low-distortion voice signal foundation for the key semantic recognition module; the key semantic recognition monitoring module relies on high-quality voice data to complete semantic understanding and correction, clarifying the precise direction of visitor needs for the voice question-and-answer response efficiency monitoring module; and the voice question-and-answer response efficiency monitoring module optimizes the retrieval strategy based on precise needs, ensuring the efficiency of question-and-answer response. The three modules work in a progressive manner, ultimately achieving full-process control of the semantic processing of voice question-and-answer in the intelligent tour guide system, effectively improving the environmental adaptability, semantic recognition accuracy, and response efficiency of complex voice questions in the intelligent tour guide voice question-and-answer optimization system.

[0021] like Figure 2 The diagram shown is a general overview flowchart of the intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data provided in this embodiment of the invention. Figure 2It can be seen that in the semantic processing of voice question-and-answer for intelligent tour guides in exhibition halls based on multimodal data, voice signal noise interference is assessed and a voice signal noise interference assessment index is obtained. It is then determined whether the voice signal noise interference assessment index is less than a preset voice signal noise interference threshold. If so, key semantic recognition analysis of the voice question is performed; otherwise, adaptive noise reduction measures are adopted. These adaptive noise reduction measures involve sequentially performing multi-band filtering noise reduction and frequency band synthesis. After the adaptive noise reduction measures are completed, if the voice signal noise interference assessment is satisfactory, key semantic recognition analysis of the voice question is performed and a semantic recognition accuracy index is obtained. It is then determined whether the semantic recognition accuracy index is greater than a preset accuracy threshold. If so, a voice question answer response is judged; otherwise, key semantic recognition correction is performed. After the key semantic recognition correction is completed, if the voice question key... If the semantic recognition analysis is successful, the voice question response is judged. It is determined whether the actual response time of the voice question reaches the preset response time threshold. If not, the current voice question response generation process continues. Otherwise, it continues to monitor whether the voice question response is output. If yes, the complete voice question response is output to the smart guide interactive terminal currently used by the user. If not, a triplet retrieval strategy is adopted. After the triplet retrieval strategy is completed, the effect of the triplet retrieval strategy is verified, and the voice question response time index is obtained. It is determined whether the voice question response time index is less than the preset qualified response threshold. If yes, the complete voice question response is output to the smart guide interactive terminal currently used by the user. Otherwise, a prompt is sent to the preset personnel to re-collect multimodal data.

[0022] Furthermore, the specific process of assessing speech signal noise interference is as follows: Speech signal noise interference assessment data items are obtained during the input of multimodal data into the exhibition hall's intelligent guide interactive terminal. These data items include a speech signal-to-noise ratio (SNR) verification value and a signal distortion verification value. The SNR verification value is the result of quantifying the proportion of the SNR to a preset noise interference assessment time period and a preset SNR threshold, and then weighting it with the SNR weight influence value. This value reflects the severity of the effective components of the speech signal being masked by background noise. The SNR is obtained through the input of multimodal data into the intelligent guide interactive terminal of the exhibition hall. The effective power is represented by the reciprocal of the ratio of effective power to background noise power. The preset speech signal-to-noise ratio (SNR) threshold is represented by the average of the SNR coefficients over a historical time period. The effective power of the speech signal is represented by the sum of the differences between the maximum and minimum intensity values ​​of the speech signal monitored by the speech signal acquisition instrument in each effective speech signal frequency band identified by the speech activity detection algorithm within the preset noise interference assessment time period, and then quantified by dividing this sum by the effective speech duration. The effective speech duration represents the effective speech duration as the proportion of each speech signal frequency band monitored by the speech signal acquisition instrument identified by the speech activity detection algorithm within the preset noise interference assessment time period. The cumulative summation of the effective speech signal durations, where the proportion quantization represents a ratio operation, is performed by summing the differences between the maximum and minimum noise signal intensity values ​​of each noise frequency band monitored by the speech signal acquisition instrument within the preset noise interference assessment time period identified by the speech activity detection algorithm, and then performing proportion quantization with the pure noise duration as the background noise power. The cumulative summation of the durations of the background noise corresponding to each speech signal frequency band monitored by the speech signal acquisition instrument, identified from the preset noise interference assessment time period by the speech activity detection algorithm, is taken as the pure noise duration; the preset noise interference assessment time period table... The time period for assessing speech signal noise interference is indicated. The signal distortion verification value is the result of quantifying the proportion of signal distortion to a preset signal distortion threshold within the preset noise interference assessment time period, and then weighting it with the signal distortion weight influence value. This value reflects the degree of impairment to the semantic integrity of the speech signal caused by background noise. The speech signal-to-noise ratio coefficient weight influence value and the signal distortion weight influence value are used to quantify the relative importance of the speech signal-to-noise ratio coefficient verification value and the signal distortion verification value in the speech signal noise interference assessment. The weighting process represents a multiplication operation. The signal distortion is represented by the ratio of the total power of the harmonic signals to the power of the fundamental signal in the speech signal.The total power of harmonic signals in a speech signal is represented by extracting integer multiples of the fundamental frequency from the speech signal after performing a Fourier transform, calculating the power of each harmonic component, and summing the results. The fundamental frequency power is represented by extracting the average power corresponding to the fundamental frequency from the speech signal after performing a Fourier transform, as monitored by the audio analyzer. The harmonic average of the speech signal noise interference evaluation data is used as the speech signal noise interference evaluation index. When the speech signal-to-noise ratio (SNR) decreases, it means that the effective components of the speech signal are less masked by background noise, and the intelligibility of the speech signal is enhanced. This reduces signal distortion, indicating a decrease in the impairment of the semantic integrity of the speech signal caused by background noise, and an improvement in the reliability of semantic transmission of the speech signal. The combined effect of these two factors leads to a reduction in the speech signal noise interference assessment index. Based on the speech signal noise interference assessment index, the following judgment is made: when the speech signal noise interference assessment index is less than a preset speech signal noise interference threshold, key semantic recognition analysis of the speech problem is performed; otherwise, it is determined that the speech signal has background noise interference and adaptive noise reduction measures are taken. The preset speech signal noise interference threshold is represented by the average value of the speech signal noise interference assessment index over a historical time period.

[0023] It is important to note that the exhibition hall intelligent guide voice question-and-answer semantic processing based on multimodal data provided in this application involves multiple mapping sets for evaluating voice signal noise interference and assessing the accuracy of key semantic recognition of voice issues. These mapping sets collectively constitute mapping groups, which are pre-set and stored in a database by professional technicians. They provide matching basis for weight coefficients and preset thresholds for calculating voice signal-to-noise ratio verification values, signal distortion verification values, and quantifying semantic recognition accuracy indicators. For example, a large amount of actual historical scenario data of exhibition hall intelligent guide voice interaction is first extracted, covering parameter combinations of two core evaluation scenarios (such as voice signal noise interference evaluation scenario, voice signal-to-noise ratio, signal distortion, and semantic recognition accuracy). The numerical combinations of effective power of the signal, background noise power, and corresponding weighted influence values ​​of the speech signal-to-noise ratio coefficient, and preset speech signal-to-noise ratio threshold; the combinations of total harmonic power of the speech signal, fundamental signal power, and corresponding weighted influence values ​​of signal distortion, and preset signal distortion threshold; in the scenario of evaluating the accuracy of key semantic recognition in speech problems, the specific combinations of the initial keyword set, preset dedicated vocabulary keywords, and corresponding weighted influence values ​​of key semantic matching rate, and the parameter combinations of the number of core semantic units in the initial keyword set, the preset total number of core semantic units, and corresponding weighted influence values ​​of semantic integrity), and assigning input parameters for each scenario based on their impact on the corresponding evaluation results. The weighted quantification values ​​of the influence of the results (speech signal noise interference assessment index, semantic recognition accuracy index) are calculated. For example, the weighted influence value of the speech signal-to-noise ratio coefficient is assigned according to its contribution to the noise interference assessment index, and the weighted influence value of the key semantic matching rate is set according to its influence on the semantic recognition accuracy index. The actual effective values ​​of the weighted influence values ​​of the speech signal-to-noise ratio coefficient, the signal distortion coefficient, and the semantic evaluation data item weight parameters in each historical scenario are recorded simultaneously. Then, through correlation analysis (such as Kendall rank correlation coefficient analysis), the abnormal values ​​caused by sudden noise in the exhibition hall, temporary failure of the speech acquisition equipment, and temporary adaptation deviation of the speech-to-text algorithm are eliminated. The system frequently correlates data, retains the correspondence between statistically significant parameter combinations and weight coefficients, and finally integrates all valid data to form a mapping group containing multiple mapping sets. The mapping relationship uses a 0-1 value range to represent the proportion of weight coefficients, realizing a one-to-one or many-to-one matching between input parameter combinations and corresponding weight coefficients under different evaluation scenarios. When the intelligent guide voice system in the exhibition hall performs voice signal noise interference assessment and voice problem key semantic recognition accuracy assessment, the corresponding weight coefficients can be quickly retrieved from this mapping group to accurately quantify the degree of influence of each parameter on the evaluation results, ensuring the accuracy and reliability of the voice signal noise interference assessment index and semantic recognition accuracy index.

[0024] In this embodiment, the assessment of noise interference in the voice signal helps to improve the accuracy of the voice signal, providing high-quality, low-distortion voice data support for the core links of the intelligent tour guide voice question and answer semantic processing, such as the key semantic recognition and analysis of voice questions and the voice question and answer response. This ensures the coherence and effectiveness of the processing logic of each link, while enhancing the adaptability and stability of the intelligent tour guide voice question and answer system in complex background noise environments (such as conversations among people and equipment operating noise), and effectively reducing the semantic understanding deviation caused by noise interference in the voice signal.

[0025] like Figure 3 The diagram shown is a logic diagram of adaptive noise reduction measures in the intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data provided in an embodiment of the present invention. Figure 3 It can be seen that the adaptive noise reduction measure involves sequentially performing multi-band filtering noise reduction and frequency band synthesis. The specific process of the multi-band filtering noise reduction measure is as follows: If the detected frequency value is greater than the preset high-frequency noise threshold, the corresponding speech signal frequency band is marked as the high-frequency band of the speech signal, and wavelet threshold filtering is applied. After the wavelet threshold filtering is completed, it is determined whether the speech signal noise interference assessment is qualified. If yes, frequency band synthesis is performed; otherwise, a wavelet processing failure alarm is sent. If the detected frequency value is greater than the preset low-frequency noise threshold but not greater than the preset high-frequency noise threshold, the corresponding... The speech signal frequency band is marked as the speech signal mid-frequency band and Wiener filtering is applied. After Wiener filtering, it is determined whether the speech signal noise interference assessment is qualified. If so, frequency band synthesis is performed; otherwise, a Wiener filtering failure alarm is sent. If the detected frequency value is not greater than the preset low-frequency noise threshold, the corresponding speech signal frequency band is marked as the speech signal low-frequency band and least mean square adaptive filtering is applied. After least mean square adaptive filtering, it is determined whether the speech signal noise interference assessment is qualified. If so, frequency band synthesis is performed; otherwise, an adaptive filtering failure alarm is sent.

[0026] Furthermore, the adaptive noise reduction measures involve sequentially performing multi-band filtering noise reduction to specifically reduce the background noise power of different frequency bands in the speech signal, and frequency band synthesis to integrate the speech signals from each frequency band after multi-band filtering noise reduction to restore the complete speech signal waveform. The specific process of the multi-band filtering noise reduction measures is as follows: if the detected frequency value is greater than the preset high-frequency noise threshold, the corresponding speech signal frequency band is marked as the high-frequency band of the speech signal and wavelet threshold filtering is applied; if the detected frequency value is greater than the preset low-frequency noise threshold but not greater than the preset high-frequency noise threshold, the corresponding speech signal frequency band is marked as the mid-frequency band of the speech signal and Wiener filtering is applied; if the detected frequency value is not greater than the preset low-frequency noise threshold, the corresponding .... For low-frequency noise thresholding, the corresponding speech signal frequency band is marked as the low-frequency band, and least mean square adaptive filtering is applied. Wavelet threshold filtering targets the high-frequency band of the speech signal, accurately capturing and suppressing high-frequency noise. Simultaneously, leveraging the time-frequency localization characteristics of wavelet transform, it maximizes the preservation of speech details in the high-frequency band, reducing distortion caused by noise reduction and improving the clarity of high-frequency speech. Wiener filtering focuses on the mid-frequency band of the speech signal, effectively reducing background noise in the mid-frequency band by dynamically adjusting the filter strength, balancing noise reduction effect with the naturalness of the speech signal. Least mean square adaptive filtering targets low-frequency noise in the speech signal, efficiently filtering out low-frequency noise by adaptively tracking noise characteristics and adjusting filter parameters in real time. Redundant noise is used to protect the low-frequency fundamental frequency components of the speech signal, preventing the loss of emotional color and emphasis information due to low-frequency noise masking, and ensuring the overall integrity of the speech signal. Frequency values ​​are represented by the frequency components of each frequency point in the spectrum monitored by the audio analyzer after the speech signal has undergone short-time Fourier transform processing. The preset high-frequency noise threshold is represented by the average power value of high-frequency noise over a historical period, and the preset low-frequency noise threshold is also represented by the average power value of low-frequency noise over a historical period. The specific process of wavelet threshold filtering is as follows: the speech signal noise interference assessment index and the high-frequency noise energy proportion value are input into the wavelet threshold parameter mapping set for querying to obtain wavelet coefficients; the high-frequency noise energy proportion value is obtained through... The ratio of high-frequency noise energy to the preset total energy of the speech signal is represented by the sum of the squared differences between the endpoint signal strength value and the starting signal strength value of each high-frequency noise band monitored by the speech signal acquisition instrument within the preset noise interference assessment period. The preset total energy of the speech signal is set in advance by the preset personnel. The adjustment step size is used as the adjustment amplitude corresponding to the wavelet coefficient, and the wavelet threshold is gradually reduced. (After each wavelet threshold reduction adjustment, the speech signal noise interference assessment index is recalculated. If the speech signal noise interference assessment index is still greater than the preset speech signal noise interference threshold, the adjusted wavelet threshold is used as the initial value for the next adjustment and the adjustment continues to be reduced step by step.)The system continuously monitors the speech signal noise interference assessment index. When the speech signal noise interference assessment index is less than the preset speech signal noise interference threshold, frequency band synthesis is performed; otherwise, wavelet threshold filtering continues. If the number of wavelet threshold filtering operations exceeds the preset maximum number of wavelet threshold filtering operations, and the speech signal noise interference assessment index is still not less than the preset speech signal noise interference threshold, a wavelet processing failure alarm is sent. The wavelet threshold is not less than the preset minimum wavelet threshold. The preset speech signal noise interference threshold is represented by the average of the speech signal noise interference assessment index over a historical time period. The preset maximum number of wavelet threshold filtering operations is preset by designated personnel.

[0027] Specifically, the Wiener filtering process is as follows: The speech signal noise interference assessment index, the proportion of mid-frequency noise energy, and the noise reference signal are input into the Wiener filtering parameter mapping set for querying to obtain the Wiener filtering coefficients. The proportion of mid-frequency noise energy is represented by the ratio of mid-frequency noise energy to the preset total speech signal energy. The mid-frequency noise energy is represented by the sum of the squared differences between the endpoint signal strength value and the starting signal strength value monitored by the speech signal acquisition instrument for each mid-frequency noise during the preset noise interference assessment time period. The noise reference signal represents the signal obtained after noise extraction processing of the speech signal based on the blocking matrix. The specific process is as follows: Packets are synchronously received through a multi-channel speech acquisition device. A mixed speech signal containing effective speech and background noise is first processed. Then, the multi-channel mixed signal is input into a preset blocking matrix. Matrix operations are used to effectively suppress the signal from the effective speech source direction, retaining only the background noise signal from the non-effective speech direction. Finally, the noise signal output from the blocking matrix undergoes frequency band filtering and signal purification to remove residual traces of effective speech, resulting in a noise reference signal. When the mid-frequency signal-to-noise ratio (SNR) is greater than a preset SNR retention threshold, the filtering intensity is gradually reduced using the adjustment amplitude corresponding to the Wiener filter coefficient as the adjustment step size. (After each adjustment of the filtering intensity, the speech signal noise interference evaluation index is recalculated. If the speech signal noise interference evaluation index is still greater than the preset speech signal noise interference evaluation index, the noise interference evaluation index will be reset.) If the noise interference threshold is not set, the adjusted filter strength will be used as the initial value for the next adjustment and gradually decreased. This helps to reduce the damage to speech details caused by filtering while ensuring that the effective speech signal in the mid-frequency band is not over-filtered, thus maintaining the naturalness and semantic integrity of the speech signal. Conversely, if the adjustment step size is set using the adjustment magnitude corresponding to the Wiener filter coefficient, the filter strength will be gradually increased. (After each adjustment, the speech signal noise interference evaluation index is recalculated. If the speech signal noise interference evaluation index is still greater than the preset speech signal noise interference threshold, the adjusted filter strength will be used as the initial value for the next adjustment and gradually increased.) This helps to more effectively suppress noise interference. Mid-frequency redundant noise is reduced to decrease the masking effect of noise on the effective components of the speech signal, thereby improving the purity and intelligibility of the mid-frequency speech signal. The speech signal noise interference assessment index is continuously monitored. When the speech signal noise interference assessment index is less than the preset speech signal noise interference threshold, frequency band synthesis is performed. Otherwise, Wiener filtering is continued. When the number of Wiener filtering operations exceeds the preset maximum number of Wiener filtering operations, if the speech signal noise interference assessment index is still not less than the preset speech signal noise interference threshold, a Wiener filtering failure alarm is sent. The preset signal-to-noise ratio retention threshold is represented by the average value of the mid-frequency signal-to-noise ratio over a historical period. The preset maximum number of Wiener filtering operations is preset by the personnel.The mid-frequency signal-to-noise ratio (SNR) is represented by the ratio of effective mid-frequency speech energy to mid-frequency noise energy. Effective mid-frequency speech energy is calculated by squaring the differences between the endpoint and starting signal strength values ​​of each effective mid-frequency speech signal monitored by the speech signal acquisition instrument within a preset noise interference assessment period, and then summing these sums. Mid-frequency noise energy is calculated by squaring the differences between the endpoint and starting signal strength values ​​of each mid-frequency noise signal monitored by the speech signal acquisition instrument within a preset noise interference assessment period, and then summing these sums. The filtering intensity is not less than the preset minimum filtering intensity and not greater than the preset maximum filtering intensity. Both the preset minimum and maximum filtering intensities are pre-set by the designated personnel.

[0028] Specifically, the process of least mean square adaptive filtering is as follows: The speech signal noise interference assessment index and the proportion of low-frequency noise energy are input into the least mean square adaptive filtering parameter mapping set for querying, and adaptive filtering coefficients are obtained. The proportion of low-frequency noise energy is represented by the ratio of low-frequency noise energy to the preset total speech signal energy. The low-frequency noise energy is represented by the sum of the squared differences between the endpoint signal strength value and the starting signal strength value monitored by the speech signal acquisition instrument for each low-frequency noise level within the preset noise interference assessment time period. The adjustment step size is used to gradually reduce the low-frequency noise gain (after each adjustment of the low-frequency noise gain, the speech signal noise interference assessment index is recalculated; if the speech signal noise interference...). If the noise interference assessment index is still greater than the preset speech signal noise interference threshold, the low-frequency noise gain adjusted in this instance will be used as the initial value for the next adjustment and will be gradually reduced. This helps to accurately control the attenuation of low-frequency noise, reduce the false filtering of effective low-frequency speech components due to excessive noise reduction in one instance, and reduce the loss or misjudgment of semantic information due to noise residue. The speech signal noise interference assessment index is continuously monitored. When the speech signal noise interference assessment index is less than the preset speech signal noise interference threshold, frequency band synthesis is performed. Otherwise, the least mean square adaptive filtering is continued. If the number of least mean square adaptive filtering processes is greater than the preset maximum number of adaptive filtering processes, and the speech signal noise interference assessment index is still not less than the preset speech signal noise interference threshold, an adaptive filtering failure alarm is sent. The preset maximum number of adaptive filtering processes is set in advance by a preset user.

[0029] Specifically, the process of frequency band synthesis is as follows: The frame overlap ratio is adjusted to reduce splicing distortion between frequency bands. Specifically, the frequency band boundary frequency difference and the noise energy proportion of each frequency band are queried in the frame overlap ratio parameter mapping set to obtain the optimal frame overlap ratio. The frequency band boundary frequency difference is obtained by accumulating the difference between the boundary frequency values ​​of two adjacent speech signal frequency bands monitored by an audio analysis instrument, and then quantifying the result by proportionalizing it with the number of speech signal frequency bands. The boundary frequency value represents the critical signal frequency value used to divide the frequency band range of two adjacent speech signals, and is the dividing point between adjacent signal frequency bands. The noise energy proportion of each frequency band includes the high-frequency noise energy proportion value, mid-frequency noise energy proportion value, and low-frequency noise energy proportion value re-obtained after reducing speech signal noise through multi-band filtering and noise reduction measures. Based on the optimal frame overlap ratio, the overlap region of each frequency band signal frame is set: the speech signal in each overlap region is processed by weighted averaging, and the final synthesized qualified noise-reduced speech signal is output.

[0030] In this embodiment, adaptive noise reduction measures are used to effectively suppress the complex background noise of the exhibition hall. Dynamic adjustment logic is introduced during the filtering process, and the speech signal noise interference assessment index is used as the real-time judgment basis. This avoids the loss of speech details and impaired semantic expression caused by over-filtering, while also preventing noise residue caused by insufficient filtering. It balances the noise reduction effect with the integrity of the speech signal, significantly improving the clarity and recognizability of the acquired speech signal. It reduces speech distortion caused by noise interference from the source, providing reliable basic data support for subsequent key semantic recognition analysis of speech problems and optimization of speech question-and-answer response efficiency. This greatly enhances the anti-interference capability and service stability of the intelligent guide voice system in complex environments.

[0031] Furthermore, the key semantic recognition analysis of speech problems includes sequential key semantic extraction and accuracy evaluation of key semantic recognition in speech problems. Key semantic extraction refers to extracting key semantics based on speech-to-text methods and natural language processing algorithms to obtain an initial keyword set. The specific process of key semantic extraction based on speech-to-text methods and natural language processing algorithms is as follows: The noise-reduced speech signal is input into the automatic speech recognition (ASR) system, and the speech waveform is converted into the corresponding text sequence. At the same time, preliminary corrections are made to the punctuation, interjections, and unclear pronunciations in the text. Then, text cleaning techniques in natural language processing are used to remove text from the converted text. Redundant information (such as modal particles, repeated words, and meaningless filler words) in this text is broken down into independent words using word segmentation tools (such as dictionary-based forward maximum matching) to obtain an initial keyword set. The specific process for evaluating the accuracy of key semantic recognition in speech problems is as follows: Semantic evaluation data items are obtained to measure the accuracy of semantic recognition. These data items include the key semantic matching rate, which measures the degree of fit between the extracted keywords and the keywords in the preset dedicated vocabulary, and the semantic completeness, which measures the comprehensiveness of the coverage of core semantic information. The key semantic matching rate is determined by the number of keywords in the initial keyword set that successfully match the keywords in the preset dedicated vocabulary, as monitored by a text analysis monitoring instrument. The number of keywords is represented by the ratio of the total number of keywords in the initial keyword set. The pre-set proprietary keyword library is pre-defined by designated personnel. Semantic completeness is represented by the ratio of the number of core semantic units in the initial keyword set, monitored by text analysis instruments, to the total number of pre-defined core semantic units. Core semantic units represent key information corresponding to voice questions in the exhibition hall tour, including but not limited to exhibit identification (such as showcase number, exhibit name), attribute characteristics (such as age, material, category), and core user needs (such as explanation content, route guidance), etc. The total number of pre-defined core semantic units is pre-defined by designated personnel. Semantic evaluation data items and their corresponding semantic evaluation data items are then used to evaluate the semantic completeness. The result of weighted coupling processing of the weight parameters is used as the semantic recognition accuracy index. Among them, the weight parameters of the semantic evaluation data items are used to quantify the relative importance of key semantic matching rate and semantic completeness in the semantic recognition accuracy evaluation. By assigning different weight ratios to the two, the contribution of each to the semantic recognition accuracy index is dynamically adjusted, including the weight influence value of key semantic matching rate and the weight influence value of semantic completeness. It is determined whether the semantic recognition accuracy index is greater than the preset accuracy qualification threshold. If it is, a voice question answer response judgment is performed; otherwise, key semantic recognition correction is performed. The preset accuracy qualification threshold is represented by the average value of the semantic recognition accuracy index over a historical time period.

[0032] In this embodiment, by analyzing key semantic recognition of voice questions, accurate conversion from voice signals to core semantics and closed-loop semantic processing are achieved. First, initial keywords are extracted using speech-to-text and natural language processing technologies. Then, the accuracy of semantic recognition is quantified by evaluating both key semantic matching rate and semantic completeness, combined with weight parameters. This process helps to filter recognition errors from a semantic perspective, providing accurate demand guidance for subsequent voice question responses, reducing irrelevant answers due to key semantic deviations, and improving the depth of understanding and accuracy of response to complex voice questions in the intelligent tour guide voice system of the exhibition hall.

[0033] Furthermore, the specific process of key semantic recognition correction is as follows: First, determine if the semantic recognition accuracy index is less than the preset deviation subdivision threshold. If so, perform keyword extraction deviation correction. This involves retrieving a preset dedicated thesaurus from the database, comparing the initial keyword set with the keywords in the preset dedicated thesaurus, marking keywords that fail the comparison as deviation keywords, and then performing secondary word segmentation on the obtained deviation keywords using a preset word segmentation method (such as jieba segmentation) before re-performing keyword extraction deviation correction to complete the replacement and correction of erroneous keywords. Otherwise, perform semantic chain logic completion correction. Full correction involves retrieving a preset semantic association library from the database and supplementing the missing logical nodes in the semantic chain based on a knowledge graph path reasoning method. The preset deviation subdivision threshold is represented by the average of the semantic recognition accuracy indicators that are not greater than the preset accuracy pass threshold over a historical time period. The preset deviation subdivision threshold is less than the preset accuracy pass threshold. After the keyword semantic recognition correction is completed, the semantic recognition accuracy indicator is retrieved again. If the semantic recognition accuracy indicator is still not greater than the preset accuracy pass threshold, a recognition correction failure alarm is sent. Otherwise, the corresponding corrected keyword set is marked as the corrected qualified keyword set, and a voice question answer response judgment is performed.

[0034] In this embodiment, by correcting key semantic recognition, precise grading of semantic recognition deviations of different degrees is achieved. This reduces resource waste caused by overcorrection and accurately solves different types of recognition problems. At the same time, the re-evaluation and failure alarm mechanism after correction further ensures the reliability of the correction effect, helps to reduce omissions or errors in key semantic extraction from the root, ensures the accurate transmission of the core needs of voice questions, and improves the fault tolerance and correction capability of the intelligent guide voice system in the exhibition hall for complex semantics, providing a semantic foundation for subsequent efficient and accurate voice question and answer responses.

[0035] Furthermore, the specific process for judging the voice question response is as follows: During the voice question response process, if the actual response time of the voice question does not reach the preset response time threshold, the current voice question response generation process continues. The preset response time threshold is set in advance by preset personnel. If the actual response time of the voice question has reached the preset response time threshold, monitoring continues to determine whether a voice question response is output. If a voice question response is output, the complete voice question response is output to the smart guide interaction terminal currently used by the user in the exhibition hall. If no voice question response is output, a triplet retrieval strategy is adopted to help reduce the retrieval delay caused by the voice question response, ensuring the immediacy and smoothness of human-computer interaction, and effectively solving the problem of low response timeliness caused by the lag or failure of the conventional response generation process. The actual response time of the voice question is represented by the time interval between the end time of the user's voice input monitored by the timestamp recorder and the time when no response has been received.

[0036] Specifically, the process of the triplet retrieval strategy is as follows: The modified qualified keyword set is input into the database to retrieve a preset question-and-answer triplet library. Based on a preset similarity retrieval algorithm, the question-and-answer triplet with the highest semantic matching degree in the modified qualified keyword set is retrieved to obtain the optimal question-and-answer match. The semantic matching degree of the qualified keyword set indicates the degree of overlap and close association between the keywords in the modified qualified keyword set and each triplet in the preset question-and-answer triplet library at the semantic level (including synonym association, hierarchical relationship, and scene adaptability). The preset question-and-answer triplet library is pre-set by designated personnel. The preset answer template in the optimal question-and-answer match is then fused and filled with the specific entity information in the modified qualified keyword set using a slot-filling algorithm and entity linking method to generate a complete voice question answer that conforms to the context. The specific entity information represents the exhibition-related entities in the modified qualified keyword set that have clear directional information (such as exhibit names, exhibition...). The data includes cabinet number, year attribute, user's current location, etc. The preset answer template is pre-set by preset personnel. The triplet retrieval strategy also includes triplet retrieval strategy effectiveness verification. The specific process of triplet retrieval strategy effectiveness verification is as follows: Obtain the voice question answer response time index; the voice question answer response time index is represented by the deviation quantification result between the completion time of key semantic extraction monitored by the timestamp recorder and the generation time of the complete voice question answer, where the deviation quantification represents the difference calculation; determine whether the voice question answer response time index is less than the preset qualified response threshold. If so, the triplet retrieval strategy is deemed effective, and the complete voice question answer is output to the smart guide interactive terminal currently used by the user in the exhibition hall; otherwise, a prompt is sent to preset personnel to re-collect multimodal data, where the preset qualified response threshold is represented by the average value of the voice question answer response time index over a historical time period.

[0037] In this embodiment, the coordinated use of voice question answer response judgment and triplet retrieval strategy enables dynamic control and emergency optimization of the voice question answering process. The voice question answer response judgment monitors the progress of regular answer generation in real time based on the actual response time of the voice question answer. If no response is generated within the time limit, the triplet retrieval strategy is triggered in a timely manner to avoid long waiting times for tourists. The triplet retrieval strategy and its effectiveness verification ensure that the emergency-generated answers are both efficient and meet the response standards. This mechanism helps to effectively reduce the problem of users not responding due to the lag and excessive time consumption of the regular answer generation process, improves the response efficiency and fault tolerance of the intelligent guide voice system in the exhibition hall, and ensures that answers that meet the needs of tourists can still be output quickly and accurately when the regular question answering process is blocked.

[0038] like Figure 4 The diagram shown is a schematic representation of the question-and-answer generation model of the intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data provided in an embodiment of the present invention. Figure 4 It can be seen that the BART-based question-and-answer generation model of the intelligent exhibition hall guide voice question-and-answer optimization system based on multimodal data shows the entire process of inputting multimodal data association information obtained from correcting qualified keyword matching along with the question text into the BART model and outputting a text answer. Specifically, the data input module is divided into two core modules: a multimodal data association information module and a question text module. The multimodal data association information module uses sub-labels to mark specific data dimensions, including exhibit number, exhibit name, and year, reflecting the correlation between the data and the upstream process. The question text module marks the user's actual speech-to-text query content. Finally, the multimodal data association information and the question text are concatenated according to a preset format to form a unified input sequence, which is then input into the BART model to finally output a text answer.

[0039] In this embodiment, by structurally concatenating multimodal data association information with user question text, and leveraging the BART model encoder's accurate association between multimodal information and question semantics, as well as the decoder's efficient generation capability of natural text, the output answer is guaranteed to accurately call upon exclusive information such as exhibit numbers and core knowledge points. Furthermore, the generated text possesses conversational and coherent characteristics, making it suitable for subsequent speech synthesis. This significantly improves the fluency of user interaction with the intelligent tour guide voice system during exhibition hall tours.

[0040] In summary, this application embodiment assesses speech signal noise interference and determines whether adaptive noise reduction measures are needed based on the assessment results. This helps solve the problem of speech signal distortion caused by complex background noise in exhibition halls, improves the accuracy of speech signals, and provides basic data support for the effective extraction of multi-layer semantics. After the speech signal noise interference assessment is qualified, key semantic recognition analysis of speech problems is performed. Based on the obtained key semantic recognition analysis results, it is determined whether key semantic recognition correction is needed. This helps improve the effective association ability of multi-layer semantics of complex speech problems and the accuracy of key requirement identification. Key semantic recognition correction is used to correct omissions or errors in keyword extraction, complete missing logical nodes in the semantic chain, and improve the accuracy of key requirement identification of complex speech problems. After the key semantic recognition analysis of speech problems is qualified, speech problem response judgment is performed to determine whether a triplet retrieval strategy is needed. This helps reduce invalid question retrieval steps and improve the response efficiency of complex speech problems. The triplet retrieval strategy is used to improve the response efficiency of complex speech problems.

[0041] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data, characterized in that: include: The system includes a voice signal noise interference monitoring module, a key semantic recognition monitoring module, and a voice question-and-answer response efficiency monitoring module. The voice signal noise interference monitoring module is used to evaluate the voice signal noise interference during the semantic processing of voice question and answer in the intelligent tour guide of the exhibition hall based on multimodal data. Based on the obtained voice signal noise interference evaluation results, it is determined whether adaptive noise reduction measures need to be taken. The adaptive noise reduction measures are used to reduce the interference of complex background noise on the collected tourist voice signals and improve the clarity and recognizability of the voice signals. The key semantic recognition and monitoring module is used to perform key semantic recognition analysis of speech problems after the speech signal noise interference assessment is qualified. Based on the obtained key semantic recognition analysis results of speech problems, it is determined whether key semantic recognition correction is needed. The key semantic recognition correction is used to improve the accuracy of key requirement identification for complex speech problems. The voice question-and-answer response efficiency monitoring module is used to judge the voice question answer response after the voice question key semantic recognition and analysis is qualified, so as to determine whether the triplet retrieval strategy needs to be adopted. The triplet retrieval strategy is used to improve the response efficiency of complex voice questions.

2. The intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data according to claim 1, characterized in that, The specific process for assessing noise interference in the speech signal is as follows: The voice signal noise interference assessment data item is obtained during the process of inputting multimodal data into the intelligent guide interactive terminal of the exhibition hall to assess the degree of noise interference of the voice signal. The speech signal noise interference assessment data items include the speech signal-to-noise ratio coefficient verification value and the signal distortion verification value; The voice signal-to-noise ratio (SNR) coefficient verification value is the result of quantifying the proportion of the SNR coefficient and the preset SNR threshold during a preset noise interference assessment period, and then weighting it with the SNR coefficient weight influence value. It is used to reflect the severity of the effective components of the voice signal being masked by background noise. The signal distortion verification value is the result of quantifying the proportion of signal distortion to a preset noise interference assessment time period and a preset signal distortion threshold, and then weighting it with the signal distortion weight influence value. It is used to reflect the degree of damage to the semantic integrity of the speech signal caused by background noise. The result of harmonic averaging of the speech signal noise interference assessment data items is used as the speech signal noise interference assessment index. When the speech signal noise interference assessment index is less than the preset speech signal noise interference threshold, key semantic recognition analysis of speech problems is performed; otherwise, adaptive noise reduction measures are adopted.

3. The intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data according to claim 2, characterized in that, The adaptive noise reduction measures refer to the sequential implementation of multi-band filtering noise reduction measures to specifically reduce the background noise power of different frequency bands in the speech signal, and frequency band synthesis to integrate the speech signals of each frequency band after the multi-band filtering noise reduction measures to restore the complete speech signal waveform. The specific process of the multi-band filtering and noise reduction measures is as follows: If the detected frequency value is greater than the preset high-frequency noise threshold, the corresponding speech signal frequency band is marked as the high-frequency band of the speech signal and wavelet threshold filtering is applied. If the detected frequency value is greater than the preset low-frequency noise threshold but not greater than the preset high-frequency noise threshold, the corresponding speech signal frequency band is marked as the speech signal mid-frequency band and Wiener filtering is applied. If the detected frequency value is not greater than the preset low-frequency noise threshold, the corresponding speech signal frequency band is marked as the low-frequency band of the speech signal, and least mean square adaptive filtering is applied. The frequency value is represented by the frequency components of each frequency point in the spectrum obtained after processing the speech signal through a short-time Fourier transform. The specific process of wavelet threshold filtering is as follows: The speech signal noise interference assessment index and the high-frequency noise energy ratio are input into the wavelet threshold parameter mapping set for querying to obtain wavelet coefficients. The high-frequency noise energy ratio is represented by the ratio of high-frequency noise energy to the total energy of the preset speech signal; The wavelet threshold is gradually reduced by using the adjustment amplitude corresponding to the wavelet coefficients as the adjustment step size. The voice signal noise interference assessment index is continuously monitored. When the voice signal noise interference assessment index is less than the preset voice signal noise interference threshold, frequency band synthesis is performed. Otherwise, wavelet threshold filtering is continued. If the number of wavelet threshold filtering operations exceeds the preset maximum number of wavelet threshold filtering operations, and the voice signal noise interference assessment index is still not less than the preset voice signal noise interference threshold, a wavelet processing failure alarm is sent.

4. The intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data according to claim 3, characterized in that, The specific process of Wiener filtering is as follows: The Wiener filter coefficients are obtained by querying the speech signal noise interference assessment index, the mid-frequency noise energy ratio, and the noise reference signal into the Wiener filter parameter mapping set. The mid-frequency noise energy ratio is represented by the ratio of mid-frequency noise energy to the total energy of the preset speech signal. The mid-frequency noise energy is represented by the sum of the squared differences between the endpoint signal strength value and the starting signal strength value of each mid-frequency noise during the preset noise interference assessment time period. The noise reference signal represents the signal obtained after the blocking matrix performs noise extraction processing on the speech signal; When the mid-frequency band signal-to-noise ratio is greater than the preset signal-to-noise ratio retention threshold, the adjustment step size is used to gradually reduce the filtering strength, and vice versa. The adjustment step size is used to gradually increase the filtering strength, and the speech signal noise interference evaluation index is continuously monitored. When the speech signal noise interference evaluation index is less than the preset speech signal noise interference threshold, frequency band synthesis is performed. Otherwise, Wiener filtering is continued. When the number of Wiener filtering operations is greater than the preset maximum number of Wiener filtering operations, if the speech signal noise interference evaluation index is still not less than the preset speech signal noise interference threshold, a Wiener filtering failure alarm is sent. The mid-frequency signal-to-noise ratio is represented by the ratio of effective mid-frequency speech energy to mid-frequency noise energy. The effective mid-frequency speech energy is represented by the sum of the squared differences between the endpoint signal strength value and the starting signal strength value of each effective mid-frequency speech signal within a preset noise interference assessment time period. The mid-frequency noise energy is represented by the sum of the squared differences between the endpoint signal strength value and the starting signal strength value of each mid-frequency noise signal within a preset noise interference assessment time period.

5. The intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data according to claim 4, characterized in that, The specific process of the least mean square adaptive filtering is as follows: The speech signal noise interference assessment index and the low-frequency noise energy ratio are input into the least mean square adaptive filter parameter mapping set for querying to obtain the adaptive filter coefficients. The low-frequency noise energy ratio is represented by the ratio of low-frequency noise energy to the total energy of the preset speech signal; The adjustment step size is used to gradually reduce the noise gain in the low-frequency band by using the adjustment amplitude corresponding to the adaptive filter coefficient; The voice signal noise interference assessment index is continuously monitored. When the voice signal noise interference assessment index is less than the preset voice signal noise interference threshold, frequency band synthesis is performed. Otherwise, the least mean square adaptive filtering is continued. If the number of least mean square adaptive filtering processes is greater than the preset maximum number of adaptive filtering processes, and the voice signal noise interference assessment index is still not less than the preset voice signal noise interference threshold, an adaptive filtering failure alarm is sent.

6. The intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data according to claim 5, characterized in that, The specific process of frequency band synthesis is as follows: The frame overlap ratio is adjusted to reduce splicing distortion between frequency bands. The specific process is as follows: The optimal frame overlap ratio is obtained by querying the frequency band boundary frequency difference and the noise energy ratio of each frequency band into the frame overlap ratio parameter mapping set. The frequency band boundary frequency difference is obtained by accumulating the difference between the boundary frequency values ​​of two adjacent speech signal frequency bands, and then representing the result by quantizing the proportion of the difference with the number of speech signal frequency bands. The noise energy proportion of each frequency band includes the high-frequency noise energy proportion value, mid-frequency noise energy proportion value, and low-frequency noise energy proportion value re-obtained after multi-frequency band filtering and noise reduction measures. The overlapping regions of signal frames in each frequency band are set based on the optimal frame overlap ratio, and the speech signals in each overlapping region are processed by weighted averaging to finally output a qualified synthesized noise-reduced speech signal.

7. The intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data according to claim 1, characterized in that, The speech problem key semantic recognition and analysis includes extracting key semantics sequentially and evaluating the accuracy of speech problem key semantic recognition. The key semantic extraction is based on speech-to-text methods and natural language processing algorithms to extract key semantics and obtain an initial set of keywords. The specific process for evaluating the accuracy of key semantic recognition in speech problems is as follows: Obtain semantic evaluation data items for measuring the accuracy of semantic recognition. The semantic evaluation data items include key semantic matching rate for measuring the degree of fit between the extracted keywords and the keywords in the preset exclusive thesaurus, and semantic completeness for measuring the comprehensiveness of the coverage of core semantic information. The key semantic matching rate is represented by the ratio of the number of keywords in the initial keyword set that successfully match keywords in the preset exclusive thesaurus to the total number of keywords in the initial keyword set. The semantic completeness is represented by the ratio of the number of core semantic units in the initial keyword set to the total number of preset core semantic units; The result of weighted coupling of semantic evaluation data items and corresponding semantic evaluation data item weight parameters is used as the semantic recognition accuracy index. If the semantic recognition accuracy index is greater than the preset accuracy threshold, then the voice question answer response is judged; otherwise, key semantic recognition correction is performed.

8. The intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data according to claim 7, characterized in that, The specific process of key semantic recognition and correction is as follows: If the semantic recognition accuracy index is less than the preset deviation subdivision threshold, then keyword extraction deviation correction is performed. The keyword extraction deviation correction means retrieving the preset exclusive lexicon, comparing the initial keyword set with the keywords in the preset exclusive lexicon, obtaining the deviation keywords, performing secondary word segmentation based on the obtained deviation keywords, and then performing keyword extraction deviation correction again to complete the replacement and correction of the erroneous keywords. Conversely, semantic chain logic completion and correction are performed. The semantic chain logic completion and correction means retrieving a preset semantic association library to complete the missing logical nodes in the semantic chain; After the keyword semantic recognition correction is completed, the semantic recognition accuracy index is reacquired. If the semantic recognition accuracy index is still not greater than the preset accuracy qualification threshold, a recognition correction failure alarm is sent. Otherwise, the corresponding corrected keyword set is marked as the corrected qualified keyword set, and the voice question answer response is judged.

9. The intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data according to claim 1, characterized in that, The specific process for judging the voice question response is as follows: If the actual response time of a voice question does not reach the preset response time threshold during the voice question answering process, the current voice question answer generation process will continue. If the actual response time of the voice question has reached the preset response time threshold, continue to monitor whether to output the voice question answer; If a voice question answer is output, the complete voice question answer will be output to the smart guide interactive terminal currently used by the user in the exhibition hall. If no voice question answer is output, a triplet retrieval strategy will be adopted.

10. The intelligent exhibition hall navigation voice question-and-answer optimization system based on multimodal data according to claim 9, characterized in that, The specific process of the triplet retrieval strategy is as follows: Retrieve the preset question-and-answer triplet library, search for the question-and-answer triplet with the highest semantic matching degree in the set of qualified keywords, and obtain the optimal question-and-answer matching item; The preset answer template in the optimal question-and-answer matching item is merged and filled with the specific entity information in the corrected qualified keyword set to generate a complete voice question and answer that fits the context; The triplet retrieval strategy also includes triplet retrieval strategy effectiveness verification, and the specific process of triplet retrieval strategy effectiveness verification is as follows: Obtain the response time metric for voice question answers; If the response time of the voice question is less than the preset qualified response threshold, the triplet retrieval strategy is deemed effective, and the complete voice question answer is output to the smart guide interactive terminal currently used by the user in the exhibition hall. Otherwise, a prompt is sent to the preset personnel to re-collect multimodal data.

Images