Intelligent agent and voice interaction method for implementing voice interaction of virtual museum

Abstract

The application discloses an intelligent agent for realizing voice interaction of a virtual museum, comprising: a voice recognition output module, through which the intelligent agent realizes interaction with a user, realizes voice instruction recognition and voice output of the user; an LLM interface calling module, which is used for converting a user instruction into a standardized instruction or outputting a communication response text according to the user instruction; and an operation control execution module, which converts the standardized instruction generated by the LLM interface calling module into a specific operation action instruction, calls a browser kernel, controls conversion of a virtual museum 3D scene according to the action instruction, and drives interaction of the user with the virtual museum scene. The application solves the problem that the existing technology has the problem that the browsing process of the virtual museum depends on manual click and drag operations, and is inconvenient for people who are not good at operating, young or old. The application also discloses a guiding method for realizing voice interaction of a virtual museum.

Description

Technical Field

[0001] This invention belongs to the fields of information technology and virtual reality technology, and relates to an intelligent agent for realizing voice interaction in virtual museums. This invention also relates to a guidance method for realizing voice interaction in virtual museums. Background Technology

[0002] Virtual museums, as an emerging form of exhibition, use 3D modeling technology to extract various information from cultural relics, transforming traditional static cultural relics into dynamic digital information composed of images, texts, etc., providing visitors with an immersive and interactive experience, allowing audiences to gain a deeper understanding of the exhibits.

[0003] Intelligent agents can perform tasks independently, typically driven by one or more models. Their key functions include task decomposition and collaboration, information retrieval and processing, dynamic decision-making and feedback, making them a key research area for digital applications in daily life. For example, Microsoft launched Heston Bot, focusing on food and cooking opportunities as well as fashion; global clothing company H&M launched a chatbot that provides shopping suggestions based on photos of users' personal wardrobes; and the chatbot Replika is popular among young people aged 18 to 25. In 2019, the University of Oklahoma Library launched the intelligent robot Bizzy, which can intelligently interact with readers, retrieve information, and provide 24-hour consultation services. Xi Wang et al. developed a voice-driven interactive audio description guide, providing visual information interpretation for visually impaired visitors through voice descriptions. Shenzhen Library piloted an IM consultation robot, using a pre-set knowledge base to provide readers with real-time reference and consultation services. M Esau, D Lawo, et al. designed a voice assistant to assess the edibility of food, based on the voice agent prototype Fischer Fritz. H Lee, D Kim, et al. conducted research on consumer decision-making in voice-interactive intelligent shopping, finding that buyers experienced the most positive user experience in conversational styles that included "questioning preferences." Research by Emre Sezgin, Shona D'Arcy, and others found that voice-interactive conversational AI can improve the convenience of nursing management. S Westby, RJ Radke, and others found that the human-like quality of a voice assistant's voice negatively interacts with its helpfulness, reversing its impact on humanization and agency perception. S Park, M Whang, and others proposed an evaluation criterion that provides empirical evidence for verifying the empathic interaction of virtual intelligent agents through facial expression measurements. S Colakoglu, M Durmus, and others analyzed user interaction data with multimodal conversational agents in the Albert Health app, discovering user preferences for voice interaction and health information acquisition when interacting with the agent. Y Mekata, M Nakanishi, and others designed a multi-stage arousal level estimation intelligent system using deep learning, unaffected by individual differences.

[0004] The current virtual museum browsing process relies on manual clicking and dragging, which is inconvenient for people with mobility issues, young children, or the elderly. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent agent for realizing voice interaction in virtual museums, which solves the problem that the browsing process of virtual museums in the prior art relies on manual clicking and dragging operations, which is inconvenient for users who are not good at using their hands, or for young or elderly people.

[0006] Another object of the present invention is to provide a guidance method for realizing voice interaction in virtual museums.

[0007] The technical solution adopted in this invention is an intelligent agent for realizing voice interaction in a virtual museum, comprising:

[0008] The voice recognition output module enables interaction between the intelligent agent and the user, realizing the recognition and output of user voice commands; The LLM interface call module is used to convert user commands into standardized commands, or output communication response text based on user commands. The operation control execution module converts the standardized instructions generated by the LLM interface call module into specific operation action instructions, calls the browser kernel, and controls the transformation of the virtual museum 3D scene according to the action instructions, driving the user's interaction with the virtual museum scene.

[0009] Preferably, the voice recognition output module includes a voice recognition module and a voice output module. The voice recognition module performs dynamic user recording after being triggered by the user, collects the user's voice commands, and converts the user's voice commands into user command text before inputting it into the LLM interface call module. The voice output module receives the communication response text output by the LLM interface call module, cleans the text, and plays it using the pyttsx3 engine.

[0010] The second technical solution adopted in this invention is a guidance method for realizing voice interaction in a virtual museum, which uses the aforementioned intelligent agent for realizing voice interaction in a virtual museum and is implemented according to the following steps: Step 1: After the user starts accessing the site, the voice output module provides a voice introduction and inquires about the customer's needs. Then, the voice recognition module starts recording. The user inputs voice commands, and the voice recognition module collects the user's voice commands and converts them into user command text. Step 2: The LLM interface call module converts the user command text into standardized command text or communication response text; Step 3: The operation control execution module converts the standardized instructions generated by the LLM interface call module into specific actions, calls the browser kernel, controls the transformation of the virtual museum 3D scene, and drives the user's interaction with the virtual museum scene.

[0011] Preferably, after a user begins accessing the site, the voice output module provides a voice introduction and inquires about the customer's needs, and issues a prompt to inform the user to "speak," after which the voice recognition module starts recording. The voice output module accepts the communication response text output by the LLM interface calling module, cleans the text, removes special characters and instruction markers, merges spaces, and then plays it using the pyttsx3 engine.

[0012] Preferably, in step 1, during the recording process, the speech recognition module acquires audio in real time and determines whether the acquired audio is silent. When it is determined to be silent, the recording stops, the acquired audio is stored as a temporary WAV file with a format conforming to PCM_16 encoding, and then the stored audio is sampled and preprocessed in a mono Whisper model. Then, the audio signal preprocessed by the Whisper model is converted into text, and then the text is cleaned to form the user command text.

[0013] Preferably, determining whether the collected audio is silent specifically involves: Calculate the root mean square of audio frames As a quantized signal energy: (1) in, It is the number of sampling points in a single frame of audio. It is the first Audio data from each sampling point; Set a mute threshold when the root mean square of the audio frames is... If the noise level is below the set mute threshold, it is considered to be in silent mode.

[0014] Preferably, the Whisper model performs audio sampling rate conversion and preprocessing as follows: Perform sample rate conversion on the acquired audio: (2) in, To collect discrete time series of audio. The sampling rate for audio acquisition. This is the audio after sample rate conversion; ( ) represents the resampling operation of the signal; Calculate the power spectral density of the audio after sampling rate conversion. : (3) in, for The time-frequency domain complex representation, For frame index, Frequency index; Speech features are obtained by filtering the power spectral density using a Mel filter bank. : (4) in, =400, which is the FFT frame length, corresponding to a 25ms time window and a 16kHz sampling rate; It is the first The frequency response weights of each Mel filter; right Logarithmic compression is performed to obtain the logarithmically compressed speech features. : (5) in, It is a very small positive number; Use encoder Perform feature extraction: (6) in, H represents the encoder, and H represents the features extracted by the encoder. Generate using decoder sequences: (7) in, , These are the weight matrix and bias vector of the decoder output layer. The softmax() operation maps the input vector to a probability distribution. Is the decoder in the 1st The text token predicted at each time step; It is a sequence of historical texts that has already been generated; It is the decoder at the time step The hidden state; Generate sequences for the decoder; Decode the sequence generated by the decoder into text y: (8).

[0015] Preferably, the text cleanup in step 1 to form the user instruction text specifically involves: Text based on word error rate and character error rate The process is performed to obtain the user command text.

[0016] Preferably, step 2 specifically involves: the LLM interface calling module calling the DeepSeek model through the application programming interface (API) to achieve language understanding, knowledge fusion, instruction generation, and context association, converting user instruction text into standardized instruction text or communication response text; In the language understanding stage, the system parses the text converted from the user's speech, identifies the user's intent, and then determines whether to directly ask and answer questions or control the interface through the operation control module. If a direct answer is required, a communication response text is generated; if interface control is required, standardized instruction text is generated. During the generation of the communication response text, local museum documents are combined with online search results to perform knowledge fusion and answer query-type questions, meeting the user's questioning needs. During the instruction generation process, standardized operation instruction text is generated based on the user's browsing intent input by voice and then transmitted to the operation control module. In the context association stage, the dialogue history is stored to support continuous interaction and provide a high-quality dialogue experience.

[0017] Preferably, in step 3, the operation control execution module converts the standardized instructions generated by the LLM interface call module into specific operation actions, calls the browser kernel, controls the transformation of the 3D scene, and drives the user's interaction with the virtual museum scene; Before using the intelligent agent, a pre-check phase is performed on the operation control execution module to check the environment validity, verifying whether the browser is running, whether the program has not been terminated, and whether the existing instruction list is empty. If the check fails, the process is terminated. During the instruction traversal phase, the operation fields are obtained from the received instructions as the basis for judging subsequent operations. The instructions mainly include: undo operation, dialogue reply, program exit, open / close 3D scene, view rotation, mouse movement, target click, skip unknown instructions, and instruction delay. During the final stage, the operation control execution module terminates all threads and resources and records and provides feedback on the program's execution process.

[0018] The beneficial effects of this invention are: This invention designs a voice-interactive intelligent agent that uses natural language dialogue to drive the agent to perform functions such as explanation, museum scene switching, and selection and clicking of exhibits, thus providing greater convenience and allowing people with limited mobility to browse using voice, thereby expanding the user base. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of the intelligent agent used in this invention to realize voice interaction in a virtual museum; Figure 2 This is a logic diagram of the voice output module in the intelligent agent used to realize voice interaction in a virtual museum according to the present invention. Figure 3 This is a functional diagram of the LLM interface calling module in the intelligent agent used to realize voice interaction in a virtual museum according to the present invention; Figure 4 This is the operational logic diagram of the operation control execution module in the intelligent agent used to realize voice interaction in a virtual museum according to the present invention; Figure 5 This is a diagram showing the construction of the virtual museum in Embodiment 6 of the present invention; Figure 6 This is a schematic diagram of the virtual museum in Embodiment 6 of the present invention; Figure 7 This is a diagram of the voice interaction guidance interface in Embodiment 6 of the present invention; Figure 8 This is a curve showing the Mandarin recognition rate in Embodiment 6 of the present invention. Detailed Implementation

[0020] The following detailed description is provided in conjunction with specific implementation methods.

[0021] This invention relates to an intelligent agent for implementing voice interaction in virtual museums, the structure of which is as follows: Figure 1 As shown, it includes: The voice recognition output module enables interaction between the intelligent agent and the user, realizing the recognition and output of user voice commands; The LLM interface call module converts user commands into standardized commands by calling local and online data, or outputs communication response text based on user commands. The operation control execution module converts the standardized instructions generated by the LLM interface call module into specific operation action instructions, calls the browser kernel, and controls the transformation of the virtual museum 3D scene according to the action instructions, driving the user's interaction with the virtual museum scene.

[0022] The speech recognition output module includes a speech recognition module and a speech output module. The speech recognition module performs dynamic user recording after being triggered by the user, collects the user's voice commands, and converts the user's voice commands into user command text before inputting it into the LLM interface call module. The voice output module receives the communication response text output by the LLM interface call module, cleans the text, and plays it using the pyttsx3 engine.

[0023] Example 2 The present invention provides a guidance method for implementing voice interaction in a virtual museum. It employs the intelligent agent described in Example 1 for implementing voice interaction in a virtual museum, and is implemented according to the following steps: Step 1: After the user starts accessing the site, the voice output module provides a voice introduction and inquires about the customer's needs. Then, the voice recognition module starts recording. The user inputs voice commands, and the voice recognition module collects the user's voice commands and converts them into user command text. Step 2: The LLM interface call module converts the user command text into standardized command text or communication response text; Step 3: The operation control execution module converts the standardized instructions generated by the LLM interface call module into specific actions, calls the browser kernel, controls the transformation of the virtual museum 3D scene, and drives the user's interaction with the virtual museum scene.

[0024] Example 3 Based on Example 2, the voice output module receives the communication response text output by the LLM interface calling module, performs text cleaning, removes special characters and instruction markers, merges spaces, and then plays it using the pyttsx3 engine.

[0025] In step 1, after the user begins accessing the site, the voice output module provides a voice introduction and inquires about the customer's needs, and issues a prompt to inform the user to "speak". Subsequently, the voice recognition module starts recording.

[0026] Step 1: During the recording process, the speech recognition module acquires audio in real time at a sampling frequency of 22050Hz, a maximum sampling duration of 60s, and a minimum sampling duration of 5s. It also determines whether the acquired audio is silent. If it is silent, the recording stops, and the acquired audio is stored as a temporary WAV file in a PCM_16 encoded format. The stored audio is then subjected to sampling rate conversion and preprocessing in a mono Whisper model. The preprocessed audio signal is then converted into text, followed by text cleaning and mapping of fuzzy instructions to form the user instruction text.

[0027] The specific steps to determine whether the captured audio is muted are as follows: Calculate the root mean square of audio frames As a quantized signal energy: (1) in, It is the number of sampling points in a single frame of audio. It is the first Audio data from each sampling point; Set a mute threshold when the root mean square of the audio frames is... If the noise level is below the set mute threshold, it is considered to be in silent mode.

[0028] Example 4 Based on Example 3, in the Whisper model recognition stage, a pre-trained Whisper model in small mode is loaded to balance recognition accuracy and speed, and the WAV file is converted into 16000Hz mono audio, specifically as follows: Perform sample rate conversion on the acquired audio: (2) in, To collect discrete time series of audio. The sampling rate for audio acquisition. =16000Hz, This is the audio after sample rate conversion; ( ) represents the resampling operation of the signal; Calculate the power spectral density of the audio after sampling rate conversion. : (3) in, for The time-frequency domain complex representation, For frame index, Frequency index; Speech features are obtained by filtering the power spectral density using a Mel filter bank. : (4) in, =400, which is the FFT frame length, corresponding to a 25ms time window and a 16kHz sampling rate; It is the first The frequency response weights of each Mel filter; right Logarithmic compression is performed to obtain the logarithmically compressed speech features. : (5) in, It is a very small positive number, to prevent taking the logarithm of zero; Next, Mel spectrum features need to be extracted, and feature mapping performed on the Mel spectrum to capture the temporal and semantic information of the audio. Specifically, this involves using an encoder to... Perform feature extraction: (6) in, H represents the encoder, and H represents the features extracted by the encoder. Generate using decoder sequences: (7) in, , These are the weight matrix and bias vector of the decoder output layer. The softmax() operation maps the input vector to a probability distribution. Is the decoder in the 1st The text token predicted at each time step; It is a sequence of historical texts that has already been generated; It is the decoder at the time step The hidden state; Generate sequences for the decoder; Decode the sequence generated by the decoder into text y: (8).

[0029] Step 1 involves text cleanup to generate user instruction text, specifically: Based on word error rate and character error rate Formula to text The process is performed to obtain the user command text.

[0030] The core of the voice output module is "text cleanup + pyttsx3 engine playback", which is also linked to user visual feedback, and the process is as follows: Figure 2 As shown. During the text input and cleanup phase, the system receives input text from the LLM interface call module, removes special characters and instruction markers, and merges spaces. During the TTS engine initialization and configuration phase, the `pyttsx3.Engine` function is initialized, and core parameters such as speech rate, volume, and voice type are set. During the voice playback and status linkage phase, trigger response points are set, different state animations are switched, and exception handling and thread safety mechanisms are implemented to ensure that the TTS engine is not concurrently called in a multi-threaded environment, avoiding conflicts.

[0031] Example 5 Based on Example 4, step 2 specifically involves: the LLM interface calling module calling the DeepSeek model through the application programming interface (API) to achieve language understanding, knowledge fusion, instruction generation, and context association, converting user instruction text into standardized instruction text or communication response text; In the language understanding stage, the system parses the text converted from the user's speech, identifies the user's intent, and then determines whether to directly ask and answer questions or control the interface through the operation control module. If a direct answer is required, a communication response text is generated; if interface control is required, standardized instruction text is generated. During the generation of the communication response text, local museum documents are combined with online search results to perform knowledge fusion and answer query-type questions, meeting the user's questioning needs. During the instruction generation process, standardized operation instruction text is generated based on the user's browsing intent input by voice and then transmitted to the operation control module. In the context association stage, the dialogue history is stored to support continuous interaction and provide a high-quality dialogue experience.

[0032] The LLM interface invocation module is the "brain" of the intelligent agent, responsible for converting user text commands into executable system operations such as mouse control, view rotation, and dialogue responses. It is the core middleware connecting user input and system functions. This invention's LLM interface invocation module utilizes the DeepSeek model for natural language understanding and command generation, integrating local knowledge and contextual information to ensure accurate understanding and execution of user intent. The main functions of the LLM interface invocation module are as follows: Figure 3 As shown.

[0033] In step 3, the operating logic of the operation control execution module is as follows: Figure 4 As shown: The standardized instructions generated by the LLM interface calling module are converted into specific operation actions, which in turn call the browser kernel to control the transformation of the 3D scene and drive the user's interaction with the virtual museum scene. Before using the intelligent agent, a pre-check phase is performed on the operation control execution module to check the environment validity, verifying whether the browser is running, whether the program has not been terminated, and whether the existing instruction list is empty. If the check fails, the process is terminated. During the instruction traversal phase, the operation fields are obtained from the received instructions as the basis for judging subsequent operations. The instructions mainly include: undo operation, dialogue reply, program exit, open / close 3D scene, view rotation, mouse movement, target click, skip unknown instructions, and instruction delay. During the final stage, the operation control execution module terminates all threads and resources and records and provides feedback on the program's execution process.

[0034] Example 6 Based on Example 5, taking Fukang City Museum as an example, a corresponding virtual museum voice interaction intelligent agent is built: Fukang City Museum is a state-owned museum under the cultural relics system located at No. 421 Bofeng West Road, Fukang City, Xinjiang Uygur Autonomous Region. It was established in 1993 and was formerly known as Fukang Tianchi Museum. It shares offices with the Fukang City Cultural Relics Protection and Management Office. It currently has 5,308 pieces (sets) of collections, including 3 second-class cultural relics and 20 third-class cultural relics. The exhibits cover stone tools, bronzes, porcelain and other types, showing the historical context of Fukang from the pre-Qin period to modern times.

[0035] The Insta360 Pro2 professional panoramic camera was used as the core acquisition device to collect 360-degree, blind-spot-free images of the Fukang City Museum. After acquisition, as follows... Figure 5 As shown, Insta360 Stitcher software was used for automated image stitching to generate high-precision digital panoramic material covering the entire exhibition hall.

[0036] By combining panoramic image data, a visitor flow map was created to provide spatial coordinates for interactive scene transitions. Simultaneously, in-depth data collection of museum special exhibition information, vertical distribution, and core exhibition area data was conducted. Through image recording and video production, diverse materials were prepared for subsequent digital upgrades such as virtual museum content iteration, artifact storytelling, and the construction of intelligent guide systems. The pre-collected and processed panoramic materials were imported into Nibiru Creator 3D interactive software, and a reasonable browsing path was designed based on hand-drawn spatial flow diagrams. In the software's flowchart editing interface, two interaction logics were used to achieve scene transitions: first, based on the default transition logic mechanism, spatial switching was achieved by adding scene hotspots; second, the event-based transition function was used, setting a countdown to trigger automatic scene transitions. After completing scene connections, the hotspot icons in the user interface were finely adjusted. By optimizing icon layout and standardizing name labeling, the interactive guidance was ensured to highly match the actual museum visitor flow and exhibition content, thus constructing a virtual museum, such as... Figure 6 As shown.

[0037] Based on users' usual methods of accessing traditional virtual museums, 12 commonly used language commands have been compiled as browsing instructions, as shown in Table 1: Table 1 Browsing Instructions

[0038] The interactive intelligent agent of this invention is run in a Python environment based on an AMD R7 processor and the PyCharm compiler.

[0039] like Figure 7 The image shows a voice-guided 3D scene interface. Users can use voice commands to control the intelligent agent, changing perspectives, moving the mouse, selecting and clicking targets, and switching locations to meet their browsing needs. Users can also communicate with the intelligent agent, providing information such as opening hours, exhibit descriptions, and standard browsing routes.

[0040] More than 60 volunteers were organized and divided into four levels according to their Mandarin proficiency to conduct speech recognition tests. The following figure shows the results of the speech recognition accuracy test. When the speech interaction is within 5 characters, the average recognition accuracy is 93.75%; when the speech interaction is 5-10 characters, the average recognition accuracy is 93.25%; when the speech interaction is more than 10 characters, the average recognition accuracy is 90.25%, and the recognition accuracy for Mandarin speakers of Grade A or above is over 86%. Figure 8 It can be seen that as Mandarin proficiency improves, recognition accuracy will effectively increase; as the number of characters decreases, recognition accuracy will also increase.

[0041] Table 2 shows the time taken for voice command recognition and execution. The time taken for regular commands is less than 4 seconds, which can meet the user's real-time browsing needs.

[0042] Table 2 Instruction Execution Time

Claims

1. An intelligent agent for realizing voice interaction in a virtual museum, characterized in that, include: The voice recognition output module enables interaction between the intelligent agent and the user, realizing the recognition and output of user voice commands; The LLM interface call module is used to convert user commands into standardized commands, or output communication response text based on user commands. The operation control execution module converts the standardized instructions generated by the LLM interface call module into specific operation action instructions, calls the browser kernel, and controls the transformation of the virtual museum 3D scene according to the action instructions, driving the user's interaction with the virtual museum scene.

2. The intelligent agent for realizing voice interaction in a virtual museum according to claim 1, characterized in that, The speech recognition output module includes a speech recognition module and a speech output module. The speech recognition module performs dynamic user recording after being triggered by the user, collects the user's voice commands, and converts the user's voice commands into user command text before inputting it into the LLM interface call module. The voice output module receives the communication response text output by the LLM interface call module, cleans the text, and plays it using the pyttsx3 engine.

3. A guidance method for realizing voice interaction in virtual museums, characterized in that, The specific steps are as follows: Step 1: After the user starts accessing the site, the voice output module provides a voice introduction and inquires about the customer's needs. Then, the voice recognition module starts recording. The user inputs voice commands, and the voice recognition module collects the user's voice commands and converts them into user command text. Step 2: The LLM interface call module converts the user command text into standardized command text or communication response text; Step 3: The operation control execution module converts the standardized instructions generated by the LLM interface call module into specific actions, calls the browser kernel, controls the transformation of the virtual museum 3D scene, and drives the user's interaction with the virtual museum scene.

4. The guidance method for realizing voice interaction in a virtual museum according to claim 3, characterized in that, In step 1, after the user starts accessing the site, the voice output module provides a voice introduction and asks the customer about their needs, and then issues a prompt to inform the user to "speak". Subsequently, the voice recognition module starts recording. The voice output module accepts the communication response text output by the LLM interface calling module, cleans the text, removes special characters and instruction markers, merges spaces, and then plays it using the pyttsx3 engine.

5. The guidance method for realizing voice interaction in a virtual museum according to claim 4, characterized in that, Step 1: During the recording process, the speech recognition module collects audio in real time and determines whether the collected audio is silent. When it is determined to be silent, the recording stops, and the collected audio is stored as a temporary WAV file with a PCM_16 encoding. Then, the stored audio undergoes sampling rate conversion and preprocessing in a mono Whisper model. The audio signal preprocessed by the Whisper model is then converted into text, and further text cleaning is performed to form the user command text.

6. The guidance method for realizing voice interaction in a virtual museum according to claim 5, characterized in that, The specific steps to determine whether the captured audio is muted are as follows: Calculate the root mean square of audio frames As a quantized signal energy: （1） in, It is the number of sampling points in a single frame of audio. It is the first Audio data from each sampling point; Set a mute threshold when the root mean square of the audio frames is... If the noise level is below the set mute threshold, it is considered to be in silent mode.

7. The guidance method for realizing voice interaction in a virtual museum according to claim 6, characterized in that, The Whisper model performs audio sampling rate conversion and preprocessing as follows: Perform sample rate conversion on the acquired audio: （2） in, To collect discrete time series of audio. The sampling rate for audio acquisition. This is the audio after sample rate conversion; ( ) represents the resampling operation of the signal; Calculate the power spectral density of the audio after sampling rate conversion. : （3） in, for The time-frequency domain complex representation, For frame index, Frequency index; Speech features are obtained by filtering the power spectral density using a Mel filter bank. : （4） in, =400, which is the FFT frame length, corresponding to a 25ms time window and a 16kHz sampling rate; It is the first The frequency response weights of each Mel filter; right Logarithmic compression is performed to obtain the logarithmically compressed speech features. : （5） in, It is a very small positive number; Use encoder Perform feature extraction: （6） in, H represents the encoder, and H represents the features extracted by the encoder. Generate using decoder sequences: （7） in, , These are the weight matrix and bias vector of the decoder output layer. The softmax() operation maps the input vector to a probability distribution. Is the decoder in the 1st The text token predicted at each time step; It is a sequence of historical texts that has already been generated; It is the decoder at the time step The hidden state; Generate sequences for the decoder; Decode the sequence generated by the decoder into text y: （8）。 8. The guidance method for realizing voice interaction in a virtual museum according to claim 7, characterized in that, Step 1 involves text cleanup to generate user instruction text, specifically: Text based on word error rate and character error rate The process is performed to obtain the user command text.

9. The guidance method for realizing voice interaction in a virtual museum according to claim 8, characterized in that, Step 2 specifically involves the LLM interface calling module calling the DeepSeek model through the application programming interface (API) to achieve language understanding, knowledge fusion, instruction generation, and context association, converting user instruction text into standardized instruction text or communication response text. During the language understanding stage, the text converted from the user's speech is parsed to identify the user's intent, and then it is determined whether to directly ask and answer questions or to control the interface through the operation control module. If it is determined that a direct answer is needed, a communication response text is generated; if it is determined that interface control is needed, a standardized instruction text is generated. In the process of generating the communication response text, local museum documents are combined with online search information to perform knowledge fusion and answer query-type questions to meet the user's questioning needs. In the process of generating instructions, standardized operation instruction text is generated based on the user's browsing intent input by voice and then transmitted to the operation control module. During the context association phase, the dialogue history is stored to support continuous interaction and provide a high-quality dialogue experience.

10. The guidance method for realizing voice interaction in a virtual museum according to claim 9, characterized in that, In step 3, the operation control execution module converts the standardized instructions generated by the LLM interface call module into specific operation actions, calls the browser kernel, controls the transformation of the 3D scene, and drives the user's interaction with the virtual museum scene; Before using the intelligent agent, a pre-check phase is performed on the operation control execution module to check the environment validity, verifying whether the browser is running, whether the program has not been terminated, and whether the existing instruction list is empty. If the check fails, the process is terminated. During the instruction traversal phase, the operation fields are obtained from the received instructions as the basis for judging subsequent operations. The instructions mainly include: undo operation, dialogue reply, program exit, open / close 3D scene, view rotation, mouse movement, target click, skip unknown instructions, and instruction delay. During the final stage, the operation control execution module terminates all threads and resources and records and provides feedback on the program's execution process.

Images