Human-computer interaction method and device, electronic equipment and storage medium

Abstract

The invention discloses a man-machine interaction method and device, electronic equipment and a storage medium, and relates to the field of artificial intelligence, and when a user performs interaction in a plot scene, obtained scene metadata is analyzed to obtain interaction interface display data, initial plot target data and context historical data associated with a plot. And when a user initiates interaction, calling a natural language processing model to generate a line text, a scene instruction and a to-be-synthesized audio parameter matched with a current plot according to a plot prompt word generated by the dialogue text, the initial plot target data and the context historical data. And further generating a response audio stream according to the line text and the audio parameters. And finally, integrating the line text, the voice stream and the visual scene of the response to form a complete plot interaction scene. And the interaction process of the user is more coherent and has stronger immersion, and the user experience is remarkably improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a human-computer interaction method, device, electronic device, and storage medium. Background Technology

[0002] With the continuous development of virtual interaction technology, human-computer interaction has gradually been applied in fields such as story generation, virtual character dialogue, and interactive entertainment. Currently, most human-computer interaction scenarios primarily involve users inputting information via text or voice, with the machine providing corresponding responses to drive the plot forward. While this method is relatively simple to implement and the interaction logic is relatively clear, it lacks a sense of interactive atmosphere, resulting in insufficient immersion and gradually failing to meet user needs. Summary of the Invention

[0003] In view of the above problems, this application provides a human-computer interaction method, device, electronic device, and storage medium to enhance the immersion and atmosphere of the interaction. The specific solution is as follows:

[0004] The first aspect of this application provides a human-computer interaction method, including:

[0005] In response to the interactive scene loading signal, the acquired scene metadata is parsed to obtain interactive interface display data, initial plot target data, and contextual history data related to the plot.

[0006] In response to user interaction signals, a natural language processing model is invoked to generate dialogue text, scene instructions, and audio parameters to be synthesized that are adapted to the current plot based on plot prompt words. The plot prompt words include at least: dialogue text entered by the user, the initial plot target data, and the contextual history data.

[0007] Generate a response audio stream based on the dialogue text and the audio parameters;

[0008] The plot display interface is generated based on the data displayed on the interactive interface, and the scene of the plot display interface is switched according to the scene instructions, while the response audio stream and the dialogue text are played synchronously.

[0009] In one possible implementation, the audio parameters include at least: speech rate and pause points, and the step of generating a response audio stream based on the dialogue text and the audio parameters includes:

[0010] Based on the speech rate, time segment markers are generated for each phoneme of the dialogue text, and synchronization cue points are embedded in the phonemes according to the pause points to obtain the response audio stream.

[0011] In one possible implementation, the human-computer interaction method further includes:

[0012] Based on the first synchronization cue point obtained from the text rendering channel and the second synchronization cue point obtained from the audio buffer channel, determine whether there is a time deviation between the dialogue text and the response audio stream;

[0013] When the time deviation is determined to exist, the playback status of the response audio stream is adjusted according to the degree of time deviation.

[0014] In one possible implementation, adjusting the playback state of the response audio stream based on the degree of time deviation includes:

[0015] When the time deviation is not less than 50ms, the playback rate of the response audio stream is adjusted according to the deviation duration, or a silent frame that matches the deviation duration is inserted into the response audio stream.

[0016] In one possible implementation, the human-computer interaction method further includes:

[0017] Based on the dialogue text, the scene instructions, and the audio parameters, a context vector is generated, and the context vector is extracted and stored in the database to provide a predictive reference for the next round of dialogue.

[0018] In one possible implementation, the initial plot target data includes at least: plot target description content; the contextual history data includes at least: context vectors or historical message sequences; the step of calling the natural language processing model to generate dialogue text, scene instructions, and audio parameters to be synthesized that are adapted to the current plot based on plot cues includes:

[0019] The natural language processing model is invoked to reference the context vector or the historical message sequence, and based on the dialogue text, with the goal of achieving a plot effect consistent with the plot target description, the dialogue text, the scene instructions, and the audio parameters to be synthesized are generated.

[0020] In one possible implementation, the human-computer interaction method further includes:

[0021] Based on the current position of the plot node, predict the plot scene of the next node, and load the corresponding scene map and display style according to the prediction results.

[0022] Based on the emotion or atmosphere tags associated with the dialogue text, the character avatars and expressions are pre-loaded;

[0023] Based on the changes in the aforementioned mood or atmosphere labels, adjust the lighting effects or filter resources accordingly.

[0024] A second aspect of this application provides a human-computer interaction device, comprising:

[0025] The scene data parsing module is used to respond to the interactive scene loading signal, parse the acquired scene metadata, and obtain interactive interface display data, initial plot target data, and contextual history data related to the plot.

[0026] The plot content generation module is used to respond to user interaction signals, call the natural language processing model to generate dialogue text, scene instructions and audio parameters to be synthesized that are adapted to the current plot based on plot prompt words. The plot prompt words include at least: dialogue text entered by the user, the initial plot target data and the context history data.

[0027] The voice data generation module is used to generate a response audio stream based on the dialogue text and the audio parameters; and,

[0028] The plot effect rendering module is used to generate a plot display interface based on the interactive interface display data, switch the scene of the plot display interface according to the scene instructions, and synchronously play the response audio stream and the dialogue text.

[0029] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the human-computer interaction method described in the first aspect or any implementation thereof.

[0030] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0031] The memory is used to store computer programs;

[0032] The processor is used to execute the computer program so that the electronic device can implement the human-computer interaction method of the first aspect or any implementation thereof.

[0033] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the human-computer interaction method described in the first aspect or any implementation thereof.

[0034] By employing the aforementioned technical solution, the human-computer interaction method provided in this application, when a user interacts within a narrative scenario, parses the acquired scene metadata to obtain interactive interface display data, initial narrative target data, and contextual history data associated with the narrative. When the user initiates an interaction, a natural language processing model is invoked to generate narrative prompts based on the dialogue text, initial narrative target data, and contextual history data. This generates dialogue text, scene instructions, and audio parameters to be synthesized that are appropriate for the current narrative. Further, a response audio stream is generated based on the dialogue text and audio parameters. Finally, the response dialogue text, audio stream, and visual scene are integrated to form a complete narrative interaction scenario. This makes the user's interaction process more coherent, provides a stronger sense of immersion, and significantly improves the user experience. Attached Figure Description

[0035] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0036] Figure 1 A footer diagram of a human-computer interaction system provided in this application;

[0037] Figure 2 A flowchart of a human-computer interaction method provided in this application;

[0038] Figure 3 Another flowchart of a human-computer interaction method provided in this application;

[0039] Figure 4 A structural diagram of a human-computer interaction device provided in this application;

[0040] Figure 5 This is a structural diagram of an electronic device provided in this application. Detailed Implementation

[0041] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0042] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0043] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0044] See Figure 1 , Figure 1 A schematic diagram of a human-computer interaction system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 can provide the human-computer interaction method provided in this embodiment to one or more terminals.

[0045] The terminal 100 can be equipped with a human-computer interaction application. The application and webpage can provide an interface. The terminal 100 can receive relevant parameters (such as interactive text) input by the user on the human-computer interaction interface and send the parameters to the server 200. The server 200 can obtain the processing result based on the received parameters and return the processing result to the terminal 100.

[0046] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters on its own, without the need for the server to cooperate. This application embodiment is not limited to this.

[0047] The following description Figure 1 The product form of the mid-terminal 100;

[0048] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, laptop computer, ultra-mobile personal computer (UMPC), personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0049] Terminal 100 may include a radio frequency unit, memory, input unit, display unit, camera (optional), audio circuitry (optional), speaker (optional), microphone (optional), headphone jack (optional), processor, external interface, power supply, and other components. Those skilled in the art will understand that the above-mentioned components are merely examples and do not constitute a limitation on the terminal or multifunctional device; it may include more or fewer components, or a combination of certain components, or different components.

[0050] The input unit can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the portable multi-functional device. Specifically, the input unit may include a touchscreen (optional) and / or other input devices. Other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0051] Among them, the input device can receive input data, etc.

[0052] The display unit can be used to display information input by the user or information provided to the user, various menus of the terminal, interactive interfaces, file display, and / or playback of any multimedia file. In the embodiments of this application, the display unit can be used to display the human-computer interaction interface, processing results, etc.

[0053] The memory can be used to store software code related to the human-computer interaction method, the processor can execute the steps of the human-computer interaction method, and can also schedule other units (such as the above-mentioned input unit and display unit) to achieve the corresponding functions.

[0054] This radio frequency unit (optional) can be used to receive and send signals during information transmission or calls.

[0055] In this embodiment of the application, the radio frequency unit can send data to the server 200 and receive the processing results sent by the server 200.

[0056] It should be understood that this radio frequency unit is optional and can be replaced with other communication interfaces, such as a network port.

[0057] Terminal 100 also includes a power source (such as a battery) for supplying power to the various components.

[0058] Terminal 100 also includes an external interface, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect terminal 100 to other devices for communication or to connect a charger to charge terminal 100.

[0059] Server 200 includes a bus, a processor, a communication interface, and memory. The processor, memory, and communication interface communicate with each other via the bus.

[0060] The memory can be used to store software code related to human-computer interaction methods, the processor can execute the steps of the chip's human-computer interaction methods, and can also schedule other units to achieve corresponding functions.

[0061] With the continuous development of technology, artificial intelligence has been widely applied in fields such as story generation, virtual character dialogue, and interactive entertainment. Currently used story-based applications can be broadly categorized as follows:

[0062] Pure text-based interactive story applications: Users advance the plot by clicking options or entering text. While relatively simple to implement and with clear interaction logic, they lack voice and visual elements, resulting in insufficient immersion.

[0063] Voice-driven dialogue applications: Users interact with models via voice, and the models generate voice responses. The advantage is that the interaction is natural and close to spoken communication. However, they usually only focus on the conversion between voice and text and lack the integration of story scenarios and visual elements.

[0064] Multimedia storytelling applications: These present the plot through a combination of text, voice-over, and animation. Their advantages include strong expressiveness and a certain degree of immersion, but they lack interactivity, typically offering one-way playback and limiting user engagement.

[0065] Current AI-based human-computer interaction is limited by the restriction of free input, requiring users to input information according to a relatively fixed format. Furthermore, the processing of text, voice, and context is relatively independent, often resulting in a lack of unified coordination between these elements. It also suffers from insufficient synchronization and a lack of contextual memory, leading to logical breaks in long conversations, a lack of dynamic guidance for plot objectives, and a relatively random user experience.

[0066] To address the aforementioned problems, embodiments of this application provide a human-computer interaction method. The human-computer interaction method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0067] Reference Figure 2 , Figure 2 This application provides a flowchart illustrating a human-computer interaction method as an embodiment. Figure 2 As shown in the figure, the human-computer interaction method provided in this application embodiment may include steps S201 to S204, which are described in detail below.

[0068] S201. In response to the interactive scene loading signal, the acquired scene metadata is parsed to obtain interactive interface display data, initial plot target data, and contextual history data related to the plot.

[0069] Specifically, when a user opens the interactive application and needs to engage in an interactive storyline, the backend server sends the necessary storyline package and scene metadata to the user's frontend application. This scene metadata may include: storyline identifiers, character lists, background resources, and initial storyline objective data. The initial storyline objective data may include: a description of the storyline, its priority, and the goal to be achieved (such as getting the user to purchase relevant items).

[0070] The user's front-end application parses the scene metadata to obtain the corresponding interactive interface display data (such as character avatars, backgrounds, dialog box templates, etc.), interprets the initial plot target data to obtain the target identifier, target description content and priority level, and reads the contextual history data related to the plot. The contextual history data here can be composed of the context vector of the current plot or the historical message sequence.

[0071] S202. In response to user interaction signals, the natural language processing model is invoked to generate dialogue text, scene instructions, and audio parameters to be synthesized that are adapted to the current plot based on the plot prompts. The plot prompts include at least: the dialogue text entered by the user, the initial plot target data, and the contextual history data.

[0072] Specifically, users can input text or voice interaction content through the terminal's input module, triggering the next development of the plot, or the application can trigger the next interactive dialogue based on the progress of the plot. At this time, a natural language processing model can be invoked, which may include generative models such as large language models. This natural language processing model judges the user's intent, determines the dialogue strategy, and generates response dialogue based on prompt words composed of user-input dialogue text, initial plot target data, and contextual history data. The final output includes: dialogue text, scene instructions (such as switching expressions, character backgrounds, etc.), and audio parameters to be synthesized (such as speech rate, timbre, pitch, pause duration, emotion tags, etc.).

[0073] S203. Generate the response audio stream based on the dialogue text and audio parameters.

[0074] Specifically, the dialogue text is synthesized using TTS (Text-to-Speech) based on the above audio parameters to obtain the corresponding response audio stream.

[0075] S204. Generate a plot display interface based on the interactive interface display data, and switch the scene of the plot display interface according to the scene instructions, and synchronously play the response audio stream and dialogue text.

[0076] Finally, while playing the audio stream of the response, the dialogue is displayed synchronously word by word / sentence by sentence on the interactive interface. Character portraits, backgrounds, and dialogue styles are also switched according to scene commands.

[0077] This human-computer interaction method achieves natural and smooth immersive interaction by uniformly coordinating text, voice, and visual scenes. It utilizes a large language model to achieve intelligent linkage between voice and text, ensuring smooth interaction. Furthermore, it maintains the continuity of long dialogues by introducing a contextual memory mechanism; combined with dynamic prompts for plot objectives, it guides users deeper into the story context, thereby significantly enhancing the interactive experience and immersion.

[0078] In a specific embodiment, to facilitate synchronization between the response audio stream and the dialogue text during synchronized playback, the audio parameters include at least: speech rate and pause points. The generation of the response audio stream based on the dialogue text and audio parameters may specifically include:

[0079] Based on the speech rate, time segment markers are generated for each phoneme of the dialogue text, and synchronization cue points are embedded in the phonemes according to the pause points to obtain the response audio stream.

[0080] Specifically, synchronization cues are specific time position markers embedded in the audio stream during its generation. These markers are machine-readable, non-audio signals used to pinpoint the exact timing of playback. When determining synchronization cues, they can be added using the sentence segmentation results of the response dialogue output by a natural language processing model.

[0081] Furthermore, based on the aforementioned synchronization prompts, a time discrepancy between the dialogue text and the response audio stream can be determined by using the first synchronization prompt obtained from the text rendering channel and the second synchronization prompt obtained from the audio buffer channel. If such a time discrepancy is confirmed, the playback status of the response audio stream is adjusted according to the degree of the discrepancy.

[0082] Specifically, during the TTS synthesis stage of the audio stream, time segment markers are generated for each phoneme, and the interval of synchronization cue points is adaptively adjusted based on the semantic segmentation results of the natural language processing model. During playback, playback deviations are calibrated in real time via a dual-channel synchronization bus (text rendering channel and audio buffer channel). Furthermore, when playback delay exceeds a threshold, a self-correction mechanism is triggered, automatically adjusting the playback rate or inserting a silence frame to resynchronize.

[0083] In the specific implementation of audio and text synchronization control, the playback status of the response audio stream can be adjusted according to the degree of time deviation. For example, when the time deviation is not less than 50ms, the playback rate of the response audio stream can be adjusted according to the deviation duration, or a silent frame that matches the deviation duration can be inserted into the response audio stream.

[0084] For example, when a time discrepancy is detected between audio and text rendering, the synchronization engine corrects it by slightly adjusting the audio playback rate (e.g., ±3%) or inserting 20ms–60ms silent frames, so that the timelines of the two are realigned and kept within a controllable threshold.

[0085] In some embodiments, to achieve real-time interaction, the human-computer interaction method further includes the following processing steps:

[0086] Based on the dialogue text, scene instructions, and audio parameters, a context vector is generated. The context vector is then summarized and stored in the database to provide a predictive reference for the next round of dialogue.

[0087] Specifically, the generated text, user input, scene changes, and time information can be used to update the contextual representation (e.g., updating dialogue state vectors, slots, and user preferences). These contextual vectors and summaries are written to persistent storage (local or cloud), and the scene resources needed for the next round can be predicted based on the current context, or the next TTS sentence can be pre-synthesized and handed over to the resource manager for execution (reducing latency). The updated contextual snapshot, target state, playback / interaction logs, and preloaded task list are persisted and can be used for the next round triggering. If an anomaly occurs (network, synthesis failure), a fallback strategy is triggered (displaying placeholder text, retrying, or providing a prompt).

[0088] In other embodiments, the initial plot target data includes at least: a plot target description, and the contextual history data includes at least: a context vector or a sequence of historical messages. The aforementioned invocation of a natural language processing model to generate dialogue text, scene instructions, and audio parameters to be synthesized that are adapted to the current plot based on plot cues may specifically include:

[0089] By invoking the natural language processing model and referencing the context vector or the historical message sequence, and based on the dialogue text, with the goal of achieving a plot effect consistent with the plot objective description, the system generates dialogue text, scene instructions, and audio parameters to be synthesized.

[0090] When constructing the prompt words, one can base them on the idea of ​​guiding a natural language processing model to make predictions. For example, the prompt words may specifically include:

[0091] "Based on the context vector or historical message sequence, as well as the dialogue text entered by the user, infer the user's intention, and combine it with the description of the plot goal to generate dialogue text, scene instructions, audio parameters to be synthesized, etc. to guide the user toward the development of the target plot."

[0092] It is understood that those skilled in the art can select and adjust the above-mentioned prompt word construction process as needed, and no restrictions are imposed here.

[0093] To further reduce processing latency and improve user experience, this human-computer interaction method also includes:

[0094] Based on the current position of the plot node, the plot scene of the next node is predicted, and the corresponding scene map and display style are loaded according to the prediction results.

[0095] Based on the emotion or atmosphere tags associated with the dialogue text, character avatars and expressions are pre-loaded.

[0096] Adjust the lighting effects or filter resources accordingly based on changes in mood or atmosphere labels.

[0097] Specifically, when predicting the resources needed for the next round of interaction, the following actions can be performed based on the current context vector (plot nodes, character states, emotion tags, scene changes, etc.):

[0098] Plot node prediction: Based on the current plot position, infer the possible next scene (such as changes in background or location), and preload relevant background images and interactive interface styles.

[0099] Role participation prediction: Based on role prompts or semantic tags output by the natural language processing model, preload the avatars and facial expressions of possible speaking roles.

[0100] Atmosphere change prediction: If the context indicates that the emotion or atmosphere will change (tension, anger, turning point, etc.), the system will preload the corresponding lighting effects or filter resources.

[0101] Context-based prediction: possible speaker roles, emotions, speech rate, tone, and voice model parameters. Then, it executes: pre-constructs a synchronous cue point template for the next sentence (phoneme time-slice structure), eliminating TTS cold start time and allowing the next sentence to play with almost no waiting time.

[0102] As a specific application and implementation of the above-mentioned human-computer interaction method, refer to Figure 3 The data flow shown indicates the direction of the human-computer interaction method, which may specifically include the following processing steps:

[0103] Initialization and scene loading (inputs: plot description, scene configuration, target configuration):

[0104] The backend / server sends the story package and scene metadata (story ID, character list, background resources, initial story objective) to the frontend / server.

[0105] Receiving module: story display module, target prompt module, historical message module.

[0106] Processing: The story display module parses the plot metadata and prepares UI resources (avatars, backgrounds, dialog templates); the target prompt module reads and initializes the target state; the history message module reads the contextual memories related to the plot from storage.

[0107] Output / Transmit Data: Ready scene resources (resource path / binary), initial target state (target ID, description, priority), and context snapshot (context vector or historical message sequence) are passed to the next step.

[0108] Interactive triggering and AI intelligent decision-making (input: triggering event = system-driven or user input), where AI refers to the processing and generation of the natural language processing model mentioned above:

[0109] Triggering conditions: The system triggers the next line according to the plot progression, or the user actively triggers it through the user input module (text / voice). Intelligent decision-making module (simultaneously reads historical context and current target).

[0110] Data Processing: The AI ​​generation and intelligent decision-making module feeds the input (user text / transcription, plot context, target state) into the NLP model for: intent determination, dialogue strategy decision, and dialogue generation. The generated output includes: dialogue text, semantic tags, emotion / speech rate suggestions, and character action tags (such as switching facial expressions or background instructions).

[0111] Output / transmit data: {text dialogue, metadata (speech rate, pause points), scene instructions, audio parameters to be synthesized} Assigned to: voice linkage module (synthesis / playback), scene-driven module (avatar / background update), and historical message module (write draft).

[0112] Multimodal rendering and real-time interaction (input text dialogue + scene commands + voice parameters):

[0113] The speech synthesis module receives text and metadata parameters, performs TTS synthesis (or calls the pre-synthesis buffer), generates an audio stream, and outputs playback instructions (timestamp, cue point). Simultaneously, the story display module receives text and renders it word-by-word / sentence-by-sentence on the interface.

[0114] The scene update module switches the avatar, background, or dialogue style according to scene instructions, and synchronizes the animation / transition.

[0115] Synchronization Control: The synchronization engine (a scheduling subsystem deployed within the voice-linked module) ensures precise time-axis alignment between audio playback and text rendering. This engine employs a dynamic synchronization cue point generation algorithm, generating time segment markers for each phoneme during the TTS synthesis stage and adaptively adjusting the synchronization cue point interval based on the semantic segmentation results from the AI ​​generation module. During playback, the system calibrates playback deviations in real time (accuracy ≤50ms) via a dual-channel synchronization bus (text rendering channel and audio buffer channel). If playback delay exceeds a threshold, a self-correction mechanism is triggered, automatically adjusting the playback rate or inserting a silence frame for resynchronization. Furthermore, a predictive rendering scheduler preloads the synchronization cue points for the next line of dialogue and caches the corresponding rendering data, achieving smooth transitions across modalities.

[0116] Record: Playback events, user interruptions, delays, etc. are written to the interaction log for use by the historical message module and the analysis module.

[0117] Output / transmit data: Synthetic audio (playback) presented to the user, visible text, updated scene status, and updated interaction logs.

[0118] Context update, persistence and preparation for the next round (input: current round of interaction artifacts + operation log):

[0119] The update and preparation module receives the text generated in this round, user input, scene changes, and time information, and updates the contextual representation (e.g., updating the dialogue state vector, slots, and user preferences). These contextual vectors and summaries are written to persistent storage (local or cloud).

[0120] Adjust the status of plot objectives (complete / pending / branch) based on the progress of the dialogue and AI strategies, and return the new objectives to the story display module for real-time display.

[0121] Preloading / Optimization: The system can predict the scene resources needed for the next round or pre-synthesize the next TTS based on the current context and hand it over to the resource manager for execution (reducing latency).

[0122] Final output: The updated context snapshot, target state, playback / interaction log, and preloaded task list are persisted and can be used for the next round of triggering. If an exception occurs (network, synthesis failure), a fallback strategy is triggered (displaying text placeholders, retrying, or prompting).

[0123] This human-computer interaction method integrates text generation, speech synthesis, scene-driven interaction, and contextual memory through a natural language processing model-driven interaction approach, forming a unified interaction system. This results in breakthroughs in immersion, multimodal collaboration, long dialogue context maintenance, and plot goal guidance, significantly improving the user experience.

[0124] The above describes a human-computer interaction method provided by the embodiments of this application. The following will describe the apparatus for performing the above human-computer interaction method.

[0125] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a human-computer interaction device provided in an embodiment of this application. Figure 4 As shown, the human-computer interaction device includes:

[0126] The scene data parsing module 401 is used to parse the acquired scene metadata in response to the interactive scene loading signal, and obtain interactive interface display data, initial plot target data, and contextual history data related to the plot.

[0127] The plot content generation module 402 is used to respond to user interaction signals, call the natural language processing model to generate dialogue text, scene instructions and audio parameters to be synthesized that are adapted to the current plot based on plot prompt words. The plot prompt words include at least: dialogue text entered by the user, initial plot target data and contextual history data.

[0128] The voice data generation module 403 is used to generate a response audio stream based on the dialogue text and audio parameters; and,

[0129] The plot effect rendering module 404 is used to generate a plot display interface based on the interactive interface display data, and switch the scene of the plot display interface according to the scene instructions, and synchronously play the response audio stream and dialogue text.

[0130] In one possible implementation, the audio parameters include at least: speech rate and pause points. The process by which the speech data generation module 403 generates a response audio stream based on the dialogue text and audio parameters includes:

[0131] Based on the speech rate, time segment markers are generated for each phoneme of the dialogue text, and synchronization cue points are embedded in the phonemes according to the pause points to obtain the response audio stream.

[0132] In one possible implementation, the voice data generation module 403 is also used for:

[0133] Based on the first synchronization cue point obtained from the text rendering channel and the second synchronization cue point obtained from the audio buffer channel, determine whether there is a time deviation between the dialogue text and the response audio stream;

[0134] When a time deviation is determined, the playback status of the response audio stream is adjusted according to the degree of time deviation.

[0135] In one possible implementation, the process by which the voice data generation module 403 adjusts the playback state of the response audio stream based on the degree of time deviation includes:

[0136] When the time deviation is not less than 50ms, the playback rate of the response audio stream is adjusted according to the deviation duration, or a silent frame that matches the deviation duration is inserted into the response audio stream.

[0137] In one possible implementation, the plot effect rendering module 404 is also used for:

[0138] Based on the dialogue text, scene instructions, and audio parameters, a context vector is generated. The context vector is then summarized and stored in the database to provide a predictive reference for the next round of dialogue.

[0139] In one possible implementation, the initial plot target data includes at least: plot target description content; the contextual history data includes at least: context vectors or historical message sequences; the plot content generation module 402 calls a natural language processing model to generate dialogue text, scene instructions, and audio parameters to be synthesized that are adapted to the current plot based on plot prompts, including:

[0140] By invoking the natural language processing model to reference context vectors or historical message sequences, and based on the dialogue text, with the goal of achieving a plot effect consistent with the content described in the plot objectives, the system generates dialogue text, scene instructions, and audio parameters to be synthesized.

[0141] In one possible implementation, the plot content generation module 402 is also used for:

[0142] Based on the current position of the plot node, predict the plot scene of the next node, and load the corresponding scene map and display style according to the prediction results.

[0143] Based on the emotion or atmosphere tags associated with the dialogue text, character avatars and expressions are pre-loaded;

[0144] Adjust the lighting effects or filter resources accordingly based on changes in mood or atmosphere labels.

[0145] This application also provides an electronic device in its embodiments. (See reference...) Figure 5 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0146] like Figure 5As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. When the electronic device is powered on, the RAM 503 also stores various programs and data required for the operation of the electronic device. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0147] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, memory cards, hard drives, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0148] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the human-computer interaction methods provided in this application.

[0149] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the human-computer interaction methods provided in this application.

[0150] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0151] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0152] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0153] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A human-computer interaction method, characterized in that, include: In response to the interactive scene loading signal, the acquired scene metadata is parsed to obtain interactive interface display data, initial plot target data, and contextual history data related to the plot. In response to user interaction signals, a natural language processing model is invoked to generate dialogue text, scene instructions, and audio parameters to be synthesized that are adapted to the current plot based on plot prompt words. The plot prompt words include at least: dialogue text entered by the user, the initial plot target data, and the contextual history data. Generate a response audio stream based on the dialogue text and the audio parameters; The plot display interface is generated based on the data displayed on the interactive interface, and the scene of the plot display interface is switched according to the scene instructions, while the response audio stream and the dialogue text are played synchronously.

2. The human-computer interaction method according to claim 1, characterized in that, The audio parameters include at least: speech rate and pause points. Generating a response audio stream based on the script text and the audio parameters includes: Based on the speech rate, time segment markers are generated for each phoneme of the dialogue text, and synchronization cue points are embedded in the phonemes according to the pause points to obtain the response audio stream.

3. The human-computer interaction method according to claim 2, characterized in that, Also includes: Based on the first synchronization cue point obtained from the text rendering channel and the second synchronization cue point obtained from the audio buffer channel, determine whether there is a time deviation between the dialogue text and the response audio stream; When the time deviation is determined to exist, the playback status of the response audio stream is adjusted according to the degree of time deviation.

4. The human-computer interaction method according to claim 3, characterized in that, The adjustment of the playback status of the response audio stream based on the degree of time deviation includes: When the time deviation is not less than 50ms, the playback rate of the response audio stream is adjusted according to the deviation duration, or a silent frame that matches the deviation duration is inserted into the response audio stream.

5. The human-computer interaction method according to any one of claims 1 to 4, characterized in that, Also includes: Based on the dialogue text, the scene instructions, and the audio parameters, a context vector is generated, and the context vector is extracted and stored in the database to provide a predictive reference for the next round of dialogue.

6. The human-computer interaction method according to claim 1, characterized in that, The initial plot target data includes at least: plot target description content; the contextual history data includes at least: context vectors or historical message sequences; the step of calling the natural language processing model to generate dialogue text, scene instructions, and audio parameters to be synthesized that are adapted to the current plot based on plot cues includes: The natural language processing model is invoked to reference the context vector or the historical message sequence, and based on the dialogue text, with the goal of achieving a plot effect consistent with the plot target description, the dialogue text, the scene instructions, and the audio parameters to be synthesized are generated.

7. The human-computer interaction method according to claim 5, characterized in that, Also includes: Based on the current position of the plot node, predict the plot scene of the next node, and load the corresponding scene map and display style according to the prediction results. Based on the emotion or atmosphere tags associated with the dialogue text, the character avatars and expressions are pre-loaded; Based on the changes in the aforementioned mood or atmosphere labels, adjust the lighting effects or filter resources accordingly.

8. A human-computer interaction device, characterized in that, include: The scene data parsing module is used to respond to the interactive scene loading signal, parse the acquired scene metadata, and obtain interactive interface display data, initial plot target data, and contextual history data related to the plot. The plot content generation module is used to respond to user interaction signals, call the natural language processing model to generate dialogue text, scene instructions and audio parameters to be synthesized that are adapted to the current plot based on plot prompt words. The plot prompt words include at least: dialogue text entered by the user, the initial plot target data and the context history data. The voice data generation module is used to generate a response audio stream based on the dialogue text and the audio parameters; and, The plot effect rendering module is used to generate a plot display interface based on the interactive interface display data, switch the scene of the plot display interface according to the scene instructions, and synchronously play the response audio stream and the dialogue text.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the human-computer interaction method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the human-computer interaction method as described in any one of claims 1 to 7.

Images