Man-machine cooperation method and system, electronic equipment and storage medium

Abstract

The invention provides a man-machine cooperation method and system, electronic equipment and a storage medium, and the method comprises the steps: enabling a server to carry out the dual-channel parallel coding of an audio data stream of a user through a dual-channel real-time voice encoder when a control instruction sent by a console is in a voice drive mode, and obtaining a first text and a rhythm-emotion vector; segmenting the first file to obtain a plurality of text fragments, and generating a structured data packet corresponding to each text fragment according to each text fragment and the rhythm-emotion vector; generating corresponding first audio data according to the structured data packet and the target timbre of the preset role, and sending a first audio data stream to the intelligent device, so that the intelligent device receives and outputs the first audio data; if the mode is the AI autonomous interaction mode, content of the content fragments output by the AI service is audited; and if the content verification is passed, generating second audio data according to the content segment, sending the second audio data to the intelligent device, and enabling the intelligent device to output the second audio data.

Description

Technical Field

[0001] This invention relates to the field of human-machine collaboration technology, and more specifically, to a human-machine collaboration method, system, electronic device, and storage medium. Background Technology

[0002] With the rise of Artificial Intelligence Generated Content (AIGC) and large-scale models, intelligent robots are rapidly moving from laboratories into various public interactive scenarios, such as large-scale events, live commercial broadcasts, and television programs. These interactive scenarios place unprecedented demands on the robots' interactive performance, on-site responsiveness, and content security.

[0003] In existing technologies, human-machine collaboration can be achieved through interactive systems based on preset scripts and rules, dialogue systems that directly integrate general large language models, or decentralized manual control systems. However, these methods are either too rigid and unable to cope with complex situations, or they rely too much on the autonomy of AI, which poses a huge broadcast risk.

[0004] Therefore, how to provide a way to effectively integrate AI capabilities and artificial intelligence, and offer an efficient, safe, and expressive human-machine collaboration method, is a problem that urgently needs to be solved. Summary of the Invention

[0005] In view of this, the present invention provides a human-machine collaboration method, system, electronic device and storage medium, with the aim of effectively integrating AI capabilities and artificial intelligence, and providing an efficient, safe and expressive human-machine collaboration method.

[0006] The first aspect of this application provides a human-machine collaboration method, the method comprising:

[0007] The system receives control commands sent from the console and determines the corresponding interaction mode based on the control commands; wherein the control commands are generated by the console based on user operations.

[0008] If the interaction mode is a voice-driven mode, the user's audio data stream is encoded in parallel using a dual-channel real-time voice encoder to obtain the first text and the prosody-emotion vector.

[0009] The first file is segmented to obtain multiple text fragments, and a structured data packet corresponding to each text fragment is generated based on each text fragment and the prosody-sentiment vector;

[0010] For each structured data packet, a corresponding first audio data is generated based on the structured data packet and the target timbre of the preset role, and the first audio data stream is sent to the smart device, so that the smart device receives and outputs the first audio data;

[0011] If it is an AI autonomous interaction mode, obtain the content fragments output by the AI ​​service and review the content fragments; wherein, the content fragments are generated by the AI ​​service based on the user's questions;

[0012] If the content is approved, second audio data is generated based on the content fragment, and the second audio data is sent to the smart device so that the smart device outputs the second audio data.

[0013] Optionally, the dual-channel real-time speech encoder includes channel one and channel two, wherein an encoder and its connected target decoder are deployed in channel one;

[0014] If the interaction mode is a voice-driven mode, a dual-channel real-time voice encoder is used to perform dual-channel parallel encoding on the user's audio data stream to obtain a first text and a prosody-emotion vector, including:

[0015] If the interaction mode is voice-driven mode, the audio data stream input by the user is collected;

[0016] The audio data stream is encoded using the encoder in channel one to obtain a hidden state sequence, and the hidden state sequence is decoded using the target decoder to obtain the first text;

[0017] Using the second channel, the prosody-sentiment regression network outputs the corresponding prosody-sentiment vector based on the hidden state sequence.

[0018] Optionally, the step of segmenting the first file to obtain multiple text fragments, and generating a structured data packet corresponding to each text fragment based on each text fragment and the prosody-sentiment vector, includes:

[0019] The first text is segmented to obtain multiple text fragments;

[0020] Extract the prosody-sentiment sub-vector matching each text segment from the prosody-sentiment vector;

[0021] Each text segment and its matching prosodic sentiment subvector are encapsulated to obtain a structured data packet for each text segment.

[0022] Optionally, for each structured data packet, generating corresponding first audio data based on the structured data packet and the target timbre of a preset role includes:

[0023] For each structured data packet, the embedding matrix of the text segments in the structured data packet is extracted;

[0024] Obtain the target timbre of the preset character and extract the embedding vector of the target timbre;

[0025] An input vector is generated based on the embedding matrix, the embedding vector, and the prosody-sentiment subvector in the structured data packet, and distribution parameters of the corresponding latent representation are generated based on the input vector;

[0026] Based on the distribution parameters of the latent representation and the embedding vector, a corresponding audio waveform is generated, and first audio data is generated based on the audio waveform.

[0027] Optionally, the method further includes:

[0028] If the interaction mode is a manual text takeover mode, extract the text from the control command;

[0029] A third audio data is generated based on the text and the target timbre, and the third audio data is sent to the smart device, so that the smart device receives and outputs the third audio data.

[0030] Optionally, the method further includes:

[0031] The target text and target speech prosody of the target audio data are obtained, wherein the target audio data is the first audio data, the second audio data, or the third audio data;

[0032] A contextual feature vector is generated based on the target text and the target speech prosody, and a target multi-dimensional label is selected from a set of pre-set multi-dimensional labels based on the contextual feature vector;

[0033] A target action sequence matching the target multi-dimensional label is determined, and the target action sequence is sent to the smart device, so that the smart device performs an action matching the target audio data according to the target action sequence.

[0034] A second aspect of this application provides a human-machine collaborative system, the system including a console, a server, and intelligent devices;

[0035] The console is used to generate control commands based on user operations and send the control commands to the server.

[0036] The server is configured to receive control commands sent by the console and determine the corresponding interaction mode based on the control commands. If the interaction mode is a voice-driven mode, a dual-channel real-time voice encoder is used to perform dual-channel parallel encoding on the user's audio data stream to obtain a first text and a prosody-emotion vector. The first file is segmented to obtain multiple text segments, and a structured data packet corresponding to each text segment is generated based on each text segment and the prosody-emotion vector. For each structured data packet, a corresponding first audio data is generated based on the structured data packet and the target timbre of a preset role, and the first audio data stream is sent to the smart device. If the interaction mode is an AI autonomous interaction mode, a content segment output by the AI ​​service is obtained, and the content segment is reviewed. The content segment is generated by the AI ​​service based on the user's question. If the content review is passed, second audio data is generated based on the content segment, and the second audio data is sent to the smart device.

[0037] The smart device is configured to output the first audio data when it receives the first audio data, and to output the second audio data when it receives the second audio data.

[0038] Optionally, the dual-channel real-time speech encoder includes channel one and channel two, wherein an encoder and its connected target decoder are deployed in channel one;

[0039] If the interaction mode is a voice-driven mode, a server that uses a dual-channel real-time voice encoder to perform dual-channel parallel encoding on the user's audio data stream to obtain the first text and prosody-emotion vector is specifically used for:

[0040] If the interaction mode is voice-driven mode, the audio data stream input by the user is collected;

[0041] The audio data stream is encoded using the encoder in channel one to obtain a hidden state sequence, and the hidden state sequence is decoded using the target decoder to obtain the first text;

[0042] Using the second channel, the prosody-sentiment regression network outputs the corresponding prosody-sentiment vector based on the hidden state sequence.

[0043] A third aspect of this application provides an electronic device, including: a processor and a memory, the processor and the memory being connected via a bus; wherein, the processor is used to call and execute a program stored in the memory; the memory is used to store the program, the program being used to implement the human-machine collaboration method provided in the first aspect of this application.

[0044] A fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions for performing the human-computer collaboration method provided in the first aspect of this application.

[0045] This application provides a human-machine collaboration method, system, electronic device, and storage medium. It receives control commands sent from a console and determines the corresponding interaction mode based on the control commands. The control commands are generated by the console based on user operations. If the interaction mode is voice-driven, a dual-channel real-time voice encoder is used to perform dual-channel parallel encoding on the user's audio data stream to obtain a first text and a prosody-emotion vector. The first file is segmented to obtain multiple text fragments, and a structured data packet corresponding to each text fragment is generated based on each text fragment and the prosody-emotion vector. For each structured data packet, corresponding first audio data is generated based on the structured data packet and the target timbre of a preset role, and the first audio data stream is sent to a smart device, enabling the smart device to receive and output the first audio data. If the interaction mode is AI-driven, content fragments output by an AI service are obtained, and the content fragments are reviewed. The content fragments are generated by the AI ​​service based on user questions. If the content review is passed, second audio data is generated based on the content fragments and sent to the smart device, enabling the smart device to output the second audio data. Therefore, the technical solution provided in this application, in voice-driven mode, allows the server to utilize a dual-channel real-time voice encoder to generate corresponding text and its prosody-emotion vector based on the user's input audio data stream, and to generate content-rich and emotional audio data based on the text and its prosody-emotion vector. This enables the control of intelligent devices to output content-rich and emotional audio data, thereby solving the problem of overly rigid output audio in existing technologies that cannot cope with complex situations. Furthermore, in AI autonomous interaction mode, this application can also review the content segments output by the AI ​​service. Only after the content review is passed will the corresponding audio data be output, thereby solving the problem of excessive reliance on AI autonomy in existing technologies, which poses a significant broadcast risk. Thus, adopting this application can effectively integrate AI capabilities and artificial intelligence, and provide an efficient, safe, and expressive human-machine collaboration method. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1This is a schematic diagram of the structure of a human-machine collaborative system provided in an embodiment of this application;

[0048] Figure 2 A schematic diagram of a human-machine collaboration process provided for an embodiment of this application;

[0049] Figure 3 An example diagram of a console provided for an embodiment of this application;

[0050] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0053] As can be seen from the above background technology, human-machine collaboration can currently be achieved through interactive systems based on preset scripts and rules, dialogue systems that directly integrate general large language models, or decentralized manual control systems.

[0054] Specifically, the core of an interactive system based on pre-set scripts and rules is to pre-write a large number of dialogue scripts and behavioral rules for the robot (such as "If the user asks 'Hello,' the robot answers 'Hello, how can I help you?'") so that corresponding human-machine collaboration can be achieved according to the pre-set dialogue scripts and behavioral rules. Although this method is completely controllable and will not have unexpected situations, and the technical implementation is relatively simple, the interactive capabilities are extremely limited and rigid. The robot can only understand and answer questions within the preset range. Once it exceeds the script, it cannot respond, cannot conduct multiple rounds of in-depth dialogue, and cannot deal with unexpected situations on site. For public performance scenarios that pursue program effects and natural interaction, this solution is seriously inadequate in terms of expressiveness and cannot meet the growing expectations of users.

[0055] Dialogue systems that directly integrate a general-purpose large language model possess powerful natural language understanding and generation capabilities, resulting in fluent dialogue, broad knowledge base, and an interactive experience far superior to script-based systems. However, due to the "unpredictability" inherent in general-purpose large language models, while the generated content is mostly appropriate, there is still a probability of factual errors, logical fallacies, or even statements that contradict social values. In public settings such as live broadcasts and program recordings where content is subject to strict censorship, this "black box" approach to content output is unacceptable and poses a significant risk of broadcast mishaps. Furthermore, since the dialogue flow is entirely AI-driven, it is difficult for backend operators to provide real-time, precise intervention and guidance.

[0056] The essence of a decentralized human control system is to materialize robots as "puppets" requiring multiple operators to control them, resulting in extremely low human-robot collaboration efficiency. First, when real-time dialogue generation is needed, operators typically use keyboard input, whose typing speed cannot keep up with the rapid changes in the environment, leading to high robot response latency and missed opportunities for optimal interaction. Second, fragmented multimodal control, with separate control of voice and motion, results in uncoordinated and unnatural robot performance. This model is costly and heavily reliant on the individual operator's skills, making it difficult to scale and standardize.

[0057] This shows that existing human-machine collaboration methods are either too rigid and unable to cope with complex situations, or they rely too much on the autonomy of AI, which poses a huge broadcast risk.

[0058] Therefore, this application provides a human-machine collaboration method, system, electronic device, and storage medium. In voice-driven mode, the server can use a dual-channel real-time voice encoder to generate corresponding text and its prosody-emotion vector based on the user's input audio data stream, and generate content-rich and emotional audio data based on the text and its prosody-emotion vector, so as to control the intelligent device to output content-rich and emotional audio data, thereby solving the problem that the output audio in the prior art is too rigid and cannot cope with complex situations. Furthermore, in AI autonomous interaction mode, this application can also review the content segments output by the AI ​​service. Only when the content review is passed will the corresponding audio data be output, thereby solving the problem that the prior art relies too much on the autonomy of AI and has huge broadcast risks. It can be seen that the technology provided by this application can effectively integrate AI capabilities and artificial intelligence, and provide an efficient, safe, and expressive human-machine collaboration method.

[0059] See Figure 1 The diagram shows a structural schematic of a human-machine collaborative system provided in an embodiment of this application. The human-machine collaborative system includes a server, a console, and intelligent devices.

[0060] The console is used to generate control commands based on user actions and send these commands to the server.

[0061] The server receives control commands from the console and determines the corresponding interaction mode based on the commands. If the interaction mode is voice-driven, it uses a dual-channel real-time voice encoder to perform dual-channel parallel encoding on the user's audio data stream, obtaining a first text and a prosody-emotion vector. The first file is segmented to obtain multiple text fragments, and a structured data package corresponding to each text fragment is generated based on each text fragment and the prosody-emotion vector. For each structured data package, the server generates corresponding first audio data based on the structured data package and the target timbre of the preset role, and sends the first audio data stream to the smart device. If the interaction mode is AI-driven, the server obtains content fragments output by the AI ​​service and reviews the content fragments. The content fragments are generated by the AI ​​service based on the user's questions. If the content review is passed, second audio data is generated based on the content fragments and sent to the smart device.

[0062] A smart device is used to output first audio data when it receives first audio data, and to output second audio data when it receives second audio data.

[0063] This application provides a human-machine collaborative system. A server receives control commands sent from a console and determines the corresponding interaction mode based on the commands. The control commands are generated by the console based on user actions. If the interaction mode is voice-driven, a dual-channel real-time voice encoder performs parallel dual-channel encoding on the user's audio data stream to obtain a first text and a prosody-emotion vector. The first file is segmented to obtain multiple text fragments, and a structured data packet corresponding to each text fragment is generated based on each text fragment and the prosody-emotion vector. For each structured data packet, corresponding first audio data is generated based on the structured data packet and the target timbre of a preset role, and the first audio data stream is sent to a smart device, enabling the smart device to receive and output the first audio data. If the interaction mode is AI-driven, content fragments output by an AI service are obtained, and the content fragments are reviewed. The content fragments are generated by the AI ​​service based on user questions. If the content review is passed, second audio data is generated based on the content fragments and sent to the smart device, enabling the smart device to output the second audio data. Therefore, the technical solution provided in this application, in voice-driven mode, allows the server to utilize a dual-channel real-time voice encoder to generate corresponding text and its prosody-emotion vector based on the user's input audio data stream, and to generate content-rich and emotional audio data based on the text and its prosody-emotion vector. This enables the control of intelligent devices to output content-rich and emotional audio data, thereby solving the problem of overly rigid output audio in existing technologies that cannot cope with complex situations. Furthermore, in AI autonomous interaction mode, this application can also review the content segments output by the AI ​​service. Only when the content review is passed will the corresponding audio data be output, thereby solving the problem of excessive reliance on AI autonomy in existing technologies, which poses a significant broadcast risk. Thus, the technical means provided in this application can effectively integrate AI capabilities and artificial intelligence, and provide an efficient, safe, and expressive human-machine collaboration method.

[0064] Optionally, based on the human-machine collaborative real-time director system shown in the above embodiments of the present invention, the dual-channel real-time speech encoder includes channel one and channel two, and channel one is equipped with an encoder and its connected target decoder.

[0065] If the interaction mode is voice-driven, a dual-channel real-time voice encoder is used to perform dual-channel parallel encoding on the user's audio data stream, resulting in a server with the first text and prosodic-emotion vectors, specifically used for:

[0066] If the interaction mode is voice-driven, the audio data stream input by the user is collected; the encoder in channel one is used to encode the audio data stream to obtain the hidden state sequence, and the target decoder is used to decode the hidden state sequence to obtain the first text; channel two is used to output the corresponding prosody-sentiment vector based on the hidden state sequence through the prosody-sentiment regression network.

[0067] Optionally, based on the human-computer collaborative real-time director system shown in the above embodiments of the present invention, a server that segments the first file to obtain multiple text fragments and generates a structured data packet corresponding to each text fragment based on each text fragment and a prosody-emotion vector is specifically used for:

[0068] The first text is segmented into multiple text fragments; the prosody-sentiment sub-vector matching each text fragment is extracted from the prosody-sentiment vector; each text fragment and its matching prosody-sentiment sub-vector are encapsulated to obtain a structured data packet for each text fragment.

[0069] Optionally, based on the human-computer collaborative real-time director system shown in the above embodiments of the present invention, a server that generates corresponding first audio data for each structured data packet according to the structured data packet and the target timbre of a preset role is specifically used for:

[0070] For each structured data packet, the embedding matrix of the text segment in the structured data packet is extracted; the target timbre of the preset role is obtained, and the embedding vector of the target timbre is extracted; an input vector is generated based on the embedding matrix, the embedding vector and the prosody-emotion sub-vector in the structured data packet, and the distribution parameters of the corresponding latent representation are generated based on the input vector; the corresponding audio waveform is generated based on the distribution parameters of the latent representation and the embedding vector, and the first audio data is generated based on the audio waveform.

[0071] Optionally, based on the human-machine collaborative real-time director system shown in the above embodiments of the present invention, the server is further used for:

[0072] If the interaction mode is manual text takeover mode, extract the text from the control commands;

[0073] A third audio data is generated based on the text and the target timbre, and then sent to the smart device so that the smart device can receive and output the third audio data.

[0074] Optionally, based on the human-machine collaborative real-time director system shown in the above embodiments of the present invention, the server is further used for:

[0075] The target text and target speech prosody of the target audio data are obtained, wherein the target audio data is the first audio data, the second audio data, or the third audio data;

[0076] Contextual feature vectors are generated based on the target text and target speech prosody, and target multi-dimensional labels are selected from pre-set multi-dimensional labels based on the contextual feature vectors.

[0077] The system determines the target action sequence that matches the target's multi-dimensional labels and sends the target action sequence to the smart device, enabling the smart device to perform actions that match the target audio data based on the target action sequence.

[0078] See Figure 2 The diagram illustrates a flowchart of a human-machine collaboration method provided in an embodiment of this application, which is applied to... Figure 1 The server in the human-computer collaboration system shown here, the human-computer collaboration method specifically includes the following steps:

[0079] S201: Receive control commands sent by the console and determine the corresponding interaction mode based on the control commands; wherein, the control commands are generated by the console based on user operations performed by the user.

[0080] In this embodiment, the console is responsible for establishing a corresponding real-time communication channel with the server; the console includes an instruction control area and an information presentation area, such as... Figure 3 As shown; from Figure 3 As can be seen, the console's command control area has functions such as core interaction mode selection, role switching control, real-time intervention control, preset content, and action library invocation; among them, the core interaction modes can include AI autonomous mode, human text takeover mode, and voice-driven mode; real-time intervention control includes an emergency cutoff button and a TTS sending button; the information presentation area includes a real-time interactive subtitle area, an AI response streaming display area (the AI ​​response streaming display area supports warning highlighting), a multi-task time queue, and the current role and system status.

[0081] It should be noted that the console can be the director's console; the console can present multi-dimensional real-time information sent from the server in a structured manner to the operator, which may include real-time interactive captions, AI content being generated in a streaming manner, and so on.

[0082] During the specific execution of step S201, when the user needs to perform human-computer interaction, the user can perform corresponding operations through the console. The console will then serialize the user's operations to generate corresponding control commands and send the control commands to the server. Upon receiving the control commands, the server will determine the interaction mode that the user wants to perform.

[0083] It should be noted that the console can be used to develop single-page applications based on HTML and Javascript; it can establish an encrypted, full-duplex, persistent communication channel with the server via the WSS protocol (WebSocket over SSL); and all commands and status data are serialized and transmitted in JSON format.

[0084] In this embodiment, the server includes a central interactive processing module, a content security and intervention module, a real-time voice processing pipeline, a role resource manager, and a smart device hardware structure module, etc. The smart device can be a robot, and correspondingly, the smart device hardware structure module can be a robot hardware device module.

[0085] It should be noted that the central interactive processing module, as the core logic hub and task scheduler of the server, is responsible for managing communication with the front end (console), parsing the control commands sent by the front end, and coordinating the invocation of other modules to complete the corresponding tasks based on the control commands and the current system status.

[0086] The Content Security and Intervention module is a dedicated functional module for real-time monitoring and immediate intervention of streaming AI content. Its core is a "human-machine collaborative streaming content filtering" mechanism. This mechanism simultaneously performs two operations when AI generates text in a streaming manner: a) automated content monitoring and b) real-time human monitoring and intervention. a) Automated content monitoring: Automatic detection of keywords and semantic patterns in the real-time text stream (content fragments output by the AI ​​service) is performed through a background filter. b) Real-time human monitoring and intervention: The text stream can be pushed to the console for synchronous display, allowing users to monitor the "thinking process" of the AI ​​service throughout and have the authority to issue "truncation" commands at any time.

[0087] The real-time speech processing pipeline consists of multiple components, a set of functions specifically designed to process real-time audio streams, including: a speech recognition module, an intelligent sentence segmentation module, and a speech synthesis module. The speech recognition module is responsible for converting input audio data into text; the intelligent sentence segmentation module is responsible for real-time analysis of the text and dynamically segmenting meaningful text segments based on acoustic or semantic features; and the speech synthesis module is responsible for receiving text and a specified timbre identifier (target timbre) and generating the corresponding audio data.

[0088] The Character Resource Manager is used to manage and configure the background services of the "Character Script". Specifically, it atomically encapsulates the multi-dimensional resources of the character (voice, language style, behavior pattern, etc.) and provides interfaces to support the background hot reloading of these resources, thereby realizing the instant switching of the robot's (smart device's) "personality".

[0089] In practical applications, users can select the new role to switch to on the console, which will then send a "role switching command" to the central interactive processing module of the server. When the central interactive processing module of the server receives the "role switching command", it will notify the role resource manager to update the role ID of the currently active preset role in the database.

[0090] The Role Resource Manager uses Redis's publish / subscribe (Pub / Sub) functionality to publish a message containing the new role configuration to a global channel (such as role_update_channel). This involves calling the message bus - Pub / Sub to publish a "role update" event, broadcasting the corresponding "role update" event to the speech synthesis module and AI service. Upon receiving the update message, both the speech synthesis module and AI service execute their internal hot reload logic to complete the update. For example, the speech synthesis module reloads the model file for the new voice (e.g., changing the preset role to the new role and loading the new voice); the AI ​​service switches its Prompt template. This publish / subscribe-based decoupled design ensures that role switching takes effect quickly and consistently across all modules.

[0091] The smart device hardware interface module is a dedicated software module deployed on a server. Its core is an integrated official software development kit (SDK) for the smart device model. This module acts as a "translation layer" or "driver layer," providing a set of standardized internal interfaces (such as `play_audio(audio_data)`, `execute_action(action_id)`) that are easily invoked by upper-layer business logic. Internally, it sends these high-level instructions to the smart device via the local area network by calling functions provided by the SDK.

[0092] In practical applications, when broadcasting precise and objective text content, users can enter the corresponding text in the command text box of the console and trigger the corresponding sending command. After detecting the command sent by the user, the director controller generates the corresponding control command, such as: {"command":"SEND_TTS","voice":"happy-male","payload":{"text":"Welcome all guests"}}, and sends the generated control command to the central interaction module of the server via WebSocket. After receiving the control command, the central interaction module can analyze the control command and determine that it carries the text entered by the user. At this point, it can determine that the manual text takeover mode is the current interaction mode.

[0093] In scenarios requiring rapid response or spontaneous, performative dialogue that conveys distinctive emotions and tone, users can activate this voice-driven mode via UI controls or keyboard shortcuts on the console to send the command {"command":"START_VOICE_DRIVE"} to the server. Upon receiving this command, the server's central interaction processing module identifies it to determine if it was triggered by the user through UI controls or keyboard shortcuts, thus confirming that voice-driven mode is the current interaction mode.

[0094] In non-core scenarios such as general Q&A and warm-up interactions, users can directly trigger the AI ​​autonomous mode through the console to send corresponding control commands to the central control processing module of the server. At this time, the central control processing module identifies the control commands to determine that the control commands were sent by the user through the console to trigger the AI ​​autonomous mode, and thus determines that the AI ​​autonomous mode is the current interaction mode.

[0095] S202: Determine whether the interaction mode is voice-driven mode, AI autonomous mode, or human text takeover mode; if it is voice-driven mode, proceed to step S203; if it is AI autonomous interaction mode, proceed to step S206; if it is human text takeover mode, proceed to step S208.

[0096] S203: The user's audio data stream is encoded in parallel using a dual-channel real-time speech encoder to obtain the first text and prosody-emotion vector.

[0097] In the specific execution step S203, a pre-configured dual-channel real-time speech encoder can be pre-deployed in the server so that after the central interactive processing module of the server determines that the interaction mode is speech-driven mode, the speech recognition module of the channel server collects the audio data stream input by the user and uses channel one and channel two in the dual-channel real-time speech encoder to process the audio data stream in parallel, so as to decouple and extract the corresponding first text and prosody-emotion vector.

[0098] It should be noted that prosodic-emotion vectors are an important research direction in the field of speech synthesis and emotion computing. They mainly achieve controllable expression of speech emotion by modeling the association between prosodic features and emotion features.

[0099] In this embodiment, the dual-channel real-time speech encoder includes channel one and channel two. Channel one is equipped with an encoder and a target decoder connected thereto, and channel two is equipped with a pre-trained prosody-emotion regression network. The target decoder can be a CTC decoder.

[0100] It should be noted that the prosodic-emotional regression network is a neural network model that combines speech prosodic features with emotion recognition.

[0101] If the interaction mode is voice-driven, the process of using a dual-channel real-time voice encoder to perform dual-channel parallel encoding on the user's audio data stream to obtain the first text and prosody-emotion vector can be as follows: If the interaction mode is voice-driven, collect the audio data stream input by the user; use the encoder in channel one to encode the audio data stream to obtain the hidden state sequence, and use the target decoder to decode the hidden state sequence to obtain the first text; use channel two to output the corresponding prosody-emotion vector based on the hidden state sequence through the prosody-emotion regression network.

[0102] It should be noted that after the central interactive processing module determines that the interactive mode is voice-driven mode, it can notify the server's speech recognition module to start using a dedicated microphone channel to acquire PCM audio data streams at a sampling rate of 16kHz and a bit depth of 16bit, that is, to acquire the audio data streams input by the user through the dedicated microphone channel.

[0103] In some embodiments, a CTC decoder can be connected to the upper layer of the encoder (such as a multi-layer CNN and Transformer encoder) deployed in channel one; after receiving the incoming audio data stream, the dual-channel real-time speech encoder performs deep feature extraction through the encoder, and the resulting high-dimensional feature sequence containing rich acoustic information is the hidden state sequence H. The hidden state sequence H is then decoded using the CTC decoder to obtain the corresponding text sequence (first text).

[0104] In practical applications, the conditional probability of the CTC decoder outputting the first text Text for a given hidden state sequence H can be expressed as: P(Text|X)=∑_{π∈Φ(Text)}∏_{t=1}^{T'}P_t(π_t|X); where X is the input audio data stream, π is all possible paths corresponding to the first text Text, Φ(Text) is the set of paths, P_t(π_t|X) is the probability that the CTC outputs the label π_t at time step t, and T is the length of the hidden state sequence.

[0105] In some embodiments, Channel 1 and Channel 2 are executed in parallel. After obtaining the corresponding hidden state sequence H using Channel 1, the obtained hidden state sequence H can be further input into the prosody-sentiment regression network deployed in Channel 2, so that the prosody-sentiment regression network performs global average pooling and fully connected layer calculation on the hidden state sequence H to obtain the corresponding prosody-sentiment vector.

[0106] Specifically, the mathematical expression for global average pooling and fully connected layer computation of the hidden state sequence H in the prosody-sentiment regression network can be expressed as follows:

[0107] H_avg=(1 / T')*∑_{t=1}^{T'}h_t;

[0108] v_p=W2*σ(W1*H_avg+b1)+b2;

[0109] Where H={h_1,...,h_T'} is the hidden state sequence, H_avg is its mean vector in the time dimension; W1∈R^{H_d×E}` and W2∈R^{P×H_d}` are the weight matrices of the fully connected layer, b1 and b2 are bias vectors, σ is a non-linear activation function, and finally the prosody-emotion vector v_p∈R^P is generated.

[0110] It should be noted that taking the "intermediate layer hidden state output" of the encoder of channel one (selecting the output sequence of a certain intermediate layer (e.g., the second to third penultimate layer)) as the input of the prosody-sentiment regression network of channel two is because, according to deep learning theory and personal practical experience, the intermediate layer features of the model usually retain richer and more general original signal information (such as prosody and sentiment), while higher layers focus more on abstract, task-oriented information (such as the content itself). Therefore, using this as the input of the prosody-sentiment regression network can enable the prosody-sentiment regression network to output a more accurate prosody-sentiment vector.

[0111] S204: Segment the first file to obtain multiple text segments, and generate a structured data package corresponding to each text segment based on each text segment and the prosody-sentiment vector.

[0112] In the specific execution step S204, after obtaining the first text and the prosody-sentiment vector, the first text can be segmented to obtain multiple text segments; the prosody-sentiment sub-vector matching each text segment can be extracted from the prosody-sentiment vector; each text segment and its matching prosody-sentiment sub-vector can be encapsulated to obtain a structured data packet for each text segment.

[0113] In practical applications, after obtaining the first text and the prosody-sentiment vector, the server can use the intelligent short sentence module to dynamically analyze the first text using acoustic or semantic features to segment the first text into multiple meaningful text segments. For each text segment, the prosody-sentiment sub-vector matching the text segment is extracted from the prosody-sentiment vector in a time-aligned manner, and the text segment and its matching prosody-sentiment sub-vector are encapsulated into a structured data package containing "content + style".

[0114] It should be noted that the text switching method based on acoustic features can be as follows: continuously monitor the speech stream corresponding to the first text output by channel one. When a speech stream exceeding a preset threshold (e.g., 350 milliseconds, which can be defined in the system configuration file) is detected, and the speech stream is determined to be a silent segment, it is judged as the end of a natural language sentence. Then, the text segment corresponding to the speech stream can be segmented until the complete speech stream of the first text is detected.

[0115] The text segmentation method based on semantic features can be as follows: use a lightweight language model to analyze the first text output by channel one, find syntactic end markers (such as common conjunctions, modal particles or complete semantic units), and segment according to each end marker to obtain multiple text fragments.

[0116] It should also be noted that after obtaining the first text, the content security and intervention module can push the first text to the console in real time as a text stream for synchronous display.

[0117] S205: For each structured data packet, generate corresponding first audio data based on the structured data packet and the target timbre of the preset role, and send the first audio data stream to the smart device so that the smart device can receive and output the first audio data.

[0118] In the specific execution step S205, after obtaining the structured data packet for each text segment, the embedding matrix of the text segment in the structured data packet is extracted for each structured data packet; the target timbre of the preset role is obtained, and the embedding vector of the target timbre is extracted; an input vector is generated based on the embedding matrix, the embedding vector, and the prosody-emotion sub-vector in the structured data packet, and the distribution parameters of the corresponding latent representation are generated based on the input vector; the corresponding audio waveform is generated based on the distribution parameters of the latent representation and the embedding vector, and the first audio data is generated based on the audio waveform.

[0119] In practical applications, after obtaining the structured data packets for each text segment through the intelligent short sentence module, each structured data packet can be transmitted to the central interactive processing module; the central interactive processing module immediately calls the speech synthesis module and inputs the structured data packets and the target timbre of the currently configured preset character.

[0120] The process by which the speech synthesis module generates the first audio data based on the structured data packet and the target timbre can be as follows: The embedding vectors of the text segments in the structured data packet and the embedding vector of the target timbre are extracted; a conditional prior encoder is used to receive the embedding matrix Z_text∈R^{L×D_text}, the embedding vector v_s∈R^{D_s} of the target timbre, and the prosodic-emotional sub-vector v_p∈R^P, which serves as the core style condition. These three are then concatenated to input the concatenated input vector into a flow-based model, which generates the distribution parameters of the corresponding latent representation z based on the input vector; finally, a decoder (such as a HiFi-GAN vocoder) is used to generate the final audio waveform based on the latent representation z and the embedding vector of the target timbre.

[0121] It should be noted that f_prior can be defined as the flow-based model, whose input is a fused conditional tensor C (input vector), and whose output is the distribution parameters of the latent representation z:

[0122] C=Concat(Expand(v_s),Expand(v_p))

[0123] μ_z,log(σ_z)=f_prior(Z_text,C)

[0124] Here, Expand means expanding the vector to match the length of the text sequence.

[0125] In some embodiments, after obtaining the corresponding audio waveform, first audio data can be generated based on the audio waveform. The first audio data obtained at this time has the target timbre of the preset character and highly replicates the user's original expression in terms of emotion and rhythm. Finally, the first audio data is sent to the smart device through the smart device hardware interface module, so that the smart device can receive and play the first audio data. At the same time, the text of the first audio data can also be recognized by the speech recognition module and sent back to the console for display via WebSocket, so that the user can check the subtitles in real time.

[0126] S206: Obtain the content fragment output by the AI ​​service and review the content fragment; the content fragment is generated by the AI ​​service based on the user's question. If the content review is passed, proceed to step S207.

[0127] In the specific execution step S206, when the interaction mode is determined to be AI autonomous mode, the AI ​​service automatically generates corresponding candidate responses (content fragments) based on the user's questions. In order to avoid the problem that the content output of the AI ​​service autonomous response is unacceptable and poses a huge risk of broadcast accidents, the content fragments output by the AI ​​service can be monitored and intervened through the content security and intervention module. Only when it is not truncated (approved) will it be sent to the speech synthesis module for processing.

[0128] In some embodiments, after acquiring a content fragment, the fragment can be processed simultaneously through two parallel processing paths. Specifically, processing path one involves an automated content filter determining if the fragment contains pre-defined high-risk sensitive words. If so, the filter immediately sends a stop signal to the AI ​​service, forcibly interrupting the generation task and ending the corresponding human-machine collaboration process. If not, the fragment is sent to the speech synthesis module, enabling the model to generate second audio data based on the fragment. Processing path two involves pushing the content fragment to the console via WebSocket, allowing the console to append the fragments one by one to the AI-responding county / district, forming a real-time subtitle stream for users to view in real time.

[0129] It should be noted that if the pushed content fragment contains pre-set medium-risk keywords, the pushed content fragment will be accompanied by a "warning": true flag. When the front end receives the word flag, it will change the background of the currently displayed content fragment to yellow to issue a visual warning to the user.

[0130] Furthermore, in this embodiment of the application, during the AI ​​autonomous interaction mode, that is, during the entire AI streaming output (from the first word to the last word), there is always an active "emergency truncation" button on the console. When monitoring the real-time subtitle stream, if the user finds any unreasonable content (logical error, factual error, or the content does not fit the program's tone), they can press this button at any time to send a corresponding immediate truncation command to the server. After receiving the immediate truncation command, the server sends a stop signal to the AI ​​service and speech synthesis module through the central interactive processing module to instantly stop the AI ​​service from continuing to push the corresponding content segment and the already generated speech synthesis.

[0131] S207: Generate second audio data based on the content fragment, and send the second audio data to the smart device so that the smart device outputs the second audio data.

[0132] In the specific execution step S207, after confirming that the content segment output by the AI ​​service has passed the review, the target timbre of the preset character is obtained, and the second audio data is generated by the speech synthesis module based on the content segment and the target timbre. Finally, the second audio data is sent to the smart device through the smart device hardware interface module, so that the smart device receives and plays the second audio data. At the same time, the text of the second audio data can also be recognized by the speech recognition module, and the text can be sent back to the console for display via WebSocket, so that the user can check the subtitles in real time.

[0133] S208: Extract text from control commands, generate third audio data based on the text and target timbre, and send the third audio data to the smart device so that the smart device can receive and output the third audio data.

[0134] During the specific execution of step S208, when the interaction mode is determined to be the manual text takeover mode, the target timbre of the preset role is obtained through the central interaction processing module, and the third audio data is generated by the speech synthesis module based on the content segment and the target timbre. Finally, the third audio data is sent to the smart device by calling the corresponding audio playback function provided by SK through the smart device hardware interface module, so that the smart device receives and plays the third audio data. At the same time, the text of the third audio data can also be recognized by the speech recognition module, and the text can be sent back to the console for display via WebSocket for the user to check the subtitles in real time.

[0135] It should be noted that the human text takeover mode refers to a mode in which a real person intervenes in real time and takes over the generation task during the AI-generated content process.

[0136] Furthermore, in this embodiment, the action library corresponding to the smart device can be pre-configured on the console, and an action library management interface can be provided on the console for the corresponding smart device personnel to configure each action sequence in the action library, and to label each action sequence with corresponding multi-dimensional tags. For example, the multi-dimensional tags can be: [emotion: positive, intention: agree, intensity: strong].

[0137] Optionally, this application may also acquire the target text and target speech prosody of the target audio data, wherein the target audio data is first audio data, second audio data, or third audio data; generate a context feature vector based on the target text and target speech prosody, and filter out target multi-dimensional labels from pre-set multi-dimensional labels based on the context feature vector; determine the target action sequence matching the target multi-dimensional labels, and send the target action sequence to the smart device, so that the smart device performs an action matching the target audio data according to the target action sequence.

[0138] In practical applications, the central interactive processing module converts the target audio data into the corresponding target text and obtains the speech prosody corresponding to the target audio data. It extracts a comprehensive contextual feature vector v_context∈R^D from the target text and speech prosody through a multimodal feature encoder. The contextual feature vector and various multi-dimensional labels are input into a multi-label classification model, which consists of multiple hidden layers and a final output layer. The multi-label classification model selects the target multi-dimensional label from the pre-set multi-dimensional labels based on the contextual feature vector.

[0139] It should be noted that a multi-label classification model with N_labels possible labels outputs an N_labels-dimensional probability P_labels. Therefore, the multi-label classification model calculates the output probability of each label in the corresponding activation label set based on the context feature vector. The output probability calculation process can be: h1=σ(W_h1*v_context+b_h1), P_labels=Sigmoid(W_out*h_n+b_out), where W and b are the weight matrix and bias vector of each layer, σ is the activation function of the hidden layer (such as ReLU), and Sigmoid is the activation function of the output layer, ensuring that the output probability of each label is between (0,1). h1 is the output vector of the first hidden layer in the multi-label classification model, and h_n is the output vector of the nth hidden layer in the multi-label classification model.

[0140] It should also be noted that in a standard feedforward neural network (multi-label classification model), information is passed forward layer by layer. The formula h1=σ(W_h1*v_context+b_h1) describes the process of calculating the output vector of the first hidden layer. Specifically, the input vector v_context is first multiplied by the weight matrix W_h1 of the first hidden layer; then the bias vector b_h1 of the first hidden layer is added; finally, the result is passed through a non-linear activation function σ (such as ReLU) to obtain the output vector h1 of the first hidden layer. If this feedforward neural network has n hidden layers, the output h1 of the first hidden layer will be used as the input of the second hidden layer to calculate h2, and so on, until the output vector h_n of the last (nth) hidden layer is calculated.

[0141] Therefore, h_n is the final, highest-level feature representation extracted from the original input v_context after undergoing multiple nonlinear transformations before the output layer of the entire neural network. This h_n is then fed into the final Sigmoid output layer to calculate the probabilities P_labels of each label.

[0142] In some embodiments, the predicted "activation label set" (i.e. the label corresponding to the probability that the value of each P_label is greater than the threshold) is used to query the action library. Specifically, the label with the highest output probability is selected according to the output probability of each label, and the similarity between the label with the highest output probability and each multi-dimensional label is calculated so that the multi-dimensional label with the highest similarity is determined as the target multi-dimensional label, and the action sequence corresponding to the target multi-dimensional label is determined as the target action sequence.

[0143] It should be noted that the similarity between Jaccard tags and multi-dimensional tags can be used, where the similarity can be calculated as: Score(i)=|L_pred∩L_i| / |L_pred∪L_i|, where L_pred is the tag, L_i is the multi-dimensional tag, and Score(i) is the similarity.

[0144] In summary, addressing the core issues of existing human-computer collaboration methods, such as rigid interaction, uncontrollable content, and low efficiency, this application can provide rich and natural interaction modes while avoiding the risk of AI services publishing inappropriate or erroneous content fragments in public places. It reduces the response latency of real-time human intervention from the "second" level to the "sub-second" level and completely eliminates the limitations of keyboard input, enabling the output audio data to carry rich emotion and natural rhythm. This allows intelligent devices to do more than just change a voice or name; it enables a thorough and consistent transformation from voice, language style, behavioral patterns to knowledge scope. Furthermore, it allows for instant switching of "personality"; it presents concurrent information from multiple sources (such as questions from multiple audience members or instructions from the director) clearly, orderly, and with priority to the corresponding user, providing them with the most efficient decision-making and execution tools to handle complex situations (such as instantly truncating inappropriate content segments); while outputting audio data, it can also output corresponding action sequences, thereby achieving control over multiple modalities such as voice, facial expressions, and actions of smart devices, allowing the physical actions of smart devices (such as gestures and postures) to match the dialogue content and emotional atmosphere in real time and intelligently, rather than executing isolated, preset, and rigid actions.

[0145] This application also provides a storage medium storing program instructions that, when loaded and executed by a processor, implement any of the above-described embodiments of the method for predicting the usage of important blank vouchers.

[0146] This application also provides an electronic device, such as Figure 4 As shown, the device includes a processor 401 and a memory 402, which are connected via a bus; the memory stores program instructions; the processor calls the program instructions in the memory to execute any of the above-described embodiments of the method for predicting the usage of important blank vouchers.

[0147] The processor mentioned in this article can be the terminal's CPU, an integrated MCU within the terminal, or a combination of a CPU and an MCU. Furthermore, the processor contains a kernel that retrieves the corresponding program from memory; one or more kernels can be configured.

[0148] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0149] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0150] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0151] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0152] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A human-machine collaboration method, characterized in that, The method includes: The system receives control commands sent from the console and determines the corresponding interaction mode based on the control commands; wherein the control commands are generated by the console based on user operations. If the interaction mode is a voice-driven mode, the user's audio data stream is encoded in parallel using a dual-channel real-time voice encoder to obtain the first text and the prosody-emotion vector. The first file is segmented to obtain multiple text fragments, and a structured data packet corresponding to each text fragment is generated based on each text fragment and the vector. For each structured data packet, a corresponding first audio data is generated based on the structured data packet and the target timbre of the preset role, and the first audio data stream is sent to the smart device, so that the smart device receives and outputs the first audio data; If it is an AI autonomous interaction mode, obtain the content fragments output by the AI ​​service and review the content fragments; wherein, the content fragments are generated by the AI ​​service based on the user's questions; If the content is approved, second audio data is generated based on the content fragment, and the second audio data is sent to the smart device so that the smart device outputs the second audio data.

2. The method according to claim 1, characterized in that, The dual-channel real-time speech encoder includes channel one and channel two, and channel one contains an encoder and a target decoder connected to it. If the interaction mode is a voice-driven mode, a dual-channel real-time voice encoder is used to perform dual-channel parallel encoding on the user's audio data stream to obtain a first text and a prosody-emotion vector, including: If the interaction mode is voice-driven mode, the audio data stream input by the user is collected; The audio data stream is encoded using the encoder in channel one to obtain a hidden state sequence, and the hidden state sequence is decoded using the target decoder to obtain the first text; Using the second channel, the prosody-sentiment regression network outputs the corresponding prosody-sentiment vector based on the hidden state sequence.

3. The method according to claim 1, characterized in that, The step of segmenting the first file to obtain multiple text fragments, and generating a structured data packet corresponding to each text fragment based on each text fragment and the prosody-sentiment vector, includes: The first text is segmented to obtain multiple text fragments; Extract the prosody-sentiment sub-vector matching each text segment from the prosody-sentiment vector; Each text segment and its matching prosodic sentiment subvector are encapsulated to obtain a structured data packet for each text segment.

4. The method according to claim 3, characterized in that, For each structured data packet, generating corresponding first audio data based on the structured data packet and the target timbre of the preset role includes: For each structured data packet, the embedding matrix of the text segments in the structured data packet is extracted; Obtain the target timbre of the preset character and extract the embedding vector of the target timbre; An input vector is generated based on the embedding matrix, the embedding vector, and the prosody-sentiment subvector in the structured data packet, and distribution parameters of the corresponding latent representation are generated based on the input vector; Based on the distribution parameters of the latent representation and the embedding vector, a corresponding audio waveform is generated, and first audio data is generated based on the audio waveform.

5. The method according to claim 1, characterized in that, The method further includes: If the interaction mode is a manual text takeover mode, extract the text from the control command; A third audio data is generated based on the text and the target timbre, and the third audio data is sent to the smart device, so that the smart device receives and outputs the third audio data.

6. The method according to claim 5, characterized in that, The method further includes: The target text and target speech prosody of the target audio data are obtained, wherein the target audio data is the first audio data, the second audio data, or the third audio data; A contextual feature vector is generated based on the target text and the target speech prosody, and a target multi-dimensional label is selected from a set of pre-set multi-dimensional labels based on the contextual feature vector; A target action sequence matching the target multi-dimensional label is determined, and the target action sequence is sent to the smart device, so that the smart device performs an action matching the target audio data according to the target action sequence.

7. A human-machine collaborative system, characterized in that, The system includes a console, a server, and smart devices; The console is used to generate control commands based on user operations and send the control commands to the server. The server is configured to receive control commands sent by the console and determine the corresponding interaction mode based on the control commands. If the interaction mode is a voice-driven mode, a dual-channel real-time voice encoder is used to perform dual-channel parallel encoding on the user's audio data stream to obtain a first text and a prosody-emotion vector. The first file is segmented to obtain multiple text segments, and a structured data packet corresponding to each text segment is generated based on each text segment and the prosody-emotion vector. For each structured data packet, a corresponding first audio data is generated based on the structured data packet and the target timbre of a preset role, and the first audio data stream is sent to the smart device. If the interaction mode is an AI autonomous interaction mode, a content segment output by the AI ​​service is obtained, and the content segment is reviewed. The content segment is generated by the AI ​​service based on the user's question. If the content review is passed, second audio data is generated based on the content segment, and the second audio data is sent to the smart device. The smart device is configured to output the first audio data when it receives the first audio data, and to output the second audio data when it receives the second audio data.

8. The system according to claim 7, characterized in that, The dual-channel real-time speech encoder includes channel one and channel two, and channel one contains an encoder and a target decoder connected to it. If the interaction mode is a voice-driven mode, a server that uses a dual-channel real-time voice encoder to perform dual-channel parallel encoding on the user's audio data stream to obtain the first text and prosody-emotion vector is specifically used for: If the interaction mode is voice-driven mode, the audio data stream input by the user is collected; The audio data stream is encoded using the encoder in channel one to obtain a hidden state sequence, and the hidden state sequence is decoded using the target decoder to obtain the first text; Using the second channel, the prosody-sentiment regression network outputs the corresponding prosody-sentiment vector based on the hidden state sequence.

9. An electronic device, characterized in that, include: A processor and a memory are connected via a bus; wherein the processor is used to call and execute a program stored in the memory; The memory is used to store a program for implementing the human-machine collaboration method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for performing the human-machine collaboration method as described in any one of claims 1-6.

Images