Voice interaction recovery method and device, robot, and storage medium

Abstract

The embodiment of the application discloses a voice interaction recovery method and device, a robot, and a storage medium, which are applied to the robot and include the following steps: in the case that a target voice being played is interrupted, a played segment and an unplayed segment of the target voice are acquired; in the case that the target voice is recovered, a recovery voice of the target voice is output, the recovery voice comprises a transition segment and the unplayed segment of the target voice, and the transition segment is generated according to the played segment and the unplayed segment. The recovery voice can be output more smoothly and naturally, and the user can avoid missing or repeatedly listening to the interrupted target voice.

Description

Technical Field

[0001] This application relates to human-computer interaction technology, including but not limited to a method and apparatus for restoring voice interaction, a robot, and a storage medium. Background Technology

[0002] In robot-guided tour scenarios, robots typically provide guided tours and introductions to users. These scenarios include museums, exhibition halls, or any type of venue that displays items.

[0003] In the above scenario, the robot can introduce the corresponding products to the user and guide the user to the corresponding location. During the product introduction, the user may have questions and may interrupt the robot's introduction to ask questions. The robot will answer the questions and then resume the product introduction after answering.

[0004] In related technologies, when resuming the product introduction, the audio that was interrupted by the user during the previous introduction is usually replayed. However, due to the different interruption points, the replay process may be abrupt, and the user may not be able to immediately understand the content that the robot is trying to introduce. Furthermore, resuming directly from the interruption point may also result in repetition or omission of the robot's guided introduction, leading to a poor user experience. Summary of the Invention

[0005] The voice interaction recovery method, apparatus, robot, and storage medium provided in this application are implemented as follows: One aspect of this application provides a method for restoring voice interaction, applied to a robot, comprising: If the playback of the target audio is interrupted, obtain the played and unplayed segments of the target audio. In the case of target speech recovery, the recovered speech of the target speech is output. The recovered speech includes the transition segment and the unplayed segment of the target speech, wherein the transition segment is generated based on the played segment and the unplayed segment.

[0006] In one embodiment, before outputting the recovered speech of the target speech, the method further includes: Generate transition segments based on played and unplayed segments; The transition segment and the unplayed segment are spliced ​​together to obtain the recovered speech of the target speech.

[0007] In one embodiment, generating a transition segment based on the played segment and the unplayed segment includes: Transition segments are generated based on the played and unplayed segments and a pre-defined large language model, which is a model obtained after training on the played sample segments, unplayed sample segments, and transition sample segments.

[0008] In one embodiment, generating a transition segment based on the played segment and the unplayed segment, as well as a preset large language model, includes: Determine the topic information of the target speech based on the already played segments; The target prompt word is determined from multiple preset prompt words based on the topic information of the target speech; The target prompt and the unplayed segment are input into the large language model to obtain the transition segment.

[0009] In one embodiment, obtaining the played and unplayed segments of the target speech includes: Obtain the interruption position of the target audio; Based on the completeness of the following statement at the interruption position, determine the played and unplayed segments of the target speech.

[0010] In one embodiment, determining the played and unplayed segments of the target speech based on the completeness of the following statement at the interruption location includes: If the statement following the interruption point is a complete statement, the statement before the interruption point is determined to be a played segment, and the statement after the interruption point is determined to be an unplayed segment. If the statement following the interruption point is incomplete, the first statement before the interruption point is determined to be the played segment, and the second statement before the interruption point and the statement after the interruption point are determined to be the unplayed segment. The first statement is the complete part of the statement preceding the interruption point, and the second statement is the incomplete part of the statement preceding the interruption point.

[0011] In one embodiment, the method further includes: If a target interrupt command is received during the playback of the target audio, or if voice information input by the target user is obtained, the target audio will be interrupted.

[0012] In one embodiment, the method further includes, in the event that the target speech is interrupted: If a target recovery command is received, or if no inserted voice input from the target user is obtained within the target duration, the target voice is recovered.

[0013] In one embodiment, the method further includes, in the event that the target speech is interrupted: Obtain the inserted voice input from the target user; Determine the target user's intent based on inserted voice; Resume the target voice if the user intends to stop the conversation.

[0014] Another aspect of this application embodiment also provides a voice interaction recovery device, applied to a robot, including: an acquisition module and an output module; The acquisition module is used to acquire the played and unplayed segments of the target audio when the playback of the target audio is interrupted. The output module is used to output the recovered speech of the target speech when the target speech is recovered. The recovered speech includes the transition segment and the unplayed segment of the target speech, wherein the transition segment is generated based on the played segment and the unplayed segment.

[0015] The robot provided in this application includes a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement the method of this application.

[0016] The computer-readable storage medium provided in this application embodiment stores a computer program thereon, which, when executed by a processor, implements the method provided in this application embodiment.

[0017] The voice interaction recovery method, apparatus, robot, and storage medium provided in this application embodiment can acquire the played and unplayed segments of the target voice when playback is interrupted; and output the recovered voice when the target voice is recovered. The recovered voice includes a transition segment and an unplayed segment of the target voice, wherein the transition segment is generated based on the played and unplayed segments. The transition segment can summarize the played segments and serve as an introduction to the unplayed segments, thereby achieving a natural transition in the recovered voice. This allows users to understand the recovered voice output by the robot more naturally and smoothly. Furthermore, since the recovered voice is composed of the transition segment and the unplayed segment, it can also avoid repetition or omission in the robot's output voice, further improving the user's experience of voice interruption recovery during robot use. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating the application scenario provided in the embodiments of this application; Figure 2This is a flowchart illustrating the voice interaction recovery method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the process for obtaining the recovered speech of the target speech provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the large language model provided in the embodiments of this application; Figure 5 This is a schematic diagram of the process for obtaining the transition segment provided in the embodiments of this application; Figure 6 This is a logical diagram illustrating the determination of played and unplayed segments provided in the embodiments of this application; Figure 7 This is another logical diagram illustrating the determination of played and unplayed segments provided in the embodiments of this application; Figure 8 This is a schematic diagram illustrating the process of interrupting or resuming target speech provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the voice interaction recovery device provided in the embodiments of this application; Figure 10 This is a schematic diagram of the robot provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0023] It should be noted that the terms "first, second, third" used in the embodiments of this application are used to distinguish similar or different objects and do not represent a specific order of objects. It can be understood that "first, second, third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0024] To more accurately illustrate the voice interaction recovery method provided in the embodiments of this application, a practical application scenario of the method will be explained below.

[0025] Figure 1 This is a schematic diagram of the application scenario provided in the embodiments of this application. Please refer to it. Figure 1 This scenario may include: robot 110 and user 120.

[0026] It should be noted that robot 110 and user 120 can be in the same scene, such as a museum, exhibition hall, factory, or any scene that requires robot guidance and introduction, without specific restrictions.

[0027] Robot 110 can be a humanoid robot, a robot of a specific shape, or a robot without a physical form. For example, it can be artificial intelligence integrated into electronic devices, without any specific restrictions.

[0028] If robot 110 is artificial intelligence integrated into an electronic device, the electronic device may include, but is not limited to, mobile phones, wearable devices (such as smartwatches, smart bracelets, smart glasses, etc.), tablets, laptops, in-vehicle terminals, PCs (Personal Computers), etc. The functions implemented by this method can be achieved by the processor in the electronic device calling program code. Of course, the program code can be stored in computer storage media. It can be seen that the electronic device includes at least a processor and a storage medium.

[0029] It should be noted that the robot 110 may have the ability to interact with the user 120, such as voice interaction.

[0030] User 120 can be a specific person, such as a tourist visiting a museum or an inspector checking work in a factory, etc., without any specific restrictions.

[0031] The type of robot 110 can be matched with the type of user 120. For example, if user 120 is a visitor in a museum, then robot 110 can be a robot that introduces the exhibits in the museum; if user 120 is an inspector who checks work in a factory, then robot 110 can be a robot that reports on the working conditions of the factory.

[0032] In the aforementioned scenarios, robot 110 will interact with user 120. The robot can be a robot with intelligent response capabilities, such as outputting preset voice messages to the user and collecting the user's questions to provide corresponding answers.

[0033] During the process of the robot introducing exhibits or work, users may have questions and may interrupt the robot's presentation to ask questions. The robot will then provide answers and resume its product introduction after answering.

[0034] In related technologies, when resuming the product introduction, the audio that was interrupted by the user during the previous introduction is usually replayed. However, due to the different interruption points, the replay process may be abrupt, and the user may not be able to immediately understand the content that the robot wants to introduce. Furthermore, resuming directly from the interruption point may also result in repetition or omission of the content that the robot wants to introduce, leading to a poor user experience.

[0035] To avoid the above situation, this application provides a method for restoring voice interaction. The following explains one feasible implementation process of this method.

[0036] Figure 2 This is a flowchart illustrating the voice interaction recovery method provided in the embodiments of this application. Please refer to... Figure 2 Methods for restoring voice interaction include: S210: If the playback of the target audio is interrupted, obtain the played and unplayed segments of the target audio.

[0037] The robot described above can be the subject of this method.

[0038] Optionally, the target speech can be the speech output by the robot. For example, if the robot is introducing the content of the current scene to the user, then that speech can be used as the target speech.

[0039] The target speech can be the speech used by the robot to introduce the content of the current scene to the user. For example, in the above museum scene, the robot can be a tour guide robot in the museum. The robot can introduce the relevant information of the exhibit to the user at the exhibit in the museum. The speech output by the robot to introduce the exhibit is the target speech mentioned above.

[0040] For the robot, the output target speech can be obtained by processing text-to-speech technology, where target text can be converted into target speech, and the target text can be pre-set text, or text generated by a large language model; in another embodiment, the target speech can also be pre-set speech, without specific limitations.

[0041] For target speech with corresponding target text, the played and unplayed segments can be determined based on the target text during the acquisition of played and unplayed segments. For target speech without target text, the target speech can be converted into target text using speech-to-text conversion technology, and then the played and unplayed segments can be determined based on the target text.

[0042] In one embodiment, the played segment and the unplayed segment can be two segments of the target text corresponding to the target speech, wherein the played segment can be a segment that has already been output as speech, and the unplayed segment can be a segment that has not yet been output as speech.

[0043] For example: If the target text corresponding to the target speech is "When it comes to electric vehicles, what everyone is generally most concerned about is charging speed and range, for which we have launched the Super Electric System." If the robot's output speech is interrupted after "range," then the already played segment will be "When it comes to electric vehicles, what everyone is generally most concerned about is charging speed and range," and the unplayed segment will be "For which we have launched the Super Electric System."

[0044] The played and unplayed segments can be divided by recording the robot's output speech, thus obtaining the two segments mentioned above. These two segments can be composed of text.

[0045] S220: If the target speech is recovered, output the recovered speech of the target speech.

[0046] It should be noted that after the target speech is interrupted, it can be resumed under certain conditions. That is, the part of the target speech that was not played in the previous speech can be resumed. During this process, the robot can output the resumed speech of the target speech.

[0047] The restored speech includes transition segments and unplayed segments of the target speech, with the transition segments generated based on the played and unplayed segments.

[0048] It should be noted that transitional segments can serve to connect the preceding text to the following text, summarize the already played segments, or introduce the unplayed segments.

[0049] For example, continuing with the target speech example above, the transition segment could be "Let's continue. We just talked about the charging speed issue, and in order to solve these problems..." Here, "Let's continue" could be a pre-set connecting statement in the transition segment, "We just talked about the charging speed issue" could be a summary of the previously played segment, and "In order to solve these problems" could be an introduction to the previously played segment.

[0050] Correspondingly, the text content corresponding to the restored voice message could be, "Let's continue. We just discussed the charging speed issue, and to address these problems, we launched the Super Electric System." In other words, the restored voice message consists of a transitional segment and an unplayed segment.

[0051] In one embodiment, the robot can output the restored speech if the speech is restored.

[0052] The transition segments can be generated based on a preset neural network model or multiple preset language templates.

[0053] The voice interaction recovery method provided in this application embodiment can acquire the played and unplayed segments of the target voice when playback is interrupted; and output the recovered voice when the target voice is recovered. The recovered voice includes a transition segment and an unplayed segment, wherein the transition segment is generated based on the played and unplayed segments. The transition segment summarizes the played segments and serves as an introduction to the unplayed segments, thus achieving a natural transition in the recovered voice. This allows users to understand the recovered voice output by the robot more naturally and smoothly. Furthermore, since the recovered voice is composed of the transition segment and the unplayed segment, it avoids repetition or omissions in the robot's output, further improving the user experience of voice interruption recovery during robot use.

[0054] The following is a detailed explanation of one feasible implementation process for obtaining the recovered speech of the target speech provided in the embodiments of this application.

[0055] Figure 3 This is a schematic diagram of the process for obtaining the recovered speech of the target speech provided in the embodiments of this application. Please refer to... Figure 3 Before outputting the recovered speech of the target speech, the method further includes: S310: Generate a transition segment based on the played segment and the unplayed segment.

[0056] It should be noted that after obtaining the played and unplayed segments, transition segments can be generated based on the played and unplayed segments.

[0057] This can be achieved through neural network models, for example, by inputting the data into the aforementioned large language model to generate transition segments, or by using specific language templates.

[0058] For the large language model approach, the aforementioned transition segments can be obtained based on the played segments, unplayed segments, and the large language model. For the language template approach, multiple transition sentence patterns can be preset, and the transition segments can be obtained by filling in the corresponding transition sentence patterns according to the content of the played segments and the content of the unplayed segments.

[0059] In actual implementation, either of the two methods mentioned above can be used, without any specific restrictions, as long as the above transition segment is obtained.

[0060] S320: The transition segment and the unplayed segment are spliced ​​together to obtain the recovered speech of the target speech.

[0061] It should be noted that after obtaining the above transition segment, the transition segment and the unplayed segment can be spliced ​​together. For example, the unplayed segment can be placed after the transition segment to obtain the corresponding text. After performing text-to-speech conversion on the text, the recovered speech of the target speech can be obtained.

[0062] In the voice interaction recovery method provided in this application embodiment, a transition segment can be generated based on the played segment and the unplayed segment; the transition segment and the unplayed segment are then spliced ​​together to obtain the recovered speech of the target speech. The process of generating the transition segment and splicing the segments serves as a bridge between the preceding and following segments in the recovered speech, thereby achieving a natural transition in the recovered speech of the target speech, allowing users to understand the recovered speech output by the robot more naturally and smoothly.

[0063] The following section will explain one feasible implementation process for generating the aforementioned transition fragments using a large language model.

[0064] In one embodiment, generating a transition segment based on the played segment and the unplayed segment includes: generating the transition segment based on the played segment, the unplayed segment, and a preset large language model, wherein the large language model is a model obtained after training based on the played sample segment, the unplayed sample segment, and the transition sample segment.

[0065] One approach is to input both the unplayed and played segments into a large language model to obtain the transition segment; another approach is to extract keywords from the unplayed and played segments and input them into the large language model to obtain the transition segment. In practice, any one of these methods can be used, or a combination of both can be used. No specific restrictions are imposed here.

[0066] The specific structure of the large language model will now be explained.

[0067] Figure 4This is a schematic diagram of the structure of the large language model provided in the embodiments of this application. Please refer to... Figure 4 A large language model can be a type of artificial intelligence (AI) model. An AI model is a concrete implementation of AI technology functions, and it represents the mapping relationship between the model's input and output. AI models can be neural networks, linear regression models, decision tree models, support vector machines (SVM), Bayesian networks, Q-learning models, or other machine learning (ML) models.

[0068] Neural networks are a specific implementation of AI or machine learning techniques. According to the general approximation theorem, neural networks can theoretically approximate any continuous function, thus enabling them to learn arbitrary mappings.

[0069] Training a large language model can be achieved using a training dataset. This dataset is used for model training and can include the model's input data, or both input and target output data. Specifically, a training dataset includes one or more training data sets, which can be either input data to the model or the model's target output data. The target output data can also be referred to as labels, output label data, or output label samples. The training dataset is a crucial part of machine learning; model training essentially involves learning certain features from the training data to make the model's output data as close as possible to the target output data, minimizing the difference between them. The composition and selection of the training dataset can, to a certain extent, determine the performance of the trained model.

[0070] In one embodiment, the played sample segments, unplayed sample segments, and transitional sample segments can be used as the training dataset described above.

[0071] Furthermore, a loss function can be defined during the training process of a model (such as a neural network). The loss function describes the difference or discrepancy between the model's output value and the target output value. This application does not limit the specific form of the loss function. The model training process involves adjusting the model parameters to make the loss function value less than a threshold, or to make the loss function value meet the target requirements.

[0072] The model parameters can include one or more of the following: structural parameters of the model (e.g., the number of layers, and / or weights), for example, if the model is a neural network, the structural parameters of the neural network include at least one of the following: the number of layers, width, weights of neurons, or parameters in the activation function of neurons; input parameters of the model (e.g., input dimension, number of input ports); and output parameters of the model (e.g., output dimension, number of output ports). It can be understood that the input dimension refers to the size of an input data set; for example, when the input data is a sequence, the input dimension corresponding to that sequence can indicate the length of the sequence. The number of input ports can refer to the quantity of input data. Similarly, the output dimension can refer to the size of an output data set; for example, when the output data is a sequence, the output dimension corresponding to that sequence can indicate the length of the sequence. The number of output ports can refer to the quantity of output data.

[0073] Furthermore, neural networks can process data in batches, enabling parallel computation to accelerate training. For example, multiple training data sets can be selected to form a batch, which is then input into the neural network to obtain output data. The output data and output label data are then input into a loss function to calculate the loss for this round. This loss is then matched with the step size parameter to update each structural parameter of the neural network, completing one iteration of the training process.

[0074] Inference data can be used as input to a trained model for inference, validation, or monitoring of model performance. During model inference, inputting inference data into the model yields the corresponding output, which is the inference result. Optionally, the model's input data, included in the training dataset, can also be used as inference data for model inference, validation, or monitoring of model performance.

[0075] Figure 4The model structure shown is that of a large language model. In the aforementioned data collection phase, the data source provides training and inference data. In the model training phase, the AI ​​model is obtained by analyzing or training the training data provided by the data source. The AI ​​model represents the mapping relationship between the model's input and output. Learning the AI ​​model through model training nodes is equivalent to learning the mapping relationship between the model's input and output using the training data. In the model inference phase, the AI ​​model trained in the model training phase is used to perform inference based on the inference data provided by the data source, obtaining the inference result. This phase can also be understood as: inputting inference data into the AI ​​model, obtaining output data through the AI ​​model, which is the inference result. This inference result can indicate the configuration parameters used (executed) by the execution object, and / or the operations performed by the execution object. In the inference result application phase, the inference result is published. For example, the inference result can be uniformly planned by the actor entity, which can send the inference result to one or more execution objects (e.g., core network devices, access network devices, terminal devices, or network management systems) for execution. For example, the execution entity can also provide feedback on the model's performance to the data source, facilitating subsequent model updates.

[0076] It is understood that, in the embodiments of this application, the large language model can be deployed on a robot or on a server. If deployed on a robot, the robot can call the large language model according to actual usage needs; if deployed on a server, the robot can interact with the server, sending the data that needs to be input into the model to the server, and receiving the corresponding result after the large language model on the server outputs the result.

[0077] The robot or server equipped with the large language model may include network elements with artificial intelligence capabilities. The AI ​​model design-related steps described above can be performed by one or more network elements with artificial intelligence capabilities. In one possible design, AI functions (such as AI modules or AI entities) can be configured within existing network elements in the robot or server equipped with the large language model to implement AI-related operations, such as AI model training and / or inference. For example, these existing network elements could be access network devices (such as gNBs), terminal devices, core network devices, or network management systems. Operations mainly involve daily network and service analysis, prediction, planning, and configuration; maintenance mainly involves daily operational activities such as testing and fault management of the network and its services. Network management systems can detect network operating status, optimize network connectivity and performance, improve network stability, and reduce network maintenance costs. Alternatively, in another possible design, an independent network element can be introduced into the robot or server equipped with the large language model to perform AI-related operations, such as training the AI ​​model. This independent network element can be called an AI network element or an AI node, etc., and this application embodiment does not limit this name. This AI network element can connect directly to devices in robots or servers that have the large language model deployed, or it can connect indirectly through third-party network elements. These third-party network elements can be core network devices such as authentication management function (AMF) network elements and user plane function (UPF) network elements, network management systems, cloud servers, or other network elements; there are no restrictions.

[0078] The large language model described above can generate transition segments more accurately and quickly, and thus generate the recovered speech of the target speech more quickly and accurately, improving the naturalness of speech transitions.

[0079] The following section will explain the specific implementation methods used in the process of using the aforementioned large language model.

[0080] Figure 5 This is a schematic diagram of the process for obtaining the transition segment provided in the embodiments of this application. Please refer to... Figure 5 Transition segments are generated based on played and unplayed segments and a pre-defined large language model, including: S510: Determine the topic information of the target speech based on the played segment.

[0081] In one embodiment, semantic recognition technology can be used to determine the topic information of the target speech based on the already played segment.

[0082] The topic information can refer to the specific topic of the audio, such as introducing an exhibit or explaining a user's question. There are no specific restrictions here, and the topic information can be determined based on the content of the already played segment.

[0083] S520: Determine the target prompt word from multiple preset prompt words based on the topic information of the target speech.

[0084] It should be noted that after determining the above topic information, the target prompt can be determined from multiple preset prompts. These preset prompts can be multiple pre-configured prompts that cover different fields. The topic information can be used to generate corresponding prompt search instructions, and then the target prompt can be determined from multiple preset prompts based on these prompt search instructions.

[0085] S530: Input the target prompt word and the unplayed segment into the large language model to obtain the transition segment.

[0086] It should be noted that after obtaining the target prompt word, both the target prompt word and the unplayed segment can be input into the large language model. The trained large language model will process the data and produce the corresponding output, which is the transition segment.

[0087] The voice interaction recovery method provided in this application embodiment can determine the topic information of the target speech based on the played segment; determine the target prompt word from multiple preset prompt words based on the topic information of the target speech; and input the target prompt word and the unplayed segment into a large language model to obtain a transition segment. By determining the topic information of the target speech and determining the target prompt word based on the topic information, the accuracy of the large language model's processing can be improved, thereby obtaining a more accurate, reasonable, and natural transition segment.

[0088] The following is a detailed explanation of one feasible implementation method for obtaining played and unplayed segments of target speech provided in the embodiments of this application.

[0089] In one embodiment, obtaining the played and unplayed segments of target speech includes: obtaining the interruption position of the target speech; and determining the played and unplayed segments of the target speech based on the completeness of the following statement at the interruption position.

[0090] It should be noted that since the target speech is determined before playback, the entire content of the target speech can be retrieved regardless of whether the robot outputs the target speech completely. The interruption point of the target speech can be determined based on the robot's output of the target speech.

[0091] The interruption point can be the location where the target speech output is interrupted. For example, if the robot plays part of the target speech through a speaker, and another part of the target speech is not played, then the connection point between these two parts of the target speech is the interruption point mentioned above.

[0092] For example, if the target voice is "When it comes to electric vehicles, what everyone generally cares about is charging speed and range, so we have launched the Super Electric System", and the output is interrupted after reaching "range", then the interruption point is between "range" and "so we have launched the Super Electric System".

[0093] After determining the interruption position, the played and unplayed segments can be identified based on the completeness of the statements following the interruption position. The following statements refer to the statements after the interruption position, and their completeness can be determined by checking whether the statement is a complete sentence. For example, the completeness of the statement can be determined using the large language model mentioned above.

[0094] The following sections will explain the specific implementation process for determining the played and unplayed segments under two different circumstances.

[0095] Figure 6 This is a logical diagram illustrating the determination of played and unplayed segments provided in the embodiments of this application. Please refer to... Figure 6 Based on the completeness of the statement following the interruption position, the played and unplayed segments of the target speech are determined, including: if the statement following the interruption position is a complete statement, the statement before the interruption position is determined to be a played segment, and the statement after the interruption position is determined to be an unplayed segment.

[0096] One method to determine whether a segment has been played or not can be achieved by checking whether the statement following the interruption point is a complete statement.

[0097] In one embodiment, if the statement following the interruption point is a complete statement, the played segment and the unplayed segment can be determined.

[0098] For example, continuing with the above example, if the following statement "For this reason, we launched the Super Electric System" is a complete statement, then the statement before the interruption position can be considered as the played segment, and the statement after the interruption position can be considered as the unplayed segment. That is to say, the played segment could be "When it comes to electric vehicles, what everyone is generally more concerned about is charging speed and range," and the unplayed segment could be "For this reason, we launched the Super Electric System."

[0099] Figure 7 For another logical diagram illustrating the determination of played and unplayed segments provided in this application embodiment, please refer to... Figure 7 Based on the completeness of the statement following the interruption position, the played and unplayed segments of the target speech are determined, including: if the statement following the interruption position is incomplete, the first statement before the interruption position is determined as the played segment, and the second statement before the interruption position and the statement after the interruption position are determined as the unplayed segment.

[0100] The first statement part is the complete part of the pre-interrupt statement, and the second statement part is the incomplete part of the pre-interrupt statement.

[0101] If the statement following the interruption is incomplete, such as "super electric system" mentioned above, then the first and second statements preceding the interruption can be identified. The first statement is the complete part of the statement preceding the interruption, and the second statement is the incomplete part. For example, "Speaking of electric vehicles, what people generally care about is charging speed and range" is the first statement. "Therefore, we launched..." is the second statement. The first statement can be considered the already played segment, i.e., "Speaking of electric vehicles, what people generally care about is charging speed and range." The second statement and the statements following the interruption can be considered the unplayed segment, i.e., "Therefore, we launched the super electric system."

[0102] In the voice interaction recovery method provided in this application embodiment, if the statement following the interruption position is a complete statement, the statement before the interruption position is determined to be a played segment, and the statement after the interruption position is determined to be an unplayed segment. If the statement following the interruption position is an incomplete statement, the first part of the statement before the interruption position is determined to be a played segment, and the second part of the statement before the interruption position and the statement after the interruption position are determined to be unplayed segments. Based on the completeness of the statement following the interruption position, different methods can be used to determine the unplayed and played segments, thereby ensuring the integrity of the unplayed segments and improving the accuracy of transition segment generation.

[0103] The following explains several feasible implementation methods for determining whether the target speech is interrupted or resumed in the embodiments of this application.

[0104] Figure 8 This is a flowchart illustrating the interruption or resumption of target speech provided in the embodiments of this application. Please refer to... Figure 8It should be noted that during the playback of the target audio, conditions for interrupting the target audio can be set, and the target audio can be interrupted when the interruption conditions are met; correspondingly, after the target audio is interrupted, conditions for resuming the target audio can be set, and the target audio can be resumed when the resumption conditions are met.

[0105] The interruption conditions for interrupting the target speech will be explained below.

[0106] In one embodiment, the method further includes: interrupting the target audio if a target interruption command is received or if voice information input by the target user is obtained during the playback of the target audio.

[0107] It should be noted that the target interrupt instruction can be an instruction initiated by the user, such as an instruction input by the user. The instruction can be input through an input device, such as a remote control or motion capture device.

[0108] In one embodiment, a corresponding command input control can be set on the robot or the remote control, and the user can input the above-mentioned target interruption command by triggering the control; or, the user can also initiate the target interruption command to the robot through corresponding gestures, actions, etc. There are no specific restrictions here, and it can be implemented in any way.

[0109] Alternatively, in addition to using interrupt commands, the robot can also acquire the voice information input by the target user and understand the user's purpose through a large language model, thereby interrupting the target voice when the user inputs the corresponding voice information.

[0110] For example, if the user explicitly instructs to end the current task, or if they have questions about parts of the current task, the target speech can be interrupted and other speech that meets the user's current needs can be generated, or an action that meets the user's current needs can be performed.

[0111] It should be noted that, in actual implementation, either of the two methods mentioned above can be used to interrupt the target speech, thereby stopping the playback of the target speech that the robot needs to output.

[0112] The following explains one feasible implementation process for speech recovery after the target speech is interrupted.

[0113] In one embodiment, if the target speech is interrupted, the method further includes: restoring the target speech if a target recovery instruction is received, or if no inserted speech input from the target user is obtained within the target duration.

[0114] It should be noted that the target recovery command can also be a command initiated by the user, such as a command input by the user. The input command can be input through an input device, such as a remote control or a motion capture device.

[0115] In one embodiment, a corresponding command input control can be set on the robot or the remote control, and the user can input the above-mentioned target recovery command by triggering the control; or, the user can also initiate the target recovery command to the robot through corresponding gestures, actions, etc. There are no specific restrictions here, and it can be implemented in any way.

[0116] Alternatively, in addition to using a recovery command, the robot can also recover the target speech if it has not received the inserted speech input from the target user within the target duration.

[0117] Inserting voice can be a user-inputted question.

[0118] For example, if the user does not input any other question within 10 seconds, the robot can resume the target voice.

[0119] In one embodiment, when the target speech is interrupted, the method further includes: acquiring inserted speech input by the target user; determining the user intent of the target user based on the inserted speech; and restoring the target speech if the user intent is to stop the dialogue.

[0120] It should be noted that the robot can also acquire the inserted speech input by the user and input it into a large language model for semantic recognition, thereby determining the user's intent. The method used is the same as the aforementioned method for determining user intent, and will not be explained again here.

[0121] After determining the user's intent, if the user's intent is to stop the conversation with the robot, then it can be determined that the robot does not need to interrupt the target speech and can resume the target speech. Accordingly, the aforementioned steps can be performed to generate the resumed speech of the target speech, and the resumed speech can be played.

[0122] It should be understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0123] Based on the foregoing embodiments, this application provides a voice interaction recovery device, which includes various modules and units included in each module, and can be implemented by a processor; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0124] Figure 9 This is a schematic diagram of the structure of the voice interaction recovery device provided in the embodiments of this application. Please refer to... Figure 9 In another aspect of the embodiments of this application, a voice interaction recovery device is also provided, applied to a robot, including: an acquisition module 910 and an output module 920; The acquisition module 910 is used to acquire the played and unplayed segments of the target audio when the playback of the target audio is interrupted. The output module 920 is used to output the recovered speech of the target speech in the case of target speech recovery. The recovered speech includes transition segments and unplayed segments of the target speech, wherein the transition segments are generated based on the played segments and unplayed segments.

[0125] In one embodiment, the output module 920 is further configured to generate a transition segment based on the played segment and the unplayed segment; and to splice the transition segment and the unplayed segment together to obtain the restored speech of the target speech.

[0126] In one embodiment, the output module 920 is specifically used to generate a transition segment based on the played segment, the unplayed segment, and a preset large language model, wherein the large language model is a model obtained after training based on the played sample segment, the unplayed sample segment, and the transition sample segment.

[0127] In one embodiment, the output module 920 is specifically used to determine the topic information of the target speech based on the played segment; determine the target prompt word from multiple preset prompt words according to the topic information of the target speech; and input the target prompt word and the unplayed segment into a large language model to obtain a transition segment.

[0128] In one embodiment, the acquisition module 910 is specifically used to acquire the interruption position of the target speech; and determine the played segment and unplayed segment of the target speech based on the completeness of the following statement at the interruption position.

[0129] In one embodiment, the acquisition module 910 is specifically configured to determine, when the statement following the interruption position is a complete statement, that the statement before the interruption position is a played segment and the statement after the interruption position is an unplayed segment; and when the statement following the interruption position is an incomplete statement, determine that the first statement portion before the interruption position is a played segment and the second statement portion before the interruption position and the statement after the interruption position are unplayed segments, wherein the first statement portion is the complete portion of the statement preceding the interruption position and the second statement portion is the incomplete portion of the statement preceding the interruption position.

[0130] In one embodiment, the device interrupts the target audio playback if a target interruption command is received or if voice information input by the target user is obtained during the playback of the target audio.

[0131] In one embodiment, the device restores the target speech if a target speech recovery command is received when the target speech is interrupted, or if no inserted speech input from the target user is obtained within the target duration.

[0132] In one embodiment, the device acquires inserted speech input by the target user when the target speech is interrupted; determines the user's intent based on the inserted speech; and resumes the target speech if the user's intent is to stop the conversation.

[0133] The voice interaction recovery device provided in this application embodiment can acquire the played and unplayed segments of the target voice when playback is interrupted; and output the recovered voice when the target voice is recovered. The recovered voice includes a transition segment and an unplayed segment of the target voice, wherein the transition segment is generated based on the played and unplayed segments. The transition segment can summarize the played segments and serve as an introduction to the unplayed segments, thereby achieving a natural transition in the recovered voice of the target voice. This allows users to understand the recovered voice output by the robot more naturally and smoothly. Furthermore, since the recovered voice is composed of the transition segment and the unplayed segment, it can also avoid repetition or omission in the robot's output voice, further improving the user's experience of voice interruption recovery during robot use.

[0134] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0135] It should be noted that, in the embodiments of this application... Figure 9 The module division of the voice interaction recovery device shown is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or have two or more units integrated into one unit. The integrated units can be implemented in hardware, as software functional units, or a combination of both.

[0136] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0137] Figure 10 This is a schematic diagram of the robot structure provided in the embodiments of this application. Please refer to... Figure 10This application provides a robot whose internal structure diagram can be as follows: Figure 10 As shown. The computer device includes a processor 1020, memory, and a network interface 1040 connected via a system bus 1010. The processor 1020 provides computing and control capabilities. The memory includes a non-volatile storage medium 1031 and internal memory 1032. The non-volatile storage medium 1031 stores an operating system, computer programs, and a database. The internal memory 1032 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium 1031. The database is used to store data. The network interface 1040 is used to communicate with external terminals via a network connection. When the computer program is executed by the processor 1020, it implements the aforementioned methods.

[0138] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method provided in the above embodiments.

[0139] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the method provided in the above-described method embodiments.

[0140] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0141] In one embodiment, the voice interaction recovery device provided in this application can be implemented as a computer program, which can be implemented in the form of, for example... Figure 10 The device operates on the computer device shown. The memory of the computer device can store the various program modules that make up the above-described apparatus. The computer program, composed of the various program modules, causes the processor to execute the steps of the methods in the various embodiments of this application described in this specification.

[0142] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0143] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0144] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0145] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0147] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0148] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0149] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0150] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0151] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0152] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0153] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0154] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for recovering voice interaction, characterized in that, Applications in robots, including: If the playback of the target audio is interrupted, obtain the played and unplayed segments of the target audio. In the case of target speech recovery, the recovered speech of the target speech is output, the recovered speech includes a transition segment and an unplayed segment of the target speech, wherein the transition segment is generated based on the played segment and the unplayed segment.

2. The method according to claim 1, characterized in that, Before outputting the recovered speech of the target speech, the method further includes: The transition segment is generated based on the played segment and the unplayed segment; The transition segment and the unplayed segment are spliced ​​together to obtain the recovered speech of the target speech.

3. The method according to claim 2, characterized in that, The step of generating the transition segment based on the played segment and the unplayed segment includes: The transition segment is generated based on the played segment, the unplayed segment, and a preset large language model, wherein the large language model is a model obtained after training on the played sample segment, the unplayed sample segment, and the transition sample segment.

4. The method according to claim 3, characterized in that, The step of generating the transition segment based on the played segment, the unplayed segment, and a preset large language model includes: The topic information of the target speech is determined based on the already played segment; The target prompt word is determined from multiple preset prompt words based on the topic information of the target speech; The target prompt word and the unplayed segment are input into the large language model to obtain the transition segment.

5. The method according to claim 1, characterized in that, The step of obtaining the played and unplayed segments of the target audio includes: Obtain the interruption position of the target speech; Based on the completeness of the following statement at the interruption location, the played and unplayed segments of the target speech are determined.

6. The method according to claim 5, characterized in that, Determining the played and unplayed segments of the target speech based on the completeness of the following statement at the interruption position includes: If the statement following the interruption position is a complete statement, the statement before the interruption position is determined to be the played segment, and the statement after the interruption position is determined to be the unplayed segment. If the statement following the interruption position is incomplete, the first statement portion before the interruption position is determined to be the played segment, and the second statement portion before the interruption position and the statement after the interruption position are determined to be the unplayed segment. The first statement portion is the complete portion of the statement preceding the interruption position, and the second statement portion is the incomplete portion of the statement preceding the interruption position.

7. The method according to claim 1, characterized in that, The method further includes: If a target interruption command is received during the playback of the target audio, or if voice information input by the target user is obtained, the target audio will be interrupted.

8. The method according to claim 7, characterized in that, In the event that the target speech is interrupted, the method further includes: If a target recovery command is received, or if no inserted voice input from the target user is obtained within the target duration, the target voice is recovered.

9. The method according to claim 7, characterized in that, In the event that the target speech is interrupted, the method further includes: Obtain the inserted voice input from the target user; The user intent of the target user is determined based on the inserted voice; If the user intends to stop the conversation, the target speech is resumed.

10. A device for restoring voice interaction, characterized in that, Applied to robots, including: acquisition module and output module; The acquisition module is used to acquire the played and unplayed segments of the target audio when the playback of the target audio is interrupted. The output module is configured to output restored speech of the target speech when the target speech is restored. The restored speech includes a transition segment and an unplayed segment of the target speech, wherein the transition segment is generated based on the played segment and the unplayed segment.

11. A robot comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 9.

Images