Multimodal feature-based spoken dialogue processing method, device, and electronic device

Abstract

A method, device (300) and electronic device for speech dialogue processing based on multimodal features. The method includes a step (101) of acquiring first speech information including a silent section currently input by a user during a dialogue exchange with the user, a step (102) of determining semantic feature information of the text information based on text information of the first speech information and historical context information of the first speech information, a step (103) of determining speech feature information of the first speech information based on a speech fragment preceding the silent section in the first speech information, a step (104) of acquiring temporal feature information of the first speech information, and a step (105) of determining whether the user has finished speech input based on the semantic feature information, the speech feature information and the temporal feature information.

Description

[Technical Field]

[0001] This application relates to the field of computer technology, and in particular to a method, device and electronic equipment for speech interaction processing based on multimodal features.

[0002] This application is based on and claims priority from a Chinese patent application bearing application number 202111337746.8 and filed on November 9, 2021, the entire contents of which are hereby incorporated by reference into this application. [Background technology]

[0003] In a voice dialogue system, when a user speaks, the voice dialogue system needs to determine the appropriate timing to take over the speaking role, i.e., to make the human-machine interaction smooth and natural, the voice dialogue system should alternate between the listener and speaker characters.

[0004] Currently, many voice dialogue systems use voice activity detection (VAD) to detect the length of a user's silence. When the length of the user's silence exceeds a threshold (e.g., 0.8 s to 1 s), the system assumes the right to speak. However, in such a system where the length of the silence is fixed, the system's response is too fast and sensitive when the silence exceeds the threshold, even though the user has not finished speaking and is still thinking. On the other hand, if the user's exchange is fast and concise, waiting for the silence to reach a set threshold before assuming the right to speak can result in a slow system response, which may lead the user to repeat their answer. Therefore, how a voice dialogue system determines when to assume the right to speak is currently an issue that needs to be resolved as soon as possible. Summary of the Invention

[0005] This application proposes a method, device and electronic equipment for speech interaction processing based on multimodal features.

[0006] In one embodiment of the present application, there is proposed a multimodal feature-based speech dialogue processing method, including the steps of: acquiring first speech information including a silent interval currently input by a user during a dialogue exchange with the user; determining semantic feature information of the text information based on text information of the first speech information and historical context information of the first speech information; determining speech feature information of the first speech information based on a speech fragment preceding the silent interval in the first speech information; acquiring temporal feature information of the first speech information; and determining whether the user should terminate speech input based on the semantic feature information, the speech feature information, and the temporal feature information.

[0007] In one embodiment of the present application, the step of determining semantic feature information of the text information based on text information of the first speech information and historical context information of the first speech information includes: performing speech recognition on the first speech information to obtain text information of the first speech information; obtaining historical context information of the first speech information; and inputting the text information and the historical context information into a semantic representation model to obtain semantic feature information of the text information.

[0008] In one embodiment of the present application, the step of determining speech feature information of the first speech information based on a speech fragment preceding the silent section in the first speech information includes: obtaining a speech fragment of a first predetermined time length preceding the silent section in the first speech information; dividing the speech fragment according to a second predetermined time length to obtain a plurality of speech fragments; extracting acoustic feature information corresponding to each of the plurality of speech fragments; and combining the acoustic feature information corresponding to each of the plurality of speech fragments to obtain a combined feature corresponding to each of the plurality of speech fragments; and inputting the combined feature into a deep residual network to obtain the speech feature information of the first speech information.

[0009] In one embodiment of the present application, the step of obtaining temporal feature information of the first audio information includes obtaining an audio time length, a speaking rate, and a text length of the first audio information, and inputting the audio time length, the speaking rate, and the text length into a pre-trained multi-layer perceptron MLP model so as to obtain the temporal feature information of the first audio information.

[0010] In one embodiment of the present application, the step of determining whether the user has terminated speech input based on the semantic feature information, the speech feature information, and the temporal feature information includes: inputting the semantic feature information, the speech feature information, and the temporal feature information into a multimodal fusion model; and determining whether the user has terminated speech input based on an output result of the multimodal fusion model.

[0011] In one embodiment of the present application, when it is determined that the user has finished voice input, the method further includes determining first reply voice information corresponding to the first voice information and outputting the first reply voice information.

[0012] In one embodiment of the present application, if it is determined that the user has not finished voice input, the method further includes obtaining second voice information to be input again by the user, determining corresponding second reply voice information based on the first voice information and the second voice information, and outputting the second reply voice information.

[0013] In another embodiment of the present application, there is proposed a multimodal feature-based speech interaction processing device, including: a first acquisition module that acquires first speech information including a silent interval currently input by a user during a dialogue exchange with the user; a first determination module that determines semantic feature information of the text information based on text information of the first speech information and historical context information of the first speech information; a second determination module that determines speech feature information of the first speech information based on a speech fragment preceding the silent interval in the first speech information; a second acquisition module that acquires temporal feature information of the first speech information; and a third determination module that determines whether the user should terminate speech input based on the semantic feature information, the speech feature information, and the temporal feature information.

[0014] In one embodiment of the present application, the first determination module is specifically used for performing speech recognition on the first speech information, so as to obtain text information of the first speech information; obtaining historical context information of the first speech information; and inputting the text information and the historical context information into a semantic representation model, so as to obtain semantic feature information of the text information.

[0015] In one embodiment of the present application, the second determination module is specifically used for obtaining a speech fragment of a first predetermined time length before the silent section in the first speech information, dividing the speech fragment according to a second predetermined time length to obtain a plurality of speech fragments, extracting acoustic feature information corresponding to each of the plurality of speech fragments, and combining the acoustic feature information corresponding to each of the plurality of speech fragments to obtain a combined feature corresponding to each of the plurality of speech fragments, and inputting the combined feature into a deep residual network to obtain speech feature information of the first speech information.

[0016] In one embodiment of the present application, the second acquisition module is specifically used for acquiring the speech duration, speaking rate and text length of the first speech information, and inputting the speech duration, speaking rate and text length into a pre-trained multi-layer perceptron MLP model so as to obtain temporal feature information of the first speech information.

[0017] In one embodiment of the present application, the third determination module includes: a multimodal processing unit that inputs the semantic feature information, the speech feature information, and the temporal feature information into a multimodal fusion model; and a determination unit that determines whether the user should terminate speech input based on an output result of the multimodal fusion model.

[0018] In one embodiment of the present application, the device further includes a first processing module, which is used to determine first reply voice information corresponding to the first voice information and output the first reply voice information when it is determined that the user has finished voice input.

[0019] In one embodiment of the present application, the system further includes a third acquisition module that acquires second voice information input by the user again when it is determined that the user has not finished voice input, and a second processing module that determines corresponding second reply voice information based on the first voice information and the second voice information, and outputs the second reply voice information.

[0020] Another embodiment of the present application proposes an electronic device comprising a memory and a processor, the memory storing computer instructions that, when executed by the processor, implement the multimodal feature-based speech interaction processing method of the embodiment of the present application.

[0021] Another embodiment of the present application proposes a non-transitory computer-readable recording medium storing computer instructions for causing a computer to execute the multimodal feature-based speech interaction processing method disclosed in the embodiment of the present application.

[0022] Another embodiment of the present application provides a computer program product which, when executed by an instruction processor therein, implements the multimodal feature based speech interaction processing method of the embodiment of the present application.

[0023] Other effects of the above-described selectable configurations will be described below in conjunction with specific examples.

[0024] The accompanying drawings are provided for a better understanding of the present technical solution, but are not intended to limit the present application. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a flowchart illustrating a multimodal feature-based speech dialogue processing method according to one embodiment of the present application. [Figure 2] FIG. 2 is a diagram illustrating a voice dialogue processing method in accordance with a model framework diagram according to one specific embodiment of the present application. [Figure 3] FIG. 3 is a diagram showing the configuration of a speech dialogue processing apparatus based on multimodal features according to one embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing the configuration of a speech dialogue processing apparatus based on multimodal features according to another embodiment of the present invention. [Figure 5] FIG. 5 is a diagram illustrating an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0026]

[0023] The following detailed description of the embodiments of the present disclosure is provided below. Examples of such embodiments are shown in the accompanying drawings, and identical or similar elements, or elements having identical or similar functions, are designated by identical or similar symbols throughout. The embodiments described below with reference to the accompanying drawings are illustrative and are used to explain the present application, but are not to be construed as limiting the present application.

[0027] Hereinafter, a multimodal feature-based speech interaction processing method, device, and electronic device according to embodiments of the present application will be described with reference to the accompanying drawings.

[0028] 1 is a flowchart showing a multimodal feature-based speech dialogue processing method according to one embodiment of the present application. The main body that executes the multimodal feature-based speech dialogue processing method proposed in the embodiment of the present application is a multimodal feature-based speech dialogue processing device, which can be realized by software and / or hardware. In the speech dialogue system of the multimodal feature-based speech dialogue processing device according to the embodiment of the present application, the speech dialogue system can be installed in an electronic device. Examples of the electronic device include a terminal device and a server.

[0029] As shown in FIG. 1, the multimodal feature-based speech interaction processing method may include steps 101 to 105.

[0030] In step 101, first voice information including a silent section currently input by the user is acquired during a dialogue exchange with the user.

[0031] In step 102, semantic feature information of the text information is determined based on the text information of the first speech information and historical context information of the first speech information.

[0032] In one embodiment of the present application, speech recognition may be performed on the first speech information to obtain text information of the first speech information, and historical context information of the first speech information may be obtained; and the text information and historical context information may be input into a semantic representation model to obtain semantic feature information of the text information.

[0033] In some embodiments, the semantic representation model includes a Transformer model based on a self-attention mechanism to capture long-distance dependencies between text information and historical context information and accurately determine semantic feature information of the text information based on the long-distance dependencies.

[0034] In some embodiments, a Transformer model may include multiple coding layers, each of which includes a Transformer-based coding structure, encodes input content using the corresponding coding structure, and then inputs the output result to the corresponding next coding layer for processing.

[0035] In some embodiments, an exemplary embodiment of obtaining historical context information of the first voice information includes obtaining a plurality of historical voice dialogue pieces prior to the first voice information, and obtaining the historical context information of the first voice information based on the plurality of historical voice dialogue pieces.

[0036] In step 103, audio feature information of the first audio information is determined based on an audio fragment before the silent section in the first audio information.

[0037] In some embodiments, a speech fragment of a first predetermined time length before a silent section in the first speech information is obtained, the speech fragment is divided according to a second predetermined time length to obtain a plurality of speech fragments, acoustic feature information corresponding to each of the plurality of speech fragments is extracted, the acoustic feature information corresponding to each of the plurality of speech fragments is combined to obtain a combined feature corresponding to each of the plurality of speech fragments, and the combined feature is input into a deep residual network to obtain speech feature information of the first speech information.

[0038] In some embodiments, the first predetermined time length is preset. For example, the first predetermined time length may be 2 seconds. That is, a 2-second audio fragment may be cut out before a silent section in the first audio information.

[0039] In some embodiments, the second predetermined length of time is preset. The first predetermined length of time is longer than the second predetermined length of time, for example, if the first predetermined length of time is 2 seconds, the second predetermined length of time may be 50 milliseconds (ms). In some embodiments, after cutting an audio segment having a length of 2 seconds, the audio segment can be divided by 50 ms to obtain multiple audio segments, each having a length of 50 ms.

[0040] In some embodiments, the acoustic feature information includes, but is not limited to, power, volume, pitch, number of zero crossings, and the like.

[0041] In step 104, the time feature information of the first audio information is obtained.

[0042] In some embodiments, a speech duration, a speaking rate and a text length of the first speech information are obtained, and the speech duration, the speaking rate and the text length are input into a pre-trained Multi Layer Perceptron (MLP) model to obtain temporal feature information of the first speech information.

[0043] In some implementations, the text length is determined based on text information corresponding to the first audio information.

[0044] In step 105, it is determined whether the user has finished the speech input based on the semantic feature information, the speech feature information, and the time feature information.

[0045] In some embodiments, in order to accurately determine whether a user has finished speech input, semantic feature information, speech feature information, and temporal feature information are input into a multimodal fusion model, and whether a user has finished speech input is determined based on the output result of the multimodal fusion model.

[0046] In some embodiments, when the multimodal fusion model acquires semantic feature information, speech feature information, and temporal feature information, weights corresponding to each of the above-mentioned semantic feature information, speech feature information, and temporal feature information are acquired, and weighting processing is performed on the semantic feature information, speech feature information, and temporal feature information based on the weights, and the weighting results are input into the activation function of the multimodal fusion model so as to obtain an output result of the multimodal fusion model.

[0047] In some embodiments, if the output result of the multimodal fusion model indicates that the user has finished their speech input, the dialog system determines that the user has finished their speech input, and then determines that the dialog system can take over the speaking role. In some other embodiments, if the output result of the multimodal fusion model indicates that the user has not finished their speech input, the dialog system determines that the user has not finished their speech input, and then continues listening and responds after it is determined that the user has finished their input.

[0048] According to an embodiment of the present application, a multimodal feature-based speech dialogue processing method combines text information of speech information currently input by the user and historical context information of first speech information to determine semantic feature information of the text information, determines speech feature information of the first speech information based on a speech fragment preceding a silent section in the first speech information, and obtains temporal feature information of the first speech information. Whether the user should end speech input is determined based on the semantic feature information, speech feature information, and temporal feature information. Thus, during a dialogue with the user, the system accurately determines whether the system can take over the speaking role by combining the semantic feature information, speech feature information, and temporal feature information.

[0049] Based on the above-mentioned embodiments, in order to enable the dialogue system to accurately respond to the voice information input by the user, in some embodiments, when the user determines to end the voice input, first reply voice information corresponding to the first voice information is determined and the first reply voice information is output.

[0050] In some other embodiments, when it is determined that the user has not finished the voice input, second voice information to be input by the user is acquired again, corresponding second reply voice information is determined based on the first voice information and the second voice information, and the second reply voice information is output, thereby combining the first voice information currently input by the user with the second voice information to be input again to provide an accurate reply.

[0051] In order to help those skilled in the art understand the present invention clearly, the method of this embodiment will be further described below in conjunction with FIG.

[0052] As shown in Fig. 2, in this embodiment, in the process of determining whether the user should end the speech input, features in three different dimensions, namely, speech feature information, semantic feature information, and temporal feature information, are used to determine whether the user should end the speech input. That is, in this embodiment, features in three different dimensions, namely, semantic feature information, speech feature information, and temporal feature information, are used to determine whether the dialogue system can take over the speaking right, i.e., whether the dialogue system should output a corresponding reply.

[0053] The processes for acquiring semantic feature information, speech feature information, and time feature information will be described below.

[0054] 1) Acquisition of semantic feature information Among these, semantic feature information is obtained from text information after speech recognition. Considering that "semantic completeness" is a fundamental element in speaker switching, the importance of semantic feature information in determining speaker right goes without saying. This means that the system can usually take over speaker right after determining that the user has fully expressed their intention. Semantic completeness is generally determined based on the context. For example, consider the following simple example: [Table 1] In the example on the left, the user has given a clear response with clear meaning, and at this point the dialogue system can take over the speaker. In the example on the right, the user is hesitating for a moment, but based on what the user is currently typing it can be determined that the user has not finished speaking. At this point the dialogue system can choose to continue listening and wait for the user to finish speaking.

[0055] To model such semantic integrity, the process of a user's dialogue interaction with a dialogue system can be summarized. After obtaining the voice information currently input by the user, voice recognition can be performed on the voice information to obtain current text information. Then, the historical context information of the currently input voice information and the current text information can be encoded to obtain semantic feature information of the text information.

[0056] In some embodiments, a self-attention based Transformer model is used to encode historical context information of the currently input audio information and text information corresponding to the current text information.

[0057] The self-attention mechanism in the Transformer model can be understood as being able to capture long-distance dependencies between historical context and text information. The final semantic features are expressed as follows:

number

[0058] 2) Acquisition of voice feature information It is understandable that in the process of a dialogue, certain voice features, such as changes in pitch and volume, are important clues for determining whether to switch speakers. Therefore, in the process of dialogue with a user, after obtaining the voice information currently input by the user, the voice (2 seconds) before the user goes silent is cut out from the voice information, and then divided into fixed-length fragments, i.e., frame division (one frame is 50 ms). Next, for each frame of voice, corresponding acoustic features such as power, volume, pitch, and number of zero crossings are extracted and combined into a one-dimensional vector to obtain the voice feature representation f of each frame, as described below. i Finally, we obtain the sequence frame features F=[f1,f2,…,f n ] is input to a multi-layer deep residual network (ResNet). The final speech features are expressed as follows:

number

[0059] 3) Time characteristics It is important to understand that temporal features (e.g., duration, speaking rate, and text length of a speech fragment) also play a role in determining whether the speaker should take over. For example, in a system-initiated outbound dialogue scenario, the system can often take over the speaker's role after the user provides a short response. When the system needs to listen, it is usually because the user provided a long response due to hesitation or other reasons. Therefore, to accurately determine whether the dialogue system can take over the speaker's role, the system obtains the speech duration, speaking rate, and text length of the speech information currently input by the user during the dialogue exchange, performs a bucketing process on the speech duration, speaking rate, and text length, and then inputs the processed speech duration, speaking rate, and text length into an MLP model to obtain low-dimensional temporal feature information of the speech information.

[0060] Among them, the low-dimensional features extracted through the multi-layer sensing network are expressed as follows:

number

[0061] 4) Multimodal feature fusion In some implementations, after obtaining the feature representation of each modality, it is input into a multimodal fusion model, which fuses three different features to make a speaker right decision.

number

[0062] Among the above mentioned multimodal fusion models, they can be constructed based on feedforward neural networks.

[0063] In one embodiment of the present application, a multimodal feature-based speech dialogue processing device is further provided, which corresponds to the multimodal feature-based speech dialogue processing method provided in several of the above-mentioned embodiments. Since the multimodal feature-based speech dialogue processing device provided in the embodiments of the present application corresponds to the multimodal feature-based speech dialogue processing method provided in several of the above-mentioned embodiments, the embodiment of the multimodal feature-based speech dialogue processing method also applies to the multimodal feature-based speech dialogue processing device provided in the embodiments of the present application.

[0064] FIG. 3 is a diagram showing the configuration of a speech dialogue processing apparatus based on multimodal features according to one embodiment of the present invention.

[0065] As shown in FIG. 3, the multimodal feature-based spoken dialogue processing apparatus 300 includes a first acquisition module 301, a first determination module 302, a second determination module 303, a second acquisition module 304 and a third determination module 305.

[0066] The first acquisition module 301 is used to acquire first voice information including silent periods currently input by the user during a dialogue exchange with the user.

[0067] The first determination module 302 is used for determining semantic feature information of the text information based on the text information of the first speech information and the historical context information of the first speech information.

[0068] The second determination module 303 is used for determining the audio feature information of the first audio information based on the audio fragment before the silent section in the first audio information.

[0069] The second acquiring module 304 is used for acquiring the time feature information of the first audio information.

[0070] The third determination module 305 is used to determine whether the user has finished the voice input based on the semantic feature information, the speech feature information and the time feature information.

[0071] In one embodiment of the present application, the first determination module 302 is specifically used for performing speech recognition on the first speech information, so as to obtain text information of the first speech information, obtaining historical context information of the first speech information, and inputting the text information and the historical context information into a semantic representation model, so as to obtain semantic feature information of the text information.

[0072] In one embodiment of the present application, the second determination module 303 is specifically used for obtaining a speech fragment of a first predetermined time length before a silent section in the first speech information, dividing the speech fragment according to a second predetermined time length to obtain a plurality of speech fragments, extracting acoustic feature information corresponding to each of the plurality of speech fragments, and combining the acoustic feature information corresponding to each of the plurality of speech fragments to obtain a combined feature corresponding to each of the plurality of speech fragments, and inputting the combined feature into a deep residual network to obtain the speech feature information of the first speech information.

[0073] In one embodiment of the present application, the second acquisition module 304 is specifically used for acquiring the speech duration, speaking rate and text length of the first speech information, and inputting the speech duration, speaking rate and text length into a pre-trained multi-layer perceptron MLP model to obtain the temporal feature information of the first speech information.

[0074] In one embodiment of the present application, based on the embodiment of the device shown in FIG. 3, as shown in FIG. 4, the third determination module 305 includes a multimodal processing unit 3051 and a determination unit 3052.

[0075] The multimodal processing unit 3051 is used to input semantic feature information, speech feature information and temporal feature information into the multimodal fusion model.

[0076] The determining unit 3052 is used to determine whether the user has finished the voice input according to the output result of the multimodal fusion model.

[0077] In one embodiment of the present application, as shown in FIG. 4, the multimodal feature-based spoken dialogue processing apparatus 300 further includes a first processing module 306 .

[0078] The first processing module 306 is used to determine first reply voice information corresponding to the first voice information and output the first reply voice information when it is determined that the user has finished the voice input.

[0079] In one embodiment of the present application, as shown in FIG. 4, the multimodal feature-based spoken dialogue processing apparatus 300 includes a third acquisition module 307 and a second processing module 308.

[0080] The third acquiring module 307 is used to acquire second voice information input by the user again when it is determined that the user has not finished the voice input.

[0081] The second processing module 308 is used for determining corresponding second reply voice information according to the first voice information and the second voice information, and outputting the second reply voice information.

[0082] In an embodiment of the present application, a multimodal feature-based speech dialogue processing device, during a dialogue exchange with a user, combines text information of the speech information currently input by the user and historical context information of the first speech information to determine semantic feature information of the text information, determines speech feature information of the first speech information based on a speech fragment preceding a silent section in the first speech information, obtains temporal feature information of the first speech information, and determines whether the user has terminated speech input based on the semantic feature information, speech feature information, and temporal feature information. Thus, during a dialogue exchange with the user, the system accurately determines whether the system can take over the speaking right by combining the semantic feature information, speech feature information, and temporal feature information.

[0083] According to an embodiment of the present application, the present application further provides an electronic device and a readable recording medium.

[0084] FIG. 5 shows an electronic device according to one embodiment of the present invention.

[0085] As shown in FIG. 5, the electronic device includes a memory 501, a processor 502, and computer instructions stored in the memory 501 and executable on the processor 502.

[0086] When the processor 502 executes the instructions, it implements the multimodal feature based speech interaction processing method provided in the above-described embodiments.

[0087] Furthermore, the electronic device further includes a communication interface 503 used for communication between the memory 501 and the processor 502 .

[0088] The memory 501 is used to store computer instructions that can be executed on the processor 502 .

[0089] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, for example at least one disk memory.

[0090] The processor 502 is used to implement the multimodal feature-based speech interaction processing method in the above-described embodiment when executing the program.

[0091] When the memory 501, the processor 502, and the communication interface 503 are implemented independently, the communication interface 503, the memory 501, and the processor 502 are connected to one another via a bus, allowing them to communicate with one another. Examples of the bus include an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, and an Extended Industry Standard Architecture (EISA) bus. Buses are divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, Figure 5 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0092] In some embodiments, when the memory 501, the processor 502, and the communication interface 503 are integrated and implemented on a single chip, the memory 501, the processor 502, and the communication interface 503 can communicate with each other via an internal interface.

[0093] Processor 502 may include a central processing unit (CPU) or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0094] The present application further proposes a computer program product, which, when executed by an instruction processor therein, implements the multimodal feature-based speech interaction processing method of the present application embodiment.

[0095] In the description herein, a description using terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, descriptions indicating the meaning of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be appropriately combined in any one or more embodiments or examples. Furthermore, if not mutually inconsistent, a person skilled in the art may combine different embodiments or examples described herein and features of different embodiments or examples.

[0096] Additionally, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or the number of the indicated technical features. This expressly or implies that the feature qualified by "first" or "second" includes at least one of this feature. In the description of this application, "plurality" means at least two, e.g., two, three, etc., unless otherwise clearly and specifically limited.

[0097] Any process or method description depicted in a flowchart or otherwise described herein should be understood to represent a module, fragment, or portion thereof, including one or more pieces of code capable of executing instructions, for implementing customized logical functions or process steps, and the scope of the preferred embodiments of the present application includes additional embodiments that perform functions not necessarily in the order shown or discussed, but in a substantially simultaneous manner or in reverse order based on such functions, as would be understood by one skilled in the art to which the embodiments of the present application pertain.

[0098] The logic and / or steps depicted in flowcharts or otherwise described herein may, for example, be an executable sequential listing of instructions for implementing logical functions, and may be embodied in any computer-readable medium for use with or in conjunction with an instruction execution system, device, or apparatus (e.g., a computer-based system, a processor-including system, or a system that receives and executes instructions from another instruction execution system, device, or apparatus). For purposes of this specification, a "computer-readable medium" refers to a device capable of storing, storing, communicating, distributing, or transmitting a program for use with an instruction execution system, device, or apparatus, or in conjunction with such an instruction execution system, device, or apparatus. More specific examples (non-exhaustive list) of computer-readable media include electrical connections with one or more wires (electronic devices), portable computer disk cartridges (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable-writeable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disk read-only memory (CD-ROM). A computer-readable medium may also be paper or other suitable medium upon which a program may be printed. For example, paper or other media can be optically scanned, and then edited, decrypted, or processed in any other suitable manner as necessary to obtain an electronic version of the program, which can then be stored in computer memory.

[0099] It should be understood that each part of the present application can be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, when implemented in hardware, as in other embodiments, it can be implemented using one or a combination of the following techniques known in the art: discrete logic circuits having logic gate circuits that implement logic functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0100] As can be understood by those skilled in the art, all or part of the steps of the method of the above-described embodiments can be accomplished by instructing related hardware through a program, which is stored in a computer-readable medium and, when executed, performs one or a combination of the steps of the method of the embodiments.

[0101] Furthermore, each functional unit in each embodiment of the present application may be integrated into a single processing module, each unit may exist physically separately, or two or more units may be integrated into a single module. The above-mentioned integrated module may be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it may be stored in a computer-readable storage medium.

[0102] The storage medium mentioned above may be a read-only memory, a magnetic disk, an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above-described embodiments are illustrative and should not be understood as limiting the present application, and that those skilled in the art may make changes, modifications, substitutions, and variations to the above-described embodiments within the scope of the present application.

Claims

1. A method for providing a computer with a method of communicating with a user, the method comprising: acquiring first speech information currently input by the user, the first speech information including a silent section; determining semantic feature information of the text information based on text information of the first speech information and historical context information of the first speech information; determining audio feature information of the first audio information based on an audio fragment preceding the silent section in the first audio information; The computer acquires temporal feature information of the first audio information; determining by the computer whether the user has finished speech input based on the semantic feature information, the speech feature information, and the temporal feature information; A multimodal feature based speech dialogue processing method, comprising:

2. determining semantic feature information of the text information based on text information of the first speech information and historical context information of the first speech information, the computer performing phonetic identification on the first audio information to obtain text information of the first audio information; the computer obtaining historical context information of the first audio information; the computer inputs the text information and the historical context information into a semantic representation model to obtain semantic feature information of the text information; The method of claim 1 , comprising:

3. determining audio feature information of the first audio information based on an audio fragment preceding the silent section in the first audio information, the computer acquires an audio fragment of a first predetermined time length before the silent section in the first audio information; the computer dividing the audio fragment according to a second predetermined time length to obtain a plurality of audio fragments; the computer extracting acoustic feature information corresponding to each of the plurality of speech fragments, and combining the acoustic feature information corresponding to each of the plurality of speech fragments to obtain combined features corresponding to each of the plurality of speech fragments; the computer inputs the combined features into a deep residual network to obtain audio feature information of the first audio information; The method of claim 1 , comprising:

4. The step of acquiring temporal feature information of the first audio information includes: the computer acquires a speech time length, a speech rate, and a text length of the first speech information; the computer inputs the speech time length, the speaking rate, and the text length into a pre-trained multi-layer perceptron MLP model to obtain temporal feature information of the first speech information; The method of claim 1 , comprising:

5. the step of determining whether the user has finished speech input based on the semantic feature information, the speech feature information, and the time feature information, the computer inputs the semantic feature information, the speech feature information, and the temporal feature information into a multimodal fusion model; the computer determines whether the user has finished speech input based on the output result of the multimodal fusion model; The method of claim 1 , comprising:

6. When the computer determines that the user has finished voice input, it determines first reply voice information corresponding to the first voice information and outputs the first reply voice information. The method of any one of claims 1 to 5, further comprising:

7. When the computer determines that the user has not finished voice input, it obtains second voice information to be input again by the user. the computer determines corresponding second reply voice information based on the first voice information and the second voice information, and outputs the second reply voice information; The method of any one of claims 1 to 5, further comprising:

8. a first acquisition module that acquires first voice information including a silent section currently input by the user during a dialogue exchange with the user; a first determination module for determining semantic feature information of the text information based on text information of the first speech information and historical context information of the first speech information; a second determination module for determining audio feature information of the first audio information based on an audio fragment preceding the silent section in the first audio information; a second acquisition module for acquiring time feature information of the first audio information; a third determination module for determining whether the user has finished speech input based on the semantic feature information, the speech feature information, and the time feature information; A multimodal feature based speech interaction processing device, comprising:

9. The first determination module specifically comprises: performing speech recognition on the first speech information to obtain text information of the first speech information; obtaining historical context information of the first audio information; inputting the text information and the historical context information into a semantic representation model to obtain semantic feature information of the text information; The device according to claim 8,

10. The second determination module specifically comprises: acquiring a speech fragment of a first predetermined time length before the silent section in the first speech information; dividing the audio fragment according to a second predetermined time length to obtain a plurality of audio fragments; extracting acoustic feature information corresponding to each of the plurality of speech fragments, and combining the acoustic feature information corresponding to each of the plurality of speech fragments to obtain combined features corresponding to each of the plurality of speech fragments; inputting the combined features into a deep residual network to obtain speech feature information of the first speech information; The device according to claim 8,

11. The second acquisition module specifically includes: acquiring a speech time length, a speech rate, and a text length of the first speech information; inputting the speech time length, the speaking rate and the text length into a pre-trained multi-layer perceptron MLP model to obtain temporal feature information of the first speech information; The device according to claim 8,

12. The third determination module: a multimodal processing unit that inputs the semantic feature information, the speech feature information, and the temporal feature information into a multimodal fusion model; a decision unit for deciding whether the user has finished inputting speech according to an output result of the multimodal fusion model; The apparatus of claim 8 , comprising:

13. a first processing module for determining first reply voice information corresponding to the first voice information and outputting the first reply voice information when the user determines to terminate the voice input; The apparatus of any one of claims 8 to 12, further comprising:

14. a third acquisition module for acquiring second voice information input by the user again when it is determined that the user has not finished the voice input; a second processing module for determining corresponding second reply voice information based on the first voice information and the second voice information, and outputting the second reply voice information; The apparatus of any one of claims 8 to 12, further comprising:

15. a memory and a processor, The memory stores computer instructions that, when executed by the processor, implement the multimodal feature-based speech interaction processing method of claim 1 .

16. A non-transitory computer-readable recording medium storing computer instructions for causing a computer to execute the multimodal feature-based speech dialogue processing method described in claim 1.

17. 1. A computer program product comprising: A computer program product which, when executed by an instruction processor of the computer program product, implements the multimodal feature based speech interaction processing method of claim 1.

Images