Voice processing method and device, electronic equipment and storage medium

Abstract

The invention provides a voice processing method and device, electronic equipment and a storage medium, and the method is applied to the field of voice processing, and comprises the steps: receiving a first voice signal, obtaining first text content corresponding to the first voice signal, obtaining multi-mode input data if the first text content has object missing, and sending the multi-mode input data to a server; the multi-mode input data is data collected through at least one input mode, determining object content missing from the first text content based on the multi-mode input data, determining second text content based on the object content and the first text content, and executing an instruction corresponding to the second text content. According to the method, the missing content of the first voice signal can be supplemented without repeatedly acquiring the voice, so that the accuracy of voice interaction is improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing, and more specifically, to a speech processing method, device, electronic device and storage medium in the field of speech processing. Background Art

[0002] With the development of information technology, users can use electronic devices for various activities. Users can use voice assistants installed in electronic devices to make calls, order music, tell time, check weather, etc. During use, due to reasons such as voice collection equipment or environment, the voice information input by the user is sometimes missing, resulting in the electronic device being unable to perform the corresponding control operation.

[0003] In the related technical scheme, a prompt message can be issued to prompt the user to re-enter the voice information, so as to obtain the complete voice information, and then control the electronic device to perform the corresponding control operation according to the complete voice information. By controlling the electronic device in this way, there is a situation where the voice information is repeatedly obtained. Summary of the invention

[0004] The present application provides a speech processing method, device, electronic device and storage medium. The method can supplement the missing content of the first speech signal without repeatedly acquiring the speech, thereby improving the accuracy of speech interaction.

[0005] In a first aspect, a speech processing method is provided, the method comprising: receiving a first speech signal, obtaining a first text content corresponding to the first speech signal; if an object is missing in the first text content, obtaining multi-mode input data, the multi-mode input data being data collected through at least one input method; determining the object content missing from the first text content based on the multi-mode input data; determining a second text content based on the object content and the first text content, and executing instructions corresponding to the second text content.

[0006] By adopting the above scheme, when it is determined that the first text content corresponding to the received first voice signal has a missing object, it is possible to obtain multi-mode input data collected through different input methods, and then determine the missing object content of the first text content through the multi-mode input data, and finally determine the second text content through the object content and the first text content, and execute the instruction corresponding to the second text content. Since the missing object content of the first text content corresponding to the first voice signal is determined using the multi-mode input data collected through different input methods, the first text content is supplemented to obtain the second text content, and the instruction corresponding to the second text content is executed, there is no need to repeatedly obtain the first voice signal, and the corresponding instruction can be executed, which can improve the convenience of the voice interaction process.

[0007] In combination with the first aspect, in some possible implementations, the multi-mode input data includes image data, touch data, and a second voice signal adjacent to the receiving order of the first voice signal, and determining the object content missing from the first text content based on the multi-mode input data includes: converting the second voice signal into a third text content, identifying parameter items contained in the third text content, and determining the parameter items to be the object content missing from the first text content; and / or, determining a gaze area of ​​a person who sends the first voice signal based on the image data, identifying object information in the gaze area, and determining the object information to be the object content missing from the first text content; and / or, determining an application triggered by the person touching the screen based on the touch data, determining the program parameters of the application, and determining the object content missing from the first text content based on the program parameters.

[0008] By adopting the above scheme, the object content missing from the first voice signal is determined based on the multi-mode input data obtained through different input methods, thereby improving the richness of the object content missing from the first voice signal, and determining the object content missing from the first content through the multi-mode input data. The object content is associated with the first voice signal, which can improve the accuracy of supplementing the first text content through the multi-mode input data to a certain extent.

[0009] In combination with the first aspect, in some possible implementations, the second text content is determined based on the object content and the first text content, and the instructions corresponding to the second text content are executed, including: if there are multiple second text contents, then it is determined whether the step of determining the missing object content of the first text content based on the multi-modal input data is executed for the first time; based on the judgment result, a fourth text content is determined among the multiple second text contents, and the instructions corresponding to the fourth text content are executed.

[0010] In combination with the first aspect, in some possible implementations, the fourth text content is determined among multiple second text contents based on the judgment result, including: if the judgment result is the first execution, then the second text content obtained by taking the parameter item as the object content is determined to be the fourth text content; if the judgment result is not the first execution, then the input method corresponding to the object content missing from the first voice signal in the previous execution instruction is determined, and the second text content obtained by the input method is determined to be the fourth text content.

[0011] By adopting the above scheme, after obtaining the second text content that supplements the first voice signal, when there are multiple second text contents, the fourth text content is determined in the multiple second text contents based on whether the step of determining the object content missing from the first voice signal based on the multi-modal input data is the judgment result of the first execution, and the fourth text content can be accurately determined in the second text content.

[0012] In combination with the first aspect, in some possible implementations, a fourth text content is determined among multiple second text contents based on a judgment result, and an instruction corresponding to the fourth text content is executed, including: converting the fourth text content into a third voice signal and broadcasting the third voice signal; if an execution instruction triggered by a character based on the third voice signal is received, executing the instruction corresponding to the third voice signal; if a non-execution instruction triggered by a character based on the third voice signal is received, re-determining the fourth text content among multiple second text contents, converting the fourth text content into a third voice signal, and broadcasting the third voice signal until a preset condition is met, the preset condition being that an execution instruction triggered by the third voice signal is received, the instruction corresponding to the third voice signal is executed, or the fourth text content is updated and the execution instruction triggered by the third voice signal converted based on the fourth text content is not received.

[0013] By adopting the above scheme, after the third voice signal is broadcast, it is determined whether to execute the third voice signal based on the received execution instruction triggered by the third voice signal, or the fourth text content converted into the third voice signal is re-determined to obtain a new third voice signal. The third voice signal can be updated according to the received instruction, thereby improving the convenience of updating the third voice signal.

[0014] In combination with the first aspect, in some possible implementations, the second text content is determined based on the object content and the first text content, and instructions corresponding to the second text content are executed, including: inputting the second text content into a semantic model; if the output result of the semantic model is that the second text content is semantically complete, executing instructions corresponding to the second text content.

[0015] By adopting the above scheme, the second text content is input into the semantic model, and then when the output of the semantic model shows that the second text content is semantically complete, the instructions corresponding to the second text content are executed, thereby avoiding the input of semantically incomplete second text content, which results in the inability to execute the corresponding instructions.

[0016] In combination with the first aspect, in some possible implementations, after obtaining the first text content corresponding to the first voice signal, it also includes: dividing the words included in the first text content; and determining whether there is an object missing in the first text content based on the grammatical relationship between the words.

[0017] By adopting the above scheme, by dividing the words contained in the first text content and determining the grammatical relationship between the words, it is possible to determine whether there is a missing object in the first text content, which provides a basis for judging whether to subsequently supplement the first text content based on multi-modal input data.

[0018] In a second aspect, a speech processing device is provided, the device comprising:

[0019] A receiving unit, configured to receive a first voice signal and obtain a first text content corresponding to the first voice signal;

[0020] an acquisition unit, configured to acquire multi-modal input data if the first text content has an object missing, where the multi-modal input data is data collected through at least one input method;

[0021] a determination unit, configured to determine object content missing from the first text content based on the multimodal input data;

[0022] The execution unit is used to determine the second text content based on the object content and the first text content, and execute the instruction corresponding to the second text content.

[0023] In a third aspect, an electronic device is provided, comprising: a memory for storing executable program code; and a processor for calling and running the executable program code from the memory, so that the electronic device executes the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0024] In a fourth aspect, a computer program product is provided, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect.

[0025] In a fifth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on a computer, the computer executes the method in the above-mentioned first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a scenario diagram of a speech processing method provided in an embodiment of the present application;

[0027] Figure 2 It is a flowchart of a speech processing method provided in an embodiment of the present application;

[0028] Figure 3 It is a flowchart of a speech processing method provided in an embodiment of the present application;

[0029] Figure 4 It is a flowchart of a speech processing method provided in an embodiment of the present application;

[0030] Figure 5 It is a flowchart of a speech processing method provided in an embodiment of the present application;

[0031] Figure 6 is a structural schematic diagram of a speech processing device provided in an embodiment of the present application;

[0032] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The technical solution in the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0034] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0035] Figure 1 It is a scenario diagram of a speech processing method provided in an embodiment of the present application.

[0036] In an embodiment of the present application, the electronic device may be a control center of a vehicle. During the driving of the vehicle, the control center of the vehicle collects a first voice signal of a person inside the vehicle through a microphone installed inside the vehicle. The control center receives the first voice signal sent by the microphone, converts the first voice signal into text content, and performs grammatical analysis on the text content, for example, whether an execution object is missing, whether an execution action is missing, etc., to determine whether the grammatical structure of the text content is complete. If the grammatical structure of the text content is incomplete, it is determined that the first voice signal is missing an object. According to the traditional processing method, a prompt message is issued to prompt the user to input the first voice signal again, and then execute the instruction corresponding to the re-received first voice signal. It can be seen from this that in the related technical solution, there is a problem that the first voice signal needs to be repeatedly obtained in the received first voice signal, thereby reducing the user's experience.

[0037] In response to the above problems, the present application proposes a speech processing method, which can obtain multi-mode input data collected through different input methods when it is determined that the first text content corresponding to the received first voice signal has a missing object, and then determine the missing object content of the first text content through the multi-mode input data, and finally determine the second text content through the object content and the first text content, and execute the instruction corresponding to the second text content. Since the missing object content of the first text content corresponding to the first voice signal is determined using multi-mode input data collected through different input methods, the first text content is supplemented to obtain the second text content, and the instruction corresponding to the second text content is executed, there is no need to repeatedly obtain the first voice signal, and the corresponding instruction can be executed, which can improve the convenience of the voice interaction process.

[0038] based on Figure 1 The scene shown is shown below. Figure 2-Figure 7 , the speech processing method provided in the embodiment of the present application is introduced in detail.

[0039] See also Figure 2 , which is a flow chart of a speech processing method provided in an embodiment of the present application. Figure 2 As shown, the method of the embodiment of the present application may include the following steps S101-S104.

[0040] S101, receiving a first voice signal, and acquiring first text content corresponding to the first voice signal;

[0041] Specifically, the first voice signal may be a signal for controlling the execution of a specific instruction, and the first voice signal may be collected based on a microphone. The microphone may be integrated in an electronic device, or connected to the electronic device by wire or wirelessly, and then the collected first voice signal is sent to the electronic device. In an embodiment of the present application, the electronic device may be a terminal device available to a user, such as a personal computer, a laptop computer, a tablet computer, a smart phone, or a control center for controlling the driving of a vehicle. Optionally, in an embodiment of the present application, the electronic device is described by taking the control center of a vehicle as an example.

[0042] It is understandable that when the electronic device is the control center of the vehicle, microphones can be installed in multiple locations of the vehicle, so as to collect not only the first voice signal emitted by the driver, but also the first voice signal emitted by the passengers. These microphones can be placed near the driver's seat to preferentially capture the driver's voice signal; at the same time, additional microphones can be installed in the co-pilot seat, the back seat and other suitable locations in the car to ensure that the first voice signal emitted by the passengers can be clearly collected, and then the first voice signal can be converted into the first text content. Specifically, the first voice signal can be converted into the first text content by automatic speech recognition technology (Automatic Speech Recognition, ASR). For example, the first text content corresponding to the first voice signal is determined to be "open" by automatic speech recognition technology.

[0043] Optionally, when the system receives voice signals from different positions, it can distinguish and identify the voice signals of each passenger by sound source localization, voice separation, etc. Optionally, in the embodiment of the present application, receiving the first voice signal emitted by the driver (person) is taken as an example for explanation.

[0044] S102, if the first text content has an object missing, obtaining multi-modal input data, where the multi-modal input data is data collected through at least one input method;

[0045] Specifically, after converting the first voice signal into the first text content, the first text content is subjected to grammatical analysis, and the subject, predicate (execution action), and object (execution object) in the first text content are parsed to determine whether the text content lacks an object. For example, if the first text content does not have an object in the parsed result of the first text content, it is determined that the first text content is missing an object, and then the first text content needs to be supplemented based on multi-mode input data other than the first voice signal, and the multi-mode input data is data collected through at least one input method, wherein the input method includes a camera, a microphone, a touch screen, etc. It can be understood that the installation position of the camera and the touch screen is similar to the installation position of the microphone mentioned above, and will not be repeated here. In an embodiment of the present application, if multi-mode input data is obtained through a microphone, a voice signal whose collection time is adjacent to the collection time of the first voice signal is determined to be the multi-mode input data; if multi-mode input data is obtained through a shooting device, image data whose collection time is consistent with the collection time of the first voice signal is determined to be the multi-mode input data; similarly, touch data whose collection time of the touch screen is consistent with the collection time of the first voice signal is determined to be the multi-mode input data, and the multi-mode input data thus obtained corresponds to the first voice signal, thereby ensuring the accuracy of supplementing the missing content of the first voice signal.

[0046] S103, determining the object content missing from the first text content based on the multimodal input data;

[0047] In one embodiment, after receiving the multi-mode input data, the object content missing from the first text content is determined from the multi-mode input data, and the object content may be the content required to complete the first voice signal.

[0048] Exemplarily, the multi-mode input data may be image data of a person who sends the first voice signal captured by a camera, and then the image data may be used to determine the device currently viewed by the person who sends the first voice signal, and then the object content may be determined. For example, the image data may be used to determine that the device currently viewed by the user is a skylight, and then the object content may be determined to be "skylight".

[0049] S104, determining a second text content based on the object content and the first text content, and executing an instruction corresponding to the second text content.

[0050] In one embodiment, the object content obtained based on the multi-modal input data is data in text form, and the first text content corresponding to the first voice signal is supplemented by the object content in text form to obtain the supplemented second text content. Exemplarily, the first text content obtained after converting the collected first voice signal into text content may be "open", and the text content is grammatically analyzed to determine that it is a predicate, and there is no object in the first text content, so it is determined that there is an object missing in the first text content. The image data corresponding to the first voice signal determines that the user is viewing the sunroof, and the sunroof is determined to be the object content of the first text content, and then the object content in text form (skylight) and the first text content (open) are combined, as well as the grammatical structure with the subject in front and the object in the back, to obtain the second text content "open the sunroof", and the control center executes the instruction corresponding to the second text content, that is, executes the action of opening the sunroof.

[0051] In an embodiment of the present application, when it is determined that the first text content corresponding to the input first voice signal has an object missing, the object content missing from the first text content is determined through the multi-mode input data, and then the second text content is determined according to the object content and the first text content, and the instruction corresponding to the second text content is executed. Since the object content determined by the multi-mode input data collected through different input methods is used to supplement the first text content to obtain the second text content, there is no need to repeatedly obtain the first voice signal, and the corresponding instruction can be executed, which improves the convenience of the voice interaction process and the accuracy of the voice interaction.

[0052] See also Figure 3 , which is a flow chart of a speech processing method provided in an embodiment of the present application. Figure 3As shown, the method of the embodiment of the present application may include the following steps S201-S208.

[0053] S201, receiving a first voice signal, and acquiring first text content corresponding to the first voice signal;

[0054] Specifically, the specific process of receiving the first voice signal and converting it into the first text content can be found in the description of step S101 in the above embodiment of the specification, which will not be described in detail here.

[0055] S202, if the first text content has an object missing, obtaining multi-modal input data, where the multi-modal input data is data collected through at least one input method;

[0056] In one embodiment, the multi-mode input data includes image data, touch data, and a second voice signal that is adjacent to the first voice signal in a receiving order.

[0057] The image data is the data obtained by shooting with a shooting device; the touch data is the data sensed by a capacitive sensor or a resistive sensor on the touch screen, which is generated by a person's click, long press, and other operations on the touch screen, including the coordinate information of the touch position of the person on the touch screen, the touch strength, and the like; the second voice signal is the data obtained by collecting the surrounding sounds through a microphone. In an embodiment of the present application, the second voice signal is a voice signal adjacent to the receiving order of the first voice signal, that is, the second voice signal can be the previous voice signal of the first voice signal, or the next voice signal of the first voice signal. It can be understood that the second voice signal is a second voice signal collected by the person in the process of sending the first voice signal, and it is a voice signal obtained by naturally and fluently completing the speaking process without any prompt or request for repetition about the missing object of the first voice signal.

[0058] S203, converting the second voice signal into a third text content, identifying a parameter item included in the third text content, and determining that the parameter item is an object content missing from the first text content;

[0059] In one embodiment, after receiving a second voice signal that is adjacent to the first voice signal in a receiving order, the second voice signal is converted into a third text content using an automatic speech recognition technology.

[0060] Further, the parameter item contained in the third text content is identified by natural language understanding (NLU). In the embodiment of the present application, the parameter item refers to the noun in the third text content identified by natural speech understanding. For example, in the embodiment of the present application, the second voice information is converted into the third text content "the left front window is closed", and the parameter item of the third text content identified by natural speech understanding is "the left front window".

[0061] S204, determining a gaze area of ​​the person who sends the first voice signal based on the image data, identifying object information in the gaze area, and determining that the object information is object content that is missing from the first text content;

[0062] In one embodiment, after obtaining image data through a shooting device, the gaze area of ​​the person of the first voice signal in the image data can be determined by eye tracking technology, and then the gaze point coordinates in the gaze area are calculated and matched with the object position to determine the object information that the person is gazing at, and determine that the object information is the object content missing from the first voice signal. The person refers to the person who sends the first voice signal.

[0063] Optionally, when the person is a driver, the camera device can be installed on the A-pillar of the vehicle, above the dashboard, or on the interior panel on the driver's side. These installation locations can provide a good viewing angle to capture the driver's facial image without excessively interfering with the driver's sight or operation. The "A-pillar" is part of the vehicle body structure, located on both sides of the front windshield, and is used to support the roof and the front windshield. Of course, the camera device can also be installed in other locations, which are not listed here one by one.

[0064] S205, determining an application triggered by the person touching the screen based on the touch data, determining program parameters of the application, and determining object content missing from the first text content based on the program parameters;

[0065] In one embodiment, when receiving touch data sent by the touch screen, the application triggered by the person on the touch screen is determined by the touch data, and then the program parameters of the application are determined, and the program parameters are determined as the object content missing from the first voice signal, and the program parameters are parameters for controlling the application to perform corresponding operations. In the embodiment of the present application, different application programs have different program parameters.

[0066] S206, determining the second text content based on the object content and the first text content;

[0067] Specifically, after obtaining the object content, the object content and the first text content are combined to obtain the second text content. It can be understood that the object content is a noun, and according to the grammatical structure, it is located after the verb, and the second text content can be "first text content + object content".

[0068] S207, if there are multiple second text contents, determining whether the step of determining the missing object content of the first text content based on the multi-modal input data is performed for the first time;

[0069] S208: Determine a fourth text content from the plurality of second text contents based on the determination result, and execute an instruction corresponding to the fourth text content.

[0070] In one embodiment, in the embodiment of the present application, if it is determined that there is only one second text content, the instruction corresponding to the second text content is executed. If it is determined that there are multiple second text contents, it is determined whether the current step of determining the missing object content of the first text content based on the multi-modal input data is executed for the first time, and then the fourth text content is determined from the multiple second text contents based on the judgment result, and the instruction corresponding to the fourth text content is executed.

[0071] Further, in an embodiment of the present application, if the judgment result is the first execution, the second text content obtained by using the parameter item as the object content is determined to be the fourth text content, that is, the second text content after the first text content is supplemented by the object content obtained by the second voice signal is determined to be the fourth text content. It can be understood that when the first execution is based on the multi-mode input data to determine the missing object content of the first text content corresponding to the first voice signal, the second text content obtained by supplementing the first text content by using the parameter item of the third text content corresponding to the second voice signal adjacent to the first voice signal in the order of reception is the fourth text content, and the parameter item is obtained by the second voice signal adjacent to the order of reception of the first voice signal, and there is a certain correlation between adjacent voice signals, thereby improving the accuracy of supplementing the first voice signal in the first execution.

[0072] If the judgment result is that it is not the first execution, then the input method of the object content missing from the first voice signal in the preceding instruction is determined, and the second text content obtained by the input method is determined to be the fourth text content. Exemplarily, if the input method of the object content in the preceding instruction is a shooting device, then the second text content obtained by the shooting device is determined to be the fourth text content in the second text content. It can be understood that the daily actions of the characters are habitual. After determining that the preceding instruction is executed, it can be determined that the input method of the object content according to the preceding instruction meets the control requirements of the characters. Then, the first text content can continue to be supplemented according to the input method in the subsequent execution, thereby improving the accuracy of supplementing the first text content.

[0073] It is understandable that in the embodiment of the present application, the multi-mode input data may include any one or more of image data, touch data, and second voice data. The following will be described by taking the multi-mode data including image data, touch data, and second voice data as an example.

[0074] First, the vehicle's control center obtains the program parameters of the application pre-set by the driver. For example, the pre-set program parameters for the "Navigation APP" are the departure point when the navigation was last exited; the pre-set program parameters for the "Music APP" are the "Heart-beating Song List." For navigation applications, the driver sets the program parameters to the departure point when the navigation was last exited. After arriving at the destination, the vehicle can automatically navigate to the departure point based on the program parameters without manual input, thereby improving the intelligence of using the vehicle. Optionally, the driver can also set the program parameters of navigation applications to addresses that he frequently visits, such as a home address or an office address.

[0075] After the vehicle control center collects the first voice signal through the microphone, it determines that the first text content corresponding to the first voice signal is "open". It obtains the second voice signal adjacent to the first voice signal in the order of reception, and determines that the third text content corresponding to the second voice signal is "the left front window is closed", and determines that the parameter item of the first text content is "the left front window", that is, it determines that "the left front window" is the object content missing from the first text content, and then it can be determined that the second text content obtained by supplementing the first text content of the first voice signal with the object content obtained by the second voice signal is "open the left front window".

[0076] The vehicle's control center determines based on the image data that the object information in the current gaze area of ​​the person who sends the first voice signal is "skylight", then determines that the missing object content is "skylight", and further determines that the second text content obtained by supplementing the first text content of the first voice signal through the image data is "open the skylight".

[0077] When a person clicks the touch screen, the coordinate information contained in the touch data obtained by the control center of the vehicle through the capacitive sensor is (10, 12), and then the coordinates of the application icons of all applications on the touch screen are traversed to find the coordinates that match or are closest to the coordinate information, and the application icon corresponding to the coordinate is determined to be the application icon of the "Navigation APP", and then the application of the application icon is determined to be the "Navigation APP". If the navigation data of the last use of the "Navigation APP" is "from the current location (Beijing) to Tianjin", it can be determined that the program parameter of the "Navigation APP" is "Beijing", that is, the object content missing from the first text content is "Beijing". Then, it is determined that the second text content obtained by supplementing the first text content of the first voice signal with image data is "Open Beijing".

[0078] Furthermore, in an embodiment of the present application, after obtaining the second text content, it is further determined whether the semantics of the second text content are complete. If not, the second text content is discarded; if complete, the instruction corresponding to the second text content is executed.

[0079] Specifically, the second text content can be input into the semantic model, and whether the semantics of the second text content is complete can be determined based on the input result of the semantic model. The semantic model can be a bag of words model, a language representation model (Bidirectional Encoder Representations from Transformers, BERT model), etc. For the second text content "open the car window", after it is input into the semantic model, it is determined that "car window" as an object can be opened, then the second text content "open the car window" is determined to be semantically complete, and the output result is that the second text content is semantically complete; similarly, for the second text content "open the sunroof", its semantics is determined to be complete, and the output result is that the second text content is semantically complete; for the second text content "open Beijing", "Beijing" as a place name or city name is not an entity that can be "opened". From a semantic point of view, "open Beijing" does not express a complete and meaningful action or state, and lacks the necessary semantic components to form a reasonable sentence, so it is determined that the second text content "open Beijing" is semantically incomplete. In an embodiment of the present application, by inputting the second text content into the semantic model, and then executing the instructions corresponding to the second text content when the output result of the semantic model is that the second text content is semantically complete, it is avoided that the input of the semantically incomplete second text content causes the corresponding instructions to fail to execute.

[0080] Further, after obtaining multiple second text contents, if it is determined that the first execution is based on the multi-mode input data to determine the missing object content of the first text content, then the second text content "open the left front window" obtained based on the second voice signal is determined to be the fourth text content, and its semantics are determined to be complete, and the control center controls to open the left front window. If it is determined that it is not the first execution based on the multi-mode input data to determine the missing object content of the first text content, and the input method corresponding to the missing object content of the first voice signal in the execution of the previous instruction is obtained through the shooting device, then the second text content "open the sunroof" is determined to be the fourth text content, and its semantics are determined to be complete, and the control center controls to open the sunroof. If it is determined that it is not the first execution based on the multi-mode input data to determine the missing object content of the first text content, and the input method corresponding to the missing object content of the first voice signal in the execution of the previous instruction is obtained by touching the screen, then the second text content "open Beijing" is determined to be the fourth text content, but the semantics of the second text content are incomplete, and the second text content is discarded.

[0081] In an embodiment of the present application, the object content missing from the first voice signal is determined based on multi-mode input data obtained through different input methods, thereby improving the richness of the object content missing from the first voice signal, ensuring that the object content of the first voice signal is obtained, and to a certain extent, being able to improve the accuracy of the subsequent execution of the third voice signal; and after obtaining the second text content that supplements the first voice signal, when there are multiple second text contents, the fourth text content is determined from the multiple second text contents based on the judgment result of whether the step of determining the object content missing from the first voice signal based on the multi-mode input data is executed for the first time, and the fourth text content can be accurately determined from the second text contents.

[0082] See also Figure 4 , which is a flow chart of a speech processing method provided in an embodiment of the present application. Figure 4 As shown, the method of the embodiment of the present application may include the following steps S301-S303.

[0083] S301, converting the fourth text content into a third voice signal, and broadcasting the third voice signal;

[0084] In one embodiment, after obtaining the fourth text content, the fourth text content is converted into a third voice signal, and the third voice signal can be broadcasted through a speaker. The fourth text content can be converted into the third voice signal by text to speech (TTS).

[0085] S302, if an execution instruction triggered by the character based on the third voice signal is received, executing the instruction corresponding to the third voice signal;

[0086] In one embodiment, after the third voice signal is broadcast, an option of whether to execute the instruction corresponding to the third voice signal may be displayed on the touch screen. If the driver triggers the "yes" option, it is determined that an execution instruction triggered by the third voice signal has been received, then the instruction corresponding to the third voice signal is executed. Exemplarily, the instruction corresponding to the third voice signal may be: opening the sunroof of the vehicle.

[0087] In another embodiment, after the third voice signal is broadcast, the voice signal collected by the microphone can also be used to determine whether to execute the instruction corresponding to the third voice signal. For example, if an affirmative statement such as "execute" or "yes" is received from the driver after the third voice signal is broadcast, the instruction corresponding to the third voice signal is executed.

[0088] S303, if a non-execution instruction triggered by a character based on a third voice signal is received, the fourth text content is re-determined among multiple second text contents, the fourth text content is converted into a third voice signal, and the third voice signal is broadcast until a preset condition is met. The preset condition is that an execution instruction triggered by the third voice signal is received, and the instruction corresponding to the third voice signal is executed, or the fourth text content is updated and the execution instruction triggered by the third voice signal converted based on the fourth text content is not received.

[0089] In one embodiment, after the third voice signal is broadcast, if a non-execution instruction triggered by the third voice signal is received, the fourth text content is redetermined among multiple second text contents, and then the fourth text content is converted into a third voice signal, and the updated third voice signal is broadcast until a preset condition is met, and then the step of redetermining the fourth text content among the second text contents is exited.

[0090] Exemplarily, when the second text content obtained to supplement the first voice signal includes "open the left front window", "open the sunroof", and "open Beijing", if it is the first execution based on the multi-modal input data to determine the missing object content of the first text content, then "open the left front window" is determined to be the fourth text content, and its semantics are determined to be complete, and then it is converted into a third voice signal. When a non-execution instruction is received after the third voice signal is broadcast, "open Beijing" is used as the fourth text content, and its semantics are determined to be incomplete and discarded; then the second text content "open the sunroof" is updated to the fourth text content, and an execution is performed to convert it into a third voice signal, and the signal of the third voice signal is broadcast. If an execution instruction is received, it is determined that the preset conditions are met, and the control center controls the execution of opening the sunroof.

[0091] Furthermore, in an embodiment of the present application, after obtaining the first text content corresponding to the first voice signal, it is also possible to divide the words contained in the first text content and determine whether there is an object missing in the first text content based on the grammatical relationship between the words.

[0092] Exemplarily, after the first text content is divided, it is input into a syntactic analyzer of natural language processing (NLP) technology (such as Stanford Parser, Spacy, etc.) to construct grammatical relations between words, and determine that there is a lack of noun after "open", that is, it is determined that the first text content lacks an object.

[0093] In an embodiment of the present application, after broadcasting the third voice signal, it is determined whether to execute the corresponding instruction of the fourth text content according to the received execution instruction triggered by the third voice signal, or the fourth text content converted into the third voice signal is re-determined to obtain a new third voice signal. The third voice signal can be updated according to the received instruction, thereby improving the convenience of updating the third voice signal; and by dividing the words included in the first text content and determining the grammatical relationship between the words, it is possible to determine whether there is a missing object in the first text content, thereby providing a basis for judging whether to subsequently supplement the first text content based on multi-modal input data.

[0094] The following will be combined Figure 5 The speech processing method provided in the embodiment of the present application is described in detail with reference to a specific example.

[0095] S1, receiving a first voice signal; S2, whether the first voice signal has an object missing; S3, no, executing the instruction corresponding to the first voice signal; S4, yes, recalling the first voice signal; S5, acquiring touch data, image data, and third voice data; S6, determining the object content missing from the first voice signal based on the touch data, image data, and third voice data; S7, obtaining a second text content that supplements the first text content corresponding to the first voice signal based on the object content and the first voice signal; S8, at least one second text content exists; S9, no, determining that the second text content does not exist, the processing fails, and the first voice signal cannot be completed; S10, yes, determining the second There is one text content; S11, determine whether the semantics of the second text content is complete; S12, yes, convert the second text content into a third voice signal; S12, determine the fourth text content from multiple second text contents; S13, whether the semantics of the fourth text content is complete; S14, yes, convert the fourth text content into a third voice signal and broadcast the third voice signal; S15, receive an execution instruction based on the third voice signal, then execute the instruction corresponding to the fourth text content; S16, receive a non-execution instruction based on the third voice signal, then update the fourth text content; S17, if it is determined that the semantics of the second text content is incomplete, it is determined that the processing has failed and the first voice signal cannot be completed.

[0096] Optionally, when it is determined that there are multiple second text contents, the scores of the input methods (any two of image data, touch data, and second voice signals) of the second text contents in the scoring system can also be obtained, and the second text content obtained by the input method with the highest score is determined to be the fourth text content. After the fourth text content is converted into a third voice signal, if a non-execution instruction for the third voice signal is received, the score of the input method is reduced according to the preset score; if an execution instruction for the third voice signal is received, the score of the input method is maintained unchanged. When there are multiple second text contents, by determining that the second text content obtained by the input method with the highest current score is the fourth text content, the fourth text content can be determined based on the actual scoring of the input method, which improves the accuracy of the determined fourth text content, thereby improving the accuracy of the supplement to the first voice signal. In addition, no extra points are given to the input method corresponding to the execution instruction, and points are deducted from the input method corresponding to the non-execution instruction, which can avoid the problem of multiple executions of a certain input method, thereby causing the second text content obtained by other input methods to be unused.

[0097] In an embodiment of the present application, when it is determined that the first text content corresponding to the input first voice signal has an object missing, the object content missing from the first text content is determined through the multi-mode input data, and then the second text content is determined according to the object content and the first text content, and the instruction corresponding to the second text content is executed. Since the object content determined by the multi-mode input data collected through different input methods is used to supplement the first text content to obtain the second text content, there is no need to repeatedly obtain the first voice signal, and the corresponding instruction can be executed, which improves the convenience of the voice interaction process and the accuracy of the voice interaction.

[0098] based on Figure 1 The following is a schematic diagram of the scene. Figure 6 , the speech processing device provided in the embodiment of the present application is introduced in detail. It should be noted that, Figure 6 The speech processing device in the present application is used to execute Figure 2-Figure 5 For the convenience of explanation, only the part related to the embodiment of the present application is shown. For the specific technical details not disclosed, please refer to the present application. Figure 2-Figure 5 Specifically, the speech processing device 1 includes: a receiving unit 11 , an acquiring unit 12 , a determining unit 13 , and an executing unit 14 .

[0099] The receiving unit 11 is used to receive a first voice signal and obtain a first text content corresponding to the first voice signal;

[0100] An acquisition unit 12 is configured to acquire multi-modal input data if the first text content has an object missing, where the multi-modal input data is data collected through at least one input method;

[0101] A determination unit 13, configured to determine the object content missing from the first text content based on the multimodal input data;

[0102] The execution unit 14 is used to determine the second text content based on the object content and the first text content, and execute the instruction corresponding to the second text content.

[0103] Optionally, the determination unit 13 includes: an identification subunit 131 , a first determination subunit 132 , and a second determination subunit 133 .

[0104] The recognition subunit 131 is used to convert the second voice signal into a third text content, recognize a parameter item included in the third text content, and determine that the parameter item is an object content missing from the first text content;

[0105] A first determining subunit 132, configured to determine a gaze area of ​​a person who sends a first voice signal based on the image data, identify object information in the gaze area, and determine that the object information is an object content missing from the first text content;

[0106] The second determining subunit 133 is used to determine the application triggered by the character touching the screen based on the touch data, determine the program parameters of the application, and determine the object content missing from the first text content based on the program parameters.

[0107] Optionally, the execution unit 14 includes: a first execution subunit 141 and a second execution subunit 142 .

[0108] The first execution subunit 141 is used to determine whether the step of determining the missing object content of the first text content based on the multi-modal input data is executed for the first time if there are multiple second text contents;

[0109] The second execution subunit 142 is used to determine the fourth text content from the plurality of second text contents based on the determination result, and execute the instruction corresponding to the fourth text content.

[0110] Optionally, the second execution subunit 142 is specifically configured to:

[0111] If the judgment result is that it is the first execution, determining the second text content obtained by taking the parameter item as the object content as the fourth text content;

[0112] If the judgment result is that it is not the first execution, the input mode corresponding to the object content missing from the first voice signal in the previous instruction is determined, and the second text content obtained by the input mode is determined to be the fourth text content.

[0113] Optionally, the second execution subunit 142 is specifically configured to: convert the fourth text content into a third voice signal, and broadcast the third voice signal;

[0114] If an execution instruction triggered by the character based on the third voice signal is received, the instruction corresponding to the third voice signal is executed;

[0115] If a non-execution instruction triggered by a third voice signal is received by a character, the fourth text content is re-determined among multiple second text contents, the fourth text content is converted into a third voice signal, and the third voice signal is broadcast until a preset condition is met. The preset condition is that an execution instruction triggered by the third voice signal is received, and the instruction corresponding to the third voice signal is executed, or the fourth text content is updated and completed, and no execution instruction triggered by the third voice signal converted from the fourth text content is received.

[0116] Optionally, the execution unit 14 further includes: a third execution subunit 143 and a parsing subunit 144 .

[0117] The sixth execution subunit 143 is used to input the second text content into the semantic model;

[0118] The parsing subunit 144 is configured to execute instructions corresponding to the second text content if the output result of the semantic model is that the second text content is semantically complete.

[0119] Optionally, the receiving unit 11 further includes: a converting subunit 111 and a determining subunit 112 .

[0120] A division subunit 111, used to divide the words contained in the first text content;

[0121] The determination subunit 112 is used to determine whether the first text content has object missing based on the grammatical relationship between words.

[0122] See also Figure 7 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 500 includes a processor 501 and a memory 502. The processor 501 is electrically connected to the memory 502.

[0123] The processor 501 is the control center of the electronic device 500 and may include one or more processing cores. The processor 501 uses various interfaces and lines to connect various parts of the entire electronic device 500, and executes various functions and processes data of the electronic device 500 by running or calling computer programs stored in the memory 502, and calling data stored in the memory 502, so as to control the electronic device 500 as a whole. Optionally, the processor 501 can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 501 can integrate one or a combination of CPU, graphics processing unit (GPU), modem, etc. Among them, the CPU mainly processes the operating system, user pages, and applications; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 501, but may be implemented separately through a communication chip.

[0124] The memory 502 can be used to store software programs and modules, and the processor 501 executes various functional applications and data processing by running the computer programs and modules stored in the memory 502. The memory 502 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, a computer program required for at least one function, etc.; the data storage area can store data created according to the use of the electronic device 500, etc.

[0125] In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0126] In this embodiment, the processor 501 in the electronic device 500 loads instructions corresponding to the processes of one or more computer programs into the memory 502 according to the following steps, and the processor 501 runs the computer program stored in the memory 502 to implement various functions, as follows:

[0127] Receive a first voice signal, and obtain first text content corresponding to the first voice signal;

[0128] If the first text content has an object missing, then obtaining multi-modal input data, where the multi-modal input data is data collected through at least one input method;

[0129] Determining object content missing from the first text content based on the multimodal input data;

[0130] The second text content is determined based on the object content and the first text content, and an instruction corresponding to the second text content is executed.

[0131] Optionally, the multi-mode input data includes image data, touch data, and a second voice signal that is adjacent to the first voice signal in a receiving order. When the processor 501 determines the object content missing from the first text content based on the multi-mode input data, the processor 501 specifically performs:

[0132] Converting the second speech signal into a third text content, identifying a parameter item contained in the third text content, and determining that the parameter item is an object content missing from the first text content; and / or,

[0133] Determine the gaze area of ​​the person who sends the first voice signal based on the image data, identify object information in the gaze area, and determine that the object information is the object content missing from the first text content; and / or,

[0134] An application program triggered by the person touching the screen is determined based on the touch data, program parameters of the application program are determined, and object content missing from the first text content is determined based on the program parameters.

[0135] Optionally, when the processor 501 determines the second text content based on the object content and the first text content and executes an instruction corresponding to the second text content, the processor 501 specifically performs:

[0136] If there are multiple second text contents, determining whether the step of determining the object content missing from the first text content based on the multi-modal input data is performed for the first time;

[0137] Based on the determination result, a fourth text content is determined from the plurality of second text contents, and an instruction corresponding to the fourth text content is executed.

[0138] Optionally, when the processor 501 determines the fourth text content from the plurality of second text contents based on the judgment result, the processor 501 specifically performs:

[0139] If the judgment result is that it is the first execution, determining the second text content obtained by taking the parameter item as the object content as the fourth text content;

[0140] If the judgment result is that it is not the first execution, the input mode corresponding to the object content missing from the first voice signal in the previous instruction is determined, and the second text content obtained by the input mode is determined to be the fourth text content.

[0141] Optionally, when executing the instruction corresponding to the fourth text content, the processor 501 specifically performs:

[0142] converting the fourth text content into a third voice signal, and broadcasting the third voice signal;

[0143] If an execution instruction triggered by the character based on the third voice signal is received, the instruction corresponding to the third voice signal is executed;

[0144] If a non-execution instruction triggered by a third voice signal is received by a character, the fourth text content is re-determined among multiple second text contents, the fourth text content is converted into a third voice signal, and the third voice signal is broadcast until a preset condition is met. The preset condition is that an execution instruction triggered by the third voice signal is received, and the instruction corresponding to the third voice signal is executed, or the fourth text content is updated and completed, and no execution instruction triggered by the third voice signal converted from the fourth text content is received.

[0145] Optionally, when the processor 501 determines the second text content based on the object content and the first text content and executes an instruction corresponding to the second text content, the processor 501 specifically performs:

[0146] inputting the second text content into the semantic model;

[0147] If the output result of the semantic model is that the second text content is semantically complete, the instruction corresponding to the second text content is executed.

[0148] Optionally, after executing the step of acquiring the first text content corresponding to the first voice signal, the processor 501 may further execute:

[0149] dividing the words included in the first text content;

[0150] It is determined whether the first text content has object missing based on the grammatical relationship between the words.

[0151] It should be understood that the device provided in the embodiment of the present application is used to execute the above-mentioned speech processing method, and thus can achieve the same effect as the above-mentioned implementation method.

[0152] In the case of an integrated unit, the device may include a processing module and a storage module. When the device is applied to a vehicle, the processing module may be used to control and manage the actions of the vehicle. The storage module may be used to support the vehicle in executing related program codes, etc.

[0153] The processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0154] In addition, the device provided in the embodiment of the present application can specifically be a chip, component or module, and the chip may include a connected processor and memory; wherein the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute a speech processing method provided in the above embodiment.

[0155] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program code is stored. When the computer program code is executed on a computer, the computer executes the above-mentioned related method steps to implement a speech processing method provided in the above embodiment.

[0156] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement a speech processing method provided by the above embodiment.

[0157] Among them, the device, computer-readable storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0158] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0159] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0160] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A speech processing method, characterized in that: The method comprises: Receive a first voice signal, and obtain first text content corresponding to the first voice signal; If the first text content has an object missing, obtaining multi-modal input data, where the multi-modal input data is data collected through at least one input method; Determining object content missing from the first text content based on the multimodal input data; A second text content is determined based on the object content and the first text content, and an instruction corresponding to the second text content is executed.

2. The method according to claim 1, characterized in that The multi-mode input data includes any one or more of image data, touch data, and a second voice signal that is adjacent to the first voice signal in a receiving order, and determining the object content missing from the first text content based on the multi-mode input data includes: converting the second speech signal into a third text content, identifying a parameter item contained in the third text content, and determining that the parameter item is an object content missing from the first text content; and / or, Determine the gaze area of ​​the person who sends the first voice signal based on the image data, identify object information in the gaze area, and determine that the object information is the object content missing from the first text content; and / or, An application program triggered by the character touching the screen is determined based on the touch data, program parameters of the application program are determined, and object content missing from the first text content is determined based on the program parameters.

3. The method according to claim 2, characterized in that The determining the second text content based on the object content and the first text content, and executing the instruction corresponding to the second text content, includes: If there are multiple second text contents, determining whether the step of determining the missing object content of the first text content based on the multi-modal input data is performed for the first time; Based on the determination result, a fourth text content is determined from the plurality of second text contents, and an instruction corresponding to the fourth text content is executed.

4. The method according to claim 3, characterized in that The determining of the fourth text content from the plurality of second text contents based on the judgment result includes: If the judgment result is that it is the first execution, determining that the second text content obtained by taking the parameter item as the object content is the fourth text content; If the judgment result is that it is not the first execution, then determining the input method corresponding to the object content missing from the first voice signal in the previous instruction to execute, and determining the second text content obtained by the input method to be the fourth text content.

5. The method according to claim 3, characterized in that: The executing the instruction corresponding to the fourth text content includes: Converting the fourth text content into a third voice signal, and broadcasting the third voice signal; If an execution instruction triggered by the character based on the third voice signal is received, the instruction corresponding to the third voice signal is executed; If a non-execution instruction is received by the character based on the third voice signal, the fourth text content is re-determined among the multiple second text contents, the fourth text content is converted into a third voice signal, and the third voice signal is broadcast until a preset condition is met, wherein the preset condition is that an execution instruction is received based on the third voice signal, the instruction corresponding to the third voice signal is executed, or the fourth text content is updated and completed, and no execution instruction is received based on the third voice signal converted from the fourth text content.

6. The method according to claim 1, characterized in that: The determining the second text content based on the object content and the first text content, and executing the instruction corresponding to the second text content, includes: inputting the second text content into a semantic model; If the output result of the semantic model is that the second text content is semantically complete, the instruction corresponding to the second text content is executed.

7. The method according to claim 1, characterized in that After obtaining the first text content corresponding to the first voice signal, the method further includes: dividing the words contained in the first text content; It is determined whether the first text content has object missing based on the grammatical relationship between the words.

8. A speech processing device, characterized in that: The device comprises: A receiving unit, configured to receive a first voice signal and obtain a first text content corresponding to the first voice signal; an acquisition unit, configured to acquire multi-modal input data if the first text content has an object missing, wherein the multi-modal input data is data collected through at least one input method; a determining unit, configured to determine object content missing from the first text content based on the multimodal input data; An execution unit is used to determine a second text content based on the object content and the first text content, and execute an instruction corresponding to the second text content.

9. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable program codes; A processor, configured to call and run the executable program code from the memory, so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program code, and when the computer program code is executed, the method according to any one of claims 1 to 7 is implemented.

Images