A method and apparatus for speech recognition

Abstract

Embodiments of the present application provide a speech recognition method and device, the speech recognition method comprising: obtaining a first speech text; obtaining modal information matched with the first speech text; performing multi-modal semantic understanding by combining the first speech text and the modal information, and outputting an intent and a slot. The speech recognition method provided by the present application can accurately recognize the intent of a user, which helps to improve the efficiency of human-computer interaction and improve user experience.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of electronic devices, and more particularly, to a voice recognition method and device. BACKGROUND

[0002] Human-computer interaction applications are currently widely used in electronic devices such as mobile phones, tablet computers, smart speakers, etc., providing users with intelligent human-computer interaction methods. In the process of human-computer interaction, the electronic device needs to accurately recognize the user's intention and then provide intelligent services for the user. However, due to the complexity of natural language, the electronic device may misinterpret the semantics that the user wants to express, resulting in reduced efficiency of human-computer dialogue and affecting user experience. SUMMARY

[0003] Embodiments of the present application provide a voice recognition method that can accurately understand the user's intention in combination with voice text and multi-modal information.

[0004] In a first aspect, a voice recognition method is provided, comprising: obtaining a first voice text; obtaining first modal information matching the first voice text according to the first voice text, the modal indicated by the first modal information being a first modal in a plurality of preset modals; determining a first intention and a first slot indicated by the first voice text when the first voice text matches the first modal according to the first voice text and the first modal information.

[0005] When the first voice text matches the first modal, the first voice text indicates the first intention. When the first voice text indicates the first intention, the first voice text matches the first modal.

[0006] In the present application, the first modal information matching the voice text is modal information of one or more modals in the plurality of preset modals. The modal information includes modal data or modal feature encoding obtained by encoding processing the modal data.

[0007] According to this scheme, multi-modal semantic understanding can be performed in combination with voice text and modal information matching the voice text, the user's intention can be accurately understood, the efficiency of human-computer dialogue can be improved, and user experience can be improved.

[0008] In combination with the first aspect, in a possible implementation manner, the obtaining of the first modal information matching the first voice text according to the first voice text comprises: obtaining a multi-modal selection vector according to the first voice text, the multi-modal selection vector indicating a probability of the first voice text being related to each modal in the plurality of preset modals; and obtaining the first modal information according to the multi-modal selection vector.

[0009] In the embodiment of the present application, the modal information matched with the voice text can be obtained according to the probability that the voice text is related to each preset modality. In this way, in subsequent multi-modal semantic understanding, the intention indicated by the voice text can be understood in combination with the modal information, the accuracy of semantic understanding is improved, the efficiency of human-computer dialogue is improved, and the user experience is improved.

[0010] In combination with the first aspect, in a possible implementation manner, the multi-modal selection vector is obtained according to the first context category, including: determining a first context category to which the first voice text belongs; and obtaining a multi-modal selection vector according to the first context category, the multi-modal selection vector being used to indicate a probability that the first context category is related to each modality in the plurality of preset modalities.

[0011] In the embodiment of the present application, a plurality of context categories can be preset, a plurality of content different voice texts can belong to a same context category, and the voice texts belonging to the same context category are similar in semantics. For example, “today the temperature is too high” and “it is too hot” can both represent the semantic of hot, and if other factors are not considered, both of them can belong to the context category of hot.

[0012] According to the scheme, the context category of the voice text is divided, and then the multi-modal selection vector is obtained according to the context category to which the voice text belongs, and then the modal information matched with the voice text can be obtained. In this way, when the modal information matched with the voice text is determined, only the context category to which the voice text belongs needs to be determined, that is, the modal related to the voice text can be selected, and then the modal information matched with the voice text can be obtained. Therefore, the time spent for obtaining the modal information matched with the voice text can be reduced, and the efficiency of human-computer dialogue is improved.

[0013] In combination with the first aspect, in a possible implementation manner, the multi-modal selection vector is obtained according to the first context category, including: obtaining the multi-modal selection vector according to the first context category and a first mapping matrix, the first mapping matrix indicating a plurality of context categories and a plurality of multi-modal selection vectors, each of the multi-modal selection vectors indicating one or more modalities, and the plurality of context categories and the plurality of multi-modal selection vectors corresponding one by one.

[0014] The first mapping matrix can be preset. In the first mapping matrix, a plurality of context categories and a plurality of multi-modal selection vectors can be preset, each preset context category corresponding to a multi-modal selection vector, and the multi-modal selection vector indicating one or more modalities in the plurality of preset modalities. That is, in the first mapping matrix, a mapping relationship between the context category and the multi-modal selection vector is established, or a mapping relationship between the context category and the preset modality is established.

[0015] In this way, when the context category to which the first speech text belongs is determined, the multimodal selection vector or the mode corresponding to the context category to which the speech text belongs can be obtained according to the first mapping matrix, and then the modal information matched with the speech text is obtained. The time spent in obtaining the modal information matched with the speech text is reduced, and the human-computer interaction efficiency is improved.

[0016] Optionally, the modal information matched with the speech text is obtained according to the first context category and the first mapping matrix.

[0017] It should be understood that the first mapping matrix can include modal information of a plurality of preset modes. For example, after obtaining the modal information of each preset mode, the modal information is stored in the first mapping matrix.

[0018] In this way, when the context category to which the speech text belongs is determined, the modal information matched with the speech text can be obtained from the first mapping matrix. The efficiency of obtaining the modal information matched with the speech text is improved, and the time spent in speech recognition is reduced.

[0019] In combination with the first aspect, in a possible implementation manner, the method further includes: obtaining modal information of the plurality of preset modes.

[0020] In this way, the modal information of each preset mode is obtained in advance, and the modal information is obtained during human-computer interaction, which can improve the human-computer interaction efficiency and improve the user experience.

[0021] Optionally, after the multimodal selection vector or the mode related to the speech text is determined, only the modal information of the related mode is obtained.

[0022] In this way, the modal information matched with the speech text can be obtained in the human-computer interaction process, and resources and power are saved.

[0023] Optionally, the modal information of each preset mode in the plurality of preset modes is periodically obtained.

[0024] According to this scheme, it can be ensured that the modal information in the human-computer interaction process is the latest modal information, thereby helping to improve the accuracy of understanding the intention indicated by the speech text, improve the human-computer interaction efficiency, and improve the user experience.

[0025] In combination with the first aspect, in a possible implementation manner, the method further includes: obtaining the first modal information according to the multimodal selection vector and the modal information of the plurality of preset modes.

[0026] In this way, according to the modal information of the plurality of preset modalities obtained in advance and the multi-modal selection vector, the modal information matching the speech text can be obtained from the modal information of the plurality of preset modalities, avoiding obtaining the modal information through dispersed sensors and the like in the process of human-computer interaction, and the human-computer interaction efficiency can be improved, and the user experience is improved.

[0027] In combination with the first aspect, in a possible implementation manner, the determining of the first context category to which the first speech text belongs includes: determining the first context category to which the first speech text belongs according to the first speech text and / or context information of the first speech text.

[0028] When the context category to which the first speech text belongs is determined in combination with the context information, the real context category of the speech text can be determined. When the current context category cannot be indicated by the speech text, but the current context category can be indicated by the context information, the context category of the speech text can be determined. Then, correct modal information is selected for multi-modal semantic understanding, the intention indicated by the speech text is determined, the semantic recognition error is reduced, the frequency of human-computer interaction is reduced, the human-computer interaction efficiency is improved, and the user experience is improved.

[0029] In combination with the first aspect, in a possible implementation manner, the determining of the first context category to which the first speech text belongs according to the first speech text and / or context information of the first speech text includes: obtaining a text feature code of the first speech text and / or the context information of the first speech text; and determining the first context category to which the first speech text belongs according to the text feature code and a first classification layer, the first classification layer being configured to map the first speech text to one context category in a plurality of preset context categories.

[0030] In this way, the context category to which the first speech text belongs can be more accurately determined in combination with the first speech text and / or the context information, and then correct modal information is obtained for multi-modal semantic understanding, and the intention of the user is determined. The frequency of human-computer interaction is reduced, the human-computer interaction efficiency is improved, and the user experience is improved.

[0031] In combination with the first aspect, in a possible implementation manner, the first modal information includes a first modal feature code, and the determining of the first intention and the first slot indicated by the first speech text when the first speech text matches the first modal information according to the first speech text and the first modal information includes: determining the first intention and the first slot indicated by the first speech text in the first modal according to the text feature code, the first modal feature code and a second classification layer, the second classification layer being configured to map the first speech text to one intention in a plurality of preset intentions.

[0032] In this way, the text feature code combined with the voice text and the modal information matched with the voice text can accurately determine the intention indicated by the voice text. The frequency of human-computer interaction is reduced, the efficiency of human-computer interaction is improved, and the user experience is improved.

[0033] With reference to the first aspect, in a possible implementation, the method further includes: performing an operation related to the first intention.

[0034] In the present application, after determining the intention of the user, an operation related to the intention can be performed. For example, if the intention of the user is to "start adaptive cruise", an operation related to starting adaptive cruise is performed, including asking and obtaining the speed setting and following distance setting of the user to fill the slot corresponding to "starting adaptive cruise", or directly starting adaptive cruise with the current speed and the default following distance. In this way, intelligent services can be provided for the user, the needs of the user are met, and the user experience is improved.

[0035] With reference to the first aspect, in a possible implementation, the method further includes: obtaining the multi-modal selection vector according to the first voice text and a third classification layer, the third classification layer being configured to determine a probability of the first voice text being related to each of a plurality of preset modalities.

[0036] In the present application, the third classification layer is trained to learn the modal matched with the voice text. When a new voice text is input, the mapping relationship between the voice text and the modal can be automatically established according to the content or features of the voice text, the probability of the voice text being related to each preset modality is determined, and the multi-modal selection vector is used to represent the probability. In this way, accurate analysis can be provided on a very large data set, and the influence of people is reduced. Furthermore, the mapping matrix can be preset, the step of obtaining the modal information matched with the voice text is reduced, and the efficiency of human-computer interaction is improved.

[0037] Optionally, the third classification layer is trained to learn the modal matched with the voice text by using a large number of voice texts with one or more modal labels.

[0038] Optionally, the multi-modal selection vector is obtained according to the first voice text, context information of the first voice text, and a third classification layer, the third classification layer being configured to determine a probability of the first voice text being related to each of a plurality of preset modalities.

[0039] In this way, the third classification layer can accurately select a mode related to the speech text according to all features of the speech text and the context information, obtain mode information matched with the speech text, and understand an intention indicated by the speech text in combination with the mode information and the context information, so that the accuracy of semantic understanding can be improved, the efficiency of human-computer dialogue can be improved, and user experience can be improved.

[0040] With reference to the first aspect, in a possible implementation manner, the method further includes: obtaining a second speech text, the second speech text being the same as content of the first speech text; obtaining second mode information matched with the second speech text, the second mode information indicating a second mode of the plurality of preset modes, the second mode being different from the first mode; and determining, according to the second speech text and the second mode information, a second intention and a second slot indicated by the second speech text when the second speech text matches the second mode information, the second intention being different from the first intention, and / or the second slot being different from the first slot.

[0041] The second aspect provides a device for speech recognition, including:

[0042] The obtaining unit is configured to obtain a first speech text.

[0043] The processing unit is configured to: obtain, according to the first speech text, first mode information matched with the first speech text, the first mode information indicating a first mode of the plurality of preset modes; and determine, according to the first speech text and the first mode information, a first intention and a first slot indicated by the first speech text when the first speech text matches the first mode.

[0044] The speech recognition device provided by the embodiments of the present application can perform multi-modal semantic understanding in combination with a speech text and mode information matched with the speech text, and then accurately determine an intention indicated by the speech text, which is beneficial to improving the efficiency of human-computer interaction and user experience.

[0045] With reference to the second aspect, in a possible implementation manner, the processing unit is specifically configured to: obtain, according to the first speech text, a multi-modal selection vector, the multi-modal selection vector indicating a probability that the first speech text is related to each mode of the plurality of preset modes; and obtain the first mode information according to the multi-modal selection vector.

[0046] With reference to the second aspect, in a possible implementation manner, the processing unit is specifically configured to: determine a first context category to which the first speech text belongs; and obtain, according to the first context category, a multi-modal selection vector, the multi-modal selection vector being used to indicate a probability that the first context category is related to each mode of the plurality of preset modes.

[0047] In a possible implementation manner of the second aspect, the processing unit is specifically configured to obtain the multi-modal selection vector according to the first context category and a first mapping matrix, the first mapping matrix indicating a plurality of context categories and a plurality of multi-modal selection vectors, each of the plurality of multi-modal selection vectors indicating one or more modalities, the plurality of context categories and the plurality of multi-modal selection vectors being in one-to-one correspondence.

[0048] In a possible implementation manner of the second aspect, the processing unit is specifically configured to obtain the multi-modal selection vector according to the first context category and a first mapping matrix, the first mapping matrix indicating a plurality of context categories and a plurality of multi-modal selection vectors, each of the plurality of multi-modal selection vectors indicating one or more modalities, the plurality of context categories and the plurality of multi-modal selection vectors being in one-to-one correspondence. In a possible implementation manner of the second aspect, the processing unit is further configured to obtain the modal information of the plurality of preset modalities.

[0049] In a possible implementation manner of the second aspect, the processing unit is specifically configured to obtain the first modal information according to the multi-modal selection vector and the modal information of the plurality of preset modalities.

[0050] In a possible implementation manner of the second aspect, the processing unit is specifically configured to determine the first context category to which the first speech text belongs according to the first speech text and / or context information of the first speech text.

[0051] In a possible implementation manner of the second aspect, the processing unit is specifically configured to obtain a text feature code of the first speech text and / or the context information of the first speech text; and determine the first context category to which the first speech text belongs according to the text feature code and a first classification layer, the first classification layer being configured to map the first speech text to one context category in a plurality of preset context categories.

[0052] In a possible implementation manner of the second aspect, the processing unit is specifically configured to determine the first intent indicated by the first speech text in the first modality and the first slot according to the text feature code, the first modality feature code and a second classification layer, the second classification layer being configured to map the first speech text to one intent in a plurality of preset intents.

[0053] In a possible implementation manner of the second aspect, the processing unit is further configured to perform an operation related to the first intent.

[0054] With reference to the second aspect, in a possible implementation manner, the processing unit is further configured to obtain the multi-modal selection vector according to the first speech text and a third classification layer, the third classification layer being used to determine a probability that the first speech text is related to each of a plurality of preset modalities.

[0055] With reference to the second aspect, in a possible implementation manner, the processing unit is further configured to obtain a second speech text, the second speech text being the same in content as the first speech text; obtain second modal information matched with the second speech text, the second modal information indicating a second modality of the plurality of preset modalities, the second modality being different from the first modality; and determine, according to the second speech text and the second modal information, a second intent and a second slot indicated by the second speech text when the second speech text matches the second modal information, the second intent being different from the first intent, and / or the second slot being different from the first slot.

[0056] The third aspect provides a speech recognition apparatus, including: a memory configured to store a computer program; and a processor configured to execute the computer program stored in the memory, so that the apparatus performs the speech recognition method in any one of the possible implementation manners of the first aspect.

[0057] With reference to the third aspect, in a possible implementation manner, the speech recognition apparatus is an electronic device.

[0058] The fourth aspect provides a computer-readable storage medium, which stores computer-executable instructions, and when the computer-executable instructions are executed on a computer, the computer is caused to perform the speech recognition method in any one of the possible implementation manners of the first aspect.

[0059] The fifth aspect provides a computer program product, which includes computer program code, and when the computer program code is executed on a computer, the computer is caused to perform the speech recognition method in any one of the possible implementation manners of the first aspect.

[0060] The sixth aspect provides a chip system, which includes at least one processor, and when program instructions are executed in the at least one processor, the at least one processor is caused to perform the speech recognition method in any one of the possible implementation manners of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 FIG. 1 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.

[0062] Figure 2Fig. 1 is a schematic diagram of a software structure of an electronic device according to an embodiment of the present application.

[0063] Figure 3 Fig. 2 is a schematic flowchart of a human-computer interaction process according to an embodiment of the present application.

[0064] Figure 4 Fig. 3 is a schematic diagram of a voice recognition method according to an embodiment of the present application.

[0065] Figure 5 Fig. 4 is a schematic diagram of another voice recognition method according to an embodiment of the present application.

[0066] Figure 6 Fig. 5 is a schematic flowchart of a voice recognition method according to an embodiment of the present application.

[0067] Figure 7 Fig. 6 is a schematic flowchart of another voice recognition method according to an embodiment of the present application.

[0068] Figure 8 Fig. 7 is a schematic block diagram of a voice recognition device according to an embodiment of the present application.

[0069] Figure 9 Fig. 8 is a schematic diagram of a hardware structure of a voice recognition device according to an embodiment of the present application. DETAILED DESCRIPTION

[0070] The terms used in the following embodiments are only for the purpose of describing particular embodiments and are not intended to be limiting of the present application. As used in the specification and the appended claims of the application, the singular forms "a," "an" and "the" are intended to include both the singular and the plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Also, it will be understood that the term "and / or," as used herein, encompasses three possibilities: one, both, or neither of the things.

[0071] Reference to "one embodiment" or "an embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiment, although it can. The terms "including," "comprising," "having" and variations thereof are meant to encompass the items listed thereafter, but do not exclude other items from also being present. The terms "a" and "an" are meant to encompass both the singular and the plural, unless otherwise indicated.

[0072] Embodiments of electronic devices, user interfaces for such electronic devices, and associated processes for using such electronic devices are described. In some embodiments, an electronic device can be a portable electronic device, such as a mobile phone, a tablet computer, a wearable electronic device (e.g., a smart watch) with wireless communication capabilities, or the like, that also contains other functionality such as personal digital assistant and / or music player functionality. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running the iOS® operating system of Apple Inc., the Android® operating system of Google Inc., the Windows® operating system of Microsoft Corporation, or other operating systems. The portable electronic devices described above can also be other portable electronic devices, such as a laptop computer, or the like. It should also be understood that, in some other embodiments, the electronic devices described above can not be portable electronic devices, but can be desktop computers.

[0073] Exemplary, Figure 1 A structural diagram of an electronic device 100 is shown. The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a compass 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, and the like.

[0074] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0075] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent components, or can be integrated in one or more processors. In some embodiments, the electronic device 101 can also include one or more processors 110. Among them, the controller can generate operation control signals according to instruction operation codes and timing signals, complete the control of fetching instructions and executing instructions. In other embodiments, a memory can also be provided in the processor 110 for storing instructions and data. Exemplarily, the memory in the processor 110 can be a cache memory. The memory can save instructions or data that the processor 110 has just used or repeatedly uses. If the processor 110 needs to use the instructions or data again, it can directly call from the memory. In this way, repeated access is avoided, the waiting time of the processor 110 is reduced, and thus the efficiency of the electronic device 101 in processing data or executing instructions is improved.

[0076] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc. The USB interface 130 is an interface conforming to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 101, and can also be used to transmit data between the electronic device 101 and a peripheral device. The USB interface 130 can also be used to connect a headset to play audio through the headset.

[0077] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 100. In some other embodiments of the present application, the electronic device 100 can also use different interface connection methods or combinations of multiple interface connection methods.

[0078] The charging management module 140 is configured to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input through a wireless charging coil of the electronic device 100. The charging management module 140 can charge the battery 142 while also providing power to the electronic device through the power management module 141.

[0079] The power management module 141 is configured to connect the battery 142 and the charging management module 140 to the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 can also be configured to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), and the like. In some other embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0080] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, and the like.

[0081] The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be configured to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0082] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G and the like applied to the electronic device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and perform filtering, amplification, and the like on the received electromagnetic waves, and transfer the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can also amplify signals modulated by the modem processor, and convert the amplified signals into electromagnetic waves to be radiated through the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 can be disposed in the same device.

[0083] The wireless communication module 160 can provide a solution for wireless communication, including wireless local area networks (WLAN) (such as a wireless fidelity (Wi-Fi) network), Bluetooth (BT), a global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, and the like, which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency-modulates and filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 can also receive signals to be transmitted from the processor 110, frequency-modulate them, amplify them, and radiate them as electromagnetic waves via the antenna 2.

[0084] The electronic device 100 implements a display function through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0085] The display screen 194 is used to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), or the like. In some embodiments, the electronic device 100 can include one or more display screens 194.

[0086] In some embodiments of the present application, when the display panel adopts an OLED, an AMOLED, an FLED, or the like, the above-described Figure 1The display screen 194 in the electronic device 100 can be foldable. Here, the display screen 194 being foldable means that the display screen can be folded at any part to any angle and can be kept at the angle, for example, the display screen 194 can be folded left and right from the middle. It can also be folded up and down from the middle.

[0087] The display screen 194 of the electronic device 100 can be a flexible screen, which is currently attracting much attention due to its unique characteristics and great potential. Compared with traditional screens, flexible screens have the characteristics of strong flexibility and bendability, can provide users with new interaction methods based on bendable characteristics, and can meet more user needs for electronic devices. For electronic devices equipped with foldable display screens, the foldable display screens on the electronic devices can be switched between the small screen in the folded state and the large screen in the unfolded state at any time. Therefore, users use the split screen function on electronic devices equipped with foldable display screens more and more frequently.

[0088] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor.

[0089] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also optimize the algorithm of the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be arranged in the camera 193.

[0090] The camera 193 is used to capture still images or videos. Objects generate optical images through lenses and project them onto photosensitive elements. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard RGB, YUV, etc. format image signal. In some embodiments, the electronic device 100 can include one or more cameras 193.

[0091] The digital signal processor is used to process digital signals, in addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0092] A video codec is used for compressing or decompressing digital video. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, and the like.

[0093] An NPU is a neural-network (NN) computing processor. By drawing on the structure of a biological neural network, such as the transmission mode between human brain neurons, the NPU can quickly process input information and can also constantly self-learn. Through the NPU, the electronic device 100 can implement intelligent cognition applications, such as image recognition, face recognition, voice recognition, text understanding, and the like.

[0094] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, music, video, and the like are saved in the external memory card.

[0095] The internal memory 121 can be used to store one or more computer programs including instructions. The processor 110 can cause the electronic device 101 to perform the payment method provided in some embodiments of the present application, various applications, and data processing, and the like, by running the above-mentioned instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, and the program storage area can also store one or more applications (such as a gallery, contacts, and the like), and the like. The data storage area can store data (such as photos, contacts, and the like) created during use of the electronic device 101, and the like. In addition, the internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more disk storage components, flash memory components, universal flash storage (UFS), and the like. In some embodiments, the processor 110 can cause the electronic device 101 to perform the payment method provided in the embodiments of the present application, and other applications and data processing, by running the instructions stored in the internal memory 121 and / or the instructions stored in the memory disposed in the processor 110. The electronic device 100 can implement an audio function through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, and the like. For example, music playing, recording, and the like.

[0096] The sensor module 180 can include a pressure sensor 180A, a gyro sensor 180B, a barometric sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0097] The pressure sensor 180A is configured to sense a pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. The pressure sensor 180A can be of various types, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, etc. The capacitive pressure sensor can include at least two parallel plates of conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation is applied to the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with a touch operation intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view short messages is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold is applied to the short message application icon, an instruction to create a new short message is executed.

[0098] The gyro sensor 180B can be configured to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., X, Y, and Z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake photography. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of shaking of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and lets the lens offset the shaking of the electronic device 100 by reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and motion sensing game scenarios.

[0099] The acceleration sensor 180E can detect the magnitude of acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the acceleration sensor 180E can detect the magnitude and direction of gravity. The acceleration sensor 180E can also be used to identify the posture of the electronic device, applied to landscape / portrait switching, pedometer applications, etc.

[0100] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust display screen 194 brightness according to sensed ambient light brightness. Ambient light sensor 180L can also be used to automatically adjust white balance when taking a photo. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touch.

[0101] Fingerprint sensor 180H is used to collect a fingerprint. Electronic device 100 can use collected fingerprint characteristics to implement fingerprint unlocking, access application lock, take a photo with fingerprint, answer a call with fingerprint, etc.

[0102] Temperature sensor 180J is used to detect temperature. In some embodiments, electronic device 100 uses temperature detected by temperature sensor 180J to implement temperature processing strategy. For example, when temperature reported by temperature sensor 180J exceeds a threshold, electronic device 100 reduces performance of a processor located near temperature sensor 180J to reduce power consumption and implement thermal protection. In another embodiment, when temperature is lower than another threshold, electronic device 100 heats battery 142 to avoid abnormal shutdown of electronic device 100 caused by low temperature. In yet another embodiment, when temperature is lower than yet another threshold, electronic device 100 boosts output voltage of battery 142 to avoid abnormal shutdown caused by low temperature.

[0103] Touch sensor 180K, also referred to as "touch panel". Touch sensor 180K can be disposed on display screen 194, and touch sensor 180K and display screen 194 form a touch screen, also referred to as "touch panel". Touch sensor 180K is used to detect a touch operation acting on or near it. Touch sensor 180K can transmit detected touch operation to an application processor to determine touch event type. Visual output related to the touch operation can be provided through display screen 194. In another embodiment, touch sensor 180K can also be disposed on the surface of electronic device 100, which is different from the position of display screen 194.

[0104] Figure 2 is a software structure block diagram of electronic device 100 of an embodiment of the present application. The layered architecture divides software into several layers, each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, application layer, application framework layer, Android runtime and system library, and kernel layer. The application layer can include a series of application packages.

[0105] As Figure 2As shown, the application package can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, voice assistant, etc.

[0106] The application framework layer provides application programming interface (API) and programming framework for the applications of the application layer, and includes some pre-defined functions.

[0107] As shown, the application framework layer can include window manager, content provider, view system, phone manager, resource manager, notification manager, etc. Figure 2

[0108] The window manager is used to manage window programs, and can acquire the size of the display screen, determine whether there is a status bar, lock the screen, and intercept the screen, etc.

[0109] The content provider is used to store and acquire data, and make the data accessible by the applications. The data can include video, image, audio, dialed and received calls, browsing history and bookmarks, phonebook, etc.

[0110] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0111] The phone manager is used to provide the communication function of the electronic device 100. For example, the management of call state (including call connection, call hang-up, etc.).

[0112] The resource manager provides various resources for the applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0113] The notification manager makes the applications display notification information in the status bar, which can be used to convey messages of the notification type, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform the completion of download, message reminder, etc. The notification manager can also be a notification in the form of a chart or a scroll bar text appearing in the top status bar of the system, such as a notification of an application running in the background, and can also be a notification in the form of a dialog window appearing on the screen. For example, prompting text information in the status bar, issuing a prompt sound, electronic device vibration, indicator light blinking, etc.

[0114] ​The system library can include a plurality of functional modules. For example, a surface manager, media libraries, a three-dimensional graphics processing library (e.g., OpenGL ES), a 2D graphics engine (e.g., SGL), etc.

[0115] The surface manager is used to manage the display subsystem and provides fusion of 2D and 3D layers for a plurality of applications.

[0116] The media libraries support a plurality of commonly used audio, video format playback and recording, and static image files, etc. The media libraries can support a plurality of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG, etc.

[0117] The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, synthesis, and layer processing, etc.

[0118] The 2D graphics engine is a drawing engine for 2D drawing.

[0119] The kernel layer is a layer between hardware and software. The kernel layer at least includes display drivers, camera drivers, audio drivers, and sensor drivers.

[0120] The voice assistant application in the application package is a type of human-computer interaction application. The voice assistant application can also be referred to as a voice assistant application or a smart assistant application, etc.

[0121] The human-computer interaction application can also be referred to as a human-computer interaction robot, a human-computer dialogue robot, or a chat robot (ChatBOT), etc. The human-computer interaction application is currently widely used in mobile phones, tablet computers, smart sound boxes, and other types of electronic devices, and provides a smart voice interaction mode for users. At present, users can perform human-computer interaction through voice in a vehicle, which not only enables voice control of navigation, music, and other vehicle software, but also enables control of in-vehicle hardware, such as vehicle windows, air conditioners, etc.

[0122] Figure 3 A schematic flowchart showing a human-computer interaction process is shown. As shown in FIG. 1, the human-computer interaction process includes the following steps. Figure 3As shown, the whole process of human-computer interaction can be implemented through an automatic speech recognition (ASR) module, a natural language understanding (NLU) module, a dialogue manager (DM) module, a natural language generation (NLG) module, and a text to speech (TTS) module. The main function of the ASR module is to recognize user speech as speech text. The main function of the NLU module is to understand the user's intent based on the speech text and perform slot analysis. The DM module can select the next action based on the intent and slot output by the NLU, such as continuing to ask the user, executing the user's instructions, or recommending other instructions to the user. The main function of the NLG module is to generate a dialogue. The main function of the TTS module is to broadcast the dialogue to the user.

[0123] The following describes several important concepts in the human-computer interaction process in detail.

[0124] The main function of the NLU module is to convert speech text into structured information that can be understood by a machine. In a dialogue system, understanding speech text means converting speech text into executable intent and slot, which will be used to complete the user's appeal through a suitable application. A typical implementation is to classify speech text into supported intents using a classification model, and then use a sequence labeling model to label slots in the text. The results of the two are combined to output the understanding results of the intent and slot for subsequent modules.

[0125] For example, the user expresses: Help me book a ticket from Beijing to Shanghai at 10 am tomorrow. From this sentence, the NLU module can analyze the content shown in Table 1.

[0126] Table 1

[0127]

[0128] The above example mentions two concepts, intent and slot. The following explains these two concepts in detail.

[0129] Intent

[0130] An intent can be understood as an intent category. First, determine which intent category the user's statement belongs to, and then the program corresponding to that type will perform specialized parsing. In one implementation, the "program corresponding to this type" can be a bot. For example, if a user says, "Play me a happy song," the NLU module determines that the user's intent category is music, so it summons a music bot to recommend a song to the user. If the user doesn't like the song, they say, "Change it," and the same music bot continues to serve the user until the user expresses a different question and the intent is no longer music, at which point a different bot is switched to serve the user.

[0131] slot

[0132] Each intent can have one or more preset slots. For example, in the example above, the intent "book a flight" has three slots defined: "departure time," "originating location," and "destination." If we want to comprehensively consider the information users need to input when booking a flight, we can certainly think of more, such as the number of passengers, airline, departure airport, and arrival airport. For voice interaction designers, the starting point of the design is defining the slots.

[0133] Once the user's intent is determined through semantic understanding, slot information can be extracted from the speech text and used for slot filling. Slot information can be understood as information related to the slot. For example, in the "book a flight" example above, the slot information corresponding to "departure time," "origin," and "destination" are "10:00 AM tomorrow," "Beijing," and "Shanghai," respectively.

[0134] In some scenarios, the slot information provided by the user's voice text is missing.

[0135] For example, a user might express "Book me a flight for 10 AM tomorrow." Semantic understanding determines the user's intent as "book a flight," with the relevant slots being "departure time," "origin," and "destination." However, the user's statement only contains the "departure time" slot; the "origin" and "destination" slots are missing. In this case, the NLU modal output will show the intent as "book a flight," with the slots "origin = empty," "departure time = 10 AM tomorrow," and "destination = empty."

[0136] At this point, the DM module can control the NLG module to generate a dialog that asks the user for information about missing slots.

[0137] For example, a user might say: "I want to book a flight."

[0138] BOT: What is your destination?

[0139] User: Shanghai.

[0140] BOT: What time do you want to take off?

[0141] When the user fills in all the slot information in the "book a ticket" intent, the DM module can control the command execution module to execute the "book a ticket" operation. For example, the command execution module can open the ticket booking App and display the flight information from Beijing to Shanghai at 10 am (or around 10 am).

[0142] It should be understood that the slots output by the NLU module can be completely filled or missing slot information. In the embodiments of the present application, they are collectively referred to as slots.

[0143] On the basis of the full development of speech recognition technology, we can accurately recognize the user's voice and convert it into speech text. However, the electronic device may misinterpret the user's intent, that is, make a mistake in semantic understanding of the user's speech text. This will lead to the generation of incorrect instructions and the execution of operations that do not meet the user's intent. To a large extent, it reduces the efficiency of human-computer dialogue and user experience.

[0144] The semantic understanding error of the speech text may be due to the fact that the context of the user in the human-computer dialogue is not considered. Even if two speech texts have the same literal content, they may have different semantics in different contexts, so the electronic device may misinterpret the user's intent.

[0145] In the embodiments of the present application, in order to improve the accuracy of the electronic device in understanding the user's intent, multi-modal information-based semantic understanding is introduced.

[0146] Modality can have a very broad definition, and each source or form of information can be called a modality. For example, humans have touch, hearing, vision, and smell; information media include speech, video, and text; and there are a variety of sensors such as radar, infrared, and accelerometers. Each of the above can be called a modality.

[0147] Modality information refers to the specific data of the modality, and modality information can also have other forms of expression, such as modality feature encoding obtained by encoding modality data.

[0148] For example, in a vehicle driving scenario, the vehicle driving speed (e.g., 30 km / h), the air conditioning temperature state (e.g., 30°C), the music playing state (e.g., playing or not playing), the navigation volume size, and the music volume size can all be considered as modality information.

[0149] Figure 4 A schematic diagram of a voice recognition method provided by an embodiment of the present application is shown. As shown in FIG. 1, the voice recognition method provided by the embodiment of the present application includes the following steps. Figure 4 As shown, the method includes 210 to 240, which are described in detail below.

[0150] 210, input the speech text of the user into a text encoding model to obtain text feature encoding.

[0151] In the embodiments of the present application, the speech text can be a word, a phrase or a sentence. The word, the phrase or the sentence herein can be in Chinese characters, English or other languages.

[0152] Specifically, the speech text is input into a text encoding model to obtain text feature encoding of the speech text. The text encoding model can be a bidirectional encoder representation from transformers (BERT) model or a long-short term memory (LSTM) model, for example.

[0153] Taking the BERT model as an example, the BERT model can learn how to represent the speech text with feature encoding through a large amount of text data. When a new speech text is input into the BERT model, the BERT model can output text feature encoding of the speech text, which can represent the semantics of the speech text.

[0154] The text feature encoding in the present application can be a word vector. The word vector refers to a numerical vector mapped from the speech text, which can include all features and semantics from the speech text. Representing the speech text with the word vector can facilitate processing by electronic devices, and it is also easier to calculate the similarity between two texts. Generally, if the word vectors of two texts are similar, the semantics of the two texts are also similar.

[0155] Alternatively, the text feature encoding can also be context feature encoding. The text encoding model encodes feature items in the speech text that can reflect the context of the speech text to obtain context feature encoding.

[0156] In the present application, the text encoding model can extract feature information of the speech text. The basic unit used to represent the text is usually referred to as a feature or a feature item of the text. Generally, the feature item needs to meet:

[0157] (1) The feature item needs to be able to identify the content of the text;

[0158] (2) The feature item has the ability to distinguish the target text from other texts;

[0159] (3) The number of feature items cannot be too large;

[0160] (4) Feature term separation is relatively easy to implement. Words, phrases, or short phrases can be used as feature terms to represent text.

[0161] It should be understood that the text feature encoding is a word vector obtained by mapping the entire speech text, and the context feature encoding is a word vector obtained by mapping the feature terms of the speech text.

[0162] Optionally, the speech text of the user and the context information of the speech text are input into a text encoding model to obtain a text feature encoding.

[0163] It should be understood that the text feature encoding combines the feature information of the speech text and the context information, and the text feature encoding can indicate the speech text and the context information.

[0164] In some embodiments, when the user first performs human-computer interaction, the speech text of the user has no context information, and the speech text does not include context information.

[0165] Optionally, before 210, there is further included obtaining the speech text or the text of the user.

[0166] In the embodiments of the present application, the human-computer interaction application can receive the speech or text input by the user. If the speech of the user is received through speech interaction, the speech of the user will be converted into text content through a pre-processing module (such as an ASR module); if the text input by the user is received through text interaction, such as text input directly through pinyin, handwriting, etc., the conversion process is not required.

[0167] It should be understood that the speech text is the speech of the user converted into text content, or the text input by the user.

[0168] The human-computer interaction application has various ways to receive the speech or text input by the user.

[0169] For example, when the electronic device detects that the user clicks on the icon of the human-computer interaction application (such as a voice assistant) on the desktop of the electronic device, the human-computer interaction application is opened, and the speech or text input by the user is received.

[0170] For example, when the electronic device detects that the user clicks on a specific physical key of the electronic device, the human-computer interaction application is opened, and the speech or text input by the user is received.

[0171] For example, when the electronic device detects the wake-up word (such as Xiaohua Xiaohua) of the user, the human-computer interaction application is opened, and the speech or text input by the user is received.

[0172] For example, when the electronic device detects the free wake-up word (such as turning on the air conditioner) set by the user or the system, the free wake-up word is taken as the speech or text input by the user.

[0173] Specifically, when the electronic device detects the above operation of the user, the human-computer interaction application of the application layer sends the corresponding label of the human-computer interaction application and the process name corresponding to the human-computer interaction application to the system service module of the framework layer. The system service module can determine which hardware and software to start through the label and the process name, so as to receive the voice or text input by the user.

[0174] Optionally, before 210, the context information of the voice text is obtained.

[0175] In the present application, the electronic device can record the voice text of the user in each human-computer interaction process. When the user performs human-computer interaction again, the recorded voice text of the user is obtained as the context information of the current voice text of the user.

[0176] 220, according to the first context category to which the voice text belongs and the context-multimodal mapping matrix, a multimodal selection vector is determined.

[0177] In 220, first, the first context category to which the voice text belongs is determined.

[0178] Reference Figure 4 The text feature code of the voice text is input into the first classification layer, and the first context category to which the voice text belongs is determined.

[0179] The classification layer can also be referred to as a classification function or a classifier. The first classification layer can automatically map the voice text to a certain context category according to the content or features of the voice text according to a pre-specified standard.

[0180] Optionally, the first classification layer can be a machine learning model. A large number of voice text samples labeled with context categories are used to train it. After training, when a new voice text is input, the voice text can be automatically mapped to a certain context.

[0181] The context category can be pre-set by the developer. In the present application, a plurality of different context categories are pre-set for all possible voice texts. The text feature code or context feature code of the voice text belonging to the same context category is similar in semantics and numerical vector.

[0182] For example, the context category can include: (1) a user feeling category, such as cold, hot, noisy, and the like. For example, the voice text "It is too hot" belongs to the hot context. (2) a single device category, such as the voice text "air conditioner temperature", which is only related to the temperature adjustment of the air conditioner and belongs to the air conditioner temperature adjustment context. For example, the voice text "seat ventilation" is only related to the adjustment of seat ventilation and belongs to the seat ventilation adjustment context. (3) a no modality category, if the voice text cannot be classified into any context category, the voice text belongs to the no modality category. For example, the context category of the voice text "What is the weather like today" is the no modality category.

[0183] Voice texts belonging to the same context category can have different expressions.

[0184] It should be understood that one context category can correspond to multiple voice texts, each voice text belongs to only one context category, and different voice texts can belong to the same context category or different context categories.

[0185] For example, the feature information of the voice texts "It is too hot", "Today the temperature is too high" or "I am almost melted" is similar, and they can all be classified into the hot context. For example, when the voice text is "It is too hot", it belongs to the hot context in the user feeling category, and when the voice text is "It is too noisy", it belongs to the noisy context in the user feeling category.

[0186] In the embodiments of the present application, a plurality of modalities can also be preset. In some embodiments, the electronic device can be a vehicle, and the voice recognition in the vehicle is relatively more applied to modalities related to the vehicle state, the driving state, the in-vehicle entertainment, the in-vehicle call, and the like. In other embodiments, the electronic device can be a sound box, and the voice recognition of the sound box is relatively more applied to modalities related to the control of home appliances, the matching of entertainment preferences, audio and video calls, and the like.

[0187] In 220, the context-modality mapping matrix can be preset, in which a mapping relationship between a plurality of preset context categories and a plurality of preset modalities is established, and each context category corresponds to one or more modalities. Optionally, a plurality of multi-modality selection vectors are used to represent the mapping relationship between each context category and the modality, each multi-modality selection vector corresponds to each context category one by one, and each multi-modality selection vector indicates one or more modalities.

[0188] For example, referring to the context-multimodal mapping matrix shown in Table 2, the context-multimodal mapping matrix includes 4 context categories, which are hotness context, coldness context, noise context, and no multimodal context, and 9 modalities, which are air conditioner temperature state, air conditioner wind state, seat heater temperature state, seat heater wind state, voice volume state, window state, car move state, outer temperature state, and temperature difference state. In the context-multimodal mapping matrix, the modalities related to the context are marked as "1", and the modalities not related to the context are marked as "0". For example, when the context category is hotness context, the corresponding modalities are air conditioner temperature state, seat heater temperature state, seat heater wind state, window state, outer temperature state, and temperature difference state.

[0189] Table 2

[0190]

[0191] It should be understood that the above context-multimodal mapping matrix is only an example and should not constitute any limitation on the present application.

[0192] According to the first context category to which the voice text belongs and the context-multimodal mapping matrix, a multimodal selection vector corresponding to the first context category or one or more modalities corresponding to the first context category can be determined.

[0193] For example, when the context category of the voice text is hotness context, according to the above context-multimodal mapping matrix, the multimodal selection vector corresponding to the hotness context can be determined as (1, 0, 1, 1, 0, 1, 0, 1, 1). The multimodal selection vector indicates the following modalities: air conditioner temperature state, seat heater temperature state, seat heater wind state, window state, outer temperature state, and temperature difference state. The multimodal selection vector can be understood as the probabilities of air conditioner temperature state, seat heater temperature state, seat heater wind state, window state, outer temperature state, and temperature difference state being related to the hotness context being 1, and the probabilities of air conditioner wind state, voice volume state, and car move state being related to the hotness context being 0.

[0194] Optionally, according to the first context category to which the voice text belongs and the context-multimodal mapping matrix, an identifier of a modality corresponding to the first context category can be determined.

[0195] Optionally, before 220, modal information of each preset modality is obtained, and a mapping relationship between a plurality of context categories and the modal information of each preset modality is established in the context-multimodal mapping matrix.

[0196] It should be understood that the modal information includes modal data or modal feature encoding obtained by encoding processing on the modal data.

[0197] The modal data can be understood as original data of the modality, for example, a modality (for example, refrigeration, heating) of an air conditioner, a temperature (25℃) set by the air conditioner, a vehicle driving speed (40km / h), and the like. The modal feature encoding is modal feature encoding obtained by encoding processing.

[0198] In this way, according to the first context category to which the voice text belongs and the context-multimodal mapping matrix, modal information matched with the voice text can be determined.

[0199] Optionally, in addition to selecting the modality in the manner shown in 220, the preset modality can also be matched with feature encoding of the voice text to select a modality with high matching degree or high similarity as the modality matched with the voice text. Further, a multimodal selection vector is determined.

[0200] 230, obtaining modal information matched with the voice text.

[0201] In 220, the multimodal selection vector has been determined or the modality matched with the voice text has been selected according to the first context category to which the voice text belongs and the context-multimodal mapping matrix.

[0202] In 230, the modal information matched with the voice text is obtained according to the multimodal selection vector.

[0203] In a possible implementation manner, the electronic device obtains modal data of each preset modality before 230, and encodes the modal data to obtain modal feature encoding of each preset modality. Then, the modal feature encoding of the modality indicated by the multimodal selection vector is obtained.

[0204] The multimodal feature encoding is formed by fusing a plurality of modal feature encodings matched with the voice text indicated by the multimodal selection vector. The manner of fusing the modal feature encodings can be adding or splicing the plurality of modal feature encodings.

[0205] Exemplarily, the electronic device obtains user profile (UP) data, context awareness (CA) data, phone state data and the like modal data before 230, and encodes the modal data to obtain UP feature encoding, CA feature encoding, phone state feature encoding and the like modal feature encoding. The multi-modal selection vector indicates that the modal matched with the speech text of the user is the user profile and the context awareness modal, and then the UP data and the CA data or the UP modal feature encoding and the CA modal feature encoding can be obtained as the modal information matched with the speech text. The phone state data or the phone state feature encoding is not the modal information matched with the speech text.

[0206] Optionally, the fusion of the plurality of modal feature encodings matched with the speech text can also be performed in the feature fusion in 240.

[0207] It should be understood that the multi-modal feature encoding can be a form of modal information.

[0208] The modal data of the preset modal can be obtained in various ways. For example, in a vehicle driving scenario, the electronic device can be a vehicle, which can obtain environmental information around the vehicle through a radar, can obtain environmental temperature data through a temperature sensor, and the like. For example, in a daily life scenario, the electronic device can be a mobile phone, which can obtain location information through a mobile phone GPS, can obtain a mobile phone temperature through a temperature sensor, can obtain a mobile phone power, or can also obtain that the user browses a webpage or plays a game (obtains information related to the game) before human-computer interaction. For example, when obtaining context awareness data, the electronic device can "perceive" the current context through a sensor and related technologies. Exemplarily, the context can be a sports scenario, a walking scenario, a home scenario and the like.

[0209] It should be understood that the modal data can change at different times. For example, the seat heating gear can be adjusted. The modal data during human-computer interaction can reflect the context of the current human-computer dialogue.

[0210] Optionally, the modal data of each preset modal is periodically obtained, and the modal data is encoded to obtain modal feature encoding.

[0211] Optionally, the electronic device stores and updates the modal data or the modal feature encoding of each preset modal in real time.

[0212] In this way, the electronic device can obtain the modal information of each preset modal in advance, and the time spent in obtaining the modal information matched with the speech text during human-computer interaction can be reduced, the efficiency of human-computer interaction can be improved, and the user experience can be improved.

[0213] Optionally, modal data of each preset modality is acquired, and the modal data of the modal indicated by the multi-modal selection vector is encoded to obtain modal feature encoding.

[0214] In another possible implementation, the electronic device can only acquire the modal data of the modal indicated by the multi-modal selection vector, and then encode the modal data to obtain modal feature encoding. At this time, the modal feature encoding is the modal feature encoding matched with the speech text.

[0215] In this way, the electronic device can acquire modal information matched with the speech text, and does not acquire modal information not matched with the speech text, so that power or resource consumption of human-computer interaction can be reduced.

[0216] The encoding process of the modal data is described below.

[0217] In this application, different encoding methods can be selected according to the type of modal data, for example, multilayer perceptron (MLP) encoding is selected for user portrait type modal data, convolutional neural network (CNN) encoding is selected for image type modal data, and the like.

[0218] The user portrait type modal data is described.

[0219] The user portrait type modal data can be a user behavior feature, a user natural attribute, and the like, for example, the user behavior feature can be love for movies, love for music, habit of closing windows while driving, and the like, for example, the user natural attribute can be name, gender, age, zodiac, height, occupation, education, and the like. Multilayer perceptron is also called deep neural network (DNN), which can combine features of data samples of user's usual human-computer interaction to form more abstract high-level attributes or features. These features are also presented in the form of vector data.

[0220] When the user portrait type modal information is combined to perform semantic understanding on the speech text, the intention expressed by the speech text of the user can be more accurately understood, so that the needs of the user can be met.

[0221] 240, multi-modal semantic understanding is performed by combining the multi-modal feature encoding and the text feature encoding, and an intention and a slot are output.

[0222] The multi-modal feature encoding and the text feature encoding are fused, and then the fused feature encoding is input into the second classification layer to output an intention and a slot.

[0223] The second classification layer can automatically map the speech text to a certain intent according to the content or features of the speech text and the modal information matched with the speech text according to a pre-specified standard. It should be understood that the slot output by 250 can be filled with slot information or can not be filled with slot information. When the speech text includes slot-related information, the multi-modal semantic understanding can identify the slot-related information and fill the slot.

[0224] In the present application, a plurality of intents can be preset, for example, in a vehicle driving scenario, including vehicle entertainment system control, such as music, volume; navigation control; driving assistance function control, such as air conditioning adjustment, sunroof adjustment, seat adjustment, etc.; telephone communication control, such as making a call, sending a message, reading a message, reading a WeChat, etc.

[0225] And each intent corresponds to a plurality of slots, which can be pre-defined, for example, the user's speech text is "navigate to the nearest gas station", the user's intent is "navigation control", and the slots corresponding to the intent are "starting point" and "destination". Among them, the slot "starting point" can be the current location by default, and the slot "destination" is "the nearest gas station", the address of "the nearest gas station" can be obtained by searching, and the slot filling is performed.

[0226] The second classification layer can combine the speech text and the modal information matched with the speech text to perform multi-modal semantic understanding, and determine which one of the plurality of preset intents the intent indicated by the speech text belongs to.

[0227] In the embodiments of the present application, when the speech text is combined with different modal information for multi-modal semantic understanding, the output intent and slot can be different.

[0228] For example, referring to Table 3, when the user's speech text is "too hot", the user's semantic is to express that he / she feels very hot, but the user's intent can be to turn off the seat heating, or to turn on the air conditioning cooling, or to turn on the seat ventilation. In a traditional human-computer interaction application, the multi-modal information matched with the speech text cannot be obtained, which can lead to incorrect identification of the user's intent, or the user's intent needs to be determined through multiple rounds of human-computer interaction, and the human-computer interaction efficiency is low. If the multi-modal information matched with the speech text is combined, for example, the air conditioner is not turned on, the environmental temperature is 33℃, the seat ventilation is turned on, the window is closed, etc., then the user's intent can be identified as turning on the air conditioning cooling.

[0229] In some scenarios, the modal information matched with the speech text is that the seat heating is turned on, the seat heating gear is 3, and the environmental temperature is 10℃, so the user's intent can be identified as turning off the seat heating or adjusting the seat heating gear to a lower level.

[0230] Table 3

[0231]

[0232]

[0233] Of course, not all the multi-modal information matched with the voice text is shown in Table 3, for example, the multi-modal information can also include the state of the open window, the internal and external temperature difference, the ambient temperature, the seat ventilation state, etc.

[0234] For example, referring to Table 4, the voice text of the user is “play “The Ambush from Ten Sides””, when the user habit in the modal information is the preference for songs, the music “The Ambush from Ten Sides” is played, and when the user habit in the modal information is the preference for movies and TV series, the video “The Ambush from Ten Sides” is played.

[0235] Table 4

[0236]

[0237] If it is identified that the voice text “play “The Ambush from Ten Sides”” is input by the co-driver, the information related to the user portrait modal can not be included in the modal information matched with the voice text, at this time, the user's intention can be determined by the inquiry mode.

[0238] In some scenarios, if the user's intention and the slot filling are not recognized in combination with the multi-modal information matched with the voice text, it can cause incorrect intention recognition and inquiry errors. For example, if the user's statement is “it is too hot”, at this time, the air conditioning cooling is in the open state, but the temperature of the air conditioning cooling is 30℃, the traditional voice assistant does not obtain the multi-modal information matched with the voice text, and can identify the user's intention as opening the air conditioning cooling, and then inquire whether the user opens the air conditioning cooling, at this time, an invalid human-computer dialogue is generated.

[0239] Figure 5 A schematic diagram of another voice recognition method provided by an embodiment of the present application is shown. Figure 5 The illustrated embodiment includes 310 to 340. Details are described below.

[0240] 310, input the voice text of the user into the text encoding model to obtain a text feature encoding.

[0241] Step 310 is the same as step 210 in Figure 3 For details, refer to the related description of step 210.

[0242] Optionally, input the voice text of the user and the context information of the voice text into the text encoding model to obtain a text feature encoding.

[0243] It should be understood that the text feature encoding combines feature information of the speech text and the context information, and the text feature encoding can indicate the speech text and the context information.

[0244] It should also be understood that the text feature encoding or the context feature encoding is processed speech text, which is a form of the speech text.

[0245] 320, according to the text feature encoding and the third classification layer, a multi-modal selection vector is obtained.

[0246] Specifically, the third classification layer can be trained by a large number of training sample data sets, and the finally trained third classification layer can predict the relevance of a new input speech text to each preset modality. The probability of the speech text being related to each preset modality is calculated by the activation function sigmoid, and the multi-modal selection vector is obtained. The multi-modal selection vector indicates the probability of the speech text being related to each preset modality.

[0247] The training sample data set can be a large number of text feature encodings labeled with multiple modal labels. For example, the text feature encoding of the speech text "too noisy" corresponds to multiple modal labels such as window state and volume state. The text feature encoding of the speech text "what sound outside" corresponds to multiple modal labels such as window state and driving speed. By training a large number of training sample data, the third classification layer learns the ability to predict the modal label corresponding to the speech text.

[0248] The definition domain of the activation function sigmoid can take real numbers in any range or a preset range, and returns an output value in the range of 0 to 1. In the embodiments of the present application, the output value represents the probability of the speech text being related to each preset modality. For example, the multi-modal selection vector is (0.98, 0.02, 0.99), and the vector of the preset modality is (modality a, modality b, modality c). The multi-modal selection probability vector indicates that the probability of being related to modality a is 0.98, the probability of being related to modality b is 0.02, and the probability of being related to modality c is 0.99.

[0249] It should be understood that when the text feature encoding is obtained by combining the speech text and the context information, the text feature encoding includes the features of the speech text and the context information. The multi-modal selection vector indicates the probability of the speech text and the context information being related to each preset modality.

[0250] Optionally, the third classification layer can include a text encoding model. In this way, the third classification layer can be trained by a large number of speech texts labeled with multiple modal labels. Thus, when a new speech text is input, the modal related to the speech text can be predicted, and then the modal information matching the speech text is obtained.

[0251] Optionally, the third classification layer can include an activation function sigmoid, when a new speech text is input, a multi-modal selection vector can be directly output, the multi-modal selection vector is used to indicate a probability that each modality in a plurality of preset modalities is related to the speech text.

[0252] 330, obtaining modality information matched with the speech text.

[0253] 330 and Figure 4 230 in 230 in 230, and details can be referred to the related description of 230.

[0254] It should be understood that when the text feature encoding is obtained in combination with the speech text and the context information, the modality information matched with the speech text can be modality information matched with the speech text and the context information.

[0255] Optionally, the modality information matched with the speech text is obtained by multiplying the multi-modal selection vector and a vector storing the modality information.

[0256] For example, the multi-modal selection vector corresponding to the speech text is (0.98, 0.02, 0.99), the modality feature encodings of each modality are stored in a vector (modality feature encoding a, modality feature encoding b, modality feature encoding c), and (0.98 x modality feature encoding a, 0.02 x modality feature encoding b, 0.99 x modality feature encoding c) can be obtained by matrix multiplication. Then, the multi-modal feature encoding matched with the speech text can be obtained by fusing these modality feature encodings.

[0257] Optionally, the multi-modal selection vector can be represented as a vector including only "0" or "1", when the probability that the speech text is related to a preset modality is greater than a preset threshold, the speech text is related to the preset modality, that is, the element corresponding to the preset modality in the multi-modal selection vector is "1". When the probability that the speech text is related to a preset modality is greater than a preset threshold, the speech text is not related to the preset modality, that is, the element corresponding to the preset modality in the multi-modal selection vector is "0".

[0258] 440, performing multi-modal semantic understanding in combination with the multi-modal feature encoding and the text feature encoding.

[0259] 440 and Figure 3 340 in 340, and details can be referred to the related description of 340.

[0260] Figure 6 An exemplary flowchart of another method for speech recognition provided by the embodiments of the present application is shown. As shown in the figure, the method 400 includes steps S410 to S440. Each step is explained in detail as follows. Figure 6

[0261] ​S410, determine a first context category to which the voice text belongs according to the voice text and the context information of the user.

[0262] Specifically, the voice text and the context information are input into a text encoding model, for example, a BERT model or an LSTM model, to encode the text and obtain text feature encoding. Then the text feature encoding is input into a first classification layer to determine the first context category to which the voice text belongs.

[0263] Optionally, feature information in the voice text and the context information is extracted and encoded to obtain text context feature encoding.

[0264] Optionally, a first context category to which the voice text belongs is determined according to the voice text of the user.

[0265] When the context category to which the first voice text belongs is determined in combination with the context information, it is helpful to determine the real context category of the voice text. Then, correct modal information is selected for multi-modal semantic understanding to determine the intent indicated by the voice text, reduce semantic recognition errors, reduce the frequency of human-computer interaction, improve the efficiency of human-computer interaction, and improve user experience.

[0266] When the voice text cannot explicitly indicate the current context category, but the context information can indicate the current context category, it can be helpful to determine the context category of the voice text. Then, correct modal information is selected for multi-modal semantic understanding to determine the intent indicated by the voice text, reduce semantic recognition errors, reduce the frequency of human-computer interaction, improve the efficiency of human-computer interaction, and improve user experience.

[0267] Exemplarily, the human-computer interaction process is as follows:

[0268] User: "Tell me a cold joke."

[0269] BOT: "Since I have suffered from choice difficulty, I have never played a simple mode."

[0270] User: "It's so cold."

[0271] At this time, if only the voice text "It's so cold" is used to determine the text encoding feature, the voice text will be attributed to the context category of "cold". If the text encoding feature is determined according to the user input "Tell me a cold joke" and "It's so cold", it can be recognized that "It's so cold" does not mean that the user's body temperature feeling is very cold, but that the joke is very cold, so the above voice text "It's so cold" should be attributed to the context category of the no-modal category.

[0272] Exemplarily, the human-computer interaction process is as follows:

[0273] User: "What is the temperature of the air conditioner?"

[0274] BOT: "The air conditioner temperature is currently 16℃."

[0275] User: "A little cold."

[0276] At this time, if only the text encoding features are determined according to the voice text "It's so cold", the voice text will be attributed to the context category of "cold". If the text encoding features are determined according to the user input "What is the air conditioner temperature" and "A little cold", it can be identified that "A little cold" does not mean that the weather is very cold, but that the air conditioner temperature is a little low, so the voice text "A little cold" should be attributed to the context category of air conditioner temperature in the single device category. When the voice text belongs to different context categories, the modalities matched with the voice text can be different, and the identified intent can be very different when performing multi-modal semantic understanding.

[0277] In some scenarios, the voice text cannot explicitly indicate the current context category, and the context information includes features reflecting the context category. An exemplary scenario is as follows.

[0278] User: "What is the seat heating gear?"

[0279] BOT: "Seat heating gear 3."

[0280] User: "Lower the gear."

[0281] At this time, the voice text cannot explicitly indicate the current context category, but the context information can indicate that the context category of the voice text is the seat heating adjustment context in the single device category.

[0282] S420, according to the second context category to which the voice text belongs and the first mapping matrix, obtaining the modality information matched with the voice text.

[0283] In some embodiments, the first mapping matrix predefines a plurality of context categories and a plurality of multi-modal selection vectors, each multi-modal selection vector corresponds to each context category one by one, and each multi-modal selection vector indicates one or more modalities. According to the first context category to which the voice text belongs and the first mapping matrix, the multi-modal selection vector corresponding to the first context category is determined from the first mapping matrix. Then, the modality information of the modality indicated by the multi-modal selection vector is obtained.

[0284] Optionally, before step 420, the modality information of each pre-defined modality in the first mapping matrix is obtained.

[0285] It should be understood that the modality information can be modality data, or modality feature encoding obtained by encoding processing the modality data.

[0286] It should also be understood that when the acquired modal information is modal data, an encoding process needs to be performed on each modal data to determine the modal feature encoding of each modal data. When the acquired modal information is modal feature encoding, the encoding step can be selectively skipped.

[0287] In the embodiments of the present application, when the context category to which the speech text belongs is determined in combination with different context information, the context category to which the speech text belongs can be different.

[0288] In some scenarios, the context category can be more accurately determined in combination with the speech text and the dialogue context, so that the correct multi-modal information can be acquired.

[0289] For example, the human-computer interaction process is as follows:

[0290] User Q1: “Just came back from playing basketball”.

[0291] BOT: “…”

[0292] User Q2: “It's so hot”.

[0293] If only the current speech text “It's so hot” is used, the context will be classified as a hot context, and the modal information such as the ambient temperature state, the internal and external temperature difference state, the air conditioner temperature state, and the window state will be selected.

[0294] If the current speech text “It's so hot” and the dialogue context information “just came back from playing basketball” are used, the context will be classified as a hot context generated by exercise, and some modalities (for example, the seat heating state) are irrelevant, and only the window state, the air conditioner temperature state, and the ambient temperature state will be selected.

[0295] For example, the human-computer interaction process is as follows:

[0296] User Q1: “What happened outside the car”.

[0297] BOT: “…”

[0298] User Q2: “It's so noisy”.

[0299] If only the current speech text “It's so noisy” is used, the context will be classified as a noisy context, and the modal information such as the air conditioner air volume state, the window state, and the volume state will be selected.

[0300] If the current speech text “It's so noisy” and the dialogue context information “what happened outside the car” are used, the context will be classified as a noisy context outside the car, and the modalities (for example, the air conditioner air volume state) inside the car are irrelevant.

[0301] In some other embodiments, the speech text does not need to be classified into a context category, and the modal information of the modal related to the speech text can also be acquired.

[0302] For example, a third classification layer is trained to learn a mapping relationship between speech text or text feature encoding of the speech text and each preset modality through a large number of sample data sets. When a new speech text is obtained, the modality related to the speech text can be predicted, and then the modality information matched with the speech text is obtained. The speech text here can include context information.

[0303] S430, multi-modal semantic understanding is performed in combination with the speech text, the context information and the modality information matched with the speech text, and an intent and a slot are output.

[0304] Specifically, multi-modal semantic understanding is performed in combination with the speech text, the context information and the modality information matched with the speech text, and an intent and a slot are output.

[0305] In some scenarios, a user expresses his / her intent through multi-round voice interaction, and a computer cannot accurately identify the intent of the user only by relying on the current speech text and multi-modal information. The intent-related information can be supplemented in combination with the dialogue context information.

[0306] An exemplary human-computer interaction process is as follows:

[0307] User: “What is the weather like in Beijing?”

[0308] BOT: “I have reported the weather in Beijing for you.”

[0309] User: “What can I eat there?”

[0310] In the human-computer interaction process, the user first asks “What is the weather like in Beijing?” and then asks “What can I eat there?” Here, “there” refers to Beijing, and the intent indicated by the speech text is “querying food in Beijing”. At this time, food in Beijing can be searched for the user and reported to the user.

[0311] Optionally, in step 340, multi-modal semantic understanding can also be performed in combination with the speech text and the modality information matched with the speech text, and an intent and a slot are output.

[0312] Figure 7 An exemplary flowchart of another method for voice recognition provided by an embodiment of the present application is shown. As shown in the flowchart, the method 500 includes steps S510 to S530. Each step is explained in detail as follows. Figure 7

[0313] S510, a first speech text is obtained.

[0314] It should be understood that the first speech text can be a speech text obtained by recognizing the voice of a user, or a text obtained by receiving the input of a user. The first speech text may, for example, be​ Figure 4 the voice text shown in the middle 210.

[0315] At S520, first modal information matching the first voice text is obtained according to the first voice text, and a modality indicated by the first modal information is a first modality in a plurality of preset modalities.

[0316] For example, the preset modalities can be all 9 modalities in the context-multimodal mapping matrix shown in Table 2. For example, when the voice text is "It is too cold", the first modality can be the modality corresponding to the cold context in the context-multimodal mapping matrix shown in Table 2: air conditioner temperature state, seat heating state, seat ventilation state, window state, ambient temperature state, and inside-outside temperature difference state. The first modal information can be the modal information corresponding to the first modality.

[0317] It should be understood that the first modal information is modal information of one or more modalities in the plurality of preset modalities.

[0318] It should also be understood that the modal information can have various forms of expression. In some embodiments, the modal information can be modal data, for example, the modal data of the ambient temperature state is 30°C. In other embodiments, the modal information can also be a modal feature code corresponding to the modal information, that is, the modal data is encoded to convert the modal data into a modal feature code that can be processed by the electronic device. The modal information can also be in the form of a multimodal feature code, that is, the modal feature codes of a plurality of modal information are fused together to form a multimodal feature code.

[0319] In the embodiments of the present application, different modalities can be preset in different scenarios. For example, in a vehicle driving scenario, modalities related to vehicle status, driving status, in-vehicle entertainment, in-vehicle call, etc. can be preset. For example, in a home scenario, modalities related to home appliance control, entertainment preference matching, audio-video call, etc. can be preset.

[0320] Optionally, by matching each preset modality with the first voice text, a modality with high matching degree or high similarity is selected as the modality matching the first voice text.

[0321] In some embodiments, a multimodal selection vector is obtained according to the first voice text, the multimodal selection vector indicating a probability of correlation between the first voice text and each modality in the plurality of preset modalities; and the first modal information is obtained according to the multimodal selection vector.

[0322] The multi-modal selection vector indicates the probability that the first speech text is related to each of the plurality of preset modalities. For example, assuming that three modalities are preset, represented by a vector (modality a, modality b, modality c), the first speech text can or can not be related to each of the preset modalities. When the first speech text is related to modality a and not related to modalities b and c, the multi-modal selection vector is (1, 0, 0), for example Figure 4 The multi-modal selection vector in the illustrated embodiment.

[0323] For another example, assuming that three modalities are preset, represented by a vector (modality a, modality b, modality c), assuming that the probability of being related to modality a is 0.98, the probability of being related to modality b is 0.02, and the probability of being related to modality c is 0.99, the multi-modal selection vector is (0.98, 0.02, 0.99), for example Figure 5 The multi-modal selection vector in the illustrated embodiment.

[0324] In some embodiments, a first context category to which the first speech text belongs is determined; and a multi-modal selection vector is obtained according to the first context category, the multi-modal selection vector being used to indicate the probability that the first context category is related to each of the plurality of preset modalities.

[0325] In the present application, a plurality of different context categories can be preset, and texts belonging to the same context category have similar features. When determining which context category the first speech text belongs to, the first speech text can be matched with the plurality of preset context categories, and then the first context category to which the first speech text belongs is determined. Then, according to the matching degree of the first context category and the plurality of preset modalities, a modality matched with the first context category is obtained, and then modality information matched with the first context category is obtained.

[0326] In some embodiments, the multi-modal selection vector is obtained according to the first context category and a first mapping matrix, the first mapping matrix indicating a plurality of context categories and a plurality of multi-modal selection vectors, each of the multi-modal selection vectors indicating one or more modalities, the plurality of context categories and the plurality of multi-modal selection vectors corresponding one by one.

[0327] For example, the first mapping matrix can be the context-multi-modal mapping matrix shown in Table 2.

[0328] It should be understood that in the first mapping matrix, a mapping relationship of a plurality of preset context categories and a plurality of multi-modal selection vectors can be established, and each preset context category corresponds to a multi-modal selection vector. Each multi-modal selection vector can indicate one or more modalities. In this way, a mapping relationship between context categories and modalities can be established.

[0329] By first determining the context category to which the first speech text belongs, and then determining the modality that matches the first speech text based on the mapping relationship between the context category and the preset modality, the modality information that matches the first speech text can be obtained.

[0330] In some embodiments, modal information of the plurality of preset modalities is obtained.

[0331] For example, acquiring ambient temperature data through temperature sensors.

[0332] In some embodiments, the first modal information is obtained based on the multimodal selection vector and the modal information of the plurality of preset modalities.

[0333] For example, the multimodal selection vector can be multiplied by the modal information of the multiple preset modalities.

[0334] In some embodiments, a first context category to which the first speech text belongs is determined based on the first speech text and / or context information of the first speech text.

[0335] In some embodiments, text feature encoding of the first speech text and / or context information of the first speech text is obtained; based on the text feature encoding and a first classification layer, a first context category to which the first speech text belongs is determined, wherein the first classification layer is used to map the first speech text to one of a plurality of preset context categories.

[0336] For example, text feature encoding can be Figure 4 and Figure 5 The text feature encoding output by the 210 algorithm. The first classification layer can be... Figure 4 The first classification layer is shown in Figure 220. Multiple preset context categories can be the context categories in the context-multimodal mapping matrix shown in Table 2.

[0337] In some embodiments, the electronic device may perform an operation related to the first intent.

[0338] For example, if the primary intent is "book a flight," the operation related to that intent could be either the command execution module performing the flight booking operation, or it could be querying the user for more information related to "book a flight" in the relevant slots.

[0339] In some embodiments, the multimodal selection vector is obtained based on the first speech text and the third classification layer, wherein the third classification layer is used to determine the probability of the first speech text being associated with each of a plurality of preset modalities.

[0340] For example, the third classification layer could be Figure 5 The third classification layer in 220. This multimodal selection vector can be... Figure 5 a multimodal selection vector in the first modality.

[0341] It should be understood that the third classification layer can include a text encoding model or an activation function sigmoid.

[0342] In some embodiments, a second speech text is obtained, the second speech text being the same as the content of the first speech text; second modality information matching the second speech text is obtained, the second modality information indicating a second modality of the plurality of preset modalities, the second modality being different from the first modality; according to the second speech text and the second modality information, a second intent and a second slot indicated by the second speech text when the second speech text matches the second modality information are determined, the second intent being different from the first intent, and / or the second slot being different from the first slot.

[0343] For example, the second speech text and the first speech text can be "It is too hot" in Table 3, the first modality information can be "environmental state: driving, temperature 33℃, device state: air conditioner not turned on" in Table 3, the second modality information can be "environmental state: driving, temperature 10℃, device state: seat heating turned on" in Table 3, the first intent can be "turn on air conditioner cooling mode" in Table 3, and the second intent can be "turn off seat heating mode" in Table 3. The first slot and the second slot are slots corresponding to "turn on air conditioner cooling mode" and "turn on seat heating mode", respectively. For example, the slot of "turn on air conditioner cooling mode" can be "air conditioner temperature = 20℃, air conditioner air volume = 3 gears"

[0344] S530, according to the first speech text and the first modality information, determining a first intent and a first slot indicated by the first speech text when the first speech text matches the first modality information.

[0345] It should be understood that a plurality of different intents can be preset in the embodiments of the present application, and each intent corresponds to a different slot. When the intent is determined, the slot is also determined. For example, the first intent and the first slot can be the intent and the slot in the example of "booking a ticket", or can be the intent corresponding to the speech text shown in Table 4.

[0346] In some embodiments, the first modality information includes a first modality feature encoding, and according to the text feature encoding, the first modality feature encoding and a second classification layer, a first intent and the first slot indicated by the first speech text in the first modality are determined, the second classification layer being used to map the first speech text to one intent of a plurality of preset intents.

[0347] For example, the first modality feature encoding can be Figure 4 or Figure 5 the second classification layer in the multi-modal feature encoding in Figure 4 or Figure 5 the second classification layer in the multi-modal feature encoding in

[0348] The method embodiments of the embodiments of the present application are described in detail above in combination with Figures 1 to 7 The device embodiments of the embodiments of the present application are described in detail below in combination with Figure 8 It should be understood that the description of the method embodiments corresponds to the description of the device embodiments, and therefore, the parts not described in detail can be referred to the foregoing method embodiments.

[0349] Optionally, the speech recognition device provided by the present application can be an electronic device.

[0350] It can be understood that, in order to realize the above functions, the electronic device comprises hardware and / or software modules corresponding to each function. The algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is realized in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application in combination with the embodiments, but such implementation should not be considered beyond the scope of the present application.

[0351] The present embodiment can divide the functional modules of the electronic device according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated in one processing module. The integrated module can be realized in the form of hardware. It should be noted that the division of modules in the present embodiment is illustrative, and is only a logical functional division. Actual implementation can have another division manner.

[0352] In the case of dividing each functional module corresponding to each function, Figure 8 A possible composition schematic diagram of the speech recognition device 600 involved in the above embodiments is shown, as shown in Figure 8 The speech recognition device 600 can comprise an acquisition unit 610 and a processing unit 620.

[0353] The acquisition unit 610 is configured to acquire a first speech text.

[0354] The processing unit 620 is configured to:

[0355] According to the first speech text, acquire first modal information matched with the first speech text, the first modal information indicating a first modal among a plurality of preset modals;

[0356] According to the first speech text and the first modality information, a first intent and a first slot indicated by the first speech text are determined when the first speech text matches the first modality information.

[0357] The speech recognition device 600 provided by the embodiments of the present application can perform multi-modal semantic understanding in combination with the first speech text and the modality information matched with the first speech text, and accurately determine the intent indicated by the first speech text, thereby facilitating improvement of human-computer interaction efficiency and user experience.

[0358] Optionally, the processing unit 620 is specifically configured to: according to the first speech text, obtain a multi-modal selection vector, the multi-modal selection vector indicating a probability that the first speech text is related to each modality in the plurality of preset modalities; and according to the multi-modal selection vector, obtain the first modality information.

[0359] Optionally, the processing unit 620 is specifically configured to: determine a first context category to which the first speech text belongs.

[0360] According to the first context category, a multi-modal selection vector is obtained, the multi-modal selection vector being used to indicate a probability that the first context category is related to each modality in the plurality of preset modalities.

[0361] Optionally, the processing unit 620 is specifically configured to: according to the first context category and a first mapping matrix, obtain the multi-modal selection vector, the first mapping matrix indicating a plurality of context categories and a plurality of multi-modal selection vectors, each multi-modal selection vector indicating one or more modalities, and the plurality of context categories and the plurality of multi-modal selection vectors corresponding one by one.

[0362] Optionally, the processing unit 620 is further configured to: obtain modality information of the plurality of preset modalities.

[0363] Optionally, the processing unit 620 is further configured to: according to the multi-modal selection vector and the modality information of the plurality of preset modalities, obtain the first modality information.

[0364] Optionally, the processing unit 620 is specifically configured to: according to the first speech text and / or context information of the first speech text, determine a first context category to which the first speech text belongs.

[0365] Optionally, the processing unit 620 is specifically configured to: obtain a text feature code of the first speech text and / or the context information of the first speech text; and according to the text feature code and a first classification layer, determine a first context category to which the first speech text belongs, the first classification layer being used to map the first speech text to one context category in a plurality of preset context categories.

[0366] Optionally, the first modality information comprises a first modality feature code, and the processing unit 620 is specifically configured to determine the first intent indicated by the first voice text in the first modality and the first slot according to the text feature code, the first modality feature code, and a second classification layer, the second classification layer being used to map the first voice text to one of a plurality of preset intents.

[0367] Optionally, the processing unit 620 is further configured to perform an operation related to the first intent.

[0368] Optionally, the processing unit 620 is specifically configured to obtain the multi-modality selection vector according to the first voice text and a third classification layer, the third classification layer being used to determine a probability of the first voice text being related to each modality of a plurality of preset modalities.

[0369] Optionally, the processing unit 620 is specifically configured to obtain a second voice text, the second voice text having the same content as the first voice text; obtain second modality information matched with the second voice text, the second modality information indicating a second modality of a plurality of preset modalities, the second modality being different from the first modality; and determine a second intent indicated by the second voice text and a second slot when the second voice text matches the second modality information, the second intent being different from the first intent and / or the second slot being different from the first slot, according to the second voice text and the second modality information.

[0370] Figure 9 FIG. 7 is a schematic diagram of a hardware structure of a voice recognition device provided by an embodiment of the present application. Figure 9 The voice recognition device 700 shown (which can be specifically an electronic device) includes a memory 710, a processor 720, a communication interface 730, and a bus 740. The memory 710, the processor 720, and the communication interface 730 are communicatively connected to each other through the bus 740.

[0371] The memory 710 can be a ROM, a static storage device, a dynamic storage device, or a RAM. The memory 710 can store a program, and when the program stored in the memory 710 is executed by the processor 720, the processor 720 is configured to perform various steps of the voice recognition method of the embodiments of the present application.

[0372] The processor 720 can adopt a general-purpose CPU, a microprocessor, an ASIC, a GPU, or one or more integrated circuits, and is configured to execute a related program to implement the functions required by the units in the voice recognition device of the embodiments of the present application, or to execute the voice recognition method of the method embodiments of the present application.

[0373] The processor 720 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the speech recognition method of this application can be completed by the integrated logic circuitry in the hardware of the processor 720 or by software instructions. The aforementioned processor 720 can also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 710. The processor 720 reads the information in memory 710 and, in conjunction with its hardware, completes the functions required by the units included in the speech recognition device of this application embodiment, or executes the speech recognition method of the method embodiment of this application.

[0374] The communication interface 730 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 700 and other devices or communication networks. For example, a description file or a first set of paths corresponding to the description file can be obtained through the communication interface 730.

[0375] Bus 740 may include a pathway for transmitting information between various components of device 700 (e.g., memory 710, processor 720, communication interface 730).

[0376] It should be noted that, although Figure 9 The illustrated device 700 only shows the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, device 700 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that device 700 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that device 700 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 9 All the devices shown.

[0377] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0378] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0379] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0380] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0381] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0382] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0383] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A speech recognition method, characterized in that, include: Obtain the first voice text; Based on the first speech text, obtain first modal information that matches the first speech text, wherein the modality indicated by the first modal information is the first modality among a plurality of preset modalities; Based on the first speech text and the first modal information, determine the first intent and the first slot indicated by the first speech text when the first speech text matches the first modal information; The step of obtaining first modal information matching the first speech text includes: Determine the first context category to which the first spoken text belongs; Based on the first context category, a multimodal selection vector is obtained, wherein the multimodal selection vector is used to indicate the probability of the first context category being associated with each of the plurality of preset modalities; The first modality information is obtained based on the multimodal selection vector.

2. The method according to claim 1, characterized in that, Based on the first context category, obtain the multimodal selection vector, including: Based on the first context category and the first mapping matrix, the multimodal selection vector is obtained. The first mapping matrix indicates multiple context categories and multiple multimodal selection vectors. Each multimodal selection vector indicates one or more modalities. The multiple context categories and the multiple multimodal selection vectors correspond one-to-one.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Obtain modal information of the multiple preset modalities.

4. The method according to claim 3, characterized in that, The method further includes obtaining the first modality information based on the multimodal selection vector and the modality information of the plurality of preset modalities.

5. The method according to claim 1 or 2, characterized in that, Determining the first context category to which the first spoken text belongs includes: Based on the first speech text and / or the context information of the first speech text, determine the first context category to which the first speech text belongs.

6. The method according to claim 5, characterized in that, Determining the first context category to which the first speech text belongs based on the first speech text and / or the context information of the first speech text includes: Text feature encoding of the first speech text and / or the context information of the first speech text; Based on the text feature encoding and the first classification layer, the first context category to which the first speech text belongs is determined. The first classification layer is used to map the first speech text to one of a plurality of preset context categories.

7. The method according to claim 6, characterized in that, The first modal information includes the first modal feature encoding. The step of determining the first intent and first slot indicated by the first speech text when the first speech text matches the first modal information, based on the first speech text and the first modal information, includes: Based on the text feature encoding, the first modality feature encoding, and the second classification layer, the first intent indicated by the first speech text in the first modality and the first slot are determined, and the second classification layer is used to map the first speech text to one of a plurality of preset intents.

8. The method according to claim 1, characterized in that, The method further includes: Perform the operation related to the first intent.

9. The method according to claim 1, characterized in that, The step of obtaining the multimodal selection vector based on the first speech text includes: Based on the first speech text and the third classification layer, the multimodal selection vector is obtained, and the third classification layer is used to determine the probability of the first speech text being associated with each of the multiple preset modalities.

10. The method according to claim 1 or 2, characterized in that, The method further includes: Obtain a second voice text, the content of which is the same as that of the first voice text; Obtain second modal information that matches the second speech text, wherein the modality indicated by the second modal information is the second modality among a plurality of preset modalities, and the second modality is different from the first modality; Based on the second speech text and the second modal information, determine the second intent and the second slot indicated by the second speech text when the second speech text matches the second modal information, wherein the second intent is different from the first intent, and / or the second slot is different from the first slot.

11. A speech recognition device, characterized in that, include: The acquisition unit is used to acquire the first speech text; Processing unit, used for: Based on the first speech text, obtain first modal information that matches the first speech text, wherein the modality indicated by the first modal information is the first modality among a plurality of preset modalities; Based on the first speech text and the first modal information, determine the first intent and the first slot indicated by the first speech text when the first speech text matches the first modal information; The processing unit is specifically used for: Determine the first context category to which the first spoken text belongs; Based on the first context category, a multimodal selection vector is obtained, wherein the multimodal selection vector indicates the probability of the first context category being associated with each of the plurality of preset modalities; The first modality information is obtained based on the multimodal selection vector.

12. The apparatus according to claim 11, characterized in that, The processing unit is specifically used for: Based on the first context category and the first mapping matrix, the multimodal selection vector is obtained. The first mapping matrix indicates multiple context categories and multiple multimodal selection vectors. Each multimodal selection vector indicates one or more modalities. The multiple context categories and the multiple multimodal selection vectors correspond one-to-one.

13. The apparatus according to claim 11 or 12, characterized in that, The processing unit is also used to acquire modal information of the plurality of preset modalities.

14. The apparatus according to claim 13, characterized in that, The processing unit is specifically used for: The first modal information is obtained based on the multimodal selection vector and the modal information of the multiple preset modalities.

15. The apparatus according to any one of claims 11 or 12, characterized in that, The processing unit is specifically used for: Based on the first speech text and / or the context information of the first speech text, determine the first context category to which the first speech text belongs.

16. The apparatus according to claim 15, characterized in that, The processing unit is specifically used for: Text feature encoding of the first speech text and / or the context information of the first speech text; Based on the text feature encoding and the first classification layer, the first context category to which the first speech text belongs is determined. The first classification layer is used to map the first speech text to one of a plurality of preset context categories.

17. The apparatus according to claim 16, characterized in that, The first modal information includes the first modal feature encoding. The processing unit is specifically used for: Based on the text feature encoding, the first modality feature encoding, and the second classification layer, the intent indicated by the first speech text in the first modality and the slot corresponding to the intent are determined. The second classification layer is used to map the first speech text to one of a plurality of preset intents.

18. The apparatus according to claim 11, characterized in that, The processing unit is also used to perform operations related to the first intent.

19. The apparatus according to claim 11, characterized in that, The processing unit is specifically used to obtain the multimodal selection vector based on the first speech text and the third classification layer, wherein the third classification layer is used to determine the probability of the first speech text being associated with each of the multiple preset modalities.

20. The apparatus according to claim 11 or 12, characterized in that, The processing unit is also used for: Obtain a second voice text, the content of which is the same as that of the first voice text; Obtain second modal information that matches the second speech text, wherein the modality indicated by the second modal information is the second modality among a plurality of preset modalities, and the second modality is different from the first modality; Based on the second speech text and the second modal information, determine the second intent and the second slot indicated by the second speech text when the second speech text matches the second modal information, wherein the second intent is different from the first intent, and / or the second slot is different from the first slot.

21. A voice recognition device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory to cause the apparatus to perform the method as described in any one of claims 1 to 10.

22. A computer-readable medium, characterized in that, The computer-readable medium stores program code that, when run on a computer, causes the computer to perform the speech recognition method as described in any one of claims 1 to 10.

23. A computer program product, characterized in that, The computer program product includes: computer program code that, when the computer program product is run on a computer, causes the computer to perform the speech recognition method as described in any one of claims 1 to 10.

Images