Voice interaction method and device, electronic equipment and storage medium
Acquisition of icon images through the graphic matching model and cloud storage links, solving the problem of low voice interaction success rate caused by the diversity of user voice descriptions, realizing high-degree of freedom icon control matching and voice interaction, improving the accuracy of voice interaction and Success rate.
Patent Information
- Application Number
- CN202411169085.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-08
AI Technical Summary
In the prior art, the icon-based voice interaction system is difficult to correctly understand and execute user voice, mainly because the user has extremely high subjectivity and diversity in the character description of the icon, resulting in a low success rate of voice interaction.
By receiving user voice and determining the icon control displayed on the interface, the graphic and text matching model determines the icon control that matches the transcribed text based on the semantic matching degree between the transcribed text of the user voice and the icon image of the icon control, thereby generating an icon control that matches the transcribed text based on the semantic matching degree of the transcribed text by the user voice, thereby Control commands to communicate voice. The method includes extracting semantic features of speech and icon controls, using pre-trained text-image models for matching degree calculations, and obtaining icon images through cloud storage links to support high-degree of freedom icon description methods.
It improves the success rate of voice interaction, supports flexible and diverse icon controls to describe the description method, and enhances the freedom and accuracy of voice interaction.
Smart Images

Figure CN118918897A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a voice interaction method, device, electronic device and storage medium. Background Art
[0002] In order to improve the aesthetics of the human-machine interface (HMI), icons have been widely used in the interfaces of smart devices such as smartphones, tablet computers, and in-vehicle central control screens.
[0003] When performing voice interaction based on the human-computer interaction interface, users' descriptions of icon features are often highly subjective and diverse, which means that there may be many different descriptions of the same icon. This difference makes it difficult for the voice interaction system to correctly understand and execute user voice, affecting the success rate of icon-based voice interaction. Summary of the invention
[0004] The present invention provides a voice interaction method, device, electronic device and storage medium, which are used to solve the defect of low success rate of icon-based voice interaction in related technologies.
[0005] The present invention provides a voice interaction method, comprising: Receive user voice, and determine icon controls displayed on the interface when receiving the user voice; Based on the semantic matching degree between the transcribed text of the user speech and the icon images of the icon controls, determining the icon control matching the transcribed text from the icon controls; Based on the transcribed text and an icon control matching the transcribed text, voice interaction is performed.
[0006] According to a voice interaction method provided by the present invention, the semantic matching degree between the transcribed text of the user voice and the icon images of the icon controls is respectively determined from the icon controls that matches the transcribed text, including: Inputting the transcribed text of the user speech and the icon images of the icon controls into a picture-text matching model, obtaining a picture-text matching result output by the picture-text matching model based on the semantic matching degree between the transcribed text and the icon images, respectively, wherein the picture-text matching result indicates the icon control matching the transcribed text; The image-text matching model is trained based on sample texts and sample images.
[0007] According to a voice interaction method provided by the present invention, the acquisition of the semantic matching degree includes: Extracting semantic features of the transcribed text of the user's speech to obtain text semantic features; Extracting semantic features of the icon image of the icon control to obtain image semantic features, wherein the text semantic features and the image semantic features are in the same feature space; The semantic matching degree is determined based on the similarity between the text semantic feature and the image semantic feature.
[0008] According to a voice interaction method provided by the present invention, obtaining the icon image includes: Obtaining a storage link of an icon image corresponding to the icon control; Based on the storage link, the icon image is obtained.
[0009] According to a voice interaction method provided by the present invention, the step of obtaining a storage link of an icon image corresponding to the icon control includes: The storage link of the icon image is read from the control properties of the icon control.
[0010] According to a voice interaction method provided by the present invention, obtaining the icon image based on the storage link includes: Based on the storage link, the icon image is downloaded from the cloud, and the cloud is used to store the icon image of each icon control.
[0011] According to a voice interaction method provided by the present invention, the step of determining icon controls displayed on an interface when receiving the user voice includes: When a change in the interface is detected, the icon controls displayed on the interface are scanned, and the icon controls scanned after the last change in the interface when receiving the user voice are used as the icon controls displayed on the interface when receiving the user voice.
[0012] According to a voice interaction method provided by the present invention, the determining of each icon control displayed on the interface when receiving the user voice also includes: During the voice interaction process, detect whether the interface changes.
[0013] According to a voice interaction method provided by the present invention, the voice interaction based on the transcribed text and an icon control matching the transcribed text includes: Based on the transcribed text and the control identifier of the icon control matching the transcribed text, generating a control instruction corresponding to the user voice; Based on the control instruction, voice interaction is performed.
[0014] According to a voice interaction method provided by the present invention, the voice interaction based on the control instruction includes: The control identifier in the interface determines the target control to be controlled from the interface; Execute the control operation of the target control corresponding to the control instruction to realize the voice interaction.
[0015] The present invention also provides a voice interaction device, comprising: An acquisition unit, configured to receive a user voice and determine icon controls to be displayed on the interface when receiving the user voice; A matching unit, configured to determine, from among the icon controls, an icon control that matches the transcribed text of the user's speech based on the semantic matching between the transcribed text of the user's speech and the icon images of the icon controls; An interaction unit is used to perform voice interaction based on the transcribed text and an icon control matching the transcribed text.
[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the voice interaction method described above is implemented.
[0017] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the voice interaction method described in any one of the above is implemented.
[0018] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the voice interaction method described above is implemented.
[0019] The voice interaction method, device, electronic device and storage medium provided by the present invention can determine the icon control that matches the transcribed text from each icon control based on the semantic matching degree between the transcribed text of the user's voice and the icon image of each icon control, and can support a highly free, flexible and diverse reference description method of the icon control. On this basis, voice interaction is performed in combination with the transcribed text and the icon control that matches the transcribed text, which can effectively ensure the success rate of voice interaction. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0021] Figure 1 It is a schematic diagram of a human-computer interaction interface in the related technology.
[0022] Figure 2 This is one of the flow charts of the voice interaction method provided by the present invention.
[0023] Figure 3 It is a structural schematic diagram of the icon management system provided by the present invention.
[0024] Figure 4 This is the second flow chart of the voice interaction method provided by the present invention.
[0025] Figure 5 It is a structural schematic diagram of the voice interaction device provided by the present invention.
[0026] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] With the rapid development of information technology, the human-machine interface, as an important bridge between users and devices, has become a key factor in measuring user experience in terms of design aesthetics and convenience of interaction. Especially on smart devices such as smartphones, tablets, and large-screen car consoles, icons, as an important part of HMI, not only play the role of improving the aesthetics of the interface, but also directly affect the user's operating efficiency and satisfaction.
[0029] The application of icons on HMI provides convenience for users to manually click on icons to achieve human-computer interaction. However, when performing voice interaction, the current voice interaction method is to transcribe the user's voice into text, match the transcribed text with the name of each icon, or match the transcribed text with the pre-set function description text of each icon to determine whether the user's voice carries the icon. Since the user's description of the characteristics of the icon is often highly subjective and diverse, different users may use completely different words to describe the same icon, and the pre-set function description text may not necessarily reflect the visual characteristics of the icon, nor can it exhaust the user's possible visual descriptions. This makes it difficult for the voice interaction system to accurately capture the user's intentions, and therefore difficult to correctly understand and execute the user's voice.
[0030] For example, Figure 1 It is a schematic diagram of the human-computer interaction interface in the related technology. Figure 1 The homepage is represented by a house icon in the lower left corner, and the application center is represented by a grid combination icon. When the user intends to open the homepage, the user voice may describe it as "open the little house", and when the user intends to open the application center, the user voice may describe it as "open four small squares". This diverse form of expression brings huge challenges to voice understanding and voice interaction.
[0031] Related technologies attempt to solve the problem of icon-based voice interaction by using OCR (Optical Character Recognition) technology. However, since OCR technology has poor recognition effect on icons of non-text characters, the voice interaction effect based on OCR icon recognition is poor.
[0032] Therefore, in icon-based voice interaction, since voice interaction itself cannot support diverse description methods for icons, the freedom of voice interaction in icon description is low, which in turn leads to a low success rate of icon-based voice interaction.
[0033] In view of the above problems, an embodiment of the present invention provides a voice interaction method. Figure 2 is one of the flow charts of the voice interaction method provided by the present invention, such as Figure 2 As shown, the method includes: Step 210, receiving user voice, and determining icon controls displayed on the interface when receiving the user voice.
[0034] Here, user voice refers to the voice emitted by the user recorded during the voice interaction process. The user voice can be obtained through a sound pickup device, which can be a smart phone with a built-in microphone, a tablet computer, a large car central control screen, etc., or a smart appliance such as a smart speaker, a TV, and an air conditioner. After the sound pickup device obtains voice data through a microphone array, it can also amplify and reduce noise on the voice data, which is not specifically limited in the embodiment of the present invention.
[0035] In the process of voice interaction, in order to achieve device control, the user voice usually points to one or more controls displayed on the device display screen on the interface. The interface referred to here is the human-computer interaction interface. For example, the user voice "turn on the air conditioner" can correspond to the air conditioner control on the interface, and the user voice "open the small house" can correspond to the homepage control displayed in the form of a house icon on the interface. Therefore, while receiving the user voice, it is possible to determine the icon controls displayed on the interface through the display screen at this time.
[0036] Here, an icon control is a type of control that is displayed in the form of an icon. An icon control is displayed in the form of an icon, which can perform a decorative effect visually while expressing functionality. Figure 1 The home page control displayed in the style of a house icon, the navigation control displayed in the style of an arrow icon, the music control displayed in the style of a musical note icon, the application center control displayed in the style of a grid combination icon, the music collection control displayed in the style of a heart icon, etc., are all icon controls.
[0037] It is understandable that when receiving the user voice, one or more icon controls may be displayed on the interface at the same time. Furthermore, the user voice may be a voice operation instruction directed to one or more of the icon controls.
[0038] Step 220: Based on the semantic matching degree between the transcribed text of the user speech and the icon images of the icon controls, determine the icon control that matches the transcribed text from the icon controls.
[0039] Specifically, after acquiring the user voice, voice recognition may be performed on the user voice, thereby obtaining a voice recognition result of the user voice, that is, a transcribed text of the user voice.
[0040] In addition, after obtaining each icon control displayed on the interface, the icon image of each icon control can be obtained respectively. The icon image of an icon control refers to a specific image or graphic displayed in the icon control. The icon image is embedded in the icon control to provide a visual representation when the icon control is displayed. It can be understood that in order to distinguish different controls when displaying icon controls, different icon controls have different icon images, that is, the icon controls correspond to the icon images one by one.
[0041] The method for obtaining the icon image of the icon control can be to obtain a pre-stored icon image of the icon control from the local or cloud, or it can be to identify and segment the icon control in the interface image based on the target recognition algorithm, thereby obtaining the icon image of the icon control. The embodiment of the present invention does not make specific limitations on this.
[0042] After obtaining the transcribed text of the user's voice and the icon images of each icon control, the semantic matching degree between the transcribed text and the icon image of each icon control can be calculated. Here, for the transcribed text and the icon image of any icon control, the semantic features of the transcribed text and the semantic features of the icon image can be extracted, and then the similarity of the semantic features of the two can be calculated, thereby obtaining the similarity between the semantic features of the two, that is, the semantic matching degree.
[0043] Among them, the semantic features of the transcribed text can reflect the characteristics of the transcribed text in semantic expression, including the characteristics of the transcribed text describing the icon image to be controlled. The semantic features of the icon image reflect the characteristics of the content of the icon image in semantic expression, which can specifically reflect the visual information of the icon image such as color, texture, shape, etc., and can also reflect the visual information of the object, human body, scene, etc. contained in the icon image.
[0044] Therefore, with respect to the transcribed text and the icon image of any icon control, the semantic matching degree between the two can reflect the degree of matching between the description of the icon image to be controlled in the transcribed text and the content of the icon image of the icon control itself. The higher the semantic matching degree, the higher the probability that the icon control is the icon control indicated by the user's voice. The lower the semantic matching degree, the lower the probability that the icon control is the icon control indicated by the user's voice.
[0045] Thus, based on the semantic matching between the transcribed text of the user's voice and the icon images of each icon control, an icon control that matches the transcribed text can be determined from each icon control. It can be understood that the icon control that matches the transcribed text obtained here is the icon control that is controlled by the user's voice. Specifically, the icon control with the highest semantic matching degree can be selected from each icon control as the icon control that matches the transcribed text, or the icon control with a semantic matching degree higher than a preset matching degree threshold can be selected from each icon control as the icon control that matches the transcribed text, and the embodiment of the present invention does not specifically limit this.
[0046] It can be understood that compared with the method of matching the transcribed text of the user's voice with the name of each icon control or the preset description text, the probability of each icon control being the icon control indicated and controlled by the user's voice is measured based on the semantic matching between the transcribed text of the user's voice and the icon image of each icon control, and the original text-to-text matching is adjusted to text-to-image matching. Since the semantic features of the icon image itself can reflect the semantic expression characteristics of the visual content of the icon image, it can be compatible with various icon description methods. Therefore, no matter what description method the user uses to point to the icon control, the match between the user's voice and the icon control can be achieved through semantic matching. In the process of voice interaction, it can support highly free, flexible and diverse reference description methods of icon controls, thereby ensuring the success rate of voice interaction.
[0047] For example, when the transcribed text is "Open the Little House", the control name of the home page control displayed in the form of a house icon is "Home", and "Open the Little House" and "Home" cannot be successfully matched in terms of text; while the icon image of the home page control contains a house, and the semantic features of the icon image also carry the image features of the house. By matching the semantic features of "Open the Little House" with the semantic features of the icon image that carries the image features of the house, a successful match can be achieved.
[0048] Step 230, performing voice interaction based on the transcribed text and an icon control matching the transcribed text.
[0049] Specifically, after obtaining the icon control that matches the transcribed text, the transcribed text and the icon control that matches it can be combined to achieve voice interaction. For example, a control instruction for controlling the icon control that matches it can be generated based on the transcribed text, and voice interaction can be achieved by executing the control instruction; for another example, semantic understanding can be performed by combining the transcribed text and the icon control that matches it, thereby obtaining a control instruction, and voice interaction can be achieved by executing the control instruction.
[0050] In the method provided in the embodiment of the present invention, based on the semantic matching degree between the transcribed text of the user's voice and the icon images of each icon control, the icon control matching the transcribed text is determined from each icon control, which can support a highly free, flexible and diverse reference description method of the icon control. On this basis, voice interaction is performed in combination with the transcribed text and the icon control matching the transcribed text, which can effectively ensure the success rate of voice interaction.
[0051] Based on the above embodiment, in step 220, the semantic matching degree between the transcribed text of the user voice and the icon images of the respective icon controls is respectively determined, from the respective icon controls, to match the transcribed text, including: Inputting the transcribed text of the user speech and the icon images of the icon controls into a picture-text matching model, obtaining a picture-text matching result output by the picture-text matching model based on the semantic matching degree between the transcribed text and the icon images, respectively, wherein the picture-text matching result indicates the icon control matching the transcribed text; The image-text matching model is trained based on sample texts and sample images.
[0052] Specifically, after obtaining the transcribed text of the user's voice and the icon images of each icon control, a pre-trained image-text matching model can be used to achieve semantic matching between the transcribed text and each icon image, and then an icon control that matches the transcribed text can be selected from each icon control.
[0053] Here, the image-text matching model can use the transcribed text of the user's voice and the icon images of each icon control as model input, respectively calculate the semantic matching degree between the transcribed text input into the image-text matching model and each icon image, and generate the image-text matching result output based on the semantic matching degree between the transcribed text and each icon image.
[0054] Here, the image-text matching result output by the image-text matching model can be obtained. Here, the image-text matching result can include the matching results of the transcribed text and each icon control, or can also include the control identifier of the icon control matching the transcribed text, which is not specifically limited in the embodiment of the present invention.
[0055] Before applying the image-text matching model to determine the icon control that matches the transcribed text, it is also necessary to obtain the image-text matching model through model training. Here, the training of the image-text matching model can be implemented based on sample text and sample images. Furthermore, the sample text and sample images can be combined into positive samples and negative samples, where the positive samples include semantically matching sample text and sample images, and the negative samples include semantically different sample text and sample images.
[0056] In the process of training the image-text matching model, an initial model can be obtained first. The initial model here can be a model after parameter initialization, or a model that has been pre-trained and has semantic feature capabilities. Subsequently, the positive sample can be input into the initial model to obtain the image-text matching result for the positive sample output by the initial model, and the negative sample can be input into the initial model to obtain the image-text matching result for the negative sample output by the initial model. Then, based on the difference between the image-text matching result for the positive sample and the conclusion that the positive sample itself matches, and the difference between the image-text matching result for the negative sample and the conclusion that the negative sample itself does not match, the loss function can be calculated, and the parameters of the initial model can be iterated based on the loss function to obtain the image-text matching model.
[0057] In the method provided in the embodiment of the present invention, an icon control matching the transcribed text is determined from each icon control through a picture-text matching model, so that the matching between the transcribed text and the icon control is more reliable and efficient, which helps to improve the efficiency of voice interaction.
[0058] Based on any of the above embodiments, the image-text matching model is a text-image pre-training model.
[0059] Here, text-image pre-trained models are a type of pre-trained models in the intersection of natural language processing (NLP) and computer vision (CV), such as CLIP (Contrastive Language-Image Pre-training) and ViLT (Vision-and-Language Transformer).
[0060] The text-image pre-training model can map text and images into a shared embedding space by jointly training on large-scale text and image datasets. In this embedding space, the semantic features of the text and images input into the text-image pre-training model can be represented in a similar way, so that the similarity between the semantic features of the text and the semantic features of the image can be quantified. This can realize the calculation of the semantic matching degree between the transcribed text and the icon image, and then determine the icon control that matches the transcribed text from each icon control.
[0061] Based on any of the above embodiments, in step 220, obtaining the semantic matching degree includes: Extracting semantic features of the transcribed text of the user's speech to obtain text semantic features; Extracting semantic features of the icon image of the icon control to obtain image semantic features, wherein the text semantic features and the image semantic features are in the same feature space; The semantic matching degree is determined based on the similarity between the text semantic feature and the image semantic feature.
[0062] Specifically, in order to determine the icon control that matches the transcribed text from each icon control, it is necessary to first determine the semantic matching degree between the transcribed text and each icon image. In an embodiment of the present invention, for an icon image of any icon control, a method for calculating the semantic matching degree between the transcribed text and the icon image may include the following steps: First, the semantic features of the transcribed text and the semantic features of the icon image are extracted respectively. For ease of explanation, in the embodiment of the present invention, the semantic features of the transcribed text are recorded as text semantic features, and the semantic features of the icon image are recorded as image semantic features. Here, the text semantic features are used to reflect the characteristics of the transcribed text in semantic expression, and the image semantic features are used to reflect the characteristics of the content of the icon image in semantic expression.
[0063] Furthermore, in order to ensure that the similarity between the semantic features of the text and the semantic features of the image is quantifiable, when extracting the semantic features of the transcribed text and the semantic features of the icon image respectively, it is required that the extracted text semantic features and image semantic features are in the same feature space, that is, the transcribed text and the icon image need to be mapped into the same, comparable semantic space.
[0064] In order to ensure that the extracted text semantic features and image semantic features are in the same feature space, the feature extraction model can be trained in a contrastive learning manner for both text and image modalities. That is, during the training process, the distance between the semantic features of mutually matching text and images extracted by the feature extraction model is made as small as possible through contrastive learning, and the distance between the semantic features of mismatching text and images extracted by the feature extraction model is made as large as possible. In this way, the extraction of text semantic features and image semantic features can be realized through the trained feature extraction model.
[0065] In some embodiments, the text-image pre-training model may be used as a feature extraction model to extract text semantic features and image semantic features in the same feature space.
[0066] After extracting the text semantic features and the image semantic features respectively, the similarity between the text semantic features and the image semantic features can be calculated, and the calculated similarity is used as the semantic matching degree between the transcribed text and the icon image. Here, the calculation of the similarity can be achieved by cosine similarity, Euclidean distance or other distance measurement methods, which is not specifically limited in the embodiment of the present invention.
[0067] Based on any of the above embodiments, obtaining the icon image includes: Obtaining a storage link of an icon image corresponding to the icon control; Based on the storage link, the icon image is obtained.
[0068] Specifically, after determining each icon control displayed on the interface when receiving the user voice, it is necessary to obtain the icon image of each icon control to achieve a match between the transcribed text of the user voice and the icon image.
[0069] In order to obtain the icon image, the storage link of the corresponding icon image can be associated with each icon control in advance. That is, the icon image of each icon control can be stored in advance, and the storage link of the icon image is generated based on the address where the icon image is stored. It can be understood that each icon control has a corresponding storage link, and the storage link corresponding to each icon control points to the storage address of the icon image of the icon control, that is, the storage link corresponding to each icon control is used to obtain the icon image of the icon control.
[0070] The storage link of each icon control may be pre-stored locally, or the storage link of each icon control may be included in the control properties of each icon control, so that after determining each icon control displayed on the interface, the storage link corresponding to each icon control may be quickly obtained.
[0071] After obtaining the storage link corresponding to each icon control, the icon image of each icon control can be obtained based on the storage link corresponding to each icon control. Further, the icon image of each icon control can be obtained based on the storage address of the icon image pointed to by the storage link. Here, the storage link can point to a local or cloud, and the embodiment of the present invention does not specifically limit this.
[0072] In the method provided in the embodiment of the present invention, the icon image of the icon control is obtained through the storage link of the icon image corresponding to the icon control. Compared with the method of performing target recognition on the interface image to realize the recognition and segmentation of the icon image, only very little computing power is consumed to quickly obtain the icon image, which helps to ensure the battery life of the device and greatly improves the response efficiency of voice interaction.
[0073] Based on any of the above embodiments, with respect to obtaining the icon image, the step of obtaining a storage link of the icon image corresponding to the icon control includes: The storage link of the icon image is read from the control properties of the icon control.
[0074] Specifically, the storage link of the icon image can be written into the control properties of the icon control in advance. Here, the control properties are used to define the appearance, behavior and other properties of the icon control.
[0075] In the case where the storage link of the icon image is pre-written into the control property of the icon control, after the icon control is determined, the control property of the icon control can be directly accessed to thereby read the pre-written storage link of the icon image from the control property.
[0076] It should be noted that, compared with the solution of directly storing the storage link of the icon image locally, in the embodiment of the present invention, the storage link of the icon image is written into the control property of the icon control, so that the storage link of the icon image is read from the control property, which realizes direct access and reading of the storage link, without the need for relatively cumbersome operations such as opening files and querying databases, and can significantly improve the convenience and efficiency of obtaining the storage link. In addition, the control property is usually closely related to the state of the icon control itself. Reading the storage link of the icon image from the control property can avoid the problem that the storage link of the icon image stored locally fails to be updated in time or corresponds to the wrong icon control, thereby ensuring the reliability of storage link acquisition.
[0077] Based on any of the above embodiments, with respect to obtaining the icon image, obtaining the icon image based on the storage link includes: Based on the storage link, the icon image is downloaded from the cloud, and the cloud is used to store the icon image of each icon control.
[0078] Specifically, the icon images of various icon controls can be uniformly stored and managed in the cloud. The cloud can generate an independent storage link pointing to the cloud in advance for the icon image of each icon control. Therefore, after obtaining the storage link, the icon image can be downloaded from the cloud based on the storage address of the cloud pointed to by the storage link.
[0079] It should be noted that compared with the solution of directly storing icon images locally, storing icon images in the cloud can more conveniently realize real-time updates of icon images, thereby ensuring that the icon images downloaded from the cloud are updated icon images that are consistent with the interface display, avoiding the acquisition of unupdated icon images that are different from the interface display, and further ensuring the reliability of icon controls that match the transcribed text based on the semantic matching between the transcribed text and the icon image.
[0080] Based on any of the above embodiments, Figure 3 It is a structural diagram of the icon management system provided by the present invention. Figure 3 As shown, the acquisition of icon images can be achieved based on the icon management system in the cloud. Figure 3 In the example, the cloud-based icon management system can realize the unified management of the icon controls displayed on the interface. Specifically, the icon images of each icon control can be stored in the cloud-based icon management system, and independent storage links can be generated for the icon images of each icon control, and the storage links can be written into the control properties of the corresponding icon controls.
[0081] Therefore, during the voice interaction process, after determining the icon controls displayed on the interface, the storage link of the icon image of each icon control can be read from the control properties of each icon control, and based on the storage link, the icon image of each icon control can be downloaded from the icon management system in the cloud, thereby realizing the acquisition of the icon image.
[0082] Based on any of the above embodiments, in step 210, determining each icon control displayed on the interface when receiving the user voice includes: When a change in the interface is detected, the icon controls displayed on the interface are scanned, and the icon controls scanned when the interface last changed when receiving the user voice are used as the icon controls displayed on the interface when receiving the user voice.
[0083] Specifically, in order to obtain each icon control displayed on the interface when receiving the user's voice, it is possible to detect in real time whether the interface changes.
[0084] When a change in the interface is detected, the changed interface can be scanned to obtain the icon controls displayed on the changed interface. Here, the scanning method can be implemented through the Accessibility Service, or through other control scanning methods, such as a custom scanning script, or an interface testing library.
[0085] By scanning the icon controls displayed on the interface after the change is detected each time, it can be ensured that the icon controls obtained each time can reflect the latest control display status on the interface. Therefore, when receiving user voice, the icon controls scanned after the most recent interface change can be used as the icon controls displayed on the interface when receiving user voice.
[0086] In the method provided in an embodiment of the present invention, the icon control is scanned each time a change in the interface is detected, thereby avoiding frequent scanning and consumption of computing power while ensuring that the scanned icon control can reflect the latest interface control display status, so that the icon control to be matched can be directly obtained when the user voice is received.
[0087] Based on any of the above embodiments, in step 210, determining each icon control displayed on the interface when receiving the user voice also includes: During the voice interaction process, detect whether the interface changes.
[0088] Specifically, in order to avoid frequently monitoring whether the interface has changed and scanning icon controls after changes, which puts pressure on device battery life, it is possible to detect whether the interface has changed only during voice interaction.
[0089] That is, when the voice interaction starts, for example, when it is detected that the user's voice contains a wake-up word, or when it is detected that the user turns on the microphone by pressing a button or operating a control, the user's voice can be received and detection of whether the interface has changed can be started at the same time until the voice interaction ends.
[0090] Furthermore, when the voice interaction begins, the icon controls displayed on the interface can be scanned first, and then each time a change in the interface is detected, the icon controls displayed on the changed interface can be scanned to ensure that the scanned icon controls can reflect the latest interface control display status.
[0091] Based on any of the above embodiments, in step 230, performing voice interaction based on the transcribed text and an icon control matching the transcribed text includes: Based on the transcribed text and the control identifier of the icon control matching the transcribed text, generating a control instruction corresponding to the user voice; Based on the control instruction, voice interaction is performed.
[0092] Specifically, after obtaining the icon control that matches the transcribed text, the transcribed text and the control identifier of the icon control that matches it can be combined to generate a control instruction corresponding to the user's voice and used to control the icon control that matches the transcribed text. Here, the control identifier can specifically be the ID of the icon control, or the control name of the icon control, etc. The generation of the control instruction can be achieved by semantically understanding the transcribed text and the control identifier of the icon control that matches the transcribed text. For example, the transcribed text and the control ID of the icon control that matches the transcribed text can be input into a semantic understanding model, and the semantic understanding model extracts the semantics of the transcribed text and the control ID, and the extracted semantics are mapped to the control instruction output, which is not specifically limited in the embodiment of the present invention.
[0093] After the control instruction is obtained, the control instruction can be executed to realize feedback on the user's voice, that is, to realize voice interaction.
[0094] Based on any of the above embodiments, step 230, performing voice interaction based on the control instruction, includes: Based on the control identifier in the control instruction, determining a target control to be controlled from the interface; Execute the control operation of the target control corresponding to the control instruction to realize the voice interaction.
[0095] Specifically, after obtaining the control instruction, the control identifier carried in the control instruction can be matched with the identifiers of the controls existing in the interface, thereby determining the control that needs to be controlled by the control instruction from the interface, which is recorded as the target control in the embodiment of the present invention. Here, the target control is the operation object of the control instruction, and the target control may include an icon control that matches the transcribed text. In addition, the target control may also include other controls in the interface.
[0096] After determining the target control, the control instruction can be executed, thereby realizing the control operation that the user expects the device to perform on the target control in the voice. The device can respond to the operation and display the content information after the operation on the interface, thereby completing the task that the user expects the device to complete in the voice, so as to achieve feedback on the user voice, that is, realize voice interaction.
[0097] For example, when receiving the user voice "open the small house", the icon images of each icon control on the interface can be obtained, and by semantically matching "open the small house" with each icon image, the icon control "home" represented by the icon style of a house can be matched from each icon control. Subsequently, a control instruction for clicking the icon control "home" can be generated based on "open the small house" and "home", and the home control is used as the target control. By executing the control instruction, the home page is opened and displayed on the interface.
[0098] Based on any of the above embodiments, Figure 4 This is the second flow chart of the voice interaction method provided by the present invention. Figure 4 As shown, the voice interaction method may include the following steps: First, the voice interaction can be started by the user. At this time, the user's voice can be received by the voice assistant software installed on the device, and the human-computer interaction interface of the device can be detected simultaneously to see if there are any changes.
[0099] Secondly, when voice interaction is turned on, the icon controls on the interface can be scanned through the accessibility service, and each time an interface change is detected thereafter, the icon controls on the interface can be scanned through the accessibility service, thereby obtaining the icon controls displayed on the interface when receiving user voice.
[0100] Next, voice recognition is performed on the user voice received by the voice assistant software to obtain a transcribed text of the user voice.
[0101] In addition, for the icon controls displayed on the acquired interface, the storage link of the icon image can be read from the control properties of the icon control, and the icon image can be downloaded from the cloud application storage link of the voice assistant software to obtain the icon image of each icon control.
[0102] Subsequently, the transcribed text obtained by speech recognition and the icon images of each icon control can be input into the text-image pre-training model, and the text-image pre-training model calculates the semantic matching degree between the transcribed text and each icon image, thereby outputting the matching result, that is, the icon image matching the transcribed text, that is, obtaining the icon control matching the transcribed text. Here, the accuracy of the matching result can be adjusted by the threshold inside the text-image pre-training model.
[0103] Next, semantic understanding may be performed on the transcribed text and the ID of the icon control that matches the transcribed text, thereby obtaining a control instruction corresponding to the user voice.
[0104] Finally, the control instructions corresponding to the control operations of the icon controls can be executed, thereby realizing voice interaction.
[0105] In the voice interaction method provided in the embodiment of the present invention, the icon control on the interface is scanned by the accessibility service, the icon image of the icon control is obtained from the cloud through the storage link, and the transcribed text of the user voice and the icon image of the icon control are semantically matched through the text-image pre-training model to obtain an icon control matching the transcribed text, and the control instructions are obtained by semantically understanding the transcribed text and the matching icon control to realize voice interaction, thereby solving the problem of low success rate of icon-based voice interaction and improving the freedom of icon-based voice interaction.
[0106] The voice interaction device provided by the present invention is described below. The voice interaction device described below and the voice interaction method described above can be referenced to each other.
[0107] Figure 5 is a structural diagram of the voice interaction device provided by the present invention, such as Figure 5 As shown, the device comprises: The acquisition unit 510 is used to receive the user voice and determine each icon control displayed on the interface when receiving the user voice; A matching unit 520, configured to determine, from among the icon controls, an icon control that matches the transcribed text based on the semantic matching between the transcribed text of the user speech and the icon images of the icon controls; The interaction unit 530 is used to perform voice interaction based on the transcribed text and an icon control matching the transcribed text.
[0108] In the device provided by the embodiment of the present invention, based on the semantic matching degree between the transcribed text of the user's voice and the icon images of each icon control, the icon control matching the transcribed text is determined from each icon control, which can support a highly free, flexible and diverse reference description method of the icon control. On this basis, voice interaction is performed in combination with the transcribed text and the icon control matching the transcribed text, which can effectively ensure the success rate of voice interaction.
[0109] Based on any of the above embodiments, the matching unit is specifically used for: Inputting the transcribed text of the user speech and the icon images of the icon controls into a picture-text matching model, obtaining a picture-text matching result output by the picture-text matching model based on the semantic matching degree between the transcribed text and the icon images, respectively, wherein the picture-text matching result indicates the icon control matching the transcribed text; The image-text matching model is trained based on sample texts and sample images.
[0110] Based on any of the above embodiments, the matching unit is further used for: Extracting semantic features of the transcribed text of the user's speech to obtain text semantic features; Extracting semantic features of the icon image of the icon control to obtain image semantic features, wherein the text semantic features and the image semantic features are in the same feature space; The semantic matching degree is determined based on the similarity between the text semantic feature and the image semantic feature.
[0111] Based on any of the above embodiments, the device further includes a picture acquisition unit, which is used to: Obtaining a storage link of an icon image corresponding to the icon control; Based on the storage link, the icon image is obtained.
[0112] Based on any of the above embodiments, the picture acquisition unit is specifically used for: The storage link of the icon image is read from the control properties of the icon control.
[0113] Based on any of the above embodiments, the picture acquisition unit is specifically used for: Based on the storage link, the icon image is downloaded from the cloud, and the cloud is used to store the icon image of each icon control.
[0114] Based on any of the above embodiments, the acquisition unit is specifically used for: When a change in the interface is detected, the icon controls displayed on the interface are scanned, and the icon controls scanned after the last change in the interface when receiving the user voice are used as the icon controls displayed on the interface when receiving the user voice.
[0115] Based on any of the above embodiments, the acquiring unit is further used for: During the voice interaction process, detect whether the interface changes.
[0116] Based on any of the above embodiments, the interaction unit is specifically used for: Based on the transcribed text and the control identifier of the icon control matching the transcribed text, generating a control instruction corresponding to the user voice; Based on the control instruction, voice interaction is performed.
[0117] Based on any of the above embodiments, the interaction unit is specifically used for: Based on the control instruction
[0118] The control identifier in the interface determines the target control to be controlled from the interface; Execute the control operation of the target control corresponding to the control instruction to realize the voice interaction.
[0119] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the voice interaction method, which includes: Receive user voice, and determine icon controls displayed on the interface when receiving the user voice; Based on the semantic matching degree between the transcribed text of the user speech and the icon images of the icon controls, determining the icon control matching the transcribed text from the icon controls; Based on the transcribed text and an icon control matching the transcribed text, voice interaction is performed.
[0120] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0121] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the voice interaction method provided by the above methods, the method includes: Receive user voice, and determine icon controls displayed on the interface when receiving the user voice; Based on the semantic matching degree between the transcribed text of the user speech and the icon images of the icon controls, determining the icon control matching the transcribed text from the icon controls; Based on the transcribed text and an icon control matching the transcribed text, voice interaction is performed.
[0122] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the voice interaction method provided by the above methods, the method comprising: Receive user voice, and determine icon controls displayed on the interface when receiving the user voice; Based on the semantic matching degree between the transcribed text of the user speech and the icon images of the icon controls, determining the icon control matching the transcribed text from the icon controls; Based on the transcribed text and an icon control matching the transcribed text, voice interaction is performed.
[0123] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.
[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A voice interaction method, characterized in that: include: Receive user voice, and determine icon controls displayed on the interface when receiving the user voice; Based on the semantic matching degree between the transcribed text of the user speech and the icon images of the icon controls, determining the icon control matching the transcribed text from the icon controls; Based on the transcribed text and an icon control matching the transcribed text, voice interaction is performed.
2. The voice interaction method according to claim 1, characterized in that: The determining, from the icon controls, an icon control matching the transcribed text based on the semantic matching degree between the transcribed text of the user speech and the icon images of the icon controls, comprises: Inputting the transcribed text of the user speech and the icon images of the icon controls into a picture-text matching model, obtaining a picture-text matching result output by the picture-text matching model based on the semantic matching degree between the transcribed text and the icon images, respectively, wherein the picture-text matching result indicates the icon control matching the transcribed text; The image-text matching model is trained based on sample texts and sample images.
3. The voice interaction method according to claim 1, characterized in that: The acquisition of the semantic matching degree includes: Extracting semantic features of the transcribed text of the user's speech to obtain text semantic features; Extracting semantic features of the icon image of the icon control to obtain image semantic features, wherein the text semantic features and the image semantic features are in the same feature space; The semantic matching degree is determined based on the similarity between the text semantic feature and the image semantic feature.
4. The voice interaction method according to claim 1, characterized in that: The acquisition of the icon image includes: Obtaining a storage link of an icon image corresponding to the icon control; Based on the storage link, the icon image is obtained.
5. The voice interaction method according to claim 4, characterized in that: The step of obtaining a storage link of an icon image corresponding to the icon control includes: The storage link of the icon image is read from the control properties of the icon control.
6. The voice interaction method according to claim 4, characterized in that: The acquiring the icon image based on the storage link includes: Based on the storage link, the icon image is downloaded from the cloud, and the cloud is used to store the icon image of each icon control.
7. The voice interaction method according to claim 1, characterized in that: The determining of each icon control displayed on the interface when receiving the user voice includes: When a change in the interface is detected, the icon controls displayed on the interface are scanned, and the icon controls scanned after the last change in the interface when receiving the user voice are used as the icon controls displayed on the interface when receiving the user voice.
8. The voice interaction method according to claim 8, characterized in that: The step of determining icon controls to be displayed on the interface when receiving the user voice also includes: During the voice interaction process, detect whether the interface changes.
9. The voice interaction method according to any one of claims 1 to 8, characterized in that: The voice interaction based on the transcribed text and the icon control matching the transcribed text includes: Based on the transcribed text and the control identifier of the icon control matching the transcribed text, generating a control instruction corresponding to the user voice; Based on the control instruction, voice interaction is performed.
10. The voice interaction method according to claim 9, characterized in that: The performing voice interaction based on the control instruction includes: Based on the control identifier in the control instruction, determining a target control to be controlled from the interface; Execute the control operation of the target control corresponding to the control instruction to realize the voice interaction.
11. A voice interaction device, characterized in that: include: An acquisition unit, configured to receive a user voice and determine icon controls to be displayed on the interface when receiving the user voice; A matching unit, configured to determine, from among the icon controls, an icon control that matches the transcribed text based on the semantic matching between the transcribed text of the user speech and the icon images of the icon controls; An interaction unit is used to perform voice interaction based on the transcribed text and an icon control matching the transcribed text.
12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the voice interaction method as described in any one of claims 1 to 10 is implemented.
13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the voice interaction method according to any one of claims 1 to 10 is implemented.
14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the voice interaction method according to any one of claims 1 to 10 is implemented.
