Voice control method, device, electronic device, storage medium and vehicle

Abstract

The present application relates to a voice control method, device, electronic device, storage medium and vehicle. The voice control method obtains voice instructions, obtains a screen display image for the voice instructions, and performs screen recognition based on the screen display image to obtain screen content information, so as to determine the operation intention according to the voice instructions and the screen content information, and then performs operation control based on the operation intention in combination with the screen display image to obtain a voice control result corresponding to the voice instruction, so that voice control can be performed in combination with a specific screen display image, thereby solving the problem of low user voice control flexibility in the voice control method in the existing related technical means, and can effectively improve the flexibility of voice control.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a voice control method, device, electronic equipment, storage medium and vehicle. Background Art

[0002] With the development of society, voice control devices have become a trend. Nowadays, all kinds of control devices have realized voice control, such as televisions, car systems, mobile devices, etc., and the existing voice control implementation method is usually implemented by preset voice control commands. Taking vehicles as an example, the current existing vehicle car systems usually have voice control functions, which provides great convenience for users in the car during driving. However, the existing voice control technology mainly relies on a preset voice command library, and performs voice control through pre-configured voice commands and corresponding control logic, such as controlling the air conditioning switch, turning on Bluetooth, etc. This means that the car system can only recognize and execute limited pre-configured commands. If the voice spoken by the user does not match the content in the pre-made command library, the voice control will not work, that is, there is a problem of low flexibility of user voice control. Summary of the invention

[0003] The present application provides a voice control method, device, electronic device, storage medium and vehicle to solve the problem of low user voice control flexibility in the voice control methods in the existing related technical means.

[0004] In a first aspect, the present application provides a voice control method, comprising:

[0005] Get voice commands;

[0006] In response to the voice command, obtaining a screen display image;

[0007] Performing screen recognition based on the screen display image to obtain screen content information;

[0008] Determining an operation intention according to the voice command and the screen content information;

[0009] Based on the operation intention, operation control is performed in combination with the screen display image to obtain a voice control result corresponding to the voice instruction.

[0010] Optionally, acquiring a screen display image in response to the voice command includes:

[0011] Determining the command type corresponding to the voice command;

[0012] In a case where the instruction type is a preset screen operation instruction type, the screen display image is acquired.

[0013] Optionally, after determining the instruction type corresponding to the voice instruction, the method further includes:

[0014] When the instruction type is an instruction type of a preset instruction library, obtaining a preset voice response strategy;

[0015] The preset voice response strategy is adopted to respond to the voice instruction and obtain the voice control result.

[0016] Optionally, performing screen recognition based on the screen display image to obtain screen content information includes:

[0017] Extracting display elements from the screen display image to obtain element type information and position logic information corresponding to each display element;

[0018] The screen content information is generated according to the element type information and the position logic information.

[0019] Optionally, the generating the screen content information according to the element type information and the position logic information includes:

[0020] Determining an element identification strategy corresponding to the element type information;

[0021] When the element recognition strategy is a text recognition strategy, text is extracted from the screen display image according to the position logic information to obtain screen text information;

[0022] The screen content information is generated according to the screen text information.

[0023] Optionally, after determining the element identification strategy corresponding to the element type information, the method further includes:

[0024] In the case where the element recognition strategy is an image recognition strategy, performing image extraction on the screen display image according to the position logic information to obtain icon information and button information;

[0025] The screen content information is generated according to the icon information and the button information.

[0026] Optionally, determining the operation intention according to the voice command and the screen content information includes:

[0027] Performing semantic analysis on the voice command to obtain the command text information;

[0028] Extracting screen text information, icon information and button information from the screen content information;

[0029] The operation intention is determined based on the instruction text information in combination with the screen text information, the icon information and the button information.

[0030] Optionally, the determining the operation intention based on the instruction text information in combination with the screen text information, the icon information, and the button information includes:

[0031] Comparing the instruction text information with the screen text information to obtain a first comparison result;

[0032] When the first comparison result is a matching result, extracting target operation information from the instruction text information, and determining an operation object according to the screen text information;

[0033] The operation intention is generated based on the target operation information and the operation object.

[0034] Optionally, after comparing the instruction text information with the screen text information to obtain a first comparison result, the method further includes:

[0035] When the first comparison result is a mismatch result, the instruction text information is compared with the icon information and the button information to obtain a second comparison result;

[0036] If the second comparison result is a matching result, extracting target operation information from the instruction text information, and determining an operation object according to the icon information and the button information;

[0037] The operation intention is generated based on the target operation information and the operation object.

[0038] Optionally, performing operation control based on the operation intention in combination with the screen display image to obtain a voice control result corresponding to the voice command includes:

[0039] Extracting target operation information and operation object from the operation intention;

[0040] Based on the screen display image, the operation corresponding to the target operation information is performed on the operation object to obtain the voice control result.

[0041] In a second aspect, the present application provides a voice control device, comprising:

[0042] A first acquisition module, used to acquire voice instructions;

[0043] A second acquisition module is used to acquire a screen display image according to the voice command;

[0044] A recognition module, used for performing screen recognition based on the screen display image to obtain screen content information;

[0045] A determination module, used to determine the operation intention according to the voice instruction and the screen content information;

[0046] The control module is used to perform operation control based on the operation intention and in combination with the screen display image to obtain a voice control result corresponding to the voice instruction.

[0047] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0048] Memory, used to store computer programs;

[0049] The processor is used to implement the voice control method described in any one of the first aspects when executing the program stored in the memory.

[0050] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the voice control method as described in any one of the first aspects is implemented.

[0051] In a fifth aspect, a vehicle is provided, wherein the vehicle is used to execute the voice control method as described in any one of the first aspects.

[0052] The embodiment of the present application obtains a voice command, obtains a screen display image for the voice command, and performs screen recognition based on the screen display image to obtain screen content information, so as to determine the operation intention according to the voice command and the screen content information, and then performs operation control based on the operation intention in combination with the screen display image to obtain a voice control result corresponding to the voice command, so that voice control can be performed in combination with a specific screen display image, thereby solving the problem of low user voice control flexibility in the voice control method in the existing related technical means, and can effectively improve the flexibility of voice control. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A flow chart of a voice control method provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of an application scenario of a voice control method provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of the structure of a voice control device provided in an embodiment of the present application;

[0056] Figure 4A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0058] Voice control, also known as voice command or voice recognition technology, refers to the technology of controlling devices or performing specific operations through voice signals, so that users can interact with computers, smart phones, smart homes, vehicle system devices, etc. by speaking, without using keyboards, mice or other input devices. Voice control is particularly important for vehicle systems, because the driver of the vehicle needs to pay continuous and high attention. When it is necessary to interact with the vehicle system, direct manual operation will cause driving problems, so voice control becomes the driver's main control method at this time; however, the voice control of existing vehicle systems is mainly controlled by pre-set voice command libraries. Users need to remember specific voice commands and use them in appropriate scenarios. For example, users may need to say "Navigate to home" to start the navigation function, or say "Play music" to play music; however, when the user speaks a voice that does not belong to the preset voice command library, it cannot play a control role, such as "click on the yellow cat", it cannot be recognized and effectively controlled, that is, there is a problem of low flexibility of voice control.

[0059] In order to solve the problem of low user voice control flexibility in the voice control methods in the existing related technical means, the present application provides a voice control method, device, electronic device, storage medium and vehicle, by obtaining a voice command, obtaining a screen display image for the voice command, and performing screen recognition based on the screen display image to obtain screen content information, so as to determine the operation intention according to the voice command and the screen content information, and then perform operation control based on the operation intention in combination with the screen display image to obtain a voice control result corresponding to the voice command, so that voice control can be performed in combination with a specific screen display image, thereby solving the problem of low user voice control flexibility in the voice control methods in the existing related technical means and effectively improving the flexibility of voice control.

[0060] Figure 1A flow chart of a voice control method provided for an embodiment of the present application. This method can be applied to one or more electronic devices such as vehicles, vehicle systems, clients, servers, etc. In addition, the execution subject of this method can be hardware or software. When the above-mentioned execution subject is hardware, the execution subject can be one or more of the above-mentioned electronic devices. For example, a single electronic device can execute this method, or multiple electronic devices can cooperate with each other to execute this method. When the above-mentioned execution subject is software, this method can be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made here.

[0061] like Figure 1 As shown, a voice control method provided in an embodiment of the present application may specifically include the following steps:

[0062] Step S110: Acquire voice instructions.

[0063] Among them, the voice command can represent an instruction for voice control. The specific voice command can be obtained by collecting the voice spoken by the user, or can be obtained through the server. This embodiment does not make specific limitations on this.

[0064] In addition, before obtaining the voice command, it can also include determining whether the voice control function is turned on. The specific judgment method can be to determine whether the preset voice wake-up word is collected, or to determine whether the function switch corresponding to the voice control function is in the on state. Of course, other methods can also be used to determine whether the voice control function is turned on; and only when the voice control function is turned on, the step of obtaining the voice command is executed.

[0065] Step S120: Acquire a screen display image in response to the voice command.

[0066] Specifically, after obtaining the voice command, a screen display image can be obtained in response to the voice command. The screen display image can represent an image displayed on the screen of a specific device. For example, in a car system, the image displayed on the car control screen can be obtained as the screen display image, and a smart phone can obtain the image displayed on the mobile phone screen as the screen display image. Of course, it can also be an image displayed on the screen of other devices, and this embodiment does not make any specific limitations on this.

[0067] Specifically, the screen display image acquired in this embodiment can be one or more images, that is, after acquiring the voice command, multiple images in the device screen can be acquired as screen display images; and the specific acquisition method can be that after acquiring the voice command, at least two images displayed on the device screen are periodically acquired at a preset time interval, and the at least two images are determined as screen display images, so as to avoid the problem of failing to acquire accurate and effective images when the display content of the device screen changes frequently, and to achieve the acquisition of multiple images displayed on the screen at different times; it can also be that after acquiring the voice command, the current display image and the background application image in the device screen are acquired, and the current display image and the background application image can be determined as screen display images, so as to avoid the problem of large application control limitations that the user needs to manually switch or pre-voice control to call the background application before controlling the application when he expects to control the background application, and can effectively control the background application, greatly reducing the control limitations. Of course, other methods can also be used to acquire multiple images as screen display images, and this embodiment does not specifically limit this.

[0068] Step S130: Perform screen recognition based on the screen display image to obtain screen content information.

[0069] Specifically, after obtaining the screen display image, screen recognition can be performed based on the screen display image to obtain screen content information, wherein screen recognition can represent display element identification, extraction, classification, and other processing on the screen display image, and the content information obtained by performing screen recognition on the screen display image is determined as the screen content information, that is, the screen content information can represent the content information corresponding to each display element contained in the screen display image, wherein the display elements can include but are not limited to buttons, icons, text, posters, etc.

[0070] Step S140: Determine the operation intention according to the voice command and the screen content information.

[0071] Specifically, after obtaining the screen content information, since the screen content information may include content information corresponding to one or more display elements, and the voice command is an instruction to operate the content on the screen, the operation intention can be determined based on the voice command and the screen content information. The operation intention can indicate the intention to perform specific operations on certain content on the screen.

[0072] Specifically, the voice command is "click on the kitten on the screen". At this time, the screen content information includes content information corresponding to the display element being the back button and content information corresponding to the display element being the kitten icon. At this time, according to the voice command and the screen content information, the operation intention can be determined to be the intention to click on the kitten icon. Of course, the above is only an example for illustration, and can be adaptively adjusted according to the scene and needs during specific implementation.

[0073] Step S150: Based on the operation intention, the operation control is performed in combination with the screen display image to obtain a voice control result corresponding to the voice instruction.

[0074] Specifically, after determining the operation intention, operation control can be performed based on the operation intention and combined with the screen display image to obtain a voice control result corresponding to the voice instruction, where the operation control can represent the control instruction corresponding to the operation intention being input on the display interface corresponding to the screen display image, and the voice control result can represent the control result obtained after the display interface corresponding to the screen display image executes the control instruction corresponding to the operation intention.

[0075] It can be seen that after obtaining the voice command, this embodiment obtains the screen display image according to the voice command, and performs screen recognition based on the screen display image to obtain screen content information, so as to determine the operation intention according to the voice command and the screen content information, and then performs operation control based on the operation intention in combination with the screen display image to obtain the voice control result corresponding to the voice command, so that the user is no longer limited to the preset voice command library during the voice control process. Even if a voice that does not belong to the preset voice command library is spoken, the execution steps of this embodiment can also be combined with the specific screen display image for voice control, thereby solving the problem of low user voice control flexibility in the voice control method in the existing related technical means, and can effectively improve the flexibility of voice control.

[0076] In an optional embodiment of the present application, step S120 obtains a screen display image for a voice command, which may specifically include the following steps: determining the instruction type corresponding to the voice command; and when the instruction type is a preset screen operation instruction type, obtaining a screen display image.

[0077] In this embodiment, after a voice instruction, the instruction type corresponding to the voice instruction can be determined, and the instruction type indicates that the voice instruction corresponds to a preset type, such as a preset voice instruction library type, a screen operation type, etc., wherein the preset voice instruction library type indicates that the voice instruction belongs to the type of preset instructions in the preset voice instruction library, and the screen operation type can indicate that the voice instruction belongs to the type that requires an operation on the screen display content; thereby, it can be determined whether the instruction type is a preset screen operation instruction type, wherein the preset screen operation instruction type can be a pre-configured type for operating the screen, and in the case where the instruction type is the preset screen operation instruction type, it means that the voice instruction at this time needs to operate the screen display type, and then the screen display image can be obtained.

[0078] Specifically, in this implementation, deep learning models and natural language processing technologies can be used in the process of determining the instruction type corresponding to a voice instruction. Specifically, after receiving the user's voice instruction, the audio signal can be converted into text information through voice recognition technology, and then the text information can be semantically analyzed through natural language processing technology to obtain semantic information. Subsequently, a deep learning model is used to classify the semantic information to determine the instruction type corresponding to the voice instruction.

[0079] In an optional embodiment of the present application, after determining the instruction type corresponding to the voice instruction, the following steps may be specifically included: when the instruction type is an instruction type of a preset instruction library, obtaining a preset voice response strategy; using the preset voice response strategy to respond to the voice instruction and obtain a voice control result.

[0080] After determining the instruction type corresponding to the voice instruction, if the instruction type is not a preset screen operation instruction type, this embodiment can determine whether the instruction type is a preset instruction library instruction type, where the preset instruction library instruction type can be an instruction type pre-configured in the voice instruction library. Since the preset voice instruction library type indicates that the voice instruction belongs to the type of preset instructions in the preset voice instruction library, when the instruction type is the preset instruction library instruction type, a preset voice response strategy can be obtained. The preset voice response strategy can indicate a strategy for calling a preset voice instruction library to respond to the voice instruction; that is, the preset voice response strategy can be used to directly respond to the voice instruction to obtain a voice control result.

[0081] Specifically, when the voice command is "turn on the air conditioner", the preset voice command library contains the preset command corresponding to turning on the air conditioner, then it can be determined that the voice command belongs to the preset command library command type, and the preset voice response strategy can be adopted at this time, and the preset voice command library is called to determine that the preset command corresponding to the voice command is the command to turn on the air conditioner. At this time, the voice command can be responded to to execute the command to turn on the air conditioner and obtain the voice control result; when the voice command is "click on the kitten on the screen", the preset voice command library does not contain the preset command corresponding to the voice command, and at this time it can be determined that the command type is the preset screen operation command type, and the screen display image is obtained to execute the contents of subsequent steps S130-S150.

[0082] It can be seen that in this embodiment, by classifying the instruction types of voice instructions, when the instruction type is a preset instruction library instruction type, the preset voice instruction library can be used to respond to the voice instruction, and when the instruction type is a preset screen operation instruction type, the screen display image can be obtained to execute the contents of subsequent steps S130-S150, so that the user is no longer limited to the preset voice instruction library during the voice control process. Even if a voice that does not belong to the preset voice instruction library is spoken, voice control can be performed in combination with a specific screen display image through the execution steps of this embodiment, thereby solving the problem of low user voice control flexibility in the voice control method in the existing related technical means, and can effectively improve the flexibility of voice control.

[0083] In an optional embodiment of the present application, step S130 performs screen recognition based on the screen display image to obtain screen content information, which may specifically include the following steps: extracting display elements from the screen display image to obtain element type information and position logic information corresponding to each display element; generating screen content information based on the element type information and position logic information.

[0084] After obtaining the screen display image, this embodiment can extract display elements from the screen display image to obtain element type information and position logic information corresponding to each display element. The display element can represent a preset element contained in the image, such as a button, an icon, text, a poster, etc., and extracting the display elements from the screen display image is to extract data corresponding to each display element from the screen display image, thereby obtaining element type information and position logic information corresponding to each display element, wherein the element type information can represent the specific type of the display element, and the position logic information can represent the specific position information of the display element in the screen display image, such as the row and column, the first, the last, the upper left corner, and other position logic information; thereby, the element type information and the position logic information can be integrated to generate screen content information, thereby representing the data corresponding to each display element contained in the screen display image through the screen content information.

[0085] In an optional embodiment of the present application, screen content information is generated based on element type information and position logic information, which may specifically include the following steps: determining an element recognition strategy corresponding to the element type information; when the element recognition strategy is a text recognition strategy, extracting text from the screen display image based on the position logic information to obtain screen text information; and generating screen content information based on the screen text information.

[0086] In the process of generating screen content information according to element type information and position logic information, the present embodiment can first determine the element recognition strategy corresponding to the element type information. The element recognition strategy indicates the strategy for generating screen content information, that is, different element type information can correspond to different element recognition strategies, and different element recognition strategies can adopt different strategies to generate element recognition strategies. For example, when the element type information is a text type, the element recognition strategy corresponding to the element type information is a text recognition strategy. When the element type information is an image type, the element recognition strategy corresponding to the element type information is an image recognition strategy. The present embodiment does not make specific limitations on this. Then, it can be determined whether the element recognition strategy is a text recognition strategy. The text recognition strategy indicates the strategy for identifying and extracting text. When the element recognition strategy is a text recognition strategy, text extraction can be performed on the screen display image according to the position logic information to obtain screen text information. Specifically, since the position logic information can indicate the specific position information of the display element in the screen display image, when text extraction is performed on the screen display image, the specific text corresponding to the display element can be extracted at the position indicated by the position logic information, and the screen text information can be obtained according to the specific text, and then the screen content information can be generated based on the screen text information.

[0087] Specifically, the specific method of extracting text from the screen display image to obtain the screen text information may be optical character recognition technology, vectorizing the text in the screen display image to obtain text vector features, and obtaining the screen text information based on the text vector features. Of course, other methods may also be used, and this embodiment does not specifically limit this; for example Figure 2 The screen display image shown, the extracted screen text information includes the text of the poster title 1-10, and the text 1 and the text 2 in the picture.

[0088] In an optional embodiment of the present application, after determining the element identification strategy corresponding to the element type information, the following steps may be specifically included: when the element identification strategy is an image identification strategy, image extraction is performed on the screen display image based on the position logic information to obtain icon information and button information; based on the icon information and button information, screen content information is generated.

[0089] After determining the element identification strategy corresponding to the element type information, this embodiment can also determine whether the element identification strategy is an image identification strategy. The image identification strategy represents a strategy for extracting and identifying images. When the element identification strategy is the image identification strategy, the screen display image can be extracted based on the position logic information to obtain icon information and button information. The image extraction of the screen display image can be to extract and identify the specific image corresponding to the display element at the position represented by the position logic information to obtain icon information and button information. The icon information can represent visual element information that conveys specific functions or instructions through graphic symbols. The button information can represent a control used to trigger an event or action. The button information indicates that the control usually contains obvious labels or instructions to indicate the operations that the user can perform. The icon information and button information can then be integrated to generate screen content information.

[0090] It should be noted that the screen text information may also include link information corresponding to the text, the icon information may also include link information associated with the icon, and the button information may also include link information associated with the button, wherein the continuous information may be obtained from the display interface corresponding to the screen display image based on the position logic information.

[0091] Specifically, in the process of generating screen content information based on element type information and position logic information, this embodiment divides display elements into text type and image type according to element type information, and adopts different element recognition strategies according to different element type information to obtain screen content information corresponding to different display elements. Therefore, the corresponding recognition and extraction strategy can be used for each display element in the screen display image to obtain the information it contains, so as to achieve the purpose of accurately and efficiently obtaining the screen content information corresponding to each display element.

[0092] In addition, in the specific method of extracting the image displayed on the screen to obtain the icon information and button information, it can be to use the multimodal model of image language (CLIP model) to extract image features, so as to extract features of non-text elements in the screen display image, such as a movie poster, in which the features include a woman and a cat in the poster, the woman has long golden hair and glasses sitting on the steps, and the black cat is playing beside her. The above-mentioned extracted content is vectorized to obtain image vector features, and the icon information and button information are generated according to the image vector features. Of course, other methods can also be used, and this embodiment does not specifically limit this; for example Figure 2 The screen display image shown, the extracted icon information and button information may include several setting button information at the top of the screen display image and the bottom of the screen display image, and poster icon information corresponding to the picture poster titles 1-10.

[0093] In an optional embodiment of the present application, step S140 determines the operation intention based on the voice instruction and the screen content information, and may specifically include the following steps: performing semantic analysis on the voice instruction to obtain instruction text information; extracting screen text information, icon information and button information from the screen content information; determining the operation intention based on the instruction text information, combined with the screen text information, icon information and button information.

[0094] After obtaining the screen content information, this embodiment can perform semantic analysis on the voice command to obtain command text information, and the command text information can represent the command text used to control the operation of the device; because the screen content information can represent the information corresponding to each display element in the screen display image, the screen text information, icon information and button information can be extracted from the screen content information at this time, and based on the command text information, combined with the screen text information, icon information and button information, the operation intention can be determined. The specific method of determining the operation intention can be to compare the command text information with the screen text information, icon information and button information in turn, and determine the target information that matches the command text information, so as to determine the operation intention according to the target information, and the target information is one of the screen text information, icon information and button information.

[0095] In one example, for example, when the voice command is "click on the little cat icon", the command text information obtained by voice analysis may be "click on the kitten icon". At this time, by comparing the command text information with the screen text information, icon information and button information in sequence, the kitten icon information matches the command text information. At this time, the kitten icon information can be determined as the target information, and the command text information is combined with the kitten icon information to obtain the operation intention of clicking the kitten; when the voice command is "copy the words under the kitten", the command text information obtained by voice analysis may be "copy the text at position X". At this time, by comparing the command text information with the screen text information, icon information and button information in sequence, the screen text information at position X matches the command text information. At this time, the screen text information at position X can be determined as the target information, and the command text information is combined with the screen text information at position X to obtain the operation intention of copying the screen text information at position X. Of course, the above is only an example for illustration. In specific implementation, adaptive adjustment can be made according to the scene and demand, and this embodiment does not make specific limitations on this.

[0096] In an optional embodiment of the present application, the operation intention is determined based on the instruction text information, combined with the screen text information, icon information and button information, which may specifically include the following steps: comparing the instruction text information with the screen text information to obtain a first comparison result; when the first comparison result is a matching result, extracting the target operation information from the instruction text information, and determining the operation object based on the screen text information; generating the operation intention based on the target operation information and the operation object.

[0097] In the process of determining the operation intention based on the instruction text information, combined with the screen text information, icon information and button information, this embodiment can first compare the instruction text information with the screen text information to obtain a first comparison result. The specific comparison method is text content comparison or instruction logic comparison, and this embodiment does not make specific limitations on this. The first comparison result can indicate whether the instruction text information matches the screen text information; and when the first comparison result is a matching result, it means that the instruction text information matches the screen text information, so that the target operation information can be extracted from the instruction text information, and the operation object can be determined based on the screen text information. The target operation information can indicate the target operation to be performed, and the operation object can indicate the object on which the target operation needs to be performed, and the target operation information and the operation object are integrated to generate the operation intention.

[0098] In one example, in the process of comparing the instruction text information with the screen text information, text content comparison or instruction logic comparison can be used. For example, when the instruction text information is "click the text at position X", the instruction text information does not contain specific text content. Then, the instruction text information can be determined to be a click text instruction through instruction logic comparison, that is, it can be determined that the instruction text information matches the screen text information. When the instruction text information is "click the next page", the instruction text information contains specific text content. Then, the instruction text information can be determined to contain the "next page" text through text content comparison, and it can also be determined that the instruction text information matches the screen text information. Then, when the first comparison result is a matching result, the target operation information is extracted from the instruction text information. For example, when the instruction text information is "click the text at position X", the extracted target operation information is a click operation. Then, when the text at position X is determined to be "next page" based on the screen text information, the text of "next page" can be determined as the operation object. Of course, the above is only an example for illustration. In specific implementation, it can be adaptively adjusted according to the scene and needs. This embodiment does not make specific limitations on this.

[0099] In an optional embodiment of the present application, after comparing the instruction text information with the screen text information to obtain a first comparison result, the following steps may be specifically included: when the first comparison result is a mismatch result, comparing the instruction text information with the icon information and the button information to obtain a second comparison result; when the second comparison result is a match result, extracting target operation information from the instruction text information, and determining the operation object based on the icon information and the button information; and generating an operation intention based on the target operation information and the operation object.

[0100] In this embodiment, after comparing the instruction text information with the screen text information to obtain the first comparison result, if the first comparison result is a mismatch result, it can be determined that the instruction text information matches the screen text information, indicating that no relevant operation needs to be performed on the screen text information at this time. At this time, the instruction text information is then compared with the icon information and the button information to obtain a second comparison result. The second comparison result can indicate whether the instruction text information matches the icon information or the button information. If the second comparison result is a matching result, it indicates that the icon information or the button information matches the instruction text information, that is, it is necessary to perform relevant operations on the icon information or the button information at this time, so that the target operation information can be extracted from the instruction text information, and the operation object is determined based on the icon information and the button information. The operation object represents the object in the icon information and the button information that matches the instruction in the instruction text information, and the target operation information and the operation object are integrated to generate the operation intention.

[0101] In one example, in the process of comparing the instruction text information with the icon information and the button information, for example, when the instruction text information is "click on the kitten", the icon information and the button information are compared to see whether they contain a kitten icon or a kitten button. If the icon information contains a kitten icon, the second comparison result is a matching result, and the target operation information can be extracted from the instruction text information as a click operation, and the instruction text information is combined with the icon information to determine that the operation object is the kitten icon, so that the target operation information and the operation object are integrated to generate the operation intention of clicking the kitten icon; and if the icon information does not contain the kitten icon, and the button information contains the kitten button, the second comparison result can also be determined as a matching result, and the target operation information can be extracted from the instruction text information as a click operation, and the instruction text information is combined with the button information to determine that the operation object is the kitten button, so that the target operation information and the operation object are integrated to generate the operation intention of clicking the kitten button. Of course, the above is only an example for illustration, and can be adaptively adjusted according to the scene and needs during specific implementation, and this embodiment does not make specific limitations on this.

[0102] Specifically, in the present embodiment, in the process of determining the operation intention based on the instruction text information, combined with the screen text information, icon information and button information, the instruction text information will first be compared with the screen text information to obtain a first comparison result, and when the first comparison result is a mismatch, the instruction text information will be compared with the icon information and button information to obtain a second comparison result; so that in the specific implementation of the present embodiment, the text content can be operated first to avoid confusion in operation. For example, when the screen text information contains kitten text and the icon information contains a kitten icon, the instruction text information is "click on the kitten". At this time, the target operation information will be extracted from the instruction text information first, and the operation object will be determined based on the screen text information, so as to generate the operation intention based on the target operation information and the operation object; avoiding the error of multiple operation intentions caused by the simultaneous comparison of the instruction text information with the screen text information, icon information and button information, and being able to accurately and orderly determine the operation intention corresponding to the instruction text information.

[0103] In another example, for example, when the user says the voice command "click on the picture of a woman and a cat", the command text information extracted at this time is "click on a woman and a cat". First, the command text information is compared with the screen text information to determine whether there is the text of "woman and cat" in the screen display image. If so, a click operation is triggered to click on the text of "woman and cat". If not, the command text information is compared with the icon information and button information to determine whether there is an icon or button of a woman and a cat in the screen display image. If so, a click operation is triggered to click on the icon or button of the woman and cat. Of course, the above is only an example to illustrate the role, and it can be adaptively adjusted according to the scene and needs during specific implementation, and this embodiment does not make specific limitations on this.

[0104] In an optional embodiment of the present application, based on the operation intention, the operation control is performed in combination with the screen display image to obtain a voice control result corresponding to the voice command, which may specifically include the following steps: extracting target operation information and operation object from the operation intention; based on the screen display image, performing the operation corresponding to the target operation information on the operation object to obtain a voice control result.

[0105] In the process of performing operation control based on the operation intention and in combination with the screen display image, this embodiment can extract target operation information and operation object from the operation intention, and perform the operation corresponding to the target operation information on the operation object based on the screen display image to obtain a voice control result. Specifically, the operation corresponding to the target operation information can be performed on the operation object in the display interface corresponding to the screen display image. In this way, voice control can be performed in combination with the specific screen display image, thereby solving the problem of low user voice control flexibility in the voice control method in the existing related technical means and effectively improving the flexibility of voice control.

[0106] like Figure 3 As shown, the present application also discloses an embodiment, providing a voice control device, including:

[0107] A first acquisition module 310, used to acquire a voice command;

[0108] A second acquisition module 320 is used to acquire a screen display image according to the voice command;

[0109] The recognition module 330 is used to perform screen recognition based on the screen display image to obtain screen content information;

[0110] A determination module 340, configured to determine an operation intention according to the voice instruction and the screen content information;

[0111] The control module 350 is used to perform operation control based on the operation intention and in combination with the screen display image to obtain a voice control result corresponding to the voice instruction.

[0112] In an optional embodiment of the present application, the second acquisition module 320 may include:

[0113] A first determining unit, used to determine the instruction type corresponding to the voice instruction;

[0114] The first acquisition unit is used to acquire the screen display image when the instruction type is a preset screen operation instruction type.

[0115] In an optional embodiment of the present application, the second acquisition module 320 may further include:

[0116] A second acquisition unit, configured to acquire a preset voice response strategy when the instruction type is an instruction type of a preset instruction library;

[0117] The response unit is used to adopt the preset voice response strategy, respond to the voice instruction, and obtain the voice control result.

[0118] In an optional embodiment of the present application, the identification module 330 may include:

[0119] A first extraction unit, configured to extract display elements from the screen display image to obtain element type information and position logic information corresponding to each display element;

[0120] The first generating unit is used to generate the screen content information according to the element type information and the position logic information.

[0121] In an optional embodiment of the present application, the first generating unit may include:

[0122] A first determining subunit, configured to determine an element identification strategy corresponding to the element type information;

[0123] A first extraction subunit is used for, when the element recognition strategy is a text recognition strategy, performing text extraction on the screen display image according to the position logic information to obtain screen text information;

[0124] The first generating subunit is used to generate the screen content information according to the screen text information.

[0125] In an optional embodiment of the present application, after determining the element identification strategy corresponding to the element type information, the method further includes:

[0126] A second extraction subunit is used for, when the element recognition strategy is an image recognition strategy, performing image extraction on the screen display image according to the position logic information to obtain icon information and button information;

[0127] The second generating subunit is used to generate the screen content information according to the icon information and the button information.

[0128] In an optional embodiment of the present application, the determining module 340 may include:

[0129] A semantic analysis unit, used for performing semantic analysis on the voice command to obtain the command text information;

[0130] A second extraction unit, used to extract screen text information, icon information and button information from the screen content information;

[0131] The second determining unit is used to determine the operation intention based on the instruction text information in combination with the screen text information, the icon information and the button information.

[0132] In an optional embodiment of the present application, the second determining unit may include:

[0133] A first comparison subunit, used for comparing the instruction text information with the screen text information to obtain a first comparison result;

[0134] a second determination subunit, configured to extract target operation information from the instruction text information and determine an operation object according to the screen text information when the first comparison result is a matching result;

[0135] The third generating subunit is used to generate the operation intention based on the target operation information and the operation object.

[0136] In an optional embodiment of the present application, the second determining unit may further include:

[0137] A second comparison subunit is used to compare the instruction text information with the icon information and the button information to obtain a second comparison result when the first comparison result is a mismatch result;

[0138] a third determining subunit, configured to extract target operation information from the instruction text information and determine an operation object according to the icon information and the button information when the second comparison result is a matching result;

[0139] The fourth generating subunit is used to generate the operation intention based on the target operation information and the operation object.

[0140] In an optional embodiment of the present application, the control module 350 may include:

[0141] A third extraction unit, used to extract target operation information and operation objects from the operation intention;

[0142] An operation unit is used to perform the operation corresponding to the target operation information on the operation object based on the screen display image to obtain the voice control result.

[0143] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0144] like Figure 4 As shown, an embodiment of the present application provides an electronic device, including a processor 410, a communication interface 420, a memory 430 and a communication bus 430, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 430.

[0145] Memory 430, for storing computer programs;

[0146] In one embodiment of the present application, the processor 410 is used to execute the program stored in the memory 430 to implement the voice control method provided by any of the aforementioned method embodiments. After obtaining the voice command, the screen display image is obtained for the voice command, and screen recognition is performed based on the screen display image to obtain screen content information, so as to determine the operation intention according to the voice command and the screen content information, and then the operation control is performed based on the operation intention in combination with the screen display image to obtain the voice control result corresponding to the voice command, so that the user is no longer limited to the preset voice command library during the voice control process. Even if a voice that does not belong to the preset voice command library is spoken, the execution steps of this embodiment can also be combined with the specific screen display image for voice control, thereby solving the problem of low user voice control flexibility in the voice control method in the existing related technical means, and can effectively improve the flexibility of voice control.

[0147] The embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the voice control method provided in any of the aforementioned method embodiments are implemented. After obtaining the voice command, a screen display image is obtained for the voice command, and screen recognition is performed based on the screen display image to obtain screen content information, so as to determine the operation intention according to the voice command and the screen content information. Subsequently, based on the operation intention, the operation control is performed in combination with the screen display image to obtain a voice control result corresponding to the voice command, so that the user is no longer limited to the preset voice command library during the voice control process. Even if a voice that does not belong to the preset voice command library is spoken, voice control can be performed in combination with a specific screen display image through the execution steps of this embodiment, thereby solving the problem of low user voice control flexibility in the voice control method in the existing related technical means, and can effectively improve the flexibility of voice control.

[0148] An embodiment of the present application also provides a vehicle, in which hardware equipment or software is configured, and the hardware equipment or software is used to implement the steps of the voice control method provided in any of the aforementioned method embodiments, so that the vehicle can execute the steps of the voice control method provided in any of the aforementioned method embodiments.

[0149] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0151] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0152] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A voice control method, characterized in that: include: Get voice commands; In response to the voice command, obtaining a screen display image; Performing screen recognition based on the screen display image to obtain screen content information; Determining an operation intention according to the voice command and the screen content information; Based on the operation intention, operation control is performed in combination with the screen display image to obtain a voice control result corresponding to the voice instruction.

2. The voice control method according to claim 1, characterized in that: The acquiring of a screen display image in response to the voice command includes: Determining the command type corresponding to the voice command; In a case where the instruction type is a preset screen operation instruction type, the screen display image is acquired.

3. The voice control method according to claim 2, characterized in that: After determining the instruction type corresponding to the voice instruction, the method further includes: When the instruction type is an instruction type of a preset instruction library, obtaining a preset voice response strategy; The preset voice response strategy is adopted to respond to the voice instruction and obtain the voice control result.

4. The voice control method according to claim 1, characterized in that: The performing screen recognition based on the screen display image to obtain screen content information includes: Extracting display elements from the screen display image to obtain element type information and position logic information corresponding to each display element; The screen content information is generated according to the element type information and the position logic information.

5. The voice control method according to claim 4, characterized in that: The generating the screen content information according to the element type information and the position logic information includes: Determining an element identification strategy corresponding to the element type information; When the element recognition strategy is a text recognition strategy, text is extracted from the screen display image according to the position logic information to obtain screen text information; The screen content information is generated according to the screen text information.

6. The voice control method according to claim 5, characterized in that: After determining the element identification strategy corresponding to the element type information, the method further includes: In the case where the element recognition strategy is an image recognition strategy, performing image extraction on the screen display image according to the position logic information to obtain icon information and button information; The screen content information is generated according to the icon information and the button information.

7. The voice control method according to claim 1, characterized in that: The determining the operation intention according to the voice command and the screen content information includes: Performing semantic analysis on the voice command to obtain the command text information; Extracting screen text information, icon information and button information from the screen content information; The operation intention is determined based on the instruction text information in combination with the screen text information, the icon information and the button information.

8. The voice control method according to claim 7, characterized in that: The determining the operation intention based on the instruction text information, in combination with the screen text information, the icon information and the button information, includes: Comparing the instruction text information with the screen text information to obtain a first comparison result; When the first comparison result is a matching result, extracting target operation information from the instruction text information, and determining an operation object according to the screen text information; The operation intention is generated based on the target operation information and the operation object.

9. The voice control method according to claim 8, characterized in that: After comparing the instruction text information with the screen text information to obtain a first comparison result, the method further includes: When the first comparison result is a mismatch result, the instruction text information is compared with the icon information and the button information to obtain a second comparison result; If the second comparison result is a matching result, extracting target operation information from the instruction text information, and determining an operation object according to the icon information and the button information; The operation intention is generated based on the target operation information and the operation object.

10. The voice control method according to any one of claims 1 to 9, characterized in that: The performing operation control based on the operation intention and in combination with the screen display image to obtain a voice control result corresponding to the voice command includes: Extracting target operation information and operation object from the operation intention; Based on the screen display image, the operation corresponding to the target operation information is performed on the operation object to obtain the voice control result.

11. A voice control device, characterized in that: include: A first acquisition module, used to acquire voice instructions; A second acquisition module is used to acquire a screen display image according to the voice command; A recognition module, used for performing screen recognition based on the screen display image to obtain screen content information; A determination module, used to determine the operation intention according to the voice instruction and the screen content information; The control module is used to perform operation control based on the operation intention and in combination with the screen display image to obtain a voice control result corresponding to the voice instruction.

12. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the voice control method described in any one of claims 1 to 10 when executing the program stored in the memory.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the voice control method according to any one of claims 1 to 10 is implemented.

14. A vehicle, characterized in that: The vehicle is used to execute the voice control method as described in any one of claims 1-10.

Images