Voice control method and device based on interactive page, equipment and storage medium

Abstract

The present application relates to speech processing technology, disclose a kind of based on interactive page voice control method, comprising: obtaining display image, display image is carried out image segmentation, obtain multiple subgraphs;The feature code corresponding to display image and multiple subgraphs is generated respectively, and it is judged whether the feature code exists in the preset data table, and the operation text contained in display image or multiple subgraphs is obtained according to the judgment result, and the multiple operation text obtained is stored in operation text cache library in the form of cache;Receive the control voice for the interactive page, convert the control voice into control text;Determine the operation text in the operation text cache library and the matching degree of the control text satisfy the matching condition as target operation text, and control the interactive page executes the operation corresponding to the target operation text.The present application also proposes a kind of based on interactive page voice control device, equipment and medium.The present application can improve the efficiency of voice control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, in particular to a speech control method and device based on an interactive page, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, more and more smart home devices supporting voice control have emerged in people's daily life, such as smart televisions, smart sound boxes, smart air conditioners and the like. Users can realize voice control of corresponding smart devices according to the guidance text information on the display screen or appearance of the smart home device, bringing convenience to people's life.

[0003] However, in the prior art, when controlling the device according to the user's voice, the current page element where the user issues the voice instruction is usually analyzed by image analysis or text analysis, and then the corresponding operation position and operation text are found to complete the voice control of the user. However, the recognition and analysis process is too long, which will result in low efficiency of the entire voice control implementation process. SUMMARY

[0004] The present application provides a speech control method and device based on an interactive page and a computer readable storage medium, which mainly aims to solve the problem of low efficiency of voice control.

[0005] To achieve the above-mentioned purpose, the present application provides a speech control method based on an interactive page, comprising:

[0006] obtaining a display image of an interactive page, performing image segmentation on the display image to obtain a plurality of sub-images;

[0007] generating feature codes corresponding to the display image and the plurality of sub-images respectively, and determining whether the feature codes corresponding to the display image and the plurality of sub-images respectively exist in a preset data table, obtaining operation texts contained in the display image or the plurality of sub-images according to the determination result, and storing the obtained operation texts in the form of cache in an operation text cache library;

[0008] receiving a control voice for the interactive page, and converting the control voice into a control text;

[0009] determining that the operation text in the operation text cache library that meets the matching condition with the control text as a target operation text, and controlling the interactive page to execute the operation corresponding to the target operation text.

[0010] Optionally, the generating of the feature codes corresponding to the display image and the plurality of sub-images respectively comprises:

[0011] Compress the display image and the plurality of sub-images into preset sizes to obtain a plurality of compressed images, and perform gray-scale processing on the plurality of compressed images;

[0012] Calculate a gray-scale feature value of each pixel point in the plurality of compressed images, and combine the gray-scale feature values to obtain a feature code of the display image and the plurality of sub-images.

[0013] Optionally, the determining whether the feature codes corresponding to the display image and the plurality of sub-images exist in the preset data table comprises:

[0014] extracting the feature code corresponding to the display image for searching in the data table;

[0015] If the feature code corresponding to the display image is searched, it is determined that the feature code corresponding to the display image exists in the data table.

[0016] If the feature code corresponding to the display image cannot be searched, the feature codes corresponding to the plurality of sub-images are searched in the data table one by one, and it is determined whether the feature codes corresponding to all the sub-images can be searched.

[0017] If the feature codes corresponding to all the sub-images can be searched, it is determined that the feature codes corresponding to all the sub-images exist in the data table.

[0018] If the feature code corresponding to any sub-image cannot be searched, it is determined that the feature codes corresponding to the display image and the plurality of sub-images do not exist in the data table.

[0019] If the feature codes corresponding to part of the sub-images can be searched, it is determined that the feature codes corresponding to part of the sub-images exist in the data table.

[0020] Optionally,

[0021] The operation text contained in the display image or the plurality of sub-images is obtained according to the determination result, comprising:

[0022] When it is determined that the feature code corresponding to the display image exists in the data table, the text corresponding to the feature code corresponding to the display image is extracted from the data table as the operation text contained in the display image.

[0023] When it is determined that the feature codes corresponding to all the sub-images exist in the data table, the texts corresponding to the feature codes corresponding to all the sub-images are extracted from the data table as the operation texts contained in all the sub-images.

[0024] When it is determined that the feature codes corresponding to part of the sub-images exist in the data table, the sub-image that does not exist in the data table is determined as a target image, and the operation text is obtained according to the target image.

[0025] determining that the display image and the plurality of sub-images are target images when it is determined that the feature code corresponding to the display image and the plurality of sub-images does not exist in the data table;

[0026] The operation text contained in the target image is obtained according to the target image, and the operation text comprises:

[0027] The similarity score of the image stored in the data table and the target image is calculated by using a preset image similarity algorithm, and it is determined whether the similarity score is greater than a preset threshold value;

[0028] When the similarity score is greater than the preset threshold value, the text corresponding to the target image in the data table is extracted as the operation text contained in the target image;

[0029] When the similarity score is less than or equal to the preset threshold value, text recognition is performed on the target image to obtain recognition text, and the recognition text is taken as the operation text contained in the target image.

[0030] Optionally, the image segmentation of the display image to obtain a plurality of sub-images comprises:

[0031] The display image is feature-extracted by using a pre-constructed segmentation network to obtain multiple image features of the display image;

[0032] The display image is image segmented according to the multiple image features to obtain a plurality of sub-images of the display image.

[0033] Optionally, the feature extraction of the display image by using the pre-constructed segmentation network to obtain multiple image features of the display image comprises:

[0034] The display image is convolution-processed and pooling-processed by using the segmentation network to obtain a pooling image;

[0035] The pooling image is fully connected to obtain a fully connected feature map;

[0036] The fully connected feature map is multi-scale feature-extracted to obtain multiple image features of the display image.

[0037] Optionally, the operation text in the operation text cache library that matches the control text to meet the matching condition is determined as a target operation text, comprising:

[0038] The similarity of the cache operation text in the operation text cache library and the voice text is calculated one by one, and it is determined whether there is a similarity greater than a first threshold value;

[0039] If the similarity greater than the first threshold exists, the cache operation text corresponding to the maximum similarity in the similarity greater than the threshold is selected as the target operation text.

[0040] If the similarity greater than the first threshold does not exist, the control voice is subjected to pronunciation unit recognition, and the target operation text is determined according to the recognition result.

[0041] To solve the above problems, the present application further provides an interactive page-based voice control device based on an interactive page, which comprises:

[0042] An image acquisition module is configured to acquire a display image of the interactive page, perform image segmentation on the display image, and obtain a plurality of sub-images.

[0043] An image text acquisition module is configured to generate feature codes corresponding to the display image and the plurality of sub-images respectively, determine whether the feature codes corresponding to the display image and the plurality of sub-images respectively exist in a preset data table, acquire operation texts contained in the display image or the plurality of sub-images according to the determination result, and store the acquired operation texts in the form of cache in an operation text cache library.

[0044] A control voice acquisition module is configured to receive a control voice for the interactive page and convert the control voice into a control text.

[0045] A voice control module is configured to determine an operation text in the operation text cache library that satisfies a matching condition with the control text as a target operation text, and control the interactive page to perform an operation corresponding to the target operation text.

[0046] To solve the above problems, the present application further provides an electronic device, which comprises:

[0047] At least one processor; and

[0048] A memory in communication connection with the at least one processor; wherein

[0049] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the above-mentioned interactive page-based voice control method.

[0050] To solve the above problems, the present application further provides a computer readable storage medium, which stores at least one computer program, and the at least one computer program is executed by a processor in an electronic device to implement the above-mentioned interactive page-based voice control method.

[0051] The embodiment of the present application extracts the operation text contained in the display image through image segmentation and feature code query after obtaining the display image of the page interaction, and stores the operation text into the operation text cache library, so that the operation text contained in the display image of the interaction can be quickly reused. When the control text is obtained according to the control voice, the operation text matched with the control text is directly searched from the operation text cache library, the efficiency of determining the operation text corresponding to the control voice in voice control is improved, and the voice control efficiency is further improved. Moreover, the present application can quickly and accurately operate a large number of users and different pages, and the user experience is improved. Therefore, the voice control method, device, electronic equipment and computer readable storage medium based on the interactive page provided by the present application can solve the problem of low voice control efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 A flowchart of a voice control method based on an interactive page provided by an embodiment of the present application is shown in the figure.

[0053] Figure 2 A flowchart of judging whether the feature codes corresponding to the display image and the plurality of subgraphs exist in the data table is shown in the figure.

[0054] Figure 3 A flowchart of determining the target operation text is shown in the figure.

[0055] Figure 4 A function module diagram of a voice control device based on an interactive page provided by an embodiment of the present application is shown in the figure.

[0056] Figure 5 A structural diagram of an electronic equipment for implementing the voice control method based on an interactive page is shown in the figure.

[0057] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0058] It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0059] The embodiment of the present application provides a voice control method based on an interactive page. The execution subject of the voice control method based on the interactive page includes but is not limited to at least one of electronic devices such as a server, a terminal and the like which can be configured to execute the method provided by the embodiment of the present application. In other words, the voice control method based on the interactive page can be executed by software or hardware installed in a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to a single server, a server cluster, a cloud server or a cloud server cluster and the like. The server can be a stand-alone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0060] Referring to Figure 1 Fig. 1 is a flowchart of a voice control method based on an interactive page provided by an embodiment of the present application. In the embodiment, the voice control method based on the interactive page includes the following steps.

[0061] S1, obtaining a display image of an interactive page, performing image segmentation on the display image to obtain a plurality of subgraphs.

[0062] In the embodiment of the present application, the display image can be obtained by directly reading a screenshot of the current page or by calling the current page through a system function, wherein the display image includes image information, and further, the obtained display image can also include text information, icon information and the like in addition to the image information, such as icons and texts in the display image page.

[0063] In the embodiment of the present application, the display image is segmented according to the multiple image features to obtain a plurality of subgraphs of the display image.

[0064] The display image is segmented according to the multiple image features to obtain a plurality of subgraphs of the display image.

[0065] The display image is segmented according to the multiple image features to obtain a plurality of subgraphs of the display image.

[0066] In the embodiment of the present application, the segmentation network can adopt a convolutional neural network with feature extraction function, such as a Segnet network, an E-net network, a V-net network and a ResNet50 network.

[0067] In an embodiment of the present application, the segmentation network adopts a ResNet50 network, and an FPN feature pyramid structure is added to the segmentation network to realize multi-feature extraction of the display image and obtain multiple image features corresponding to the display image.

[0068] Specifically, the feature extraction of the display image by using the pre-constructed segmentation network includes:

[0069] The display image is subjected to convolution processing and pooling processing by using the segmentation network to obtain a pooled image.

[0070] The pooled image is subjected to full connection processing to obtain a full connection feature map.

[0071] The full connection feature map is subjected to multi-scale feature extraction to obtain multiple image features of the display image.

[0072] For example, there is a display image A, the display image A is subjected to convolution, pooling and full connection processing by using the segmentation network to obtain a full connection feature map corresponding to the display image A, and the full connection feature map is subjected to multi-feature extraction by using an FPN feature pyramid structure in the segmentation network including 5 layers of output to obtain 5 different sizes of image features as the multiple image features of the display image A.

[0073] Further, the image segmentation of the display image according to the multiple image features to obtain multiple sub-images of the display image includes:

[0074] A segmentation frame diagram corresponding to the multiple image features is selected from preset segmentation frame diagrams.

[0075] The display image is framed multiple times according to the segmentation frame diagram, and different images framed are taken as multiple sub-images.

[0076] For example, the multiple image features are five sizes of image features, a corresponding segmentation frame diagram is selected from preset segmentation frame diagrams according to the five sizes of image features, and the display image is framed one by one by using the corresponding segmentation frame diagram to obtain the framed sub-images.

[0077] In the embodiment of the present application, the multiple feature extraction of the display image can realize the acquisition of multiple sizes of image features, which is beneficial to improve the accuracy of the segmented sub-images.

[0078] S2, generate the feature code corresponding to the display image and the plurality of sub-images respectively, and determine whether the feature code corresponding to the display image and the plurality of sub-images respectively exists in a preset data table, obtain the operation text contained in the display image or the plurality of sub-images according to the determination result, and store the obtained plurality of operation texts in the operation text cache library in the form of cache.

[0079] In the embodiment of the application, the feature code corresponding to the plurality of sub-images of the display image can be MD5 code or hash code formed by representing the image.

[0080] In the embodiment of the application, the data table is a pre-stored image information database, and contains a large number of images and sub-images, and further includes information (such as feature code and text) corresponding to the images and sub-images.

[0081] In the embodiment of the application, the generation of the feature code corresponding to the display image and the plurality of sub-images comprises:

[0082] The display image and the plurality of sub-images are compressed to a preset size to obtain a plurality of compressed images, and the plurality of compressed images are grayscaled.

[0083] The gray feature value of each pixel point in the plurality of compressed images is calculated, and the gray feature values are combined to obtain the feature code corresponding to the display image and the plurality of sub-images.

[0084] In the embodiment of the application, the gray feature value can be a gray difference value or a gray average value; if the gray feature value is a gray difference value, the difference value of the gray of adjacent pixel points in the compressed image is calculated, the difference value is processed, that is, the gray difference value is obtained, and the feature value of the display image and the plurality of sub-images is obtained by analyzing the gray difference value (for example, a positive number or 0 is recorded as 1, and a negative number is recorded as 0); if the gray feature value is a gray average value, the average value of the gray values of all pixel points in the compressed image is calculated, that is, the gray average value, and the feature code of the image of the interactive page and the plurality of sub-images is obtained by analyzing the gray average value (for example, greater than or equal to the gray average value is recorded as 1, and less than the gray average value is recorded as 0).

[0085] In another optional embodiment of the application, the display image and the plurality of sub-images can be converted into a character array, and the character data is MD5 encoded to obtain the MD5 code of the display image and the plurality of sub-images, and the MD5 code can be used as the feature code.

[0086] In an optional embodiment of the application, before the operation text contained in the display image or the plurality of sub-images is obtained according to the determination result, the method further comprises:

[0087] The display image and the plurality of sub-images are formed into a mapping relationship with the feature codes corresponding to the display image and the plurality of sub-images.

[0088] The operation text contained in the display image and the plurality of sub-images is generated, the display image and the plurality of sub-images are formed into a mapping relationship with the text contained in the display image and the plurality of sub-images, and the mapping relationship is stored into a data table.

[0089] In the embodiment of the present application, the corresponding text can be obtained by labeling the display image and the plurality of sub-images, or by recognizing the text in the image through a pre-trained model.

[0090] In the embodiment of the present application, after the mapping relationship formed by the display image and the plurality of sub-images and the feature codes corresponding to the display image and the plurality of sub-images is filled into the data table, a corresponding list label can also be generated according to the display image and the plurality of sub-images, the feature codes corresponding to the display image and the plurality of sub-images, and the text corresponding to the display image and the plurality of sub-images. For example, the list label corresponding to the list of the display image and the plurality of sub-images is image; the list label corresponding to the feature codes of the display image and the plurality of sub-images is feature code; and the list label corresponding to the text of the display image and the plurality of sub-images is text.

[0091] In the embodiment of the present application, referring to Figure 2 As shown in the figure, the judgment of whether the feature codes corresponding to the display image and the plurality of sub-images exist in the preset data table comprises:

[0092] S21, the feature code corresponding to the display image is extracted for searching in the data table;

[0093] If the feature code corresponding to the display image is searched, S22 is executed to determine whether the feature code corresponding to the display image exists in the data table;

[0094] If the feature code corresponding to the display image cannot be searched, S23 is executed to select the feature codes corresponding to the plurality of sub-images one by one for searching in the data table, and it is determined whether the feature codes corresponding to all the sub-images can be searched;

[0095] If the feature codes corresponding to all the sub-images can be searched, S24 is executed to determine whether the feature codes corresponding to all the sub-images exist in the data table;

[0096] If the feature code corresponding to any sub-image cannot be searched, S25 is executed to determine whether the feature codes corresponding to the display image and the plurality of sub-images exist in the data table;

[0097] If the feature code corresponding to the part of the subgraph can be retrieved, S26 is performed, and it is determined that the feature code corresponding to the part of the subgraph exists in the data table.

[0098] In the embodiment of the application, the operation text contained in the display image or the plurality of subgraphs according to the determination result comprises:

[0099] When it is determined that the feature code corresponding to the display image exists in the data table, the text corresponding to the feature code corresponding to the display image is extracted from the data table as the operation text contained in the display image.

[0100] When it is determined that the feature codes corresponding to all the subgraphs exist in the data table, the text corresponding to the feature codes corresponding to all the subgraphs is extracted from the data table as the operation text contained in all the subgraphs.

[0101] When it is determined that the feature codes corresponding to part of the subgraphs exist in the data table, the subgraph that does not exist in the data table is determined as a target image, and the operation text is acquired according to the target image.

[0102] When it is determined that the feature codes corresponding to the display image and the plurality of subgraphs do not exist in the data table, the display image and the plurality of subgraphs are determined as target images.

[0103] The operation text contained in the display image or the plurality of subgraphs according to the target image comprises:

[0104] The similarity score of the image stored in the data table and the target image is calculated by using a preset image similarity algorithm, and it is determined whether the similarity score is greater than a preset threshold.

[0105] When the similarity score is greater than the preset threshold, the text corresponding to the target image is extracted from the data table as the operation text contained in the target image.

[0106] When the similarity score is less than or equal to the preset threshold, text recognition is performed on the target image, the recognized text is obtained, and the recognized text is taken as the operation text contained in the target image.

[0107] In the embodiment of the application, the number of texts in the data table is greater than the number of texts in the operation text cache library, and the texts in the operation text cache library only include the texts corresponding to the display image of the interactive page and the subgraph of the display image of the interactive page.

[0108] In the embodiment of the present application, the image, the feature code corresponding to the image and the text corresponding to the image in the data table can be stored in association, because the text corresponding to the image can be extracted according to the feature code corresponding to the image; after the extracted text is stored in the operation text cache library, all the texts corresponding to the image can be obtained from the operation text cache library.

[0109] In the embodiment of the present application, if the feature code cannot be searched in the data table, image recognition needs to be performed to determine whether there is a similar image in the data table.

[0110] In the embodiment of the present application, the image similarity algorithm includes but is not limited to SIFT algorithm, perceptual hashing algorithm and template matching algorithm; the image whose feature code cannot be recognized is calculated for similarity with the pre-stored images in the data table one by one through the image similarity algorithm to obtain a similarity value, and the image in the data table with a similarity value greater than a preset threshold is selected as the target image, and the text corresponding to the target image is the operation text.

[0111] In the embodiment of the present application, if the feature code corresponding to the image cannot be searched in the data table, and the image similarity algorithm also cannot obtain an image with a result greater than a preset threshold, the image may not exist in the data table, and therefore text recognition needs to be performed to obtain the text of the corresponding image.

[0112] In the embodiment of the present application, the text recognition on the target image to obtain the recognized text includes:

[0113] All characters in the target image are obtained through a segmentation algorithm;

[0114] The text recognition network pre-trained on the all characters is used to perform text recognition to obtain the recognized text.

[0115] Specifically, the all characters in the target image are obtained through a segmentation algorithm, and the method includes:

[0116] The corresponding image is projected horizontally, and the upper limit and the lower limit of each row after projection are obtained;

[0117] The upper limit and the lower limit are cut according to the upper limit and the lower limit;

[0118] Each row after cutting is projected vertically, and the left limit and the right limit of each character after projection are obtained;

[0119] Each character is cut according to the left limit and the right limit to obtain a plurality of characters.

[0120] In the embodiment of the present application, the text recognition network can be obtained by training a CNN neural network, inputting text training data into the CNN neural network for prediction, and optimizing the neural network by calculating the loss value of the prediction result.

[0121] In another optional embodiment of the present application, the target image can also be subjected to text detection and character segmentation by using a CTPN deep neural network and a SegLink algorithm.

[0122] S3, receiving a control voice for the interactive page, and converting the control voice into control text.

[0123] In the embodiment of the present application, the conversion of the control voice into control text comprises:

[0124] extracting speech features from the control voice to obtain a feature vector;

[0125] inputting the feature vector into a preset acoustic model to obtain phoneme information;

[0126] obtaining a plurality of phoneme segments according to a preset number of phonemes;

[0127] searching each of the phoneme segments in a preset word library one by one;

[0128] taking the text corresponding to the phoneme segment searched in the word library as the control text;

[0129] performing approximate sound conversion on the phoneme segment not searched in the word library, and re-matching the converted phoneme segment with the word library.

[0130] In the embodiment of the present application, the acoustic model includes but is not limited to HMM (Hidden Markov Model); the phoneme-by-phoneme matching according to the preset number of phonemes can be matching with the word library in the manner of single phoneme, two phonemes, three phonemes, or four phonemes; and the word library includes popular words, commonly used words, standard words divided according to parts of speech (such as adjectives, nouns, adverbs, etc.).

[0131] In the embodiment of the present application, the approximate sound conversion includes initial letter conversion and pronunciation conversion. For example, when the initial letter is F or H, the corresponding phoneme is replaced between the initial letters F and H, and the replaced part is recombined; when the initial letter is L, M, N, or R, the initial letters L, M, N, and R are replaced, and the non-replaced part is recombined; when the pronunciation includes Z, C, S, Zh, Ch, and Sh, the flat and raised tongue sounds are converted, and the non-replaced part is recombined.

[0132] S4, determine target operation text in the operation text library that matches the control text, and control the interactive page to perform an operation corresponding to the target operation text.

[0133] In the embodiment of the present application, before the operation corresponding to the target text is performed, the operation text in the text library can be matched with the corresponding operation code, and the corresponding operation can be performed through the operation code. For example, there are operation texts "movie A" and "next page", and the corresponding operation codes can be "jump to detail page" and "jump to next page", respectively.

[0134] In the embodiment of the present application, referring to Figure 3 The determination of the target operation text in the operation text library that matches the control text includes:

[0135] S41, calculate the similarity between the operation text in the operation text library and the voice text one by one, and determine whether there is a similarity greater than a first threshold value;

[0136] If there is a similarity greater than the first threshold value, perform S42, select the cached operation text corresponding to the maximum similarity greater than the threshold value as the target operation text;

[0137] If there is no similarity greater than the first threshold value, perform S43, perform pronunciation unit recognition on the control voice, and determine the target operation text according to the recognition result.

[0138] Further, the similarity between the operation text in the text library and the control text can be calculated by the following formula:

[0139]

[0140] Wherein, the cosθ is the similarity score, a is the control text, b i is the i-th operation text in the text library.

[0141] Specifically, the target operation text is determined according to the recognition result by using the control voice to perform pronunciation unit recognition, which includes:

[0142] The control voice is divided into a plurality of pronunciation units, and one of the plurality of pronunciation units is selected as a target pronunciation unit;

[0143] The minimum distance is calculated according to the target pronunciation unit and the preset standard pronunciation unit, and it is determined whether the minimum distance is less than a second threshold value;

[0144] If the minimum distance is less than a second threshold value, it is determined that the text corresponding to the standard pronunciation unit is the target operation text.

[0145] If the minimum distance is greater than or equal to the second threshold value, it is determined that the target operation text is not found, and a result of not finding the target operation text is output to a user.

[0146] Further, the embodiment of the present application can calculate the minimum distance of the target pronunciation unit and the preset standard pronunciation unit by the following formula:

[0147]

[0148] Wherein, D is the similarity, R is the target pronunciation unit, T is the standard pronunciation unit, and θ is a preset coefficient.

[0149] The embodiment of the present application can quickly extract the operation text contained in the display image and store it in the operation text cache library by image segmentation and feature code query after obtaining the display image of the page interaction, so that the operation text contained in the display image of the interaction can be quickly reused. When the control text is obtained according to the control voice, the operation text matching the control text is directly found from the operation text cache library, which improves the efficiency of determining the operation text corresponding to the control voice in voice control, and further improves the voice control efficiency. Moreover, the present scheme can quickly and accurately operate a large number of users and different pages, thereby improving the user experience. Therefore, the voice control method based on interactive pages provided by the present application can solve the problem of low efficiency in the implementation process of the whole voice control.

[0150] As shown in Figure 4 Fig. 1 is a functional module diagram of a voice control device based on interactive pages according to an embodiment of the present application.

[0151] The voice control device based on interactive pages 100 can be installed in an electronic device. According to the functions implemented, the voice control device based on interactive pages 100 can include an image acquisition module 101, an image text acquisition module 102, a control voice acquisition module 103, and a voice control module 104. The modules of the present application can also be referred to as units, which refer to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, and are stored in the memory of the electronic device.

[0152] In the present embodiment, the functions of each module / unit are as follows:

[0153] The image acquisition module 101 is configured to acquire a display image of an interactive page, perform image segmentation on the display image, and obtain a plurality of sub-images.

[0154] The image text acquisition module 102 is configured to generate feature codes corresponding to the display image and the plurality of sub-images respectively, and determine whether the feature codes corresponding to the display image and the plurality of sub-images respectively exist in a preset data table, acquire operation texts contained in the display image or the plurality of sub-images according to a determination result, and store the acquired operation texts in the operation text cache library in a cache form.

[0155] The control voice acquisition module 103 is configured to receive a control voice for the interactive page, and convert the control voice into control text.

[0156] The voice control module 104 is configured to determine that an operation text in the operation text cache library that matches the control text meets a matching condition as a target operation text, and control the interactive page to perform an operation corresponding to the target operation text.

[0157] In detail, each module in the voice control device 100 based on the interactive page in the embodiment of the present application adopts the same technical means as the voice control method based on the interactive page in the above Figures 1 to 3 , and can produce the same technical effects, which will not be described here.

[0158] As shown in Figure 5 , it is a structural schematic diagram of an electronic device for implementing the voice control method based on the interactive page according to an embodiment of the present application.

[0159] The electronic device 1 can include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and can further include a computer program (such as a voice control program) stored in the memory 11 and executable on the processor 10.

[0160] In some embodiments, the processor 10 can be composed of an integrated circuit, for example, can be composed of a single packaged integrated circuit, or can be composed of a plurality of packaged integrated circuits with the same function or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 10 is the control core of the electronic device. It connects all components of the electronic device through various interfaces and lines, executes programs or modules stored in the memory 11 (such as executing a voice control program), and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0161] The memory 11 includes at least one type of readable storage medium, such as a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 can also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 11 can include both an internal storage unit and an external storage device of the electronic device. The memory 11 can be used not only to store application software and various data installed in the electronic device, such as a code of a voice control program, etc., but also to temporarily store data that has been output or will be output.

[0162] The communication bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable connection and communication between the memory 11 and at least one processor 10, etc.

[0163] The communication interface 13 is used for communication between the electronic device and other devices, including a network interface and a user interface. Optionally, the network interface can include a wired interface and / or a wireless interface (e.g., a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the electronic device and other electronic devices. The user interface can be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface can also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch, etc. The display can also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visualized user interface.

[0164] Figure 5 Only an electronic device with components is shown, and those skilled in the art can understand that, Figure 5The structure shown does not constitute a limitation on the electronic device 1, and can include fewer or more components than shown, or combine certain components, or arrange different components.

[0165] For example, although not shown, the electronic device can also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, so that the power management device can implement functions such as charge management, discharge management, and power consumption management. The power supply can also include one or more direct current or alternating current power supplies, recharging devices, power supply fault detection circuits, power supply converters or inverters, power supply status indicators, and any other components. The electronic device can also include various sensors, Bluetooth modules, Wi-Fi modules, and the like, which are not described here.

[0166] It should be understood that the embodiments are only for illustration and are not limited in the scope of the patent application by this structure.

[0167] The voice control program stored in the memory 11 in the electronic device 1 is a combination of multiple instructions, which, when executed in the processor 10, can achieve:

[0168] Obtaining a display image of an interactive page, performing image segmentation on the display image to obtain multiple sub-images;

[0169] Generating feature codes corresponding to the display image and the multiple sub-images respectively, and determining whether the feature codes corresponding to the display image and the multiple sub-images respectively exist in a preset data table, obtaining operation texts contained in the display image or the multiple sub-images according to the determination result, and storing the obtained multiple operation texts in an operation text cache in a cache form;

[0170] Receiving a control voice for the interactive page, and converting the control voice into a control text;

[0171] Determining that an operation text in the operation text cache that meets a matching condition with the control text is a target operation text, and controlling the interactive page to perform an operation corresponding to the target operation text.

[0172] Specifically, the specific implementation method of the processor 10 on the above instructions can refer to the description of the related steps in the corresponding embodiments of the accompanying drawings, which is not described here.

[0173] Further, the modules / units integrated in the electronic device 1 are stored in a computer readable storage medium if they are realized in the form of software function units and sold or used as independent products. The computer readable storage medium can be volatile or non-volatile. For example, the computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM).

[0174] The application further provides a computer readable storage medium, which stores a computer program, and the computer program can realize the following when executed by a processor of an electronic device:

[0175] obtaining a display image of the interactive page, performing image segmentation on the display image to obtain a plurality of sub-images;

[0176] generating feature codes corresponding to the display image and the plurality of sub-images respectively, and judging whether the feature codes corresponding to the display image and the plurality of sub-images respectively exist in a preset data table, obtaining operation texts contained in the display image or the plurality of sub-images according to a judgment result, and storing the obtained operation texts in an operation text cache in a cache form;

[0177] receiving a control voice for the interactive page, and converting the control voice into a control text;

[0178] determining that an operation text in the operation text cache that meets a matching condition with the control text is a target operation text, and controlling the interactive page to perform an operation corresponding to the target operation text.

[0179] In several embodiments provided in the application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and the division of the modules is merely a logical function division, and there can be another division manner in actual implementation.

[0180] The modules described as separated components can or can not be physically separated, and the components displayed as modules can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. According to actual needs, some or all of the modules can be selected to achieve the purpose of the embodiments.

[0181] In addition, each functional module in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of hardware plus software function module.

[0182] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be realized in other specific forms without departing from the spirit or essential characteristics of the present application.

[0183] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, the scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any additional reference signs in the claims should not be considered as limiting the claims to which they relate.

[0184] The blockchain referred to in the present application is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. Blockchain, in essence, is a decentralized database, which is a series of data blocks associated using cryptographic methods, each data block containing information of a batch of network transactions, for verifying the validity (anti-fake) of the information and generating the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer, etc.

[0185] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0186] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The plurality of units or devices stated in the system claims can also be realized by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, not any particular order.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A voice control method based on an interactive page, characterized in that, The method includes: The process involves acquiring a display image of an interactive page, performing image segmentation on the display image to obtain multiple sub-images, and then: using a pre-constructed segmentation network to extract multiple features from the display image to obtain multiple image features of the display image containing five different sizes; selecting segmentation frames corresponding to the five different sizes of the multiple image features from preset segmentation frames, and repeatedly selecting the display image based on the selected segmentation frames, using the different selected images as multiple sub-images; wherein the segmentation network is a ResNet50 network and includes an FPN feature pyramid structure, and the multiple image features are multiple image features of different sizes; The process involves generating feature codes corresponding to the displayed image and the plurality of sub-images, and determining whether the feature codes corresponding to the displayed image and the plurality of sub-images exist in a preset data table. Generating the feature codes for the displayed image and the plurality of sub-images includes: compressing the displayed image and the plurality of sub-images to a preset size to obtain a plurality of compressed images, and converting the plurality of compressed images to grayscale; calculating the grayscale feature value of each pixel in the plurality of compressed images, and combining the grayscale feature values ​​to obtain the feature codes for the displayed image and the plurality of sub-images, wherein the grayscale feature values ​​represent the grayscale difference between adjacent pixels. Based on the judgment result, obtain the operation text contained in the displayed image or multiple sub-images, and store the obtained multiple operation texts in the operation text cache library in a cached form; Receiving control voice for the interactive page and converting the control voice into control text includes: inputting a voice feature vector into an acoustic model to obtain phoneme information; dividing the phoneme information into multiple phoneme segments according to single, two, three, and four sounds, and retrieving each phoneme segment in a preset dictionary; and using the text corresponding to the phoneme segments retrieved in the dictionary as the control text. The system determines the target operation text in the operation text cache library that matches the control text with a matching degree that meets the matching condition, and controls the interactive page to execute the operation corresponding to the target operation text.

2. The voice control method based on an interactive page as described in claim 1, characterized in that, The step of determining whether the preset data table contains the feature codes corresponding to the displayed image and the multiple sub-images includes: Extract the feature code corresponding to the displayed image and retrieve it from the data table; If a feature code corresponding to the displayed image is found, it is determined that the feature code corresponding to the displayed image exists in the data table; If the feature code corresponding to the displayed image cannot be retrieved, the feature codes corresponding to the multiple sub-images are selected one by one and retrieved in the data table, and it is determined whether the feature codes corresponding to all sub-images can be retrieved. If all feature codes corresponding to the subgraphs can be retrieved, it is determined that all feature codes corresponding to the subgraphs exist in the data table. If no feature code is found for any sub-image, it is determined that the display image and the feature codes corresponding to the multiple sub-images do not exist in the data table. If the feature codes corresponding to some of the subgraphs can be retrieved, it is determined that the feature codes corresponding to some of the subgraphs exist in the data table.

3. The voice control method based on an interactive page as described in claim 2, characterized in that, The step of obtaining the operation text contained in the displayed image or multiple sub-images based on the judgment result includes: When it is determined that the feature code corresponding to the displayed image exists in the data table, the text corresponding to the feature code corresponding to the displayed image is extracted from the data table as the operation text contained in the displayed image; When it is determined that all the feature codes corresponding to the subgraphs exist in the data table, the text corresponding to all the feature codes corresponding to the subgraphs is extracted from the data table as the operation text contained in all the subgraphs; When it is determined that some of the feature codes corresponding to the sub-images exist in the data table, the sub-images that do not exist in the data table are identified as target images, and the operation text is obtained based on the target images; When it is determined that the display image and the multiple sub-images do not contain a feature code in the data table, the display image and the multiple sub-images are determined to be the target image; The operation text for obtaining the display image or multiple sub-images based on the target image includes: The similarity score between the image stored in the data table and the target image is calculated using a preset image similarity algorithm, and it is determined whether the similarity score is greater than a preset threshold. When the similarity score is greater than a preset threshold, the text corresponding to the target image is extracted from the data table as the operation text contained in the target image; When the similarity score is less than or equal to a preset threshold, text recognition is performed on the target image to obtain the recognized text, and the recognized text is used as the operation text contained in the target image.

4. The voice control method based on an interactive page as described in claim 1, characterized in that, The step of using a pre-constructed segmentation network to extract features from the displayed image to obtain multiple image features of the displayed image includes: The segmentation network is used to perform convolution and pooling processing on the displayed image to obtain a pooled image; The pooled image is processed by a fully connected layer to obtain a fully connected feature map; Multi-scale feature extraction is performed on the fully connected feature map to obtain multiple image features of the displayed image.

5. The voice control method based on an interactive page as described in any one of claims 1 to 4, characterized in that, The step of determining the target operation text as the operation text in the operation text cache library whose matching degree with the control text meets the matching condition includes: Calculate the similarity between each cached operation text in the operation text cache library and the speech text, and determine whether there is a similarity greater than the first threshold. If there is a similarity greater than the first threshold, then the cached operation text corresponding to the largest similarity among the similarities greater than the threshold is selected as the target operation text. If there is no similarity greater than the first threshold, then the control speech is subjected to pronunciation unit recognition, and the target operation text is determined based on the recognition result.

6. A voice control device based on an interactive page, characterized in that, The device includes: An image acquisition module is used to acquire the display image of the interactive page, and to perform image segmentation on the display image to obtain multiple sub-images. The image segmentation to obtain multiple sub-images includes: extracting multiple features from the display image using a pre-constructed segmentation network to obtain multiple image features of the display image containing five different sizes; selecting segmentation frames corresponding to the five different sizes of the multiple image features from preset segmentation frames, and repeatedly selecting the display image based on the selected segmentation frames, using the different selected images as multiple sub-images; wherein the segmentation network is a ResNet50 network and includes an FPN feature pyramid structure, and the multiple image features are multiple image features of different sizes. An image text acquisition module is used to generate feature codes corresponding to the display image and the multiple sub-images, and to determine whether the feature codes corresponding to the display image and the multiple sub-images exist in a preset data table. The generation of feature codes for the display image and the multiple sub-images includes: compressing the display image and the multiple sub-images to a preset size to obtain multiple compressed images, and converting the multiple compressed images to grayscale; calculating the grayscale feature value of each pixel in the multiple compressed images, and combining the grayscale feature values ​​to obtain the feature codes for the display image and the multiple sub-images, where the grayscale feature value represents the grayscale difference between adjacent pixels; and acquiring the operation text contained in the display image or the multiple sub-images according to the determination result, and storing the acquired operation text in a cached form in an operation text cache library. The control voice acquisition module is used to receive control voice for the interactive page and convert the control voice into control text, including: inputting the voice feature vector into an acoustic model to obtain phoneme information; dividing the phoneme information into multiple phoneme segments in the manner of single syllable, two syllables, three syllables, and four syllables, and searching for each phoneme segment in a preset dictionary; and using the text corresponding to the phoneme segments retrieved in the dictionary as the control text. The voice control module is used to determine the operation text in the operation text cache that matches the control text as the target operation text, and to control the interactive page to perform the operation corresponding to the target operation text.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the voice control method based on an interactive page as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the voice control method based on an interactive page as described in any one of claims 1 to 5.

Images