Webpage element control method, device, equipment, storage medium and program product

Abstract

The application provides a webpage element control method and device, equipment, storage medium and program product, which can be applied to the technical field of artificial intelligence and financial technology. The method comprises the following steps: processing a plurality of candidate audio clips through a pre-trained voice activity detection model to obtain a voice existence probability corresponding to each candidate audio clip, the plurality of candidate audio clips being obtained by screening from a plurality of audio clips based on at least one of short-time energy and zero-crossing rate of each audio clip; screening the plurality of candidate audio clips based on the plurality of voice existence probabilities to obtain a target audio clip; performing voice recognition on the target audio clip to obtain voice recognition text; guiding a pre-trained large model to process the voice recognition text and an element index based on a plurality of first prompt words to obtain an element control instruction, and transmitting the element control instruction to a webpage to control the webpage to perform a webpage operation indicated by the element control instruction, the element index indicating a corresponding relationship between a user intention and a webpage element.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and financial technology, and more specifically to a method, apparatus, device, medium, and program product for controlling web page elements. Background Technology

[0002] With the development of Internet technology, browser operation control based on voice recognition has been widely used, such as using voice assistants to control web page operations.

[0003] However, in noisy environments and other scenarios, the accuracy of traditional speech recognition technology needs to be improved, and its semantic understanding ability is limited, making it difficult to accurately identify the control commands contained in speech. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a web page element control method, apparatus, device, storage medium, and program product.

[0005] According to a first aspect of this application, a webpage element control method is provided. The method includes: processing multiple candidate audio segments using a pre-trained speech activity detection model to obtain the speech presence probability corresponding to each candidate audio segment; selecting multiple candidate audio segments from among the multiple audio segments based on at least one of short-time energy and zero-crossing rate, where short-time energy indicates the amplitude strength of the audio segment and zero-crossing rate characterizes the frequency characteristics of the audio segment; selecting a target audio segment from among the multiple candidate audio segments based on the multiple speech presence probabilities; performing speech recognition on the target audio segment to obtain speech-recognized text; and guiding a pre-trained large model to process the speech-recognized text and element index based on multiple preset first prompt words to obtain element control instructions, and transmitting these instructions to a webpage to control the webpage to execute the webpage operation indicated by the element control instructions, where the element index indicates the correspondence between user intent and webpage elements.

[0006] According to an embodiment of this application, multiple candidate audio segments are obtained in the following manner: based on the amplitude of the sampling points in each audio segment, the short-time energy corresponding to each of the multiple audio segments is obtained; based on the degree of change of adjacent sampling points in each audio segment, the zero-crossing rate corresponding to each of the multiple audio segments is obtained; and audio segments with short-time energy greater than a preset energy threshold or zero-crossing rate greater than a preset zero-crossing rate threshold are selected as candidate audio segments.

[0007] According to an embodiment of this application, the method further includes: when the short-time energy is less than a first noise determination threshold, fusing the short-time energy and the energy threshold to update the energy threshold, wherein the first noise determination threshold indicates the update scale of the energy threshold; and when the zero-crossing rate is less than a second noise determination threshold, fusing the zero-crossing rate and the zero-crossing rate threshold to update the zero-crossing rate threshold, wherein the second noise determination threshold indicates the update scale of the zero-crossing rate threshold.

[0008] According to embodiments of this application, the element index is obtained in the following manner: based on preset search conditions, the document object model tree of the webpage is traversed to obtain multiple elements, each of which has attribute information. The document object model tree indicates all elements in the webpage and the relationships between elements; multiple elements are filtered based on preset filtering conditions to obtain multiple candidate elements, the filtering conditions including at least one of size, style, position, and status; multiple importance scores are obtained based on at least one of element type, attribute information, element area, and element center point of the multiple candidate elements, and multiple target elements are obtained by filtering among the multiple candidate elements based on the multiple importance scores; element identifiers of the multiple target elements are determined based on at least one of identifier, name, tag, position, and attribute of the multiple target elements, and the multiple element identifiers are associated with the positions of their respective target elements to form an element index.

[0009] According to embodiments of this application, multiple importance scores are obtained based on at least one of the element type, attribute information, element area, and element center point of multiple candidate elements, including: obtaining element type factors of multiple candidate elements based on the element type and preset scores of the element type, where the element type factor indicates the interaction probability of the element type; obtaining text existence factors of multiple candidate elements based on the proportion of non-empty characters in multiple attribute information, where the text existence factor indicates the degree of matching between the candidate element and the user intent; obtaining area normalization factors of multiple candidate elements based on the proportion of the area of ​​multiple elements relative to the viewport area of ​​the webpage, where the area normalization factor indicates the salience of the candidate element relative to the webpage; obtaining center distance factors of multiple candidate elements based on the distance between the center point of multiple elements and the viewport center of the webpage, where the center distance factor indicates the degree of visual focus of the candidate element relative to the webpage; and fusing the multiple element type factors, multiple text existence factors, multiple area normalization factors, and multiple center distance factors respectively to obtain multiple importance scores.

[0010] According to embodiments of this application, the plurality of first prompt words include a plurality of first sub-prompt words and a plurality of second sub-prompt words. The plurality of first sub-prompt words are used to indicate the role positioning of the large model, and the plurality of second sub-prompt words are used to indicate the task requirements. Based on the preset plurality of first prompt words, guiding the pre-trained large model to process speech recognition text and element indexes to obtain element control instructions includes: using the plurality of first sub-prompt words and the plurality of second sub-prompt words to guide the large model to process speech recognition text and element indexes under the role positioning and according to the task requirements to obtain element control instructions.

[0011] According to an embodiment of this application, performing speech recognition on a target audio segment to obtain speech recognition text includes: processing the target audio segment using a pre-trained decoder based on a keyword set to obtain speech recognition text. The keyword set is obtained in advance by performing at least one of keyword extraction, related word extraction, and semantic expansion on key elements of the webpage.

[0012] According to an embodiment of this application, the method further includes: guiding a large model to verify element control instructions based on a plurality of preset second prompt words, wherein the plurality of second prompt words are used to indicate the structural verification reasoning steps for the element control instructions.

[0013] According to a second aspect of this application, a webpage element control device is provided, comprising: a speech detection module, configured to process multiple candidate audio segments using a pre-trained speech activity detection model to obtain the speech presence probability corresponding to each candidate audio segment, wherein the multiple candidate audio segments are selected from the multiple audio segments based on at least one of short-time energy and zero-crossing rate, wherein short-time energy indicates the amplitude strength of the audio segment and zero-crossing rate characterizes the frequency characteristics of the audio segment; a selection module, configured to select from the multiple candidate audio segments based on the multiple speech presence probabilities to obtain a target audio segment; a speech recognition module, configured to perform speech recognition on the target audio segment to obtain speech recognition text; and an instruction parsing module, configured to guide a pre-trained large model to process the speech recognition text and element index based on a plurality of preset first prompt words to obtain element control instructions, and transmit them to the webpage to control the webpage to execute the webpage operation indicated by the element control instructions, wherein the element index indicates the correspondence between the user's intent and the webpage element.

[0014] According to a third aspect of this application, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] According to a fourth aspect of this application, a computer-readable storage medium is also provided, on which a computer program or instructions are stored, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0016] According to a fifth aspect of this application, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0017] According to the webpage element control method, apparatus, device, storage medium, and program product provided in the embodiments of this application, speech activity detection is performed only on audio segments pre-screened by zero-crossing rate or short-time energy. On the one hand, the zero-crossing rate or short-time energy is used to filter audio unrelated to user speech, thereby improving the efficiency and accuracy of speech activity detection. On the other hand, the speech activity detection model is used to perform activity detection on candidate audio segments, ensuring that the target audio segments used for speech recognition are all valid audio segments that reflect user speech. The semantic understanding capability of the large model is used to process speech recognition text and element indexes, which can parse complex instructions and improve the accuracy of element control instruction recognition. At the same time, the element index is used as the basic data for instruction matching, thereby improving recognition efficiency. Attached Figure Description

[0018] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 The illustrations depict application scenarios of web page control methods, apparatus, devices, media, and program products according to embodiments of this application.

[0020] Figure 2 A flowchart illustrating a webpage control method according to an embodiment of this application is shown schematically.

[0021] Figure 3 A flowchart illustrating the selection of candidate audio segments according to an embodiment of this application is shown schematically;

[0022] Figure 4 A flowchart illustrating the construction of an element index according to an embodiment of this application is shown schematically;

[0023] Figure 5 A flowchart illustrating the calculation of importance scores according to embodiments of this application is shown schematically.

[0024] Figure 6 A schematic diagram illustrating the structure of a webpage control device according to an embodiment of this application is shown.

[0025] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a web page control method according to an embodiment of this application. Detailed Implementation

[0026] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0031] In scenarios where personal information is used for automated decision-making, the methods, devices, and systems provided in this application all provide users with corresponding operation entry points for users to choose to agree to or reject the automated decision results; if the user chooses to reject, the process enters the expert decision-making process.

[0032] Browsers are essential tools in people's daily work and life. With the development of internet technology, in order to improve the efficiency of browser operation in mobile environments or when both hands need to perform other tasks, voice recognition-based browser operation control is being developed, such as using voice assistants to control web page operations.

[0033] Traditional speech recognition technology uses speech recognition models to identify text within a user's speech and parse it into commands. However, in noisy environments and other scenarios, the recognition accuracy needs improvement, and its semantic understanding capability is limited, making it difficult to accurately identify control commands contained in speech, and even more difficult to recognize complex commands.

[0034] To address at least one of the aforementioned problems, embodiments of this application provide a webpage element control method. This method performs speech activity detection only on audio segments pre-selected using zero-crossing rate or short-time energy, filtering out audio unrelated to user speech, thereby improving the efficiency and accuracy of speech activity detection and ensuring that all target audio segments used for speech recognition are valid audio segments reflecting user speech. Furthermore, it utilizes the semantic understanding capabilities of a large model to process speech recognition text and element indexes, enabling the parsing of complex instructions and improving the accuracy of element control instruction recognition. Simultaneously, it uses element indexes as the basic data for instruction matching, further enhancing recognition efficiency.

[0035] Figure 1 The illustration depicts an application scenario of a webpage element control method according to an embodiment of this application. For example... Figure 1 As shown, application scenario 100 according to an embodiment of this application may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables. For example, a user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send information, etc.

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be electronic devices such as smartphones, wearable devices, personal computers, intelligent voice interaction devices, smart home appliances, intelligent vehicles, in-vehicle terminals, aircraft, unmanned vending terminals, and extended reality devices. Extended reality devices can include virtual reality devices, augmented reality devices, and mixed reality devices. A client application for the target application can be installed and run on the terminal devices. This target application can include, but is not limited to, financial transaction applications, payment applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, and social media platform software (these are just examples). Furthermore, this application embodiment does not limit the form of the target application, and it can include, but is not limited to, applications, mini-programs, etc., installed on the terminal devices, and can also be in the form of web pages.

[0037] Server 105 can be a server providing various services, such as a backend management server supporting websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and basic cloud computing services such as big data. The server can be the backend server of the aforementioned target application, used to provide backend services to the clients of the target application.

[0038] It should be noted that the webpage element control method provided in this application embodiment can generally be executed by server 105 and / or terminal devices 101-103. Accordingly, the webpage element control device provided in this application embodiment can generally be set in server 105 and / or terminal devices 101-103.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0040] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0041] Figure 2A flowchart illustrating a webpage element control method according to an embodiment of this application is shown schematically. Figure 2 As shown, the web page element control method 200 according to an embodiment of this application may include steps S210 to S240.

[0042] In step S210, multiple candidate audio segments are processed by a pre-trained speech activity detection model to obtain the speech presence probability corresponding to each candidate audio segment.

[0043] According to embodiments of this application, multiple candidate audio segments are selected from multiple audio segments based on at least one of short-time energy and zero-crossing rate. Short-time energy indicates the amplitude strength of the audio segment; for example, a sudden spike in short-time energy suggests the audio segment is likely speech produced by a user. Zero-crossing rate characterizes the frequency characteristics of the audio segment; for example, unvoiced sounds in speech have a high zero-crossing rate.

[0044] The “voice activity model” can learn the relationship between input and output from training data, so that it can generate the corresponding output for a given input after training is completed.

[0045] For example, the short-time energy and zero-crossing rate of each audio segment are calculated, and audio segments with short-time energy greater than a preset energy threshold or zero-crossing rate greater than a preset zero-crossing rate threshold are selected as candidate audio segments. This filters noise while avoiding the omission of audio segments that may be suspected speech.

[0046] Here, multiple audio segments are obtained by dividing the acquired user speech into frames and arranging them based on time sequence.

[0047] For example, the input to a pre-trained speech activity detection model is candidate audio segments, and the output is the corresponding speech presence probability. The pre-trained speech activity detection model can be a pre-trained convolutional recurrent neural network, trained on a large amount of audio data labeled with speech presence probabilities.

[0048] In embodiments of this application, the user's consent or authorization can be obtained before acquiring the user's voice. For example, a request to acquire user information can be sent to the user before step S210. If the user consents or authorizes the acquisition of user information, step S210 is executed.

[0049] For example, before collecting a user's voice, the user can click the record button on the webpage. The webpage will then push a permission request page. Only after the user clicks to authorize on the page will the webpage have permission to access the microphone for recording.

[0050] In step S220, the target audio segment is obtained by filtering multiple candidate audio segments based on the probability of the presence of multiple speech sounds.

[0051] Furthermore, when the probability of speech presence in multiple consecutive candidate audio segments exceeds a preset first threshold, it is marked as the start of speech; when the probability of speech presence in multiple consecutive candidate audio segments is lower than a preset second threshold, it is marked as the end of speech; thus, candidate audio segments containing complete speech fragments are segmented as target audio segments.

[0052] For example, the first threshold can be 0.8 and the second threshold can be 0.2.

[0053] In step S230, speech recognition is performed on the target audio segment to obtain the speech recognition text.

[0054] For example, a speech recognition engine can be used to convert target audio segments into speech-recognition text.

[0055] In step S240, based on a number of preset first prompt words, the pre-trained large model is guided to process the speech recognition text and element index to obtain element control instructions, which are then transmitted to the webpage to control the webpage to perform the webpage operations indicated by the element control instructions.

[0056] According to embodiments of this application, an element index indicates the correspondence between user intent and webpage elements. For example, an element index may be an index structure that identifies user intent, corresponding elements, and element positions.

[0057] In one or more embodiments described herein, the term "large model" can refer to a deep learning model with a large number of model parameters, which can include hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. A large model can also be called a foundational model or basic model. It is pre-trained using a large-scale unlabeled corpus to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability; examples include large language models and multimodal pre-trained models.

[0058] It should be understood that in practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as natural language processing and computer vision. Specifically, they can be applied to computer vision tasks such as visual question answering, image description, and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation.

[0059] For example, multiple first prompt words, speech recognition text, and element indexes are input into a large model. Guided by the multiple first prompt words, the semantic understanding capability of the large model is used to process the speech recognition text and element indexes to obtain element control instructions.

[0060] In the embodiments of this application, candidate audio segments are obtained by screening multiple audio segments using short-time energy and zero-crossing rate, which filters noise while preventing audio segments suspected of being speech from being filtered out; effective data in multiple candidate audio segments is further screened using a speech activity detection model, which improves the processing efficiency of subsequent speech recognition; multiple first prompt words are used to guide the large model to process speech recognition text and element indexes using its own semantic understanding capabilities, which facilitates the parsing of complex instructions; at the same time, it is no longer necessary to process all information on the webpage, which improves the efficiency of obtaining element control instructions.

[0061] Figure 3 A flowchart illustrating the selection of candidate audio segments according to an embodiment of this application is shown.

[0062] Combination Figure 3 In some embodiments, multiple candidate audio segments are obtained using the following method:

[0063] In step S310, based on the amplitude of the sampling points in each audio segment, the short-time energy corresponding to each of the multiple audio segments is obtained.

[0064] According to embodiments of this application, the magnitude of short-term energy reflects the amplitude of the corresponding audio segment. When speaking, the voiced sounds produced by vocal cord vibration typically have higher energy; while pure silence and background noise have lower energy. In other words, the energy of human vocalization is usually much higher than that of ambient noise, thus enabling the distinction between noise and speech.

[0065] For example, the first Short-time energy of each audio segment It can be represented as:

[0066] ;

[0067] In the formula, Indicates the i-th audio segment. Normalized amplitude of each sampling point This represents the total number of sampling points. For example, It can be 320.

[0068] In step S320, based on the degree of change of adjacent sampling points in each audio segment, the zero-crossing rate corresponding to each of the multiple audio segments is obtained.

[0069] According to embodiments of this application, the zero-crossing rate refers to the number of times a time-domain waveform crosses the zero level (horizontal axis) within an audio frame. In other words, it's the number of times the sound waveform changes from positive to negative, or from negative to positive. The zero-crossing rate represents the frequency characteristics of an audio segment; the higher the frequency, the more times it crosses the zero point per unit time.

[0070] During human speech, voiceless consonants have very low short-time energy, and relying solely on short-time energy for judgment may result in their being filtered out. For example, the initial consonant / s / in the word "search" is a voiceless consonant with very low energy, but a high zero-crossing rate. Therefore, it is necessary to introduce the zero-crossing rate to segment and filter audio, avoiding the filtering out of suspected speech.

[0071] For example, the first Zero crossing rate of each audio segment Represented as:

[0072] ;

[0073] In the formula, This represents the sign function, with a positive value of 1, a negative value of -1, and a zero value of 0.

[0074] In step S330, audio segments with short-time energy greater than a preset energy threshold or zero-crossing rate greater than a preset zero-crossing rate threshold are selected as candidate audio segments.

[0075] Furthermore, the energy threshold It can be represented as:

[0076] ;

[0077] Zero crossing rate threshold It can be represented as:

[0078] ;

[0079] In the formula, This represents the energy threshold coefficient, which can be 2.0. Indicates average energy. This represents the zero-crossing rate threshold coefficient, which can be 1.5. This represents the average zero-crossing rate.

[0080] In the embodiments of this application, short-time energy and zero-crossing rate are used to screen candidate audio segments in order to improve the accuracy of candidate audio segment screening and avoid missing key information in the audio data.

[0081] In some embodiments, the method further includes: when the short-time energy is less than a first noise determination threshold, fusing the short-time energy and the energy threshold to update the energy threshold; and when the zero-crossing rate is less than a second noise determination threshold, fusing the zero-crossing rate and the zero-crossing rate threshold to update the zero-crossing rate threshold.

[0082] According to an embodiment of this application, a first noise determination threshold indicates the update scale of the energy threshold, and a second noise determination threshold indicates the update scale of the zero-crossing rate threshold.

[0083] For example, firstly, the short-time energy mean and zero-crossing rate threshold of the first 100 frames of audio data are used as the energy threshold and zero-crossing rate threshold, respectively. Then, for each frame processed, if the short-time energy of the frame data is less than the first noise determination threshold, the corresponding short-time energy and short-time energy threshold are weighted and summed to update the energy threshold; if the zero-crossing rate of the frame data is less than the second noise determination threshold, the corresponding zero-crossing rate and zero-crossing rate threshold are weighted and summed to update the zero-crossing rate threshold.

[0084] For example, the updated energy threshold Represented as:

[0085] ;

[0086] In the formula, This represents the smoothing factor (usually 0.95). This represents the first noise threshold. Indicates a multiple, such as 1.2.

[0087] Updated zero-crossing threshold Represented as:

[0088] ;

[0089] In the formula, This indicates the second noise threshold.

[0090] In the embodiments of this application, in order to adapt to changes in environmental noise, the energy threshold and zero-crossing rate threshold are updated according to the actual situation of the processed audio segments, so that the threshold settings fit the actual application scenario and improve the accuracy of candidate audio segment screening.

[0091] In some embodiments, performing speech recognition on a target audio segment to obtain speech-recognized text includes: processing the target audio segment using a pre-trained decoder based on a keyword set to obtain speech-recognized text.

[0092] According to embodiments of this application, the keyword set is obtained in advance by performing at least one of keyword extraction, related word extraction, and semantic expansion on key elements of a webpage.

[0093] For example, the keyword set is obtained as follows: First, keywords are extracted from the text of key elements on the webpage using regular expressions to obtain multiple primary keywords; for example, the word "search" in the search box, and the button text "buy now". Then, related words adjacent to or related to the keyword areas are extracted from the semantic map element text to obtain multiple secondary keywords; for example, the "find" button and "filter" options near the search box. Next, the aforementioned keywords and related words are semantically expanded based on a thesaurus to obtain multiple tertiary keywords, forming a keyword set with the aforementioned primary and secondary keywords; for example, "search" is expanded to "search", "retrieve", and "find"; "buy" is expanded to "buy", "place an order", and "pay". The thesaurus contains multiple words and their corresponding synonyms.

[0094] For example, the decoder can be a speech recognition engine (Vosk). The target audio segments are processed by a pre-trained decoder to obtain the speech-recognized text. During the decoder's beam search, decoding paths containing any word from the keyword set receive additional bias points, improving the recognition rate of words relevant to the webpage.

[0095] In the embodiments of this application, a set of keywords related to web pages is introduced for speech recognition to improve the accuracy of language conversion.

[0096] Figure 4 A flowchart illustrating the construction of an element index according to an embodiment of this application is shown.

[0097] Combination Figure 4 In some embodiments, the element index is obtained in the following way:

[0098] First, based on the preset search criteria, the document object model tree 401 of the webpage is traversed to obtain multiple elements 402.

[0099] According to embodiments of this application, multiple elements each have attribute information. The Document Object Model (DOM) tree 401 indicates all elements in a webpage and the relationships between them. The DOM tree 401 of the current page can be obtained through the global document object of the webpage. The DOM tree 401 can include all webpage tags, text, attributes, etc.

[0100] For example, attribute information may include the tag name, identifier, all attributes (such as name, type, value, placeholder, title, label, etc.), internal text, coordinates, and size of the corresponding element.

[0101] For example, search criteria may include buttons, links, inputs, selections, text areas, details, elements with click attributes or roles="button" or "link", elements with tag index attributes that are not -1, associated form controls, field-level elements, options, audio or video control buttons, custom components, etc.

[0102] Then, multiple elements 402 are filtered based on preset filtering conditions to obtain multiple candidate elements 403.

[0103] According to embodiments of this application, the filtering conditions include at least one of size, style, position, and status.

[0104] Furthermore, considering that there may be invisible or inoperable elements in the webpage, in order to improve the efficiency of subsequent large model processing, elements can be filtered using conditions such as size, style, position, and state.

[0105] For example, the filtering conditions may include: size-based filtering conditions, such as elements whose width or height is less than a preset size threshold, which may be 5 pixels; style-based filtering conditions, such as elements that are displayed as none, whose visibility is hidden or whose opacity is 0, and elements that have no pointer events; position-based filtering conditions, such as elements that are located outside the webpage viewport and have no possibility of being scrolled to be visible; and state-based filtering conditions, such as buttons that have disabled attributes.

[0106] Then, based on at least one of the element type, attribute information, element area, and element center point of multiple candidate elements 403, multiple importance scores are obtained, and multiple target elements 404 are obtained by filtering among multiple candidate elements 403 based on multiple importance scores.

[0107] For example, multiple importance scores are obtained by weighted fusion based on the scores of the element type, attribute information, element area, and element center point of multiple candidate elements 403. Candidate elements with importance scores greater than a preset threshold are selected as target elements.

[0108] Then, based on at least one of the identifiers, names, tags, positions, and attributes of the multiple target elements 404, the element identifiers of the multiple target elements 404 are determined, and the multiple element identifiers are associated with the positions of their respective target elements to form element index 405.

[0109] For example, the identifier or name of the target element is preferred as the element identifier. In the absence of an identifier and name, position, tag, and attribute are used as element identifiers.

[0110] In the embodiments of this application, visibility filtering and importance scoring are used to ensure that the elements in the element index are those that users are actually likely to operate on, thus avoiding irrelevant elements from interfering with the large model. At the same time, the amount of data processing is reduced, and the data transmission time and processing overhead of the large model are lowered.

[0111] Figure 5 A flowchart illustrating the calculation of importance scores according to an embodiment of this application is shown.

[0112] Combination Figure 5 In some embodiments, multiple importance scores are obtained based on at least one of the following: element type 501, attribute information, element area, and element center point of multiple candidate elements:

[0113] First, based on the element type 501 of multiple candidate elements and the preset score 502 of the element type, the element type factor of multiple candidate elements is obtained.

[0114] According to embodiments of this application, the element type factor indicates the interaction probability of an element type. Different types of elements have different interaction probabilities; for example, buttons, links, form controls, etc., are more likely to be the target of user interaction than other elements.

[0115] For example, different element types correspond to different preset branches. For instance, an element type of button corresponds to a score of 1; an element type of link corresponds to a score of 0.9; an element type of input box / selection / text field corresponds to a score of 0.8; an element type of clickable element corresponds to a score of 0.6; an element type of form auxiliary element corresponds to a score of 0.5; and an element type of ordinary container corresponds to a score of 0.1.

[0116] Then, based on the proportion of non-empty characters (503) in multiple attribute information, the text existence factor of multiple candidate elements is obtained.

[0117] According to embodiments of this application, the text presence factor indicates the degree to which candidate elements match user intent. Elements containing visible text often provide semantic cues (such as text on buttons) that aid in matching user intent and are therefore given priority.

[0118] For example, text existence factor It can be represented as:

[0119] .

[0120] Then, based on the ratio of the area of ​​multiple elements to the viewport area of ​​the webpage (504), the area normalization factor of multiple candidate elements is obtained.

[0121] According to embodiments of this application, the area normalization factor indicates the salience of a candidate element relative to the webpage. The area ratio of an element in the viewport affects its salience. Elements that are too small (such as icons) are difficult to click, while elements that are too large (such as full-screen overlays) are often not the primary target of operation; medium-sized elements (such as buttons and input boxes) best match typical interaction patterns.

[0122] For example, area normalization factor Represented as:

[0123] ;

[0124] ;

[0125] In the formula, Represents the area of ​​an element. Indicates the viewport area. Indicates the area percentage. This indicates the lower limit of the ideal area ratio, which can be 1%. This represents the minimum score when the area percentage is 0, which can be 0.1. This indicates the upper limit of the ideal area ratio, which can be 10%. This indicates the width of the attenuation range.

[0126] Then, based on the distance of 505 between the center point of multiple elements and the viewport center of the webpage, the center distance factor of multiple candidate elements is obtained.

[0127] According to embodiments of this application, the center distance factor indicates the degree of visual focus of a candidate element relative to the webpage. A user's visual focus is typically located near the center of the viewport; therefore, elements closer to the center of the viewport are more likely to be noticed and manipulated.

[0128] Furthermore, firstly, a normalized distance is obtained based on the distance between the element's center point and the viewport center of the webpage, as well as the maximum possible distance between the element's center point and the viewport center of the webpage; then, a center distance factor is obtained based on the difference between a preset constant and the normalized distance. Here, the preset constant can be 1.

[0129] For example, the distance between the center point of an element and the center of the viewport of the webpage. It can be represented as:

[0130] ;

[0131] The maximum possible distance between the center point of an element and the center of the webpage's viewport. It can be represented as:

[0132] ;

[0133] Normalized distance table It is shown as:

[0134] ;

[0135] Center distance factor Represented as:

[0136] ;

[0137] In the formula, This represents the x-coordinate of the center point of the element. This represents the x-coordinate of the viewport center point. This represents the y-coordinate of the center point of the element. This represents the y-coordinate of the viewport center point. Indicates the viewport width. Indicates the viewport height.

[0138] Then, multiple element type factors 509, multiple text existence factors 508, multiple area normalization factors 507, and multiple center distance factors 506 are fused to obtain multiple importance scores 510.

[0139] For example, multiple importance scores are obtained by weighting and summing multiple element type factors, multiple text existence factors, multiple area normalization factors, and multiple center distance factors. Here, the sum of the weight parameters is 1.

[0140] In the embodiments of this application, multiple dimensions such as visual saliency, interaction type, and semantic cues are comprehensively considered to evaluate the importance of elements, improve the comprehensiveness and accuracy of importance score calculation, ensure that the element index focuses on the element most likely to be pointed to by the user's intent, and provide high-quality context for subsequent large model instruction parsing.

[0141] In some embodiments, the plurality of first prompt words include a plurality of first sub-prompt words and a plurality of second sub-prompt words, wherein the plurality of first sub-prompt words are used to indicate the role positioning of the large model, and the plurality of second sub-prompt words are used to indicate the task requirements; based on the preset plurality of first prompt words, the pre-trained large model is guided to process speech recognition text and element indexes to obtain element control instructions, including:

[0142] By using multiple first sub-copula words and multiple second sub-copula words, the large model is guided to process speech recognition text and element indexes according to task requirements under role positioning, and obtain element control instructions.

[0143] For example, multiple first prompt words can be represented as:

[0144] "You are a browser intelligent assistant responsible for processing the input speech recognition text and element index, finding element identifiers in the element index that match the natural language instructions indicated by the speech recognition text, and converting them into actions that can be performed on the current webpage."

[0145] Multiple secondary cue words can be represented as:

[0146] "Please output the element control instructions in JSON format, including the action type (such as click, input, scroll) and the target element identifier, as well as the necessary parameters (such as input values).

[0147] When no matching element is found, the return format can be {"action":"empty"}.

[0148] In the embodiments of this application, a large model is used to process speech recognition text and element indexes under the guidance of the first and second sub-prompt words. By leveraging the semantic understanding capabilities of the large model and combining it with the contextual information of the webpage, the accuracy of element control command parsing is improved.

[0149] In some embodiments, the method further includes: guiding large model verification element control instructions based on a plurality of preset second prompt words, wherein the plurality of second prompt words are used to indicate the structural verification reasoning steps for the element control instructions.

[0150] According to embodiments of this application, the structural verification reasoning step may include the role positioning of the large model, verification rules for element control instructions, and output format requirements.

[0151] For example, element control instructions can be represented as:

[0152] "{"Action":"Click","Target":"Search Button"}.

[0153] Multiple secondary cue words can be represented as:

[0154] "You are an element control instruction verification expert. Your task is to verify and correct element control instructions."

[0155] Verification rules:

[0156] 1. Formatting rules: It must be a valid JSON object.

[0157] 2. Field Integrity:

[0158] An "Action" field is required, and the value can only be "Click", "Input", "Scroll", or "Empty".

[0159] If the action is "click", a "target" field is required.

[0160] If the action is "Input", it must have "Target" and "Value" fields.

[0161] 3. Existence rule:

[0162] The value of "target" must exist in the element index.

[0163] 4. Semantic rationality:

[0164] The type of action must match the actual type of the element (e.g., do not perform an input operation on an image).

[0165] Please perform the following operations:

[0166] 1. Check whether the instruction to be verified conforms to the above rules.

[0167] 2. If it matches, output it as is.

[0168] 3. If it does not meet the requirements, please correct it (e.g., complete the fields, correct the target) and then output it.

[0169] 4. If the error cannot be corrected (e.g., the element does not exist), output {"Action": "Empty", "Reason": "Error Reason"}.

[0170] In the embodiments of this application, the semantic understanding capability of the large model is used to perform structural verification of element control instructions, reducing the probability of the large model exhibiting illusions and ensuring the executability of element control instructions in actual web page element control.

[0171] Based on the above-described webpage element control method, embodiments of this application also provide a webpage element control device. The following will be combined with... Figure 6 The device is described in detail.

[0172] Figure 6 A schematic block diagram of a web page element control device according to an embodiment of this application is shown.

[0173] like Figure 6 As shown, the web page element control device 600 of this embodiment includes a voice detection module 610, a filtering module 620, a voice recognition module 630, and an instruction parsing module 640.

[0174] The speech detection module 610 processes multiple candidate audio segments using a pre-trained speech activity detection model to obtain the speech presence probability corresponding to each candidate audio segment. The multiple candidate audio segments are selected from among the multiple audio segments based on at least one of the short-time energy and zero-crossing rate of each audio segment. The short-time energy indicates the amplitude strength of the audio segment, and the zero-crossing rate characterizes the frequency characteristics of the audio segment. In one embodiment, the speech detection module can be used to perform step S210 described above, which will not be repeated here.

[0175] The filtering module 620 is used to filter multiple candidate audio segments based on the probability of the presence of multiple speech sounds to obtain the target audio segment. In one embodiment, the filtering module can be used to perform step S220 described above, which will not be repeated here.

[0176] The speech recognition module 630 is used to perform speech recognition on the target audio segment to obtain the speech-recognized text. In one embodiment, the speech recognition module can be used to perform step S230 described above, which will not be repeated here.

[0177] The instruction parsing module 640 is used to guide a pre-trained large model to process speech recognition text and element indexes based on a plurality of preset first prompt words, obtain element control instructions, and transmit them to the webpage to control the webpage to perform the webpage operations indicated by the element control instructions. The element index indicates the correspondence between the user's intent and the webpage elements. In one embodiment, the instruction parsing module can be used to execute step S240 described above, which will not be repeated here.

[0178] According to an embodiment of this application, the instruction parsing module 640 is specifically used to guide the large model to process speech recognition text and element indexes according to task requirements under role positioning by using multiple first sub-prompt words and multiple second sub-prompt words, so as to obtain element control instructions.

[0179] According to an embodiment of this application, the speech recognition module 630 is specifically used to process target audio segments using a pre-trained decoder based on a keyword set to obtain speech recognition text. The keyword set is obtained in advance by performing at least one of keyword extraction, related word extraction, and semantic expansion on key elements of the webpage.

[0180] According to an embodiment of this application, the device further includes: a verification module, used to guide a large model to verify element control instructions based on a plurality of preset second prompt words, wherein the plurality of second prompt words are used to indicate the structural verification reasoning steps for the element control instructions.

[0181] According to embodiments of this application, any multiple modules among the voice detection module 610, screening module 620, voice recognition module 630, and instruction parsing module 640 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the voice detection module 610, screening module 620, voice recognition module 630, and instruction parsing module 640 can be at least partially implemented as hardware circuitry, such as field-programmable gate arrays, programmable logic arrays, systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits, or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the speech detection module 610, the filtering module 620, the speech recognition module 630, and the instruction parsing module 640 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0182] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a web page element control method according to an embodiment of this application.

[0183] like Figure 7 As shown, an electronic device 700 according to an embodiment of this application includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory 702 or a program loaded from a storage portion 708 into a random access memory 703. The processor 701 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a dedicated microprocessor. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for executing different steps of the method flow according to an embodiment of this application.

[0184] Random access memory 703 stores various programs and data required for the operation of electronic device 700. Processor 701, read-only memory 702, and random access memory 703 are interconnected via bus 704. Processor 701 executes various steps of the method flow according to embodiments of this application by executing programs in read-only memory 702 and / or random access memory 703. It should be noted that the programs may also be stored in one or more memories other than read-only memory 702 and random access memory 703. Processor 701 may also execute various steps of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0185] According to embodiments of this application, the electronic device 700 may further include an input / output interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube, liquid crystal display, etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card, such as a local area network card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0186] Embodiments of this application also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0187] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include the read-only memory 702, and / or random access memory 703, and / or one or more memories other than read-only memory 702 and random access memory 703 described above.

[0188] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0189] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0190] In embodiments of this application, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by processor 701, it performs the functions defined in the system of embodiments of this application. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0191] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0193] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A method for controlling web page elements, characterized in that, The method includes: Multiple candidate audio segments are processed by a pre-trained speech activity detection model to obtain the speech presence probability corresponding to each candidate audio segment. The multiple candidate audio segments are selected from multiple audio segments based on at least one of the short-time energy and zero-crossing rate of each audio segment. The short-time energy indicates the amplitude strength of the audio segment, and the zero-crossing rate characterizes the frequency characteristics of the audio segment. The target audio segment is obtained by filtering from the multiple candidate audio segments based on the probability of the presence of multiple speech segments; Speech recognition is performed on the target audio segment to obtain the speech-recognized text; Based on a number of preset first prompt words, a pre-trained large model is guided to process the speech recognition text and element index to obtain element control instructions, which are then transmitted to the webpage to control the webpage to perform the webpage operations indicated by the element control instructions. The element index indicates the correspondence between the user's intent and the webpage elements.

2. The method according to claim 1, characterized in that, The multiple candidate audio segments are obtained using the following method: Based on the amplitude of the sampling points in each of the audio segments, the short-time energy corresponding to each of the multiple audio segments is obtained; Based on the degree of change of adjacent sampling points in each of the audio segments, the zero-crossing rate corresponding to each of the multiple audio segments is obtained; Audio segments with short-term energy greater than a preset energy threshold or zero-crossing rate greater than a preset zero-crossing rate threshold are selected as candidate audio segments.

3. The method according to claim 2, characterized in that, The method further includes: If the short-time energy is less than the first noise determination threshold, the short-time energy and the energy threshold are fused to update the energy threshold, whereby the first noise determination threshold indicates the update scale of the energy threshold. If the zero-crossing rate is less than the second noise determination threshold, the zero-crossing rate is fused with the zero-crossing rate threshold to update the zero-crossing rate threshold, where the second noise determination threshold indicates the update scale of the zero-crossing rate threshold.

4. The method according to any one of claims 1 to 3, characterized in that, The element index is obtained using the following method: Based on preset search conditions, the document object model tree of the webpage is traversed to obtain multiple elements, each of which has attribute information. The document object model tree indicates all elements in the webpage and the relationships between them. The multiple elements are filtered based on preset filtering conditions to obtain multiple candidate elements. The filtering conditions include at least one of size, style, position and status. Based on at least one of the element type, attribute information, element area, and element center point of multiple candidate elements, multiple importance scores are obtained, and multiple target elements are obtained by filtering among the multiple candidate elements based on the multiple importance scores. Based on at least one of the identifiers, names, tags, positions, and attributes of the plurality of target elements, the element identifiers of the plurality of target elements are determined, and the plurality of element identifiers are associated with the positions of their respective target elements to form an element index.

5. The method according to claim 4, characterized in that, The importance scores are obtained based on at least one of the following factors among multiple candidate elements: element type, attribute information, element area, and element center point: Based on the element types of the multiple candidate elements and the preset scores of the element types, an element type factor of the multiple candidate elements is obtained, and the element type factor indicates the interaction probability of the element type. Based on the proportion of non-empty characters in multiple attribute information, the text existence factor of the multiple candidate elements is obtained, and the text existence factor indicates the degree of matching of the candidate elements with respect to the user intent. Based on the proportion of the area of ​​multiple elements relative to the viewport area of ​​the webpage, the area normalization factor of the multiple candidate elements is obtained, and the area normalization factor indicates the significance of the candidate elements relative to the webpage. Based on the distance between the center points of multiple elements and the viewport center of the webpage, a center distance factor for the multiple candidate elements is obtained, and the center distance factor indicates the degree of visual focus of the candidate elements relative to the webpage. Multiple element type factors, multiple text existence factors, multiple area normalization factors, and multiple center distance factors are fused to obtain the multiple importance scores.

6. The method according to claim 1, characterized in that, The plurality of first prompt words include a plurality of first sub-prompt words and a plurality of second sub-prompt words. The plurality of first sub-prompt words are used to indicate the role positioning of the large model, and the plurality of second sub-prompt words are used to indicate the task requirements. The process of guiding a pre-trained large model to process the speech recognition text and element index based on a plurality of preset first prompt words to obtain element control instructions includes: Using the multiple first sub-prompt words and the multiple second sub-prompt words, the large model is guided to process the speech recognition text and the element index according to the task requirements under the role positioning, and obtain the element control instructions.

7. The method according to claim 1, characterized in that, The process of performing speech recognition on the target audio segment to obtain the speech recognition text includes: Based on the keyword set, the target audio segments are processed using a pre-trained decoder to obtain the speech recognition text. The keyword set is obtained in advance by performing at least one of keyword extraction, related word extraction, and semantic expansion on the key elements of the webpage.

8. The method according to claim 1, characterized in that, The method further includes: Based on a number of preset second prompt words, the large model is guided to verify the element control instructions. The number of second prompt words are used to indicate the structural verification reasoning steps for the element control instructions.

9. A webpage element control device, characterized in that, The device includes: The speech detection module is used to process multiple candidate audio segments through a pre-trained speech activity detection model to obtain the speech presence probability corresponding to each candidate audio segment. The multiple candidate audio segments are selected from multiple audio segments based on at least one of the short-time energy and zero-crossing rate of each audio segment. The short-time energy indicates the amplitude strength of the audio segment, and the zero-crossing rate characterizes the frequency characteristics of the audio segment. The filtering module is used to filter from the multiple candidate audio segments based on the probability of the presence of multiple speech sounds to obtain the target audio segment; A speech recognition module is used to perform speech recognition on the target audio segment to obtain the speech-recognized text; and The instruction parsing module is used to guide a pre-trained large model to process the speech recognition text and element index based on a number of preset first prompt words, obtain element control instructions, and transmit them to the webpage to control the webpage to execute the webpage operation indicated by the element control instructions. The element index indicates the correspondence between the user's intent and the webpage element.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images