Script generation method and device based on browser interaction, equipment and medium

Abstract

The invention relates to the technical field of development, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a script generation method, device and equipment based on browser interaction and a medium, the method comprises the following steps: obtaining a browser interaction demand of a target user, and carrying out demand splitting on the browser interaction demand to obtain an interaction operation step; identifying an interaction operation type corresponding to the interaction operation step, and loading target interaction extension item information by using the interaction operation type; obtaining a target markup language corresponding to the target interaction extension item information, and extracting operable marking element information in the target markup language; performing page element identification on a pre-acquired browser tab to obtain page element information; and generating an automatic interaction script corresponding to the browser interaction demand according to the interaction operation type, the operable mark element information and the page element information. According to the invention, script generation automation based on user interaction requirements can be realized, and script generation efficiency can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the development technical field, and in particular to a script generation method and device based on browser interaction, equipment and medium. BACKGROUND

[0002] In the process of using automation tools, traditional Robotic process automation (RPA) tools face the problem of high technical threshold and high learning cost in interactive applications, and it is difficult to meet the use requirements of non-technical users. Specifically, the traditional RPA tool usually requires the target user to have programming or script writing ability, needs to be familiar with programming languages such as Python and Java, or master the exclusive syntax and configuration method of a specific tool, which leads non-technical personnel (such as business personnel or ordinary users) to have to rely on the support of developers in the use process, which not only increases the human communication cost, but also reduces the efficiency of completing interactive tasks; at the same time, the interface design and operation process of the traditional RPA tool are generally complex, and the target user needs to spend a lot of time learning the use method of the function module (such as process designer, script editor, debugging tool, etc.), which further restricts the independent use of automation tools by non-technical personnel.

[0003] For example, in the medical health scene, medical staff often need to process high-frequency interactive tasks such as electronic medical record archiving and medical insurance reimbursement data verification. The traditional RPA tool requires the target user to master programming languages such as Python or exclusive syntax, and the medical staff lack relevant technical background and have to rely on developers to customize target scripts, which not only causes the patient service response to be slow due to communication delay, but also may cause related medical data entry errors due to demand transmission deviation, further reducing the medical work efficiency.

[0004] For example, in the financial technology scene, tellers and risk control personnel need to complete transaction flow reconciliation, credit data review and other highly compliant tasks. The function modules (such as exception handling and loop operation) of the traditional RPA tool are difficult to learn, and non-technical personnel such as tellers and risk control personnel cannot independently configure task scripts, but relying on technical team support will prolong the customer business handling time, and syntax errors in the task script configuration process may cause financial data leakage, further reducing the task processing efficiency.

[0005] Therefore, how to realize script generation automation based on user interaction requirements and improve script generation efficiency has become a problem to be solved. SUMMARY

[0006] The application provides a script generation method and device based on browser interaction, an electronic device and a medium.

[0007] In a first aspect, to achieve the above object, the application provides a script generation method based on browser interaction, comprising: obtaining browser interaction requirements of a target user, performing requirement decomposition on the browser interaction requirements to obtain a plurality of interaction operation steps; identifying a plurality of interaction operation types corresponding to the interaction operation steps, and loading target interaction extension item information of the browser interaction requirements by using the plurality of interaction operation types; obtaining a target markup language corresponding to the target interaction extension item information, and extracting operable markup element information in the target markup language; performing page element recognition on a browser tab page obtained in advance to obtain page element information; generating an automatic interaction script corresponding to the browser interaction requirements according to the interaction operation types, the operable markup element information and the page element information.

[0008] In a second aspect, the application further provides a script generation device based on browser interaction, comprising: An interaction requirement decomposition module is configured to obtain browser interaction requirements of a target user, perform requirement decomposition on the browser interaction requirements to obtain a plurality of interaction operation steps; An extension item loading module is configured to identify a plurality of interaction operation types corresponding to the interaction operation steps, and load target interaction extension item information of the browser interaction requirements by using the plurality of interaction operation types; A markup information extraction module is configured to obtain a target markup language corresponding to the target interaction extension item information, and extract operable markup element information in the target markup language; A page element recognition module is configured to perform page element recognition on a browser tab page obtained in advance to obtain page element information; An interaction script generation module is configured to generate an automatic interaction script corresponding to the browser interaction requirements according to the interaction operation types, the operable markup element information and the page element information.

[0009] In a third aspect, the application further provides an electronic device, comprising: at least one processor; and a memory in communication with the at least one processor; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the browser interaction-based script generation method described above.

[0010] In a fourth aspect, the present application further provides a computer readable storage medium, which stores at least one computer program, and the at least one computer program is executed by a processor in an electronic device to implement the browser interaction-based script generation method described above.

[0011] In the embodiments of the present application, by performing demand splitting on the browser interaction demand of the target user, the complex browser interaction demand can be converted into simple and orderly interaction operation steps, operation errors caused by unclear demand can be reduced, and user interaction efficiency can be improved; the multiple interaction operation types corresponding to the interaction operation steps are identified, and the target interaction extension item information is loaded accordingly, which can improve the pertinence and accuracy of the interaction, avoid unnecessary resource occupation, enhance the expansibility of the browser interaction function, and quickly adapt to diversified interaction scenarios.

[0012] Specifically, by obtaining the target markup language corresponding to the target interaction extension item information and extracting the operable markup element information, the interaction key points can be accurately located, the interaction resource utilization rate can be improved, the page element information can be obtained by performing page element recognition on the browser tab page, and the key basis can be provided for subsequent page interaction design, script generation and the like, and the stability and ease of use of script generation can be enhanced; the automatic interaction operation is realized by the automatic interaction script, the tediousness and time consumption of manual operation are avoided, the accuracy and consistency of each interaction operation are ensured, the automatic interaction script of the browser interaction demand is generated according to the interaction operation type, the operable markup element information and the page element information, and the script generation efficiency and script generation accuracy can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0014] Figure 1 An application environment diagram of a browser interaction-based script generation method in an embodiment of the present application; Figure 2 A flow diagram of a browser interaction-based script generation method provided by an embodiment of the present application; Figure 3A flowchart of a process for generating an automatic interaction script corresponding to a browser interaction requirement according to an embodiment of the present application is provided. Figure 4 A module diagram of a script generation device based on browser interaction according to an embodiment of the present application is provided. Figure 5 A structure diagram of an electronic device for implementing a script generation method based on browser interaction according to an embodiment of the present application is provided. Figure 6 Another structure diagram of an electronic device for implementing a script generation method based on browser interaction according to an embodiment of the present application is provided.

[0015] The purposes, functional features and advantages of the present application will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0016] In order to make the person in the art better understand the technical solutions of the present disclosure, and to fully understand and implement the implementation process of the present disclosure how to apply technical means to solve technical problems and achieve corresponding technical effects, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, not all embodiments. The embodiments of the present disclosure and each feature in the embodiments can be combined with each other without conflict, and the technical solutions formed thereby are all within the protection scope of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative labor should be within the scope of protection of the present disclosure.

[0017] It should be noted that the terms "first", "second" and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product or equipment including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or equipment.

[0018] The embodiment of the application provides a script generation method based on browser interaction, and the execution subject of the script generation method based on browser interaction includes but is not limited to at least one of electronic devices such as a server, a terminal and the like which can be configured to execute the device provided by the embodiment of the application. In other words, the script generation method based on browser interaction can be executed by software or hardware installed in a terminal device or a server device. The server includes but is not limited to a single server, a server cluster, a cloud server or a cloud server cluster and the like. The server can be a stand-alone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0019] The script generation method based on browser interaction can be applied in an application environment such as Figure 1 In the application environment, a client communicates with a server through a network. The server can obtain browser interaction requirements of a target user through the client, and can convert complex browser interaction requirements into simple and orderly interaction operation steps through requirement splitting of the browser interaction requirements of the target user, so as to reduce operation errors caused by unclear requirements, improve user interaction efficiency, identify a plurality of interaction operation types corresponding to the interaction operation steps and load target interaction extension item information according to the interaction operation types, so as to improve the pertinence and accuracy of interaction, avoid unnecessary resource occupation, enhance the expansibility of browser interaction functions, and quickly adapt to diversified interaction scenarios.

[0020] Specifically, by obtaining target markup language corresponding to target interaction extension item information and extracting operable markup element information, the interaction key points can be accurately positioned, the interaction resource utilization rate can be improved, page element recognition is performed on a browser tab page to obtain page element information, key basis is provided for subsequent page interaction design, script generation and the like, and the stability and ease of use of script generation are enhanced. Automatic interaction operation is realized through an automatic interaction script, the tediousness and time consumption of manual operation are avoided, the accuracy and consistency of each interaction operation are ensured, an automatic interaction script of browser interaction requirements is generated according to the interaction operation type, the operable markup element information and the page element information, the script generation efficiency and the script generation accuracy are improved, and finally the automatic interaction script is output and fed back to the client.

[0021] The client can be but is not limited to various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be realized by a stand-alone server or a server cluster composed of a plurality of servers. The application will be described in detail through specific embodiments.

[0022] Reference Figure 2 The diagram shown is a flowchart illustrating a browser-interactive script generation method according to an embodiment of the present invention. In this embodiment, the browser-interactive script generation method includes: S1. Obtain the browser interaction requirements of the target user, and break down the browser interaction requirements to obtain multiple interaction operation steps.

[0023] In this embodiment of the invention, the browser interaction requirements refer to the various interactive operations that the target user expects to perform with the browser during the use of the browser, as well as the effects and functional requirements that the interactive operations should achieve; the browser interaction requirements include the target user's needs in multiple aspects such as browsing web pages, inputting information, clicking buttons, scrolling pages, and using browser plugins.

[0024] In detail, the requirement decomposition refers to the process of breaking down browser interaction requirements into multiple relatively independent and operable interaction steps; for example, the browser interaction requirement of "implementing a convenient search function in the browser" can be broken down into the interaction steps of the target user entering search keywords, the interaction steps of the browser obtaining search keywords and sending them to the search engine, and the interaction steps of the browser receiving the search results returned by the search engine and displaying them to the target user.

[0025] In this embodiment of the invention, the step of breaking down the browser interaction requirements into multiple interaction operation steps includes: The browser interaction requirements are subjected to structured semantic parsing to obtain key interaction elements. Based on the key interactive elements, the browser interaction requirements are categorized to obtain the target requirement categories; The target requirement category is broken down into multiple preliminary interaction units; Logical correlation verification is performed on multiple preliminary interaction units to generate an ordered sequence of interaction units; Multiple interactive operation steps are generated based on the ordered sequence of interactive units.

[0026] In this embodiment of the invention, the structured semantic parsing refers to converting unstructured or semi-structured browser interaction requirements into structured data and extracting key semantic information for subsequent processing and analysis. Specifically, text segmentation technology can be used to break down the text description of browser interaction requirements into individual words, and then part-of-speech tagging technology can be used to tag each word with its part of speech, such as noun, verb, adjective, etc. Named entity recognition technology can be used to identify key entities in the tagged browser interaction requirements, such as operation objects (e.g., "search box" and "submit button").

[0027] In detail, a category division rule is defined in advance, the extracted key demand interaction elements are matched with the category division rule, and a target demand category of the browser interaction demand is determined according to a matching result; for example, if the key demand interaction elements contain a “search” related operation object and operation type, the browser interaction demand is divided into a “search type demand”; if “form filling” related elements are contained, it is divided into a “form filling type demand”, and the like.

[0028] Among them, different target demand categories are determined to be able to be disassembled into preliminary interaction units; for example, for “search type demand”, it can be disassembled into “input keywords in search box”, “click search button” and the like preliminary interaction units.

[0029] Specifically, the preliminary interaction units are sorted by analyzing the dependency relationship between the preliminary interaction units to generate an ordered interaction unit sequence; a directed acyclic graph (DAG) and the like data structure can be used to represent the logical relationship between the preliminary interaction units, and a topological sorting algorithm is used to determine the ordered interaction unit sequence.

[0030] Further, for each ordered interaction unit in the ordered interaction unit sequence, a specific operation step description is generated according to the operation object corresponding to the ordered interaction unit; for example, for the ordered interaction unit “input keywords in search box”, an interaction operation step description such as “move the mouse to the search box position, click the search box, and then input the specified keywords” is generated.

[0031] Among them, the interaction operation step descriptions of all ordered interaction units are combined in the order of the ordered interaction unit sequence to form a plurality of complete interaction operation steps, which can guide the browser to perform the corresponding interaction operation in the correct order to meet the browser interaction demand of the target user.

[0032] Exemplarily, in the field of financial technology, taking the interaction demand of a bank customer to complete the online fund purchase process through a browser as an example, first, the browser interaction demand is structured and semantically analyzed, and through natural language processing technology, the key demand interaction elements are extracted from the demand text "I want to buy a fund product of a certain fund company, and the investment amount is 5,000 yuan" input by the user, including the fund company name, the fund product name, and the investment amount; then, the browser interaction demand is classified according to these key demand interaction elements, and since it involves fund product purchase and fund operation, it is classified as a financial transaction type target demand category; then, the target demand category is decomposed into multiple preliminary interaction units, such as a bank account login unit for verifying the user's identity, a fund product selection unit for facilitating the user to select the target product from a plurality of funds, an investment amount input unit for the user to set the investment amount, and a transaction confirmation unit for the user to finally confirm the transaction information.

[0033] After that, the logical correlation of the multiple preliminary interaction units is verified, considering the business logic and operation sequence, an ordered interaction unit sequence is generated, that is, first logging into the bank account, then selecting the fund product, then inputting the investment amount, and finally confirming the transaction; finally, multiple interaction operation steps are generated according to the ordered interaction unit sequence, in the browser interface design, the first step is to pop up a login window to guide the user to input the account number and password; the second step is to display a fund product list and provide search and filtering functions; the third step is to pop up an amount input box after selecting the product; the fourth step is to display the transaction details for the user to confirm after the user inputs the amount, and the interaction operation steps for the entire fund purchase process are generated, ensuring that the user can clearly and smoothly complete the financial transaction operation on the browser.

[0034] In the embodiment of the present application, by decomposing the browser interaction demand of the target user, the complex browser interaction demand can be converted into simple and ordered interaction operation steps, reducing the operation errors caused by unclear demand and improving the user interaction efficiency.

[0035] S2, identify a plurality of interaction operation types corresponding to the interaction operation steps, and load target interaction extension item information of the browser interaction demand by using the plurality of interaction operation types.

[0036] In the embodiment of the present application, the target interaction extension item information is related information or a functional module loaded additionally to meet specific browser interaction demands; for example, when the interaction operation type is identified as an image viewing operation, the target interaction extension item information is a code plug-in of functions such as image enlargement, reduction, and rotation; if the interaction operation type involves video playback, the target interaction extension item information is a video decoder, a playback control plug-in, or the like.

[0037] The method comprises the following steps: extracting an interaction operation identifier in the interaction operation step; calculating a similarity matching value between the interaction operation identifier and an operation type label in a pre-acquired operation type feature library; taking an operation type label with a similarity matching value greater than a preset similarity threshold value as a candidate operation type of the interaction operation step; cross- verifying the candidate operation type to obtain a plurality of interaction operation types corresponding to the interaction operation step.

[0038] In the embodiment of the application, the text description of the interaction operation step is subjected to word segmentation processing, the interaction operation step after the word segmentation processing is screened by a keyword extraction algorithm to obtain keywords, then the keywords are integrated and refined by semantic analysis to form an interaction operation identifier capable of representing the interaction operation step.

[0039] For example, for the interaction operation step "click the login button on the webpage by using the left mouse button", the extracted interaction operation identifiers can be "mouse left button click" and "login button click".

[0040] The extracted interaction operation identifier and the operation type label in the operation type feature library are subjected to word vector conversion respectively, a text similarity calculation algorithm is used to calculate the similarity between the word vector of the interaction operation identifier and the word vector of each operation type label, and all operation type labels in the operation type feature library are traversed to compare the similarity matching value of each operation type label with the interaction operation identifier with the preset similarity threshold value.

[0041] Specifically, if the similarity matching value of a certain operation type label is greater than the preset similarity threshold value, the operation type label is taken as a candidate operation type; for each candidate operation type, it is checked whether the operation object of the candidate operation type matches the operation object of the interaction operation step and whether the operation environment is consistent, and then a plurality of interaction operation types are obtained.

[0042] For example, if the candidate operation type is "mouse click" and the operation object in the interaction operation step is a button that can be clicked by a mouse and the operation environment is in a webpage browsing scenario, it is considered that the candidate operation type matches the interaction operation step.

[0043] Exemplarily, in the medical health field, taking the online consultation interaction process between a patient and a doctor in a medical consultation APP as an example, after completing the splitting of the consultation interaction demand to obtain multiple interaction operation steps, the interaction operation types corresponding to the steps are identified; in the operation step of the patient sending a symptom description, the "symptom description text input box clicking and content entering" is the interaction operation identifier of this step; in the step of the doctor replying to a diagnosis suggestion, the "diagnosis suggestion text editing and sending button clicking" is the corresponding identifier; then, the similarity matching values between the interaction operation identifiers and the operation type labels in the pre-acquired operation type feature library are calculated, the operation type feature library contains labels such as "text input type", "button clicking type", "image uploading type" and features thereof, and through algorithm calculation, the similarity matching value between the "symptom description text input box clicking and content entering" and the "text input type" label is relatively high.

[0044] Then, the operation type label with a similarity matching value greater than a preset similarity threshold value (such as 0.8) is taken as the candidate operation type of the interaction operation step, and for the above-mentioned patient sending a symptom description step, the "text input type" becomes the candidate operation type; finally, the candidate operation types are cross-verified to determine that the patient sending a symptom description step not only belongs to the "text input type", but also may be indirectly related to the "image related type" operation in combination with the picture supplement demand in a specific situation, so that multiple interaction operation types corresponding to the interaction operation step are obtained, which provides a basis for subsequent optimization of interaction design and improvement of user experience.

[0045] In the embodiment of the application, the target interaction extension item information of the browser interaction demand is loaded by using multiple interaction operation types, which comprises: corresponding interaction extension item retrieval conditions are constructed according to the multiple interaction operation types; initial interaction extension item information matched with each interaction operation type is screened out from a preset interaction extension item library based on the interaction extension item retrieval conditions; compatibility verification is performed on the initial interaction extension item information to obtain target interaction extension item information.

[0046] In the embodiment of the application, semantic analysis is performed on each interaction operation type, and according to the results of the semantic analysis and the functional features, applicable scenarios and the like of the interaction extension items corresponding to different interaction operation types, interaction extension item retrieval conditions are constructed.

[0047] The constructed interaction extension item retrieval conditions are converted into database query statements, and each record in the interaction extension item library is checked by using an information matching algorithm; if the function description, applicable environment and the like of a certain record are consistent with the interaction extension item retrieval conditions, the interaction extension item information corresponding to the record is screened out as the initial interaction extension item information.

[0048] Further, the initial interactive extension item information is loaded into a preset simulation environment for compatibility verification, that is, a conflict detection algorithm is used to check whether the interactive extension item will conflict with the existing functions of the browser or other loaded interactive extension items; if the interactive extension item can normally run in the simulation environment and does not conflict with other components, it is taken as the target interactive extension item information; if there is a conflict or it cannot normally run, the interactive extension item is adjusted or excluded until an interactive extension item information meeting the compatibility requirement is found as the target interactive extension item information.

[0049] In the embodiment of the application, the plurality of interactive operation types corresponding to the interactive operation steps are identified and the target interactive extension item information is loaded accordingly, which can improve the pertinence and accuracy of the interaction, avoid unnecessary resource occupation, enhance the expansibility of the browser interaction function, and quickly adapt to diversified interaction scenarios.

[0050] S3, obtaining target markup language corresponding to the target interactive extension item information, and extracting operable markup element information in the target markup language.

[0051] In the embodiment of the application, the function to be implemented by the target interactive extension item and the presentation form in the browser are determined according to the target interactive extension item information, the template matching technology is adopted to compare the target interactive extension item information with a preset template library, and in the template library, each template corresponds to a common interactive extension item type and the corresponding language markup structure.

[0052] For example, a template for picture carousel contains language labels of a carousel container, labels of a picture list, labels of navigation buttons, etc., the most matched template with the target interactive extension item information is found by comparison, the corresponding language markup code is extracted from the template, and the target markup language is generated according to the language markup code.

[0053] In the embodiment of the application, the extraction of the operable markup element information in the target markup language includes: performing syntax verification on the target markup language to obtain a verification markup language; constructing a language object model tree according to the verification markup language; performing node attribute analysis on the language object model tree to obtain a candidate element node set; calculating an interactive feature weight of each candidate element node in the candidate element node set, and determining an operable element node set in the candidate element node set according to the interactive feature weight; extracting node information of each operable element node in the operable element node set, and summarizing all the node information as operable markup element information.

[0054] In the embodiment of the present application, the language object model tree is a tree data structure formed after parsing a target markup language (such as HTML), which organizes various elements, attributes and text content in the target markup language into a hierarchical tree structure, facilitating in-depth analysis and operation of the target markup language.

[0055] In detail, the target markup language is input into a syntax analyzer, which parses the target markup language line by line according to the syntax rules of the target markup language (such as the tag nesting rules, attribute definition rules, etc. of HTML), and at the same time, the syntax analyzer supervises and judges the parsing process according to the preset syntax rules. If it is found that there are places in the target markup language that do not conform to the syntax rules, such as tags not being closed, attribute value format errors, etc., the syntax analyzer will record these error information and attempt to make simple repairs (if the rules allow). After syntax checking, the target markup language that conforms to the syntax specification is obtained, i.e. the checked markup language.

[0056] Further, language elements of the checked markup language, such as start tags, end tags, attributes, text content, etc. are extracted, and then the language elements are added to the tree in order according to the hierarchical structure of the checked markup language using a tree construction algorithm, so as to gradually construct a language object model tree. The language object model tree accurately reflects the structure and hierarchical relationship of the checked markup language.

[0057] Specifically, starting from the root node of the language object model tree, each node in the language object model tree is traversed in a depth-first or breadth-first order. For each node, check its attribute list to find out whether there are attributes related to interactive operation, such as onclick (click event), ondrag (drag event), type (form element type, such as input's text, button, etc.).

[0058] Among them, if the node has one or more of these attributes, it is considered that the node has the potential characteristics of operability, and it is added to the candidate element node set. For each node in the candidate element node set, extract the interactive-related features, add the weights of all features of each node, and obtain the interactive feature weight of the node. The nodes with interactive feature weight higher than the preset weight threshold are determined as operable element nodes, so as to obtain the operable element node set.

[0059] In detail, for each node in the set of operable element nodes, node information of the node is extracted, including a tag name (such as div, button, input, etc.) of the node, a list of attributes (such as values of id, class, onclick, etc.), and text content; the above node information is organized and stored in a certain format, for example, the tag name, attribute name and attribute value, and text content can be stored as keys and values in the form of key-value pairs, and then, node information of all operable element nodes is summarized into a set, and this set is the operable markup element information.

[0060] In the implementation of the present application, the target markup language corresponding to the target interactive extension item information is acquired, and the operable markup element information is extracted, so that the interactive key points can be accurately located, the redundant processing of irrelevant elements is avoided, and the utilization rate of interactive resources is greatly improved.

[0061] S4, page element recognition is performed on the pre-acquired browser tab page to obtain page element information.

[0062] In the embodiment of the present application, the aggregation nodes and semantic relationships of the page elements are identified by analyzing the visual presentation and spatial layout of the browser tab page, and the structured page element information is generated according to the semantic relationships.

[0063] In the embodiment of the present application, the page element recognition is performed on the pre-acquired browser tab page to obtain page element information, including: The page source data and the rendering style data of the browser tab page are extracted; The page rendering is performed on the browser tab page according to the page source data and the rendering style data to obtain a rendered tab page; The visible element set in the rendered tab page is extracted, the attribute recognition is performed on the visible element set to obtain visual presentation attributes; The spatial layout analysis is performed on the visible element set based on the visual presentation attributes, and the aggregation nodes in the visible element set are identified according to the result of the spatial layout analysis; The semantic association analysis is performed on the aggregation nodes to determine the content attributes of the aggregation nodes and the logical relationships between the aggregation nodes; The content attributes and the logical relationships are structured and integrated to obtain the page element information.

[0064] In the embodiment of the present application, the page source data of the browser tab page is acquired by using a web page grabbing tool; for the rendering style data, the rendering style sheet in the rendering file link in the browser tab page can be analyzed to obtain the rendering style data.

[0065] In detail, the extracted page source data and rendering style data are input into a preset page rendering engine. The page rendering engine first parses the data structure of the page source data, creates a document object model (DOM) according to the data structure, then performs rendering style calculation on the document object model according to the rendering style data to determine the size, position, color and other visual properties of each data, and finally draws on the virtual canvas according to the calculation results to form a rendering tab page similar to that in the browser.

[0066] Specifically, the rendering tab page is screenshot, and then image processing algorithms are used to perform edge detection, color analysis and other operations on the screenshot to identify the boundaries of each page element. According to the shape, color distribution and other characteristics of the page elements, an element recognition algorithm is used to determine the type of the page elements, such as text, picture, button, etc. For each identified page element, further extract visual presentation properties, such as font, color of text, size, transparency of picture, etc.

[0067] Among them, the spatial analysis algorithm is used to calculate the position coordinates of each element in the visible element set, the distance and overlapping situation with other elements, and other spatial relationships. According to the spatial relationship, the clustering algorithm is used to cluster the page elements with close distance and visual association together to form an aggregation node; natural language processing technology and semantic analysis algorithm are used to perform word segmentation, part-of-speech tagging, named entity recognition and other operations on the text content in the aggregation node to determine the content properties of the aggregation node.

[0068] Then, the semantic association of the texts in different aggregation nodes is analyzed, and the semantic analysis algorithm is used to determine the logical relationship between the aggregation nodes, such as cause-effect relationship, parallel relationship, etc. The data structure and integration algorithm is used to organize the content properties and logical relationship in a certain format to form complete page element information.

[0069] Exemplarily, in the field of financial technology, taking the browser page of a certain bank online wealth management platform as an example, first, the page source data of the browser tab page is extracted, covering HTML code, JavaScript script, etc., and the rendering style data such as CSS style sheet is obtained. These data completely record the original composition and appearance setting of the page; then, the browser tab page is rendered according to the page source data and rendering style data to simulate a rendering tab page consistent with what the user actually sees, and the code is converted into intuitive page display; subsequently, the visible element set is extracted from the rendering tab page, such as buttons, text boxes, wealth management product display cards, etc. The visual presentation properties such as color, font size, shape, etc. are obtained by identifying the properties of these visible elements to clearly define the characteristics of each element in the visual layer.

[0070] Based on the visual presentation attribute, the spatial layout analysis is performed on the visible element set, the aggregated nodes closely arranged and having correlation on the page are found out, for example, a group of adjacent financial product recommendation cards, then, the semantic correlation analysis is performed on the aggregated nodes, the content attribute of the aggregated nodes is determined, such as the yield of the financial product, the risk level, and the logical relationship between the aggregated nodes, such as the matching logic of the recommended product and the risk preference of the user, finally, the content attribute and the logical relationship are structured and integrated to present the page element information in a clear format, which provides strong support for subsequent page optimization, interactive design adjustment and accurate recommendation of financial products, and improves the operation experience of the user on the financial technology platform.

[0071] In the embodiment of the application, the page element recognition is performed on the browser tab page to obtain the page element information, which can avoid irrelevant information interference, facilitate timely adjustment of element position and style, and provide a key basis for subsequent page interactive design, script generation and the like, and enhance the stability and ease of use of script generation.

[0072] S5, generating an automatic interactive script corresponding to the browser interactive requirement according to the interactive operation type, the operable mark element information and the page element information.

[0073] In the embodiment of the application, the automatic interactive script refers to a program or instruction set written for the automatic execution of specific interactive operation between the browser and the target user.

[0074] As shown in Figure 3 In the embodiment of the application, the automatic interactive script corresponding to the browser interactive requirement is generated according to the interactive operation type, the operable mark element information and the page element information, including: performing element fusion on the operable mark element information and the page element information to obtain fusion element information; determining a target operation template according to the interactive operation type, and filling the fusion element information into the target operation template to generate an initial automatic unit control; performing effectiveness verification on the initial automatic unit control to obtain a target automatic unit control; performing timing arrangement on the target automatic unit control to obtain a control execution flow; converting the control execution flow into a target script language, and generating the automatic interactive script corresponding to the browser interactive requirement according to the target script language.

[0075] In the embodiment of the present application, the operable mark element information and the page element information are combined to ensure that the relevant information of each element is integrated together, and the corresponding relationship between the operable mark element and the page element is found according to the attributes of the element (such as the function of the element, the positional relationship in the page, etc.) by using the correlation analysis technology.

[0076] For example, if an operable mark element is the identification of a button, and there is a button element in the page element information that matches the position and style, they are associated, and finally the fusion element information is obtained, which contains the complete attributes and the association relationship of the operable mark element and the page element.

[0077] Among them, the target operation template can be determined by operating the template library according to the interactive operation type, and the operation template library contains templates of various common interactive operation types, such as click operation template, input operation template, scroll operation template, etc. When the interactive operation type is determined, the target operation template matching the interactive operation type is found in the operation template library by using the template matching technology, and the related element attributes (such as the identification and position of the button) in the fusion element information are filled into the corresponding parameter position of the target operation template to obtain the initial automation unit control.

[0078] Further, the initial automation unit control is checked by using the logic verification technology, for example, whether the element identification in the initial automation unit control exists, whether the position of the element is within the page range, whether the operation parameter conforms to the logic, etc. If a logic error is found, the initial automation unit control is adjusted and corrected accordingly to obtain the target automation unit control.

[0079] Specifically, according to the dependency relationship of the browser interactive requirement, the target automation unit control is arranged in the correct order by using the process arrangement technology, and at the same time, the target automation unit control is optimized and adjusted considering the time interval and possible delay of the execution of the target automation unit control to obtain the control execution flow.

[0080] Among them, each control operation in the control execution flow is converted into a corresponding code statement according to the syntax rules and code generation specifications of the target script language. In the conversion process, it is ensured that the generated code conforms to the syntax requirements of the target script language, such as variable naming rules, function call format, etc. At the same time, the timing information in the control execution flow is also converted into time control statements in the code. Finally, all the generated code statements are combined in the order of the control execution flow to form an automation interactive script, which can be executed in the running environment of the target script language to realize the specific interactive operation between the browser and the target user.

[0081] Exemplarily, in the field of financial technology, taking the generation of an automatic transaction script of a browser page of a certain securities transaction APP as an example, the element fusion is performed on the operable mark element information (such as the positioning identification of a buy button and a sell button) and the page element information (such as the visual attributes of the button color and size), to obtain fusion element information containing the positioning and appearance characteristics of the buttons; according to the interaction operation type, such as a "stock buying operation", a corresponding target operation template is determined, and the fusion element information is filled into the template to generate an initial automatic unit control, which contains the parameters such as the coordinates and timing of clicking the buy button; the initial control is subjected to effectiveness verification, to check whether the clicking position is accurate and whether the transaction data format is correct, to obtain a target automatic unit control; then, the target automatic unit control is subjected to timing arrangement, to determine the control execution flow of logging in an account first, then selecting a stock, and finally clicking the buy button; finally, the control execution flow is converted into a Python script language, to generate an automatic interaction script capable of automatically executing the stock buying operation in the browser, to help an investor to quickly complete the transaction and reduce manual operation errors and time costs.

[0082] Exemplarily, in the field of medical health, taking the generation of an automatic consultation script of a browser page of a certain online consultation platform as an example, the fusion element information is obtained by fusing the operable mark element information (such as the positioning information of a symptom description input box and a send button) and the page element information (such as input box prompt text and button style); according to the interaction operation type "symptom information submission", a target operation template is determined, the fusion element information is filled into the template to generate an initial automatic unit control, and the text format of inputting the symptoms and the timing of clicking the send button are specified; the initial control is subjected to effectiveness verification, to ensure that the input content meets the requirements and the send function is normal, to obtain a target automatic unit control; the target control is subjected to timing arrangement, to form the control execution flow of filling in the symptoms first, then uploading a test report (if any), and finally clicking the send button. The control execution flow is converted into a JavaScript script language, to generate an automatic interaction script, to help a patient to efficiently complete the online consultation and improve the medical experience.

[0083] In the embodiments of the present application, the automatic interaction script is used to realize the automatic interaction operation, to avoid the tediousness and time consumption of manual operation, to improve the interaction efficiency, and to ensure the accuracy and consistency of each interaction operation; the automatic interaction script of the browser interaction requirement is generated according to the interaction operation type, the operable mark element information and the page element information, to improve the script generation efficiency and the script generation accuracy.

[0084] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0085] For example, Figure 4As shown, it is an embodiment of the present application to provide a kind of function module diagram of script generation device based on browser interaction.

[0086] In the embodiment of the present disclosure, a script generation device based on browser interaction is provided, which corresponds to the script generation method based on browser interaction in the above embodiment. As shown in the figure, Figure 4 As shown, the script generation device based on browser interaction 100 can be installed in an electronic device, according to the function to be realized, the script generation device based on browser interaction 100 includes interaction requirement splitting module 101, extension item loading module 102, mark information extraction module 103, page element identification module 104 and interaction script generation module 105. The detailed description of each functional module is as follows: The interaction requirement splitting module 101 is used to obtain the browser interaction requirement of the target user, split the browser interaction requirement, and obtain a plurality of interaction operation steps; The extension item loading module 102 is used to identify a plurality of interaction operation types corresponding to the interaction operation steps, and load the target interaction extension item information of the browser interaction requirement by using a plurality of interaction operation types; The mark information extraction module 103 is used to obtain the target markup language corresponding to the target interaction extension item information, and extract the operable markup element information in the target markup language; The page element identification module 104 is used to identify the page elements of the pre-obtained browser tab page, and obtain the page element information; The interaction script generation module 105 is used to generate the automatic interaction script corresponding to the browser interaction requirement according to the interaction operation type, the operable markup element information and the page element information.

[0087] In an embodiment, when the interaction requirement splitting module 101 performs the requirement splitting on the browser interaction requirement to obtain a plurality of interaction operation steps, it is used for: Performing structured semantic analysis on the browser interaction requirement to obtain key requirement interaction elements; Classifying the browser interaction requirement according to the key requirement interaction elements to obtain a target requirement category; Splitting the target requirement category to obtain a plurality of preliminary interaction units; Logical correlation verification is performed on a plurality of preliminary interaction units to generate an ordered interaction unit sequence; A plurality of interaction operation steps are generated according to the ordered interaction unit sequence.

[0088] In an embodiment, the extension loading module 102, when performing the step of identifying a plurality of interaction operation types corresponding to the interaction operation, is configured to: extract an interaction operation identifier in the interaction operation step; calculate a similarity matching value between the interaction operation identifier and an operation type label in a pre-acquired operation type feature library; identify an operation type label with a similarity matching value greater than a preset similarity threshold as a candidate operation type of the interaction operation step; cross-verify the candidate operation type to obtain a plurality of interaction operation types corresponding to the interaction operation step.

[0089] In an embodiment, the extension loading module 102, when performing the step of loading the target interaction extension information of the browser interaction requirement by using a plurality of the interaction operation types, is configured to: construct a corresponding interaction extension retrieval condition according to a plurality of the interaction operation types; filter out initial interaction extension information matching each of the interaction operation types from a preset interaction extension library based on the interaction extension retrieval condition; perform compatibility verification on the initial interaction extension information to obtain target interaction extension information.

[0090] In an embodiment, the mark information extraction module 103, when performing the step of extracting operable mark element information in the target markup language, is configured to: perform syntax verification on the target markup language to obtain a verification markup language; construct a language object model tree according to the verification markup language; perform node attribute analysis on the language object model tree to obtain a candidate element node set; calculate an interaction feature weight of each candidate element node in the candidate element node set, and determine an operable element node set in the candidate element node set according to the interaction feature weight; extract node information of each operable element node in the operable element node set, and aggregate all the node information as operable mark element information.

[0091] In an embodiment, the page element identification module 104, when performing the step of identifying a page element in a pre-acquired browser tab page to obtain page element information, is configured to: extract page source data and rendering style data of the browser tab page; perform page rendering on the browser tab page according to the page source data and the rendering style data to obtain a rendered tab page; extracting a visible element set in the rendering tab page, performing attribute recognition on the visible element set to obtain a visual presentation attribute; performing spatial layout analysis on the visible element set based on the visual presentation attribute, and recognizing an aggregation node in the visible element set according to a result of the spatial layout analysis; performing semantic association analysis on the aggregation node to determine a content attribute of the aggregation node and a logical relationship between the aggregation nodes; performing structured integration on the content attribute and the logical relationship to obtain page element information.

[0092] In an embodiment, the interaction script generation module 105, when generating the automatic interaction script corresponding to the browser interaction requirement according to the interaction operation type, the operable mark element information and the page element information, is configured to: performing element fusion on the operable mark element information and the page element information to obtain fused element information; determining a target operation template according to the interaction operation type, and filling the fused element information into the target operation template to generate an initial automatic unit control; performing effectiveness verification on the initial automatic unit control to obtain a target automatic unit control; performing timing arrangement on the target automatic unit control to obtain a control execution flow; converting the control execution flow into a target script language, and generating the automatic interaction script corresponding to the browser interaction requirement according to the target script language.

[0093] In the present application, the specific limitations of the script generation device based on browser interaction can be referred to the limitations of the script generation method based on browser interaction in the foregoing, which will not be repeated here. Each module in the script generation device based on browser interaction can be realized by software, hardware and combinations thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0094] In one embodiment, a computer device is provided, which can be a server, and its internal structure diagram can be as shown in Figure 5As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes non-volatile and / or volatile storage medium, internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external client through the network connection. The computer program is executed by the processor to realize the functions or steps of the server side of the browser interaction-based script generation method.

[0095] In one embodiment, a computer device is provided, which can be a client, and its internal structure diagram can be as shown in the figure. Figure 6 As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes non-volatile storage medium, internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external server through the network connection. The computer program is executed by the processor to realize the functions or steps of the client side of the browser interaction-based script generation method.

[0096] In one embodiment, a computer device is provided, including a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to realize the following steps: Obtain the browser interaction requirement of the target user, split the browser interaction requirement, and obtain a plurality of interaction operation steps; Identify a plurality of interaction operation types corresponding to the interaction operation steps, and load target interaction extension item information of the browser interaction requirement using a plurality of the interaction operation types; Obtain the target markup language corresponding to the target interaction extension item information, and extract the operable markup element information in the target markup language; Page element recognition is performed on the pre-acquired browser tab page to obtain page element information; According to the interaction operation type, the operable markup element information and the page element information, an automatic interaction script corresponding to the browser interaction requirement is generated.

[0097] In several embodiments of the present application, it should be understood that the disclosed devices, apparatuses can be implemented in other manners. For example, the division of the above-described system embodiments is merely a logical function division, and there can be other division manners in actual implementation.

[0098] In addition, each function module in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or in the form of hardware plus software function module.

[0099] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any additional reference signs in the claims should not be considered as limiting the claims involved.

[0100] In some embodiments of the present embodiment, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium. The computer program is characterized in that when executed by a processor, it implements the steps of the method described in the above embodiments.

[0101] The readable storage medium described in the present application stores a computer program, and the computer program can realize the following functions when executed by a processor of an electronic device: Obtain the browser interaction requirement of the target user, perform requirement splitting on the browser interaction requirement, and obtain a plurality of interaction operation steps; Identify a plurality of interaction operation types corresponding to the interaction operation steps, and load target interaction extension item information of the browser interaction requirement by using a plurality of the interaction operation types; Obtain a target markup language corresponding to the target interaction extension item information, and extract operable markup element information in the target markup language; Perform page element recognition on a pre-obtained browser tab page to obtain page element information; Generate an automatic interaction script corresponding to the browser interaction requirement according to the interaction operation type, the operable markup element information, and the page element information.

[0102] It should be noted that the functions or steps that the computer readable storage medium or the computer device can realize correspond to the descriptions of the server side and the client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0103] The computer-readable storage medium can also store at least one computer-executable program / instruction, such as computer-readable instructions. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory, and the like. The computer-readable storage medium may, for example, include read-only memory (ROM), a hard disk, flash memory, and the like. For example, the non-transitory computer-readable storage medium can be connected to a computing device, such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0104] In addition, the computer device can also include, but is not limited to, a data bus, an input / output (I / O) bus, a display, and an input / output device (for example, a keyboard, a mouse, a speaker, and the like), and the like.

[0105] In one embodiment, the at least one computer-executable instruction can also be compiled or composed into a software product / computer program product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods described in the embodiments of the present technology.

[0106] A person of ordinary skill in the art can understand that all or part of the processes in the above-described embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium and can include the processes of the above-described embodiments when executed. Any reference to memory, storage, database, or other medium used in the embodiments provided herein can include non-volatile and / or volatile memory.

[0107] It can be clearly understood by a person of ordinary skill in the art that, for the convenience and brevity of description, only the division of the above-described functional units and modules is exemplified, and in actual applications, the above-described functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0108] In the embodiments provided by the present disclosure, it should be understood that the disclosed apparatus and method can also be implemented in other manners. The embodiments described above are merely exemplary for describing the present disclosure. For example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the apparatus, method and computer program product according to the embodiments of the present disclosure. In this regard, each block in the flowcharts and block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that, in some alternative implementations, the functions noted in the blocks can occur in a different order from that noted in the accompanying drawings. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a special-purpose hardware-based system that performs the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0109] The above-described embodiments are merely used to illustrate the technical solutions of the present disclosure, rather than limit the present disclosure; although the present disclosure has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some of the technical features can be replaced equivalently; and the modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure.

[0110] It should be noted that, if a software tool or component that is not from the company appears in the embodiments of the present disclosure, it is merely used for example introduction, and does not represent actual use.

Claims

1. A method for generating a script based on browser interaction, characterized by, The method comprises: obtaining the browser interaction requirement of the target user, performing requirement decomposition on the browser interaction requirement to obtain a plurality of interaction operation steps; identifying a plurality of interaction operation types corresponding to the interaction operation steps, and loading target interaction extension item information of the browser interaction requirement by using the plurality of interaction operation types; obtaining target markup language corresponding to the target interaction extension item information, and extracting operable markup element information in the target markup language; performing page element recognition on a pre-obtained browser tab page to obtain page element information; generating an automatic interaction script corresponding to the browser interaction requirement according to the interaction operation type, the operable markup element information and the page element information.

2. The browser interaction based script generation method of claim 1, wherein, The requirement decomposition on the browser interaction requirement to obtain a plurality of interaction operation steps comprises: performing structured semantic analysis on the browser interaction requirement to obtain a key requirement interaction element; performing category division on the browser interaction requirement according to the key requirement interaction element to obtain a target requirement category; performing requirement decomposition on the target requirement category to obtain a plurality of preliminary interaction units; performing logical correlation verification on the plurality of preliminary interaction units to generate an ordered interaction unit sequence; generating a plurality of interaction operation steps according to the ordered interaction unit sequence.

3. The browser interaction based script generation method of claim 1, wherein, The identification of a plurality of interaction operation types corresponding to the interaction operation steps comprises: extracting an interaction operation identifier in the interaction operation step; calculating a similarity matching value between the interaction operation identifier and an operation type label in a pre-obtained operation type feature library; taking an operation type label with a similarity matching value greater than a preset similarity threshold as a candidate operation type of the interaction operation step; performing cross verification on the candidate operation type to obtain a plurality of interaction operation types corresponding to the interaction operation step.

4. The browser interaction based script generation method of claim 1, wherein, The loading of target interaction extension item information of the browser interaction requirement by using a plurality of interaction operation types comprises: constructing a corresponding interaction extension item retrieval condition according to a plurality of interaction operation types; screening initial interaction extension item information matched with each interaction operation type from a preset interaction extension item library based on the interaction extension item retrieval condition; performing compatibility verification on the initial interaction extension item information to obtain target interaction extension item information.

5. The browser interaction based script generation method of claim 1, wherein, The extraction of operable markup element information in the target markup language comprises: performing syntax verification on the target markup language to obtain a verification markup language; constructing a language object model tree according to the verification markup language; performing node attribute analysis on the language object model tree to obtain a candidate element node set; calculating an interaction feature weight of each candidate element node in the candidate element node set, and determining an operable element node set in the candidate element node set according to the interaction feature weight; extracting node information of each operable element node in the operable element node set, and summarizing all the node information as operable markup element information.

6. The browser interaction based script generation method of claim 1, wherein, The page element recognition on a pre-obtained browser tab page to obtain page element information comprises: extract page source data and rendering style data of the browser tab page; perform page rendering on the browser tab page according to the page source data and the rendering style data, to obtain a rendered tab page; extract a set of visible elements in the rendered tab page, perform attribute identification on the set of visible elements, and obtain visual presentation attributes; perform spatial layout analysis on the set of visible elements based on the visual presentation attributes, and identify aggregated nodes in the set of visible elements according to a result of the spatial layout analysis; perform semantic association analysis on the aggregated nodes, to determine content attributes of the aggregated nodes and a logical relationship between the aggregated nodes; perform structured integration on the content attributes and the logical relationship, to obtain page element information.

7. The browser interaction based script generation method of claim 1, wherein, The generating, according to the interactive operation type, the operable markup element information, and the page element information, of the automatic interactive script corresponding to the browser interactive requirement comprises: performing element fusion on the operable markup element information and the page element information, to obtain fused element information; determining a target operation template according to the interactive operation type, and filling the fused element information into the target operation template, to generate an initial automatic unit control; performing effectiveness verification on the initial automatic unit control, to obtain a target automatic unit control; performing timing arrangement on the target automatic unit control, to obtain a control execution flow; converting the control execution flow into a target script language, and generating the automatic interactive script corresponding to the browser interactive requirement according to the target script language.

8. An apparatus for generating a script based on browser interaction, characterized by The apparatus comprises: an interactive requirement splitting module configured to obtain a browser interactive requirement of a target user, split the browser interactive requirement, and obtain a plurality of interactive operation steps; an extension item loading module configured to identify a plurality of interactive operation types corresponding to the interactive operation steps, and load target interactive extension item information of the browser interactive requirement by using the plurality of interactive operation types; a markup information extraction module configured to obtain a target markup language corresponding to the target interactive extension item information, and extract operable markup element information in the target markup language; a page element identification module configured to perform page element identification on a pre-obtained browser tab page, and obtain page element information; an interactive script generation module configured to generate an automatic interactive script corresponding to the browser interactive requirement according to the interactive operation type, the operable markup element information, and the page element information.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the script generation method based on browser interaction according to any one of claims 1 to 7.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the script generation method based on browser interaction according to any one of claims 1 to 7.

Images