Automatic screen recording script generation method and device, storage medium and computer equipment

Abstract

The invention discloses an automatic screen recording script generation method and device, a storage medium and computer equipment, and relates to the technical field of automatic testing, the technical field of medical treatment and health and the technical field of finance. Obtaining page structured data, first gesture operation data and a first type parameter of the target client; performing page control element identification on the page structured data based on the first gesture operation data to obtain a target control element of the target page operated by the user; performing calculation processing based on the first model parameter, the first gesture operation data and a second model parameter of the to-be-tested client to obtain second gesture operation data corresponding to the to-be-tested client; and performing automatic script generation based on the second gesture operation data and the target control element to obtain a target automatic screen recording script. According to the method, the automatic screen recording script generation can be realized, and the test efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of automated testing, medical and health technology, and financial technology, and particularly relates to a method and device for generating an automated screen recording script, a storage medium, and a computer device. Background Art

[0002] With the popularization and increasing complexity of mobile applications, the demand for automated testing and automated screen recording is growing. In the field of medical and health technology, automated screen recording and automated testing of medical and health software can be carried out to support functions such as disease-assisted diagnosis, health management, and remote consultation; in the field of financial technology, automated testing and automated screen recording technologies can support functions such as shopping, social interaction, interactive games, and resource transfer; existing automated screen recording tools mainly rely on manual operations. Users need to click and drag the mouse to record the operation steps and then generate the corresponding automated script. This method is not only time-consuming and laborious but also error-prone. Especially when facing complex user interfaces and changing device environments, the manually recorded scripts often need to be debugged multiple times to achieve the expected effect. In addition, existing automated screen recording tools have significant problems in adapting to different models. Due to differences in screen resolutions and sizes of different devices, when the manually recorded scripts are run on different devices, problems such as inaccurate element positioning and operation failures may occur. This not only affects the efficiency of testing but also increases the development and maintenance costs. Summary of the Invention

[0003] In view of this, the present invention provides a method and device for generating an automated screen recording script, a storage medium, and a computer device, mainly aiming to solve the problem of low efficiency in manually recording screen recording scripts currently.

[0004] To solve the above problems, the present application provides a method for generating an automated screen recording script, including:

[0005] Responding to a page operation of a user on a target page of a target client, obtaining page structured data, first gesture operation data, and first model parameters of the target client;

[0006] Identifying page control elements for the page structured data based on the first gesture operation data to obtain target control elements for the user to operate the target page;

[0007] Performing calculation processing based on the first model parameters, the first gesture operation data, and second model parameters of a client to be tested to obtain second gesture operation data corresponding to the client to be tested;

[0008] Generating an automated script based on the second gesture operation data and the target control elements to obtain a target automated screen recording script.

[0009] Optionally, before identifying page control elements for the page structured data based on the gesture operation data, the method further includes:

[0010] Querying each model parameter in a preset database based on the first model parameter to obtain a query result;

[0011] When the query result indicates that the first model parameter is not included in each of the model parameters, saving the first model parameter in each of the model parameters in the preset database;

[0012] The model parameter includes one or several of a mobile phone system parameter, a mobile phone model parameter, a mobile phone version parameter, a screen resolution parameter, and a size parameter.

[0013] Optionally, the identifying, based on the gesture operation data, page control elements for the page structured data to obtain target control elements of the target page for the user operation specifically includes:

[0014] Performing coordinate matching based on the first gesture operation coordinate position in the first gesture operation data and the page structured data to obtain several control elements within the range of the area where the first gesture operation coordinate position is located when the user operates the target page;

[0015] Identifying each of the control elements by using a preset control element recognition model based on the first gesture operation data to obtain target control elements of the target page for the user operation.

[0016] Optionally, the identifying, based on the first gesture operation data, each of the control elements by using a preset control element recognition model to obtain target control elements of the target page for the user operation specifically includes:

[0017] Performing data processing on the gesture operation data to obtain a natural language description statement;

[0018] Identifying based on the natural language description statement by using the preset control element recognition model to obtain a predicted control coordinate position;

[0019] Screening each of the control elements based on the predicted control coordinate position to obtain target control elements of the target page for the user operation.

[0020] Optionally, before identifying each of the control elements by using a preset control element recognition model based on the gesture operation data, the method further includes: constructing a preset control element recognition model;

[0021] The constructing of the preset control element recognition model specifically includes:

[0022] Obtain historical gesture operation data and historical page structured data, where the gesture operation data includes gesture type, gesture operation touch point coordinates, and time stamps; the historical page structured data includes control position coordinates, control types, and control content;

[0023] Perform data preprocessing on the historical gesture operation data and the historical page structured data to obtain gesture operation data description statements and historical page structured data description statements;

[0024] Based on the gesture operation data description statements and the historical page structured data description statements, load a pre-trained NLP model for model training to obtain the preset control recognition model.

[0025] Optionally, the calculation process based on the first model parameters, the first gesture operation data, and the second model parameters of the client to be tested to obtain the second gesture operation data corresponding to the client to be tested specifically includes:

[0026] Perform calculation processing based on the first abscissa in the first gesture operation parameters, the first horizontal pixel number in the first model parameters, and the second horizontal pixel number in the second model parameters to obtain the second abscissa of the second gesture operation data corresponding to the client to be tested;

[0027] Perform calculation processing based on the first ordinate in the first gesture operation parameters, the first vertical pixel number in the first model parameters, and the second vertical pixel number in the second model parameters to obtain the second ordinate of the second gesture operation data corresponding to the client to be tested.

[0028] Optionally, the generation of the target automated screen recording script based on the second gesture operation data and the target control element specifically includes:

[0029] Query a preset code template library based on the target control element to obtain the template code corresponding to the target control element;

[0030] Generate an automated script based on the second gesture operation data and the template code to obtain the target automated screen recording script;

[0031] Execute the target automated screen recording script to perform test verification on the client to be tested.

[0032] To solve the above problems, the present application provides an automated screen recording script generation device, including:

[0033] An acquisition module, configured to acquire page structured data, first gesture operation data, and first model parameters of the target client in response to a page operation of a user on a target page of the target client;

[0034] An identification module, configured to identify page control elements of the page structured data based on the first gesture operation data, so as to obtain target control elements for the user to operate the target page;

[0035] A calculation module, configured to perform calculation processing based on the first model parameters, the first gesture operation data, and second model parameters of a client to be tested, so as to obtain second gesture operation data corresponding to the client to be tested;

[0036] A generation module, configured to generate an automated script based on the second gesture operation data and the target control elements, so as to obtain a target automated screen recording script.

[0037] To solve the above problems, the present application provides a storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned automated screen recording script generation method are implemented.

[0038] To solve the above problems, the present application provides a computer device including at least a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program on the memory, the steps of the above-mentioned automated screen recording script generation method are implemented.

[0039] The beneficial effects in the present application: The present application receives page structured data, first gesture operation data, and first model parameters of the target client obtained by listening using a plug-in with a listening function installed on the target client; based on the first gesture operation data and the page structured data, a preset control identification model is used to identify page control elements, so as to obtain target control elements for the user to operate the target page; using the preset control identification model to identify page controls can improve the efficiency of generating automated scripts; calculation processing is performed based on the first model parameters, the first gesture operation data, and second model parameters of the client to be tested, so as to obtain second gesture operation data corresponding to the client to be tested; an automated script is generated based on the second gesture operation data and the target control elements, so as to obtain a target automated screen recording script. The present application can automatically generate an automated script according to gesture operations, without manually configuring scripts for different mobile phone models, and the present application can adapt to as many models in the market as possible, without adapting according to various models, saving manpower.

[0040] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable, the specific embodiments of the present invention are hereinafter specifically exemplified. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of illustrating the preferred embodiments and are not considered to be a limitation of the present invention. And throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0042] Figure 1 It shows a schematic diagram of the application environment of an automated screen recording script generation method provided by an embodiment of the present application;

[0043] Figure 2 It shows a schematic flow chart of an automated screen recording script generation method provided by an embodiment of the present application;

[0044] Figure 3 It shows a schematic flow chart of an automated screen recording script generation method provided by another embodiment of the present application;

[0045] Figure 4 It shows a structural block diagram of an automated screen recording script generation device provided by another embodiment of the present application;

[0046] Figure 5 It shows a schematic structural diagram of a computer device in an embodiment of the present application;

[0047] Figure 6 It shows another schematic structural diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] Reference is made herein to the various aspects and features of the present application as illustrated in the accompanying drawings.

[0049] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above specification should not be construed as a limitation, but only as an example of the embodiments. Those skilled in the art will envision other modifications within the scope and spirit of the present application.

[0050] The drawings included in the specification and forming a part of the specification illustrate the embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, are used to explain the principles of the present application.

[0051] These and other features of the present application will become apparent from the following description of the preferred forms of the embodiments given as non - limiting examples with reference to the accompanying drawings.

[0052] It should also be understood that although the present application has been described with reference to some specific examples, those skilled in the art can surely implement many other equivalent forms of the present application.

[0053] When combined with the accompanying drawings, the above - mentioned and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description.

[0054] Hereinafter, specific embodiments of the present application will be described with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of the present application and can be implemented in various ways. Well - known and / or repetitive functions and structures are not described in detail to avoid obscuring the present application with unnecessary or redundant details. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but are merely used as a basis and representative basis for the claims to teach those skilled in the art to use the present application in substantially any suitable detailed structure in a variety of ways.

[0055] This specification may use the phrases "in one embodiment", "in another embodiment", "in yet another embodiment" or "in other embodiments", each of which may refer to one or more of the same or different embodiments according to the present application.

[0056] The automated screencast script generation method provided by the embodiments of the present application can be applied in an application environment such as Figure 1 where the client communicates with the server through a network. The server can obtain page structured data, first gesture operation data, and the first model parameters of the target client by responding to a page operation of the user on a target page of the target client; identify the target control elements of the target page operated by the user based on the first gesture operation data for the page structured data; perform calculation processing based on the first model parameters, the first gesture operation data, and the second model parameters of the client to be tested to obtain second gesture operation data corresponding to the client to be tested; generate an automated script based on the second gesture operation data and the target control elements to obtain a target automated screencast script, and feedback the target automated screencast script to the client. The present application can automatically generate an automated script according to gesture operations without manually configuring scripts for different mobile phone models, and the present application can adapt to as many models in the market as possible without adapting according to various models, saving manpower.

[0057] The embodiments of the present application provide an automated screencast script generation method, as Figure 2 shown, including:

[0058] Step S101: In response to a page operation of a user on a target page of a target client, obtain page structured data, first gesture operation data, and first model parameters of the target client.

[0059] In the specific implementation process of this step, the first gesture operation data includes gesture types (such as click, slide, long press, etc.), touch point coordinates of the gesture operation (such as (x, y)), and time stamps. The page structured data includes the positions (such as bounding box coordinates), types (such as buttons, text boxes, etc.), and contents (such as button texts) of each control. Specifically, the page structured data, the first gesture operation data, and the first model parameters of the target client can be obtained by acquiring a plug-in with a listening function installed on the target client. In the field of medical and health technologies, the first gesture operation data can be a drag operation on a medical control, a long press operation on a medical control, a slide operation on a medical control, etc. performed by the user; for example, gesture operation data such as the user clicking on a medical appointment button, the user clicking on a disease consultation button, and the user clicking on a reservation registration button. In the field of financial technologies, the first gesture operation data can be a gesture operation of the user clicking on a payment button, an operation of the user clicking on a product view button, etc.

[0060] Step S102: Based on the first gesture operation data, perform page control element recognition on the page structured data to obtain a target control element of the target page operated by the user.

[0061] In the specific implementation process of this step, based on the first gesture operation data and the page structured data, use a preset control recognition model to perform control recognition to obtain a target control element of the target page operated by the user. The target control element can be a control element of types such as buttons, links, input boxes, sliders, switches, etc.

[0062] Step S103: Based on the first model parameters, the first gesture operation data, and second model parameters of a client to be tested, perform calculation processing to obtain second gesture operation data corresponding to the client to be tested.

[0063] In the specific implementation process of this step, based on the first abscissa in the first gesture operation parameters, the first horizontal pixel number in the first model parameters, and the second horizontal pixel number in the second model parameters, perform calculation processing to obtain a second abscissa of the second gesture operation data corresponding to the client to be tested; based on the first ordinate in the first gesture operation parameters, the first vertical pixel number in the first model parameters, and the second vertical pixel number in the second model parameters, perform calculation processing to obtain a second ordinate of the second gesture operation data corresponding to the client to be tested.

[0064] Step S104: Generate an automated script based on the second gesture operation data and the target control element to obtain a target automated screen recording script.

[0065] In the specific implementation process of this step, query a preset code template library based on the target control element to obtain the template code corresponding to the target control element; generate an automated script based on the second gesture operation data and the template code to obtain a target automated screen recording script; execute the target automated screen recording script to perform test verification on the client to be tested.

[0066] This application receives the page structured data, the first gesture operation data, and the first model parameter of the target client listened by a plug-in with a listening function installed on the target client; uses a preset control recognition model to identify the page control element for the user to operate the target page based on the first gesture operation data and the page structured data, and obtains the target control element for the user to operate the target page; using the preset control recognition model to identify the page control can improve the efficiency of generating the automated script; perform calculation processing based on the first model parameter, the first gesture operation data, and the second model parameter of the client to be tested to obtain the second gesture operation data corresponding to the client to be tested; generate an automated script based on the second gesture operation data and the target control element to obtain a target automated screen recording script. This application can automatically generate an automated script according to the gesture operation, without manually configuring the script for different mobile phone models, and this application can adapt to as many models as possible in the market, without adapting according to various models, saving manpower.

[0067] An embodiment of this application provides a method for generating an automated screen recording script, as Figure 3 shown, including:

[0068] Step S201: In response to a user's page operation on a target page of a target client, obtain page structured data, first gesture operation data, and the first model parameter of the target client;

[0069] In the specific implementation process of this step, the first gesture operation data includes the gesture type, the touch point coordinates of the gesture operation, and the time stamp; the gesture type includes types such as click, slide, long press, etc.; the touch point coordinates of the gesture operation (such as (x, y)). The page structured data includes the positions of various controls (such as bounding box coordinates), types (such as buttons, text boxes, etc.), and contents (such as button texts). Specifically, the page structured data, the first gesture operation data, and the first model parameters of the target client can be obtained by using a plug-in with a listening function installed on the target client. For example: the target client can be a mobile client such as a mobile phone. The plug-in with the listening function is installed on the mobile phone. The plug-in reads the structured data of the application UI page and the gesture operation data of the current operation, and then sends them to the application service platform of the automated script generation method of this application through an HTTP request. In the field of medical and health technology, the application service platform of this application can be a medical cloud platform. A medical cloud refers to a medical and health service cloud platform created by using "cloud computing" on the basis of new technologies such as cloud computing, mobile technology, multimedia, 4G communication, big data, and the Internet of Things, combined with medical technology, realizing the sharing of medical resources and the expansion of the medical scope. Due to the application and combination of cloud computing technology, the medical cloud improves the efficiency of medical institutions and facilitates residents' medical treatment. For example, the current hospital appointment registration, electronic medical records, medical insurance, etc. are all the products of the combination of cloud computing and the medical field. The medical cloud also has the advantages of data security, information sharing, dynamic expansion, and overall layout. In the field of financial technology, the application service platform of this application can be a platform with functions such as applying for loans, credit cards, or purchasing insurance, financial products, etc. The application service platform can be an insurance system, a bank system, a trading system, an order system, etc.

[0070] Step S202: Query the model parameters in the preset database based on the first model parameter to obtain a query result;

[0071] In the specific implementation process of this step, the preset database pre-stores the model parameters of each client. Specifically, query the model parameters in the preset database based on the first model parameter; when the first model parameter is included in each of the model parameters, an automated script can be generated based on the first gesture operation data and the target control element to obtain a target automated screen recording script corresponding to the client to be tested. Specifically, query the preset code template library based on the target control element to obtain the template code corresponding to the target control element; generate an automated script based on the first gesture operation data and the template code to obtain the automated screen recording script.

[0072] Step S203: When the query result indicates that the first model parameter is not included in each of the model parameters, save the first model parameter in each of the model parameters in the preset database;

[0073] In the specific implementation of this step, when the query result indicates that the first model parameter is not included in each of the model parameters, save the first model parameter in each of the model parameters in the preset database. For example: The mobile phone model parameters can be entered on the PC side and stored in the preset database; there are many models of mobile clients. Once a new mobile client model appears, the parameters of the new mobile client model can be entered into the system. The purpose is to make the screencast script automatically generated by this application adapt to the new mobile client model and enable functional verification on the new mobile client model.

[0074] Step S204: Based on the first gesture operation coordinate position in the first gesture operation data and the page structured data, perform coordinate matching to obtain several control elements within the area range where the first gesture operation coordinate position is located when the user operates the target page;

[0075] In the specific implementation of this step, traverse all the control elements in the page structured data. Check whether the coordinate range of each control element contains the coordinate of the first gesture operation coordinate position. If it contains, add the control element to the result list to obtain several control elements.

[0076] Step S205: Construct a preset control recognition model;

[0077] In the specific implementation of this step, historical gesture operation data and historical page structured data are obtained. The gesture operation data includes gesture type, gesture operation touch point coordinates, and time stamp. Specifically, the target client can be a mobile client. For the mobile client, a plug-in with a listening function can be installed on the mobile client to listen for the user's gesture operations on the mobile client, and then send the obtained gesture operation data to the system or platform to which the automated screen recording script generation method of this application is applied. In the field of medical technology, the system of this application can be... The gesture operation data can also be data collected by a touch screen, including gesture type (such as click, slide, zoom, etc.), gesture operation touch point coordinates (x, y), and time stamp. The historical page structured data includes control position coordinates, control type, and control content. Data preprocessing is performed on the historical gesture operation data and the historical page structured data to obtain gesture operation data description statements and page structured data description statements. Specifically, the gesture operation data is converted into a structured format, for example, a description statement is generated, such as "The user performed a click operation at time stamp 1234567890, and the coordinates are (100, 200)". The page structured data is also converted into a description statement, such as "There is a button on the page, the position coordinates are (50, 100), the type is button, and the content is 'Submit'". Based on the historical gesture operation data and the historical page structured data, a pre-trained NLP model is loaded for model training to obtain the preset control recognition model. A pre-trained natural language processing model is used. The pre-trained natural language processing model can be a model with the ability to calculate and reason for business logic. For example, the pre-trained natural language processing model can be models such as BERT and GPT. These models have been pre-trained on large-scale text data and can understand the semantics of natural language. The preprocessed gesture operation data description statements and page structured data description statements are used as inputs to train the model to recognize and understand the relationship between gesture operations and page controls. Through training, the model can accurately identify the target control of the user's operation based on the gesture operation data and the page structured data. The preset control recognition model can be applied to a variety of different fields, such as speech recognition, medical diagnosis, testing of application programs, etc.

[0078] Step S206: Based on the first gesture operation data, use the preset control recognition model to recognize each of the control elements to obtain the target control element of the target page operated by the user.

[0079] In the specific implementation process of this step, the gesture operation data is processed to obtain a natural language description statement; based on the natural language description statement, the preset control recognition model is used for recognition to obtain the predicted control coordinate position; based on the predicted control coordinate position, each of the control elements is filtered to obtain the target control element of the target page operated by the user. Specifically, the Euclidean distance between the predicted control coordinate position and each of the control elements can be calculated to obtain each Euclidean distance value; the spatial element corresponding to the smallest Euclidean distance value among the filtered Euclidean distance values is determined as the target control element. The target control element can be a control element of types such as buttons, links, input boxes, sliders, switches, etc.

[0080] Step S207: Based on the first model parameter, the first gesture operation data, and the second model parameter of the client to be tested, calculation and processing are performed to obtain the second gesture operation data corresponding to the client to be tested;

[0081] In the specific implementation process of this step, based on the first abscissa in the first gesture operation parameter, the first horizontal pixel number in the first model parameter, and the second horizontal pixel number in the second model parameter, calculation and processing are performed to obtain the second abscissa of the second gesture operation data corresponding to the client to be tested; based on the first ordinate in the first gesture operation parameter, the first vertical pixel number in the first model parameter, and the second vertical pixel number in the second model parameter, calculation and processing are performed to obtain the second ordinate of the second gesture operation data corresponding to the client to be tested. By this method, the automated script of the present application can be adapted to multiple different devices.

[0082] Step S208: Query the preset code template library based on the target control element to obtain the template code corresponding to the target control element;

[0083] In the specific implementation process of this step, the preset code template library is a collection containing various control template codes, and these template codes define the code structure of the controls. The template library can be a local file, a database, or a remote service; by querying the preset code template library, the template code corresponding to the target control element can be obtained.

[0084] Step S209: Based on the second gesture operation data and the template code, an automated script is generated to obtain a target automated screen recording script;

[0085] In the specific implementation process of this step, the second gesture operation data is converted into a format suitable for the template code. This may include extracting key information (such as gesture type, coordinates, timestamp) and formatting it into parameters required by the template code. The preprocessed gesture operation data is replaced at the placeholder positions in the template code to generate specific automated script code. The replaced code is saved as an automated script file.

[0086] Step S210: Execute the target automated screen recording script to perform test verification on the client to be tested.

[0087] In the specific implementation process of this step, the generated automated script is run to simulate the user's gesture operations on the client to be tested for test verification. In the field of medical and health, the automated screen recording script generation method of the present application can be applied to application scenarios such as the testing of electronic medical record systems, medical imaging systems, medical and health applications, medical device management systems, and telemedicine systems. For example, for the testing of an electronic medical record system, the daily operations of doctors and nurses are simulated through an automated script to verify whether the system functions properly, such as the functions of medical record entry, query, and modification; the response time and stability of the system are tested under high load to ensure normal operation during peak hours; it is verified whether the system complies with medical data security regulations to ensure the confidentiality and integrity of patient information, etc. In the field of fintech, the automated screen recording script generation method of the present application can be applied to the testing of real-time trading systems, etc. The automated script can simulate real trading traffic, verify the exception handling mechanism and disaster recovery plan, quickly locate problems, and improve the efficiency of fault resolution.

[0088] This application receives the first gesture operation data monitored by a plug-in with a monitoring function installed on the target client, and uses a preset control element recognition model pre-trained by natural language processing technology to identify the space of the user operation to obtain the target control element of the user operation. Using the preset control element recognition model can save calculation time and accurately find the target control element, improving the generation efficiency of the automation script. And when the first model parameter is not included in the preset database, the first model parameter is stored in the database, so that the automatically generated script can be adapted to more models. This application can match the operation data of the target client's first model parameter and the first gesture operation data for operating on the target client to obtain the second gesture operation data corresponding to the to-be-tested client. Finally, based on the second gesture operation data and the template code, an automation script is generated to obtain a target automation screen recording script, and the target automation screen recording script is executed to test and verify the to-be-tested client. Users only need to perform simple operations on the target client to generate the automation script, without the need for developers to record the operation steps by clicking and dragging the mouse for gesture operations, and then generate the script for the gesture operation, saving manpower and material resources and being less prone to errors. At the same time, the automation screen recording script of this application can be adapted to different models and screen recording tools of different sizes under the same model, solving the problems that may occur when manually recorded scripts are run on different devices due to differences in screen resolutions and sizes of different devices, such as inaccurate element positioning and operation failures, improving the test efficiency, and reducing the development and maintenance costs.

[0089] An embodiment of this application provides an automation screen recording script generation method, as Figure 4 shown, including:

[0090] An acquisition module 1, configured to obtain page structured data, first gesture operation data, and the first model parameter of the target client in response to a page operation of a user on a target page of the target client;

[0091] An identification module 2, configured to perform page control element identification on the page structured data based on the first gesture operation data to obtain the target control element of the user operating the target page;

[0092] A calculation module 3, configured to perform calculation processing based on the first model parameter, the first gesture operation data, and the second model parameter of the to-be-tested client to obtain the second gesture operation data corresponding to the to-be-tested client;

[0093] A generation module 4, configured to generate an automation script based on the second gesture operation data and the target control element to obtain a target automation screen recording script.

[0094] In the specific implementation process, the device further includes a storage module, and the storage module is specifically configured to: query each model parameter in a preset database based on the first model parameter to obtain a query result; when the query result indicates that the first model parameter is not included in each of the model parameters, save the first model parameter in each of the model parameters in the preset database; the model parameters include one or several of mobile phone system parameters, mobile phone model parameters, mobile phone version parameters, screen resolution parameters, and size parameters.

[0095] In the specific implementation process, the recognition module 2 is specifically configured to: perform coordinate matching based on the first gesture operation coordinate position in the first gesture operation data and the page structured data to obtain a plurality of control elements within the range of the area where the first gesture operation coordinate position is located when the user operates the target page; use a preset control recognition model to recognize each of the control elements based on the first gesture operation data to obtain the target control element of the target page operated by the user.

[0096] In the specific implementation process, the recognition module 2 is further configured to: perform data processing on the gesture operation data to obtain a natural language description statement; use the preset control recognition model to perform recognition based on the natural language description statement to obtain a predicted control coordinate position; screen each of the control elements based on the predicted control coordinate position to obtain the target control element of the target page operated by the user.

[0097] In the specific implementation process, the device further includes a model construction module, and the model construction module is specifically configured to: obtain historical gesture operation data and historical page structured data, where the gesture operation data includes gesture type, gesture operation touch point coordinates, and time stamp; the historical page structured data includes control position coordinates, control type, and control content; perform data preprocessing on the historical gesture operation data and the historical page structured data to obtain a gesture operation data description statement and a historical page structured data description statement; load a pre-trained NLP model based on the gesture operation data description statement and the historical page structured data description statement to perform model training to obtain the preset control recognition model.

[0098] In the specific implementation process, the calculation module 3 is specifically configured to: perform calculation processing based on the first abscissa in the first gesture operation parameter, the first horizontal pixel number in the first model parameter, and the second horizontal pixel number in the second model parameter to obtain the second abscissa of the second gesture operation data corresponding to the client to be tested; perform calculation processing based on the first ordinate in the first gesture operation parameter, the first vertical pixel number in the first model parameter, and the second vertical pixel number in the second model parameter to obtain the second ordinate of the second gesture operation data corresponding to the client to be tested.

[0099] In the specific implementation process, the generation module 4 is specifically configured to: query a preset code template library based on the target control element to obtain the template code corresponding to the target control element; generate an automated script based on the second gesture operation data and the template code to obtain a target automated screen recording script; execute the target automated screen recording script to perform test verification on the client to be tested.

[0100] This application receives the page structured data, the first gesture operation data, and the first model parameter of the target client listened by a plug-in with a listening function installed on the target client; uses a preset control recognition model to identify the page control element based on the first gesture operation data and the page structured data to obtain the target control element for the user to operate the target page; using the preset control recognition model to identify page controls can improve the efficiency of generating automated scripts; perform calculation processing based on the first model parameter, the first gesture operation data, and the second model parameter of the client to be tested to obtain the second gesture operation data corresponding to the client to be tested; generate an automated script based on the second gesture operation data and the target control element to obtain a target automated screen recording script. This application can automatically generate an automated script according to the gesture operation without manually configuring the script for different mobile phone models, and this application can adapt to as many models as possible in the market without adapting according to various models, saving manpower.

[0101] Another embodiment of this application provides a storage medium, which stores a computer program. When the computer program is executed by a processor, the following method steps are implemented:

[0102] Step 1: In response to a user's page operation on a target page of a target client, obtain page structured data, first gesture operation data, and the first model parameter of the target client;

[0103] Step 2: Identify the page control element for the page structured data based on the first gesture operation data to obtain the target control element for the user to operate the target page;

[0104] Step 3: Perform calculation and processing based on the first model parameters, the first gesture operation data, and the second model parameters of the client to be tested, and obtain the second gesture operation data corresponding to the client to be tested;

[0105] Step 4: Generate an automated script based on the second gesture operation data and the target control element to obtain a target automated screen recording script.

[0106] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0107] Those skilled in the art can clearly understand that for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0108] For the specific implementation process of the above method steps, reference can be made to the embodiments of any of the above automated screen recording script generation methods, and this embodiment will not be repeated here.

[0109] This application receives page structured data, first gesture operation data, and first model parameters of the target client monitored by a plug-in with a monitoring function installed on the target client; based on the first gesture operation data and the page structured data, a preset control element recognition model is used to identify page control elements, obtaining the target control elements for the user to operate the target page; using the preset control element recognition model to identify page controls can improve the efficiency of automated script generation; based on the first model parameters, the first gesture operation data, and second model parameters of the client to be tested, calculation and processing are performed to obtain second gesture operation data corresponding to the client to be tested; based on the second gesture operation data and the target control elements, an automated script is generated to obtain a target automated screen recording script. This application can automatically generate an automated script according to gesture operations without manually configuring scripts for different mobile phone models, and this application can adapt to as many models in the market as possible without adapting according to various models, saving manpower.

[0110] Another embodiment of this application provides a computer device, which can be a server, and its internal structure diagram can be as Figure 5 shown. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer device program is executed by the processor, it realizes the functions or steps on the server side of an automated screen recording script generation method.

[0111] In one embodiment, a computer device is provided, which can be a client. Its internal structure diagram can be as Figure 6 shown. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer device program is executed by the processor, it realizes the functions or steps on the client side of an automated screen recording script generation method.

[0112] Another embodiment of the present application provides a computer device, which at least includes a memory and a processor. A computer program is stored on the memory, and when the processor executes the computer program on the memory, the following method steps are implemented:

[0113] Step 1: In response to a page operation of a user on a target page in a target client, obtain page structured data, first gesture operation data, and first model parameters of the target client.

[0114] Step 2: Based on the first gesture operation data, perform page control element recognition on the page structured data to obtain a target control element for the user to operate the target page.

[0115] Step 3: Perform calculation processing based on the first model parameters, the first gesture operation data, and second model parameters of a client to be tested to obtain second gesture operation data corresponding to the client to be tested.

[0116] Step 4: Based on the second gesture operation data and the target control element, generate an automated script to obtain a target automated screen recording script.

[0117] For the specific implementation process of the above method steps, reference can be made to the embodiments of any of the above automated screen recording script generation methods, and details will not be repeated in this embodiment.

[0118] The present application obtains page structured data, first gesture operation data, and first model parameters of the target client by receiving and listening through a plug-in with a listening function installed on the target client; based on the first gesture operation data and the page structured data, perform page control element recognition using a preset control recognition model to obtain a target control element for the user to operate the target page; performing page control recognition using a preset control recognition model can improve the efficiency of automated script generation; perform calculation processing based on the first model parameters, the first gesture operation data, and second model parameters of a client to be tested to obtain second gesture operation data corresponding to the client to be tested; based on the second gesture operation data and the target control element, generate an automated script to obtain a target automated screen recording script. The present application can automatically generate an automated script according to gesture operations without manually configuring scripts for different mobile phone models, and the present application can adapt to as many models as possible in the market without adapting according to various models, saving manpower.

[0119] The above embodiments are only exemplary embodiments of the present application and are not used to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art can make various modifications or equivalent replacements within the essence and protection scope of the present application, and such modifications or equivalent replacements should also be regarded as falling within the protection scope of the present application.

Claims

1. An automated screen recording script generation method, characterized in that, Including: In response to a page operation of a user on a target page of a target client, obtaining page structured data, first gesture operation data, and first model parameters of the target client; Based on the first gesture operation data, identifying page control elements in the page structured data to obtain target control elements for the user to operate the target page; Based on the first model parameters, the first gesture operation data, and second model parameters of a client under test, performing calculation processing to obtain second gesture operation data corresponding to the client under test; Based on the second gesture operation data and the target control elements, generating an automated script to obtain a target automated screen recording script.

2. The method according to claim 1, characterized in that, Before identifying page control elements in the page structured data based on the gesture operation data, the method further includes: Querying each model parameter in a preset database based on the first model parameters to obtain a query result; When the query result indicates that the first model parameter is not included in each of the model parameters, saving the first model parameter in each of the model parameters of the preset database; The model parameters include one or several of mobile phone system parameters, mobile phone model parameters, mobile phone version parameters, screen resolution parameters, and size parameters.

3. The method according to claim 1, wherein The identifying page control elements in the page structured data based on the gesture operation data to obtain target control elements for the target page operated by the user specifically includes: Based on the first gesture operation coordinate positions in the first gesture operation data and the page structured data, performing coordinate matching to obtain several control elements within the area range where the first gesture operation coordinate positions are located when the user operates the target page; Based on the first gesture operation data, using a preset control element recognition model to recognize each of the control elements to obtain target control elements for the target page operated by the user.

4. The method according to claim 3, wherein The using a preset control element recognition model to recognize each of the control elements based on the first gesture operation data to obtain target control elements for the target page operated by the user specifically includes: Performing data processing on the gesture operation data to obtain a natural language description statement; Based on the natural language description statement, using the preset control element recognition model to perform recognition to obtain predicted control coordinate positions; Based on the predicted control coordinate positions, screening each of the control elements to obtain target control elements for the target page operated by the user.

5. The method according to claim 4, characterized in that, Before using a preset control element recognition model to recognize each of the control elements based on the gesture operation data, the method further includes: constructing a preset control element recognition model; The constructing a preset control element recognition model specifically includes: Obtaining historical gesture operation data and historical page structured data, where the gesture operation data includes gesture type, gesture operation touch point coordinates, and time stamp; the historical page structured data includes control position coordinates, control type, and control content; Performing data preprocessing on the historical gesture operation data and the historical page structured data to obtain a gesture operation data description statement and a historical page structured data description statement; Based on the gesture operation data description statement and the historical page structured data description statement, a pre-trained NLP model is loaded for model training to obtain the preset control recognition model.

6. The method according to claim 1, wherein The calculation and processing based on the first model parameter, the first gesture operation data, and the second model parameter of the client to be tested to obtain the second gesture operation data corresponding to the client to be tested specifically includes: Performing calculation and processing based on the first abscissa in the first gesture operation parameter, the first horizontal pixel number in the first model parameter, and the second horizontal pixel number in the second model parameter to obtain the second abscissa of the second gesture operation data corresponding to the client to be tested; Performing calculation and processing based on the first ordinate in the first gesture operation parameter, the first vertical pixel number in the first model parameter, and the second vertical pixel number in the second model parameter to obtain the second ordinate of the second gesture operation data corresponding to the client to be tested.

7. The method according to claim 1, wherein The generation of the target automated screen recording script based on the second gesture operation data and the target control element specifically includes: Querying a preset code template library based on the target control element to obtain the template code corresponding to the target control element; Generating an automated script based on the second gesture operation data and the template code to obtain the target automated screen recording script; Executing the target automated screen recording script to perform test verification on the client to be tested.

8. An automated screen recording script generation device, characterized in that Including: An acquisition module, configured to acquire page structured data, first gesture operation data, and the first model parameter of the target client in response to a page operation of a user on a target page of the target client; An identification module, configured to perform page control element identification on the page structured data based on the first gesture operation data to obtain the target control element for the user to operate the target page; A calculation module, configured to perform calculation and processing based on the first model parameter, the first gesture operation data, and the second model parameter of the client to be tested to obtain the second gesture operation data corresponding to the client to be tested; A generation module, configured to generate an automated script based on the second gesture operation data and the target control element to obtain the target automated screen recording script.

9. A storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the automated screen recording script generation method according to any one of claims 1-7 above are implemented.

10. A computer device, characterized in that, At least including a memory and a processor, a computer program is stored on the memory, and when the processor executes the computer program on the memory, the steps of the automated screen recording script generation method according to any one of claims 1-7 above are implemented.

Images