Automatic dial testing method and device, equipment, storage medium and product
Through the terminal automatic dialing method based on image detection model and state machine, the problems of poor compatibility and complex logic control in the prior art are solved, and efficient and intelligent terminal automatic dialing is realized, reducing costs and testing time.
Patent Information
- Application Number
- CN202411496681.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2025-02-14
AI Technical Summary
The existing terminal automatic dialing technology has poor compatibility and cannot cope with complex logic controls such as mid-way exceptions. The control logic is deeply bound to the target application code and UI level. After modification, the logic control code needs to be reconstructed, resulting in high usage costs.
By obtaining the screenshot information of the terminal to be dialed, state recognition is performed based on the pre-trained image detection model, and combining the pre-trained state machine to generate and send control instructions to realize automatic dialing of the terminal.
It improves the intelligence level of automatic dialing at terminals, reduces dialing costs, enhances the real-time and testing efficiency of automated dialing attestation, and can cope with complex logic control and reduces restructuring costs.
Smart Images
Figure CN119450562A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminal testing, and in particular to an automatic dialing test method, device, equipment, storage medium and product. Background Art
[0002] In order to ensure the healthy operation of services on the terminal, it is often necessary to test the service quality of the services running on the terminal. For example, with the development of 5G communication technology, 5G new call services have been widely used. The 5G new call is a new generation of call products based on the 5G network, including but not limited to: 5G VoNR ultra-clear calls, smart translation (real-time translation between Chinese and English, voice to large characters, etc.), fun calls, smart customer service, content sharing, remote assistance and other functions. In order to ensure the healthy development of the business, it is necessary to continuously test the service quality.
[0003] In the related art, the automatic dialing test of the terminal mainly includes the following two methods: method 1, automatic dialing test based on the position of the operation element; method 2, automatic dialing test based on the ID of the operation element.
[0004] The above method 1 first uses manual operation of the terminal; then during the operation, the terminal's built-in display pointer position is used to record the position of the operating elements in each operation step in turn; then a control script is written to generate a series of timing control instructions in accordance with the operation sequence; finally, the script is executed to execute the control commands of the script in turn.
[0005] The second method mentioned above first opens the target application and manually executes the control process; then uses some layout display tools such as UIAutormatorViewer or AppiumInspector to record the page controllable element IDs in each operation step of the target application in turn; finally, writes the control logic code to determine whether there is the page control element ID on the page, and if so, operates these control elements.
[0006] It should be pointed out that the above method 1 is limited by the size of different terminals and the position of the operating elements, has poor compatibility, and is based on the recorded operating steps for sequential control, and cannot cope with complex logic control such as midway exceptions. The above method 2 can only control a single target application. If multiple applications jump, it cannot be automatically controlled and requires manual intervention; secondly, the control logic is deeply bound to the target application code and UI level. If the name of the operable element of the target application is modified, the entire control logic will become invalid and the logic control code needs to be reconstructed, resulting in high usage costs for ordinary users. Summary of the invention
[0007] In view of this, the embodiments of the present application provide an automatic dialing method, apparatus, device, storage medium and product, which aim to improve the intelligence level of terminal automatic dialing and reduce the dialing cost.
[0008] The technical solution of the embodiment of the present application is implemented as follows:
[0009] In a first aspect, an embodiment of the present application provides an automatic dialing method, including:
[0010] Get the screenshot information of the terminal to be tested;
[0011] Processing the screenshot information based on a pre-trained image detection model to obtain a current state of the terminal to be tested;
[0012] Based on the current state and the pre-trained state machine, generate a control instruction;
[0013] The control instruction is sent to the terminal to be tested; wherein the control instruction is used for the terminal to be tested to execute the corresponding setting action in the current state and automatically enter the next state.
[0014] In the above scheme, the pre-trained image detection model is used to process the screenshot information to obtain the current state of the terminal to be tested, including:
[0015] Based on the pre-trained image detection model, the page of the screenshot information and the elements in the page are identified to obtain the page identification result and the element identification result of the terminal to be tested; wherein the current state includes the page identification result and the element identification result.
[0016] In the above solution, the pre-trained image detection model includes: a page classification model for page recognition and an object detection model for element recognition in a page, and the method further includes:
[0017] Constructing a first training sample set based on screenshot information in different page states, and training the page classification model based on the first training sample set;
[0018] A second training sample set is constructed based on bounding box information of different elements in the screenshot information, and the object detection model is obtained by training based on the second training sample set.
[0019] In the above solution, the generating of control instructions based on the current state and the pre-trained state machine includes:
[0020] Based on the current state, querying the operation state table of the pre-trained state machine for a set action corresponding to the current state;
[0021] generating the control instruction based on the setting action;
[0022] Wherein, each operation state table in the pre-trained state machine includes: current state, set action and next state.
[0023] In the above scheme, the method further includes:
[0024] Processing the screenshot information of the terminal during the manual dialing test based on the pre-trained image detection model to obtain the states of multiple key frames in the manual dialing test process and the corresponding page description files;
[0025] Outputting a visual user interface based on the page description file;
[0026] Configuring the operation status table based on multiple visual interfaces;
[0027] The pre-trained state machine is obtained or updated based on the configured operation state table.
[0028] In the above scheme, before obtaining the screenshot information of the terminal to be tested, the method further includes:
[0029] Establishing an application-based communication connection with the terminal to be tested; wherein the communication connection is used for the subsequent transmission of screenshot information and control instructions.
[0030] In the above solution, establishing an application-based communication connection with the terminal to be tested includes:
[0031] Obtaining terminal model information of the terminal to be tested;
[0032] Determine a target installation package based on the terminal model information;
[0033] The target installation package is sent to the terminal to be tested, so that the terminal to be tested starts the server end of the communication connection after running the target installation package, and establishes a connection with the server end based on the client end of the communication connection.
[0034] In the above scheme, the method further includes:
[0035] A state machine matching the terminal to be tested is determined based on the terminal model information.
[0036] In the above scheme, the method further includes:
[0037] Receive the screen recording video data of the corresponding dialing test process sent by the terminal to be dialed after the dialing test is completed.
[0038] In a second aspect, an embodiment of the present application provides an automatic dialing device, including:
[0039] An acquisition module is used to obtain the screenshot information of the terminal to be tested;
[0040] A first processing module, configured to process the screenshot information based on a pre-trained image detection model to obtain a current state of the terminal to be tested;
[0041] A second processing module, used for generating a control instruction based on the current state and a pre-trained state machine;
[0042] The sending module is used to send the control instruction to the terminal to be tested; wherein the control instruction is used for the terminal to be tested to execute the corresponding setting action in the current state and automatically enter the next state.
[0043] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor, when used to run the computer program, executes the steps of the method described in the first aspect of the embodiment of the present application.
[0044] In a fourth aspect, an embodiment of the present application provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect of the embodiment of the present application are implemented.
[0045] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect of the embodiment of the present application.
[0046] The technical solution provided by the embodiment of the present application obtains the screenshot information of the terminal to be tested; processes the screenshot information based on the pre-trained image detection model to obtain the current state of the terminal to be tested; generates a control instruction based on the current state and the pre-trained state machine; sends the control instruction to the terminal to be tested; wherein the control instruction is used for the terminal to be tested to execute the corresponding set action in the current state and automatically enter the next state. In this way, the state of the screenshot information of the terminal to be tested can be identified based on the image detection model, and the control instruction can be generated and sent to the terminal to be tested based on the identified current state and the pre-trained state machine, thereby realizing automatic dialing of the terminal, effectively saving the labor cost and test time of terminal dialing. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 This is a flow chart of the automatic dialing method according to an embodiment of the present application;
[0048] Figure 2 A schematic diagram of a visual user interface generated based on a page description file in an application example of the present application;
[0049] Figure 3 A flowchart of an automatic dialing method according to an embodiment of the present application;
[0050] Figure 4 This is a schematic diagram of the structure of the automatic dialing device according to an embodiment of the present application;
[0051] Figure 5 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0052] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments.
[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.
[0054] An embodiment of the present application provides an automatic dialing method, which can be independently applied to electronic devices with data processing capabilities, such as terminal-side electronic devices, servers, etc., and can also be implemented by the cooperation between terminal-side electronic devices and servers; wherein the terminal-side electronic devices can specifically be computers, smart phones, personal digital assistants (PDA), etc.; the server can specifically be an application server or a Web server. In actual deployment, the server can be an independent server or a cluster server.
[0055] like Figure 1 As shown, the automatic dialing method includes:
[0056] Step 101: Obtain screenshot information of the terminal to be tested.
[0057] Here, after the electronic device establishes a connection with the terminal to be tested, the terminal to be tested can reflect the current state based on the screenshot information and send the screenshot information to the electronic device. The terminal to be tested can be understood as a terminal that needs to perform a test on the service running on the terminal, including but not limited to the 5G new call service that has been launched and is about to be launched.
[0058] Step 102: Process the screenshot information based on a pre-trained image detection model to obtain the current state of the terminal to be tested.
[0059] Here, the pre-trained image detection model can be understood as a model with computer vision recognition capabilities. The electronic device can perform status recognition on the screenshot information of the terminal to be tested based on the model to obtain the current status of the terminal to be tested.
[0060] Step 103: Generate a control instruction based on the current state and the pre-trained state machine.
[0061] Here, the electronic device may pre-train the state machine and the current state of the terminal to be tested, determine the setting action corresponding to the terminal to be tested, and generate a control instruction corresponding to the setting action.
[0062] Step 104, sending the control instruction to the terminal to be tested; wherein the control instruction is used for the terminal to be tested to execute the corresponding setting action in the current state and automatically enter the next state.
[0063] Here, the electronic device sends a control instruction to the terminal to be tested. After receiving the control instruction, the terminal to be tested can execute the corresponding set action in the current state and automatically enter the next state. In this way, the state recognition can be performed based on the image detection model based on the screenshot information of the terminal to be tested, and the control instruction can be generated and sent to the terminal to be tested based on the recognized current state and the pre-trained state machine, thereby realizing automatic terminal dialing test, effectively saving the labor cost and test time of terminal dialing test.
[0064] It should be noted that using the screenshot information of the terminal to be tested as the input of the dialing test control can enhance the real-time performance of the automated control of the dialing test and improve the test efficiency of the dialing test; in addition, based on the current state and the pre-trained state machine, control instructions are generated to achieve automated control with feedback adjustment, that is, by updating the control strategy of the state machine, feedback adjustment of the dialing test can be achieved, thereby meeting the needs of dynamic adjustment of the dialing test logic and reducing the cost of the dialing test.
[0065] Exemplarily, the processing of the screenshot information based on the pre-trained image detection model to obtain the current state of the terminal to be tested includes:
[0066] Based on the pre-trained image detection model, the page of the screenshot information and the elements in the page are identified to obtain the page identification result and the element identification result of the terminal to be tested; wherein the current state includes the page identification result and the element identification result.
[0067] It should be pointed out that the image detection model of the embodiment of the present application introduces image detection in two dimensions, namely, image detection in the page dimension and image detection in the dimension of controllable elements on the page, to obtain page recognition results and element recognition results of the terminal to be tested. In this way, the accuracy of state recognition of the terminal to be tested can be improved, and the accuracy of automated dialing can be effectively guaranteed; in addition, through the coordination of image detection in two dimensions, the efficiency of state recognition of the terminal to be tested can be improved, and the real-time performance of automated dialing can be improved.
[0068] Exemplarily, the pre-trained image detection model includes: a page classification model for page recognition and an object detection model for element recognition in a page, and the method further includes:
[0069] Constructing a first training sample set based on screenshot information in different page states, and training the page classification model based on the first training sample set;
[0070] A second training sample set is constructed based on bounding box information of different elements in the screenshot information, and the object detection model is obtained by training based on the second training sample set.
[0071] Exemplarily, screenshot information corresponding to key nodes in the manual dialing process of the terminal can be obtained, and a first training sample set can be constructed based on the classification of page states of the screenshot information, so that the classification model learns the image features of each page state to obtain a trained page classification model.
[0072] In an application example, the first training sample set includes screenshot information of multiple page states, taking call dialing as an example, including but not limited to: initialization page (init), dialing interface (dialer_home), dial pad interface (dialpad_page), number input interface (number_entered), dialing interface (calling), call interface (call_in_progress), call end interface (call_ended). Each page state may correspond to multiple screenshot information, and a deep learning framework may be used for training to obtain a page classification model, for example, Pytorch (an open source deep learning framework for machine learning and deep learning) and the first training sample set may be used for training to obtain a page classification model.
[0073] Exemplarily, the screenshot information of key nodes can be used to construct a second training sample set based on the bounding box of each element and the annotation of the element category, so that the target detection model learns the image features of the screenshot area of each element to obtain a trained target detection model.
[0074] In an application example, a labeling tool can be used to label specific elements in the screenshot. For example, a tool such as LabelImg can be used to draw the bounding box of each element on each screenshot and label the corresponding category label. For example, a bounding box of dialpad_button_1 is drawn and its category label (for example, 'dialpad_button_1') is labeled. In this way, the screenshots of each element on the screenshot information and the corresponding category labels can be obtained, and then the second training sample set can be obtained. Further, the target detection algorithm YOLO and the second training sample set can be used for training to obtain a target detection model for page element detection.
[0075] Exemplarily, the generating a control instruction based on the current state and the pre-trained state machine includes:
[0076] Based on the current state, querying the operation state table of the pre-trained state machine for a set action corresponding to the current state;
[0077] generating the control instruction based on the setting action;
[0078] Wherein, each operation state table in the pre-trained state machine includes: current state, set action and next state.
[0079] Here, the pre-trained state machine is used to manage the control logic of the dial test, wherein the operation state table included in the state machine can be configured based on a visual interface, thereby realizing complex logic control in the dial test process. For example, different from learning the sequential control logic of manual dial test, different jump logics can be configured through the visual interface to realize personalized dial test control. Moreover, the process of logic reconstruction does not require the user to master programming technology, and the operation is simple, which can effectively reduce the dial test cost.
[0080] Exemplarily, in order to implement visual configuration of the operation status table, the method further includes:
[0081] Processing the screenshot information of the terminal during the manual dialing test based on the pre-trained image detection model to obtain the states of multiple key frames in the manual dialing test process and the corresponding page description files;
[0082] Outputting a visual user interface based on the page description file;
[0083] Configuring the operation status table based on multiple visual interfaces;
[0084] The pre-trained state machine is obtained or updated based on the configured operation state table.
[0085] Here, the page recognition results and element recognition results of the screenshots of each key frame can be identified based on the aforementioned page classification model and target detection model, and a page description file can be generated based on the page recognition results and element recognition results. The page description file can be understood as a file that describes the page interface of the key frame. In an application example, the page description file can adopt the YAML file format. YAML is a highly readable format used to express data serialization.
[0086] For example, a page interface of a screenshot of multiple key frames can be represented as follows:
[0087] Pages:
[0088] Name: "init"
[0089] Elements:
[0090]
[0091] Based on the page description file of the page interface of the above multiple key frames, the following can be generated: Figure 2 The visual user interface shown, on this basis, the user can perform connection operations on the visual user interface, configure the jump logic of the operable elements, obtain the configuration file of the operation state table, and then obtain the pre-trained state machine or update the pre-trained state machine. Here, if the user configures the jump logic of the operable elements on the page interface of the key frame for the first time, the pre-trained state machine can be obtained based on the configuration file of the operation state table. If the jump logic of the operable elements on the page interface of the key frame is configured based on the existing state machine, the state machine can be updated based on the newly generated configuration file of the operation state table, thereby realizing the dynamic adjustment of the control logic of the dial test.
[0092] In an application example, the configuration file of the operation status table may adopt a YAML file format, as shown below:
[0093]
[0094] It is understandable that the above operation state table includes: current state, set action, control instruction, next state. The pre-trained state machine can configure multiple operation state tables based on the connection operation on the visual user interface, thereby realizing the logic switching control of the dialing process.
[0095] Exemplarily, before obtaining the screenshot information of the terminal to be tested, the method further includes:
[0096] Establishing an application-based communication connection with the terminal to be tested; wherein the communication connection is used for the subsequent transmission of screenshot information and control instructions.
[0097] Here, in order to improve the reliability of the dial test control, before the dial test control is started, an application-based communication connection can be established between the electronic device and the terminal to be dialed, thereby ensuring the reliability of the subsequent transmission of the screenshot information and the control command.
[0098] Exemplarily, establishing an application-based communication connection with the terminal to be tested includes:
[0099] Obtaining terminal model information of the terminal to be tested;
[0100] Determine a target installation package based on the terminal model information;
[0101] The target installation package is sent to the terminal to be tested, so that the terminal to be tested starts the server end of the communication connection after running the target installation package, and establishes a connection with the server end based on the client end of the communication connection.
[0102] Here, the electronic device can first obtain the terminal model information of the terminal to be tested, and search for a matching target installation package based on the terminal model information, for example, search for a jar package that can be run on the terminal to be tested, and send the jar package to the terminal to be tested. After receiving the jar package, the terminal to be tested will start a server based on TCP / IP protocol socket communication, and at the same time, the terminal to be tested will start a client based on TCP / IP protocol socket communication, and then establish a connection between the two.
[0103] Exemplarily, the method further includes:
[0104] A state machine matching the terminal to be tested is determined based on the terminal model information.
[0105] Exemplarily, in order to enhance the accuracy of dialing control, the pre-trained state machine can correspond to the terminal model information of the terminal. After the electronic device obtains the terminal model information of the terminal to be dialed, it can query the matching state machine based on the terminal model information, and then obtain the state machine that matches the terminal to be dialed, so that the terminal of this model can perform automatic dialing control based on the corresponding state machine.
[0106] Exemplarily, the method further includes:
[0107] Receive the screen recording video data of the corresponding dialing test process sent by the terminal to be dialed after the dialing test is completed.
[0108] Here, the terminal to be tested can generate and send screen recording video data to the electronic device after the dialing test is completed. In this way, the electronic device can analyze the screen recording video data through the test algorithm to obtain the test results. In addition, the data of the dialing process can also be traced, which is convenient for subsequent manual review.
[0109] The present application is further described in detail below in conjunction with application examples.
[0110] Figure 3 A flow chart of the automatic dialing method of the present application embodiment is shown, wherein the terminal to be dialed is an Android terminal, and the electronic device includes a terminal interaction module and an instruction generation module, wherein the terminal interaction module is used to collect the real-time status of the terminal, and then pass the real-time status to the instruction generation module, after the instruction generation module generates a control instruction, the control instruction is sent to the terminal through the terminal interaction module.
[0111] like Figure 3 As shown, the method of this application embodiment includes the following steps:
[0112] Step 301: Connect the Android terminal to the PC terminal via USB.
[0113] Step 302: The terminal interaction module obtains the terminal model information, which mainly includes Android operating system information, screen height information, and screen width information.
[0114] Step 303: The terminal interaction module obtains the terminal model information and sends it to the instruction generation module.
[0115] Step 304: The instruction generation module performs initialization work according to the received terminal model information and selects a suitable state machine.
[0116] Step 305: the terminal interaction module pushes the jar package executable by the Android terminal to the Android terminal and runs it. This step avoids the need to install the App on the Android terminal and requires root permission.
[0117] Step 306 to step 308: The running jar package will start a server based on TCP / IP protocol socket communication on the Android terminal, and the terminal interaction module will start a client based on TCP / IP protocol socket communication, and then establish a connection between the two.
[0118] Step 309: The server started in step 307 will continuously record the terminal screen.
[0119] Step 310: The server started in step 307 sends the screen content to the terminal interaction module in the form of a video stream.
[0120] Step 311: The terminal interaction module extracts key frame images based on the received video stream information.
[0121] Step 312: The terminal interaction module sends the key frame image to the instruction generation module.
[0122] Step 313: The instruction generation module recognizes the terminal status based on the key frame image, queries the current status and the corresponding next operation in the operation status table, and then generates corresponding control instructions, such as pressing the dial button on the dialing completion page; pressing the answer button on the answering page, etc.
[0123] Step 314 to step 315: the instruction generation module sends the control instruction to the terminal interaction module, and then the terminal interaction module converts the control instruction into a control instruction that can be recognized by the terminal, continuously interacts and controls the terminal, and realizes automatic call making.
[0124] Steps 316 to 318: After the automated dialing is completed, the screen recording video data generated by the automated dialing of the calling and called ends is sent to the terminal interaction module, and the test results are generated through algorithm analysis.
[0125] It should be noted that the instruction generation module in this application embodiment can be deployed on a PC terminal or on a remote server, that is, the effect of remote automatic dialing can be achieved based on the interaction between the PC terminal and the remote server.
[0126] It can be understood that the present application embodiment does not require the user to write the logic code for automatic dialing. The control logic of the dialing can be implemented by a state machine. The state machine is written in YAML language, which is similar to natural language, simple to understand and easy to modify. In addition, real-time control is achieved by real-time recognition of the terminal screen image. For example, real-time input of the dialing test can be achieved based on the control instructions generated by the state machine. For example, in addition to terminal operations such as answering and dialing buttons, third-party input can also be included. For example, in the voice transcription scenario, the set voice data is input, thereby ensuring the real-time nature of the dialing test. Thirdly, feedback adjustment is achieved by introducing a state machine, and the dialing state of the terminal can be obtained in real time, effectively improving the control accuracy and real-time nature. By real-time acquisition of the terminal screen image, the image is converted into a state in the state machine, and the next operation instruction is obtained according to the state machine, thereby achieving automatic control with feedback adjustment.
[0127] In order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an automatic dialing device, which corresponds to the above-mentioned automatic dialing method, and each step in the above-mentioned automatic dialing method embodiment is also completely applicable to the embodiment of the present automatic dialing device.
[0128] like Figure 4 As shown, the automatic dialing device includes: an acquisition module 401, a first processing module 402, a second processing module 403 and a sending module 404; the acquisition module 401 is used to acquire the screenshot information of the terminal to be dialed; the first processing module 402 is used to process the screenshot information based on a pre-trained image detection model to obtain the current state of the terminal to be dialed; the second processing module 403 is used to generate a control instruction based on the current state and a pre-trained state machine; the sending module 404 is used to send the control instruction to the terminal to be dialed; wherein, the control instruction is used for the terminal to be dialed to execute the corresponding set action in the current state and automatically enter the next state.
[0129] Exemplarily, the first processing module 402 is specifically configured to:
[0130] Based on the pre-trained image detection model, the page of the screenshot information and the elements in the page are identified to obtain the page identification result and the element identification result of the terminal to be tested; wherein the current state includes the page identification result and the element identification result.
[0131] Exemplarily, the pre-trained image detection model includes: a page classification model for page recognition and a target detection model for element recognition in a page. The automatic dialing device also includes: a training module 405, which is used to construct a first training sample set based on screenshot information in different page states, and obtain the page classification model based on the first training sample set; and to construct a second training sample set based on the bounding box information of different elements in the screenshot information, and obtain the target detection model based on the second training sample set.
[0132] Exemplarily, the second processing module 403 is specifically configured to:
[0133] Based on the current state, querying the operation state table of the pre-trained state machine for a set action corresponding to the current state;
[0134] generating the control instruction based on the setting action;
[0135] Wherein, each operation state table in the pre-trained state machine includes: current state, set action and next state.
[0136] Exemplarily, the automatic dialing device includes: a configuration module 406, which is used to:
[0137] Processing the screenshot information of the terminal during the manual dialing test based on the pre-trained image detection model to obtain the states of multiple key frames in the manual dialing test process and the corresponding page description files;
[0138] Outputting a visual user interface based on the page description file;
[0139] Configuring the operation status table based on multiple visual interfaces;
[0140] The pre-trained state machine is obtained or updated based on the configured operation state table.
[0141] Exemplarily, the automatic dialing device further includes: a connection module 407, which is used to establish an application-based communication connection with the terminal to be dialed before acquiring the screenshot information of the terminal to be dialed; wherein the communication connection is used for subsequent transmission of screenshot information and control instructions.
[0142] Exemplarily, the connection module 407 is specifically used for:
[0143] Obtaining terminal model information of the terminal to be tested;
[0144] Determine a target installation package based on the terminal model information;
[0145] The target installation package is sent to the terminal to be tested, so that the terminal to be tested starts the server end of the communication connection after running the target installation package, and establishes a connection with the server end based on the client end of the communication connection.
[0146] Exemplarily, the acquisition module 401 is further used for:
[0147] A state machine matching the terminal to be tested is determined based on the terminal model information.
[0148] Exemplarily, the acquisition module 401 is further used for:
[0149] Receive the screen recording video data of the corresponding dialing test process sent by the terminal to be dialed after the dialing test is completed.
[0150] In actual application, the acquisition module 401, the first processing module 402, the second processing module 403, the sending module 404, the training module 405, the configuration module 406 and the connection module 407 can be implemented by a processor in the electronic device. Of course, the processor needs to run the computer program in the memory to realize its function.
[0151] It should be noted that: the automatic dialing device provided in the above embodiment only uses the division of the above program modules as an example when performing automatic dialing. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the automatic dialing device provided in the above embodiment and the automatic dialing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0152] Based on the hardware implementation of the above program modules and in order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an electronic device. Figure 5 Only an exemplary structure of the electronic device is shown, not all structures, and can be implemented as needed. Figure 5 Partial or complete structure shown.
[0153] like Figure 5 As shown, the electronic device 500 provided in the embodiment of the present application includes: at least one processor 501, a memory 502, a user interface 503 and at least one network interface 504. The various components in the electronic device 500 are coupled together through a bus system 505. It can be understood that the bus system 505 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 505 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 5 Various buses are labeled as bus system 505 .
[0154] The user interface 503 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.
[0155] The memory 502 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program used to operate on the electronic device.
[0156] The automatic dialing method disclosed in the embodiment of the present application can be applied to the processor 501, or implemented by the processor 501. The processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the automatic dialing method can be completed by the hardware integrated logic circuit in the processor 501 or the instructions in the form of software. The above-mentioned processor 501 can be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 501 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, and the storage medium is located in the memory 502. The processor 501 reads the information in the memory 502 and completes the steps of the automatic dialing method provided in the embodiment of the present application in combination with its hardware.
[0157] In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA), general processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned method.
[0158] It can be understood that the memory 502 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.
[0159] In an exemplary embodiment, the present application also provides a computer storage medium, which can be a computer-readable storage medium, for example, a memory 502 storing a computer program, and the computer program can be executed by a processor 501 of an electronic device 500 to complete the steps described in the method of the present application embodiment. The computer-readable storage medium can be a memory such as a ROM, a PROM, an EPROM, an EEPROM, a Flash Memory, a magnetic surface memory, an optical disk, or a CD-ROM.
[0160] In an exemplary embodiment, the embodiment of the present application further provides a computer program product, including a computer program, which can be executed by the processor 501 of the electronic device 500 to complete the steps described in the method of the embodiment of the present application.
[0161] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0162] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.
[0163] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. An automatic dialing method, characterized in that: include: Get the screenshot information of the terminal to be tested; Processing the screenshot information based on a pre-trained image detection model to obtain a current state of the terminal to be tested; Based on the current state and the pre-trained state machine, generate a control instruction; The control instruction is sent to the terminal to be tested; wherein the control instruction is used for the terminal to be tested to execute the corresponding setting action in the current state and automatically enter the next state.
2. The method according to claim 1, characterized in that The step of processing the screenshot information based on the pre-trained image detection model to obtain the current state of the terminal to be tested includes: Based on the pre-trained image detection model, the page of the screenshot information and the elements in the page are identified to obtain the page identification result and the element identification result of the terminal to be tested; wherein the current state includes the page identification result and the element identification result.
3. The method according to claim 2, characterized in that The pre-trained image detection model includes: a page classification model for page recognition and an object detection model for element recognition in a page, and the method further includes: Constructing a first training sample set based on screenshot information in different page states, and training the page classification model based on the first training sample set; A second training sample set is constructed based on bounding box information of different elements in the screenshot information, and the object detection model is obtained by training based on the second training sample set.
4. The method according to claim 1, characterized in that: The generating of control instructions based on the current state and the pre-trained state machine includes: Based on the current state, querying the operation state table of the pre-trained state machine for a set action corresponding to the current state; generating the control instruction based on the setting action; Wherein, each operation state table in the pre-trained state machine includes: current state, set action and next state.
5. The method according to claim 4, characterized in that The method further comprises: Processing the screenshot information of the terminal during the manual dialing test based on the pre-trained image detection model to obtain the states of multiple key frames in the manual dialing test process and the corresponding page description files; Outputting a visual user interface based on the page description file; Configuring the operation status table based on multiple visual interfaces; The pre-trained state machine is obtained or updated based on the configured operation state table.
6. The method according to claim 1, characterized in that Before obtaining the screenshot information of the terminal to be tested, the method further includes: Establishing an application-based communication connection with the terminal to be tested; wherein the communication connection is used for the subsequent transmission of screenshot information and control instructions.
7. The method according to claim 6, characterized in that The establishing of an application-based communication connection with the terminal to be tested includes: Obtaining terminal model information of the terminal to be tested; Determine a target installation package based on the terminal model information; The target installation package is sent to the terminal to be tested, so that the terminal to be tested starts the server end of the communication connection after running the target installation package, and establishes a connection with the server end based on the client end of the communication connection.
8. The method according to claim 7, characterized in that The method further comprises: A state machine matching the terminal to be tested is determined based on the terminal model information.
9. The method according to claim 1, characterized in that: The method further comprises: Receive the screen recording video data of the corresponding dialing test process sent by the terminal to be dialed after the dialing test is completed.
10. An automatic dialing device, characterized in that: include: An acquisition module is used to obtain the screenshot information of the terminal to be tested; A first processing module, configured to process the screenshot information based on a pre-trained image detection model to obtain a current state of the terminal to be tested; A second processing module, used for generating a control instruction based on the current state and a pre-trained state machine; The sending module is used to send the control instruction to the terminal to be tested; wherein the control instruction is used for the terminal to be tested to execute the corresponding setting action in the current state and automatically enter the next state.
11. An electronic device, characterized in that: include: A processor and a memory for storing a computer program that can be executed on the processor, wherein: The processor is used to execute the steps of the method according to any one of claims 1 to 9 when running a computer program.
12. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.
13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.