Multi-machine synchronous control method, device, equipment and medium based on cloud real machine

Abstract

The present application discloses a multi-machine synchronous control method, device, equipment and medium based on cloud real machine, which relates to the field of application testing. The first control instruction of the user to the master cloud real machine is intercepted through the browser, including the first control event and the first control position information; the current screen pictures of the master cloud real machine and the slave cloud real machine are respectively subjected to text recognition based on text recognition technology to obtain text recognition results; the second control position information of each slave cloud real machine is determined according to the first control position information and the text recognition result; a message body is generated according to the first control event, the first control position information and the second control position information; the message body is sent to the server, so that the server can synchronously control the master cloud real machine and the slave cloud real machine according to the message body. The present application can not only be applied across platforms, but also can realize real-time synchronous response of the master cloud real machine and the slave cloud real machine.

Description

Technical Field

[0001] The present application relates to the field of application testing technology, and in particular to a multi-machine synchronization control method, device, equipment and medium based on a cloud real machine. Background Art

[0002] Smart terminal application software testing is a key task in the smart terminal software development life cycle. The main task is to test the compatibility and test efficiency of unreleased tested software on different test models and devices. In order to facilitate the testing of smart terminal application software, a cloud real machine control platform is usually used for testing.

[0003] In order to improve the test efficiency of smart terminal application software testing, realize parallel testing and enhance test coverage, automated testing tools that support one-machine-multiple-control mode are usually used, such as SoloPi, Zezhongyun real machine, etc. to implement testing of application software in mobile terminals.

[0004] SoloPi is based on Android AccessibilityService system service and only supports Android devices. When in use, it needs to use the mobile phone system accessibility function and adb service. The preparation steps are cumbersome and time-consuming. Each operation will dump the current control element, and the overall response is slow. The real machine of Zezhong Zhongyun synchronizes the click position of the device based on the coordinate conversion of the device resolution, which has too many limitations. For example, at the same resolution, the user sets a different system text size, and the click position is greatly offset and cannot be clicked correctly; at different resolutions, the position based on coordinate conversion is greatly offset and cannot be clicked correctly. Summary of the invention

[0005] In view of the shortcomings of the prior art mentioned above, the purpose of the present application is to provide a multi-machine synchronous control method, device, equipment and medium based on cloud real machine, which is used to solve the problems in the prior art that the one-machine multi-control mode of the automated testing tool cannot be used across platforms, has slow response and cannot be accurately synchronized.

[0006] To achieve the above-mentioned purpose and other related purposes, the present application provides a multi-machine synchronization control method based on a cloud real machine, which is applied to the Web end of a cloud real machine testing platform, and the method includes:

[0007] intercepting, by a browser, a first control instruction of a user on a master cloud real machine, wherein the first control instruction includes a first control event and first control position information;

[0008] Obtaining current screen images of the master cloud real machine and at least one slave cloud real machine;

[0009] Performing text recognition on the current screen images of the master cloud real machine and the slave cloud real machine respectively to obtain text recognition results;

[0010] Determine the second control position information of each slave control cloud real machine according to the first control position information and the text recognition result;

[0011] Generate a message body according to the first manipulation event, the first manipulation position information and the second manipulation position information;

[0012] The message body is sent to the server, so that the server synchronously controls the master cloud real machine and the slave cloud real machine according to the message body.

[0013] In an optional embodiment of the present application, the server synchronously controls the master cloud real machine and the slave cloud real machine according to the message body, including:

[0014] The server parses the message body to obtain a second control instruction for controlling the master cloud real machine and a third control instruction for controlling the slave cloud real machine, wherein the second control instruction includes a second control event and the first control position information, and the third control instruction includes a third control event and the second control position information;

[0015] The server sends the second control instruction to the master cloud real machine, so that the master cloud real machine executes the second control instruction;

[0016] The server sends the third control instruction to the corresponding slave cloud real machine, so that the slave cloud real machine executes the third control instruction.

[0017] In an optional embodiment of the present application, the step of sending the message body to the server includes:

[0018] The message body is sent to the server via the EventBus bus.

[0019] In an optional embodiment of the present application, the step of performing text recognition on the current screen images of the master cloud real machine and the slave cloud real machine respectively to obtain text recognition results includes:

[0020] Performing text recognition on the current screen image of the master cloud real machine using the OCR text detection model to obtain all texts and the area and specific position of the text in the current screen image of the master cloud real machine as a first text recognition result;

[0021] The OCR text detection model is used to perform text recognition on the current screen image of the slave control cloud real machine to obtain all texts and the area and specific position where the text is located in the current screen image of the slave control cloud real machine as the second text recognition result.

[0022] In an optional embodiment of the present application, the OCR text detection model is a fine-tuned PaddleOCR model.

[0023] In an optional embodiment of the present application, determining the second control position information of each slave control cloud real machine according to the first control position information and the text recognition result includes:

[0024] Acquire a first target text area corresponding to the first manipulation position information according to the first text recognition result;

[0025] Acquire first target text content corresponding to the first target text area according to the first text recognition result;

[0026] The second control position information of each of the slave control cloud real machines is determined according to the first target text content and the second text recognition result.

[0027] In an optional embodiment of the present application, the step of determining the second control position information of each of the slave control cloud real machines according to the first target text content and the second text recognition result includes:

[0028] Acquire second target text content matching the first target text content according to the second text recognition result;

[0029] Acquire a second target text area corresponding to the second target text content according to the second text recognition result;

[0030] The specific position of the second target text area is acquired according to the second text recognition result as the second manipulation position information.

[0031] To achieve the above-mentioned purpose and other related purposes, the present application provides a multi-machine synchronization control device based on a cloud real machine, which is applied to the Web end of a cloud real machine testing platform, and the device includes:

[0032] The command interception module is used to intercept the user's control command on the master cloud real machine, where the control command includes the control event and the first control position information;

[0033] A picture acquisition module, used to acquire current screen pictures of the master cloud real machine and at least one slave cloud real machine;

[0034] A text recognition module is used to perform text recognition on the current screen images of the master cloud real machine and the slave cloud real machine respectively to obtain text recognition results;

[0035] A position acquisition module, used to determine the second control position information of each slave control cloud real machine according to the first control position information and the text recognition result;

[0036] a message encapsulation module, configured to generate a message body according to the manipulation event, the first manipulation position information and the second manipulation position information;

[0037] The message distribution module is used to send the message body to the server, so that the server can synchronously control the master cloud real machine and the slave cloud real machine according to the message body.

[0038] To achieve the above objectives and other related objectives, the present application provides an electronic device, the electronic device comprising:

[0039] one or more processors;

[0040] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the electronic device implements any of the above-mentioned cloud real machine-based multi-machine synchronous control methods.

[0041] To achieve the above-mentioned purpose and other related purposes, the present application provides a computer-readable storage medium, characterized in that a computer program is stored thereon, and when the computer program is executed by the processor of a computer, the computer executes the multi-machine synchronization control method based on cloud real machine as described in any one of the above claims.

[0042] As described above, the multi-machine synchronous control method, device, equipment and medium based on cloud real machine of the present application have the following beneficial effects:

[0043] This application intercepts the user's first control instruction on the master cloud real machine through a browser, and the first control instruction includes a first control event and a first control position information; obtains the current screen image of the master cloud real machine and at least one slave cloud real machine; performs text recognition on the current screen image of the master cloud real machine and the slave cloud real machine respectively to obtain text recognition results; determines the second control position information of each slave cloud real machine according to the first control position information and the text recognition results; generates a message body according to the first control event, the first control position information and the second control position information; sends the message body to the server, so that the server synchronously controls the master cloud real machine and the slave cloud real machine according to the message body. This application can realize cross-platform automated testing and other operations through event interception and message encapsulation, and accurately determine the control position of the slave cloud real machine through text recognition and matching, thereby realizing millisecond-level real-time synchronization and precise response of the master cloud real machine and the slave cloud real machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a flow chart of a cloud-based real machine synchronization control method provided in an embodiment of the present application;

[0045] Figure 2 Schematic diagram of text recognition on screen images based on the fine-tuned OCR text detection model.

[0046] Figure 3 Shown is a structural block diagram of a cloud-based real machine synchronization control device provided by an embodiment of the present application;

[0047] Figure 4 Shown is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0049] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0050] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0051] Cloud real machine refers to running a physical device remotely and operating the physical device through network access to improve operational efficiency. Cloud real machine is a remote physical device, a real machine running remotely, such as a mobile phone, tablet computer, etc.; Android cloud real machine is a real machine running remotely with the Android operating system, iOS cloud real machine is a real machine running remotely with the iOS operating system, and HarmonyOS cloud real machine is a real machine running remotely with the HarmonyOS operating system.

[0052] In order to solve the problems in the prior art that the one-machine multi-control mode of the automated testing tools cannot be used across platforms, the response is slow, and the synchronization cannot be accurate, the embodiments of the present application provide a multi-machine synchronization control method, device, equipment and medium based on cloud real machine, which can be applied to cloud real machine testing scenarios, such as smart terminal application software compatibility testing scenarios, smart terminal application software new demand testing scenarios, online problem reproduction, development self-testing, product and operation acceptance, etc. Through event interception and message encapsulation, cross-platform automated testing and other operations can be realized, and the control position information of the slave cloud real machine can be determined by text recognition and matching, so as to achieve millisecond-level real-time synchronization and accurate response of the master cloud real machine and the slave cloud real machine.

[0053] Figure 1 A multi-machine synchronous control method based on a cloud real machine in an exemplary embodiment of the present application is shown, including steps S100-S600, which can be executed by the Web end of the cloud real machine testing platform.

[0054] Among them, the cloud real machine testing platform includes a Web end, an Agent server end and an OCR server. The Web end is communicated with the Agent server end and the OCR server respectively. The Agent server end is used to connect multiple cloud real machines. The Web client can be a client that interacts with the Agent server end through a web page, such as a browser in a computer. The user's operation based on the Web end can be a conventional web page operation, such as a web page operation realized by input devices such as a mouse, keyboard, and touch screen. The Web end controls the connection with the Agent server end to multiple cloud real machines to realize automated testing by interacting with the Agent server end.

[0055] First, step S100 is executed to intercept a first control instruction of a user on a master cloud real machine through a browser, where the first control instruction includes a first control event and first control position information.

[0056] Since the web terminal can display the screen images of the master cloud machine and several slave cloud machines through the browser page, the user's control instructions on the master cloud machine can be intercepted through the browser. As an example, there is 1 master cloud machine and 4 slave cloud machines.

[0057] Taking the mouse as an input device on the Web side as an example, the Web side can intercept the user's first operation instruction on the main control cloud real machine through the browser. The first control instruction includes mouse actions such as mouse up, mouse down, mouse leave, mouse move, etc. Different combinations of these mouse actions will form different mouse control instructions, that is, the first control instructions. These mouse control instructions include mouse control events (that is, the first control events) and the specific coordinate position of the user when operating on the main control cloud real machine (that is, the first control position information). Among them, mouse control events include click (tap), slide (swipe), long press (longpress), back, HOME key and other events.

[0058] Next, execute step S200 to obtain the current screen images of the master cloud real machine and at least one slave cloud real machine.

[0059] When the Web client intercepts the user's first operation instruction on the master cloud real machine through the browser, the current screen images of the master cloud real machine and all the slave cloud real machines can be obtained by screenshot. As an example, if there is one master cloud real machine and four slave cloud real machines, a total of five current screen images can be obtained.

[0060] Next, step S300 is executed to perform text recognition on the current screen images of the master cloud real machine and the slave cloud real machine respectively to obtain text recognition results.

[0061] In this embodiment, since the slave cloud real machine and the master cloud real machine have differences in screen size, resolution, interface layout, etc., the same operation has different positions on the master cloud real machine and the slave cloud real machine. OCR text recognition technology can be used to perform text recognition on the current screen images of the master cloud real machine and the slave cloud real machine to obtain text recognition results as a basis for subsequently determining the control position information of the slave cloud real machine.

[0062] To this end, an OCR text detection model can be prepared in advance, and then deployed on a dedicated OCR server or cloud platform so that the OCR server or cloud platform can provide OCR services for the Web end to request the OCR service to perform text recognition on the current screen images of the master cloud real machine and the slave cloud real machine.

[0063] After obtaining the current screen pictures of the master cloud real machine and all the slave cloud real machines, the OCR text detection model can be used to perform text recognition on the current screen picture of the master cloud real machine to obtain all the texts and the area and specific location where the texts are located in the current screen picture of the master cloud real machine as the first text recognition result; at the same time, the OCR text detection model is used to perform text recognition on the current screen picture of the slave cloud real machine to obtain all the texts and the area and specific location where the texts are located in the current screen picture of the slave cloud real machine as the second text recognition result.

[0064] like Figure 2 As shown in the figure, when using the OCR text detection model to perform text recognition on the current screen image of the master cloud real machine, the current screen image of the master cloud real machine can be sent to the OCR service. The OCR service will first perform denoising, grayscale, binarization and other operations on the received current screen image of the master cloud real machine, and then mark the position of the image text ( Figure 2 The value of the rightmost parameter "pos") and outputs the boundary matrix ( Figure 2 The value of the parameter "elem_det_regeion" on the far right), and finally feature extraction, using the OCR text detection model to recognize text characters ( Figure 2 The value of the parameter "text" on the far right) is used as the first text recognition result. The text recognition of the current screen image of the slave cloud real machine using the OCR text detection model is basically the same as the text recognition logic of the current screen image of the master cloud real machine, so it will not be repeated.

[0065] It should be noted that in order to improve model performance and reduce development costs, the OCR text detection model can be obtained by fine-tuning the OCR text detection pre-trained model. Fine-tuning is performed by further training the training model on a small amount of annotated data of the new task, so that the model can learn the specific features and rules related to the test scene image text recognition task of this application, so as to better adapt to the test scene image text recognition task of this application. In this embodiment, the OCR text detection pre-trained model can be, for example, a PaddleOCR pre-trained model.

[0066] When fine-tuning the OCR text detection pre-trained model, test scene images in real test scenarios can be collected through real UI automation tests as image materials, and the image materials can be uploaded to the OSS service or saved locally, and a suitable number (for example, at least 500) of test scene images can be manually screened out; then the text areas of the screened test scene images are annotated, and finally the OCR text detection pre-trained model is fine-tuned using the annotated test scene graphics to obtain the OCR text detection model, wherein the test scene images can be, for example, screen images of real cloud machines in real test scenarios.

[0067] Next, step S400 is executed to determine the second control position information of each of the slave control cloud real machines according to the first control position information and the text recognition result.

[0068] In one embodiment of the present application, when determining the second control position information of each of the slave control cloud real machines based on the first control position information and the text recognition result, the first target text area corresponding to the first control position information can be first obtained based on the first text recognition result; then the first target text content corresponding to the first target text area can be obtained based on the first text recognition result; finally, the second control position information of each of the slave control cloud real machines can be determined based on the first target text content and the second text recognition result.

[0069] In one embodiment of the present application, when determining the second control position information of each of the slave control cloud real machines based on the first target text content and the second text recognition result, the second target text content matching the first target text content can be first obtained based on the second text recognition result; then, the second target text area corresponding to the second target text content is obtained based on the second text recognition result; finally, the specific position of the second target text area is obtained based on the second text recognition result as the second control position information.

[0070] Specifically, since the first text recognition result includes all texts and the area and specific location of the current screen image of the master cloud real machine, the second text recognition result includes all texts and the area and specific location of the current screen image of the slave cloud real machine. Therefore, the first control position information can be matched with the first text recognition result to determine the text area corresponding to the first control position information, that is, the first target text area, and then the text content corresponding to the first target text area can be obtained, that is, the first target text content; after obtaining the first target text content, the text content matching the first target text content can be found in the second text recognition result of the current screen image of each slave cloud real machine as the second target text content, and then the text area to which the second target text content belongs can be obtained, that is, the second target text area, and then the specific location of the second target text area can be obtained, that is, the second control position information. It should be noted that each slave cloud real machine can determine a second control position information.

[0071] Since the determination of the second control position information in this application does not rely on spatial elements and device resolution, but is based on OCR technology to identify the current screen images of the master control cloud real machine and the slave control cloud real machine to obtain accurate and unbiased actual position data, that is, the obtained second control position information is accurate and unbiased, so that the slave control cloud real machine can be accurately controlled based on the second control position information.

[0072] It should be noted that there may be multiple text areas with the same text content in the screen image of the cloud real machine, such as Figure 2 In the screen image, two text areas with the text content "Address Book" will be identified. This will result in multiple text contents being matched in the second text recognition result based on the first target text content, making it impossible to accurately locate the control position information in the slave cloud real machine. Although the different screen layouts of different cloud real machines will cause the position of the text area to change, the order of multiple text areas with the same text content usually does not change, for example Figure 2 Of the two "address books" text areas, the text area of ​​the "address book" corresponding to the loading place is always before the "address book" corresponding to the unloading place. For this reason, sequential numbers can be added to the text contents of multiple text areas with the same text content in the first text recognition result and the second text recognition result obtained by recognizing the screen image to distinguish them.

[0073] Next, step S500 is executed to generate a message body according to the first manipulation event, the first manipulation position information and the second manipulation position information.

[0074] In order to realize cross-platform application, it is necessary to convert the first control event, the first control position information and each second control position information into a message body in a unified format, so that the Agent server can parse the message body into control instructions required by cloud real machines on different platforms after receiving it. It should be noted that when encapsulating the message, the control event and related event data including the control position information will be encapsulated into the message body.

[0075] As an example, we will encapsulate the tap event:

[0076] type: 'debug',

[0077] detail: 'tap',

[0078] point: '`${pointA_X},${pointA_Y}`',

[0079] }

[0080] Encapsulate the swipe event: {

[0081] type: 'debug',

[0082] detail: swipe,

[0083] pointA: '`${pointA_X},${pointA_Y}`',

[0084] pointA: '`${pointB_X},${pointB_Y}`',

[0085] }

[0086] Similar to this message body, there are also long press, back, HOME key and other events.

[0087] Finally, step S600 is executed to send the message body to the server, so that the server synchronously controls the master cloud real machine and the slave cloud real machine according to the message body.

[0088] Specifically, all message bodies can be synchronously sent to the server through the EventBus bus. When the Agent server synchronously controls the master cloud real machine and the slave cloud real machine according to the message body, the Agent server first parses the message body to obtain the second control instruction for controlling the master cloud real machine and the third control instruction for controlling each of the slave cloud real machines. Then, the server sends the second control instruction to the master cloud real machine so that the master cloud real machine executes the second control instruction; at the same time, the server also sends the third control instruction to the corresponding slave cloud real machine so that the slave cloud real machine executes the third control instruction. Among them, the second control instruction includes the second control event, the first control position information and other event data, and the third control instruction includes the third control event, the second control position information and other event data.

[0089] Through event interception and message encapsulation, cross-platform automated testing and other operations can be achieved. The control position of the slave cloud real machine can be accurately determined through text recognition and matching, thereby achieving millisecond-level real-time synchronization and precise response between the master cloud real machine and the slave cloud real machine.

[0090] Based on the same concept, Figure 3 As shown, the present application also provides a multi-machine synchronization control device 11 based on a cloud real machine, and the multi-machine synchronization control device 11 based on a cloud real machine includes an instruction interception module 111, an image acquisition module 112, a text recognition module 113, a location acquisition module 114, a message encapsulation module 115 and a message distribution module 116.

[0091] The command interception module 111 is used to intercept the user's control command on the master cloud real machine, and the control command includes a control event and first control position information;

[0092] The picture acquisition module 112 is used to acquire the current screen pictures of the master cloud real machine and at least one slave cloud real machine;

[0093] The text recognition module 113 is used to perform text recognition on the current screen images of the master cloud real machine and the slave cloud real machine respectively to obtain text recognition results;

[0094] The position acquisition module 114 is used to determine the second control position information of each slave control cloud real machine according to the first control position information and the text recognition result;

[0095] The message encapsulation module 115 is used to generate a message body according to the manipulation event, the first manipulation position information and the second manipulation position information;

[0096] The message distribution module 116 is used to send the message body to the server, so that the server can synchronously control the master cloud real machine and the slave cloud real machine according to the message body.

[0097] It should be noted that the multi-machine synchronous control device 11 based on a cloud real machine provided in the above embodiment and the multi-machine synchronous control method based on a cloud real machine provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In actual applications, the multi-machine synchronous control device 11 based on a cloud real machine provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0098] Figure 4 A structural schematic diagram of an electronic device that implements the multi-machine synchronous control method based on cloud real machine in the present application is shown.

[0099] The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a multi-machine synchronization control program based on a cloud real machine.

[0100] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 12 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 1, such as codes for multi-machine synchronous control based on cloud real machines, but also can be used to temporarily store data that has been output or is to be output.

[0101] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1, and uses various interfaces and lines to connect the various components of the entire electronic device 1, and executes or executes programs or modules stored in the memory 12 (such as multi-machine synchronization control programs based on cloud real machines, etc.), and calls data stored in the memory 12 to execute various functions of the electronic device 1 and process data.

[0102] The processor 13 executes the operating system and various installed applications of the electronic device 1. The processor 13 executes the applications to implement the steps in the multi-machine synchronous control method based on cloud real machines.

[0103] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into an instruction interception module 111, a picture acquisition module 112, a text recognition module 113, a location acquisition module 114, a message encapsulation module 115, and a message distribution module 116.

[0104] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium, and the computer-readable storage medium can be non-volatile or volatile. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to perform part of the functions of the multi-machine synchronization control method based on a cloud real machine described in each embodiment of the present application.

[0105] In summary, the multi-machine synchronous control method, device, equipment and medium based on cloud real machine disclosed in this application can realize cross-platform automated testing and other operations through event interception and message encapsulation, and accurately determine the control position of the slave cloud real machine through text recognition and matching, thereby realizing real-time millisecond-level synchronous and precise response of the master cloud real machine and the slave cloud real machine. Therefore, this application effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0106] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A multi-machine synchronous control method based on cloud real machine, applied to the Web end of the cloud real machine testing platform, characterized in that: The method comprises: intercepting, by a browser, a first control instruction of a user on a master cloud real machine, wherein the first control instruction includes a first control event and first control position information; Obtaining current screen images of the master cloud real machine and at least one slave cloud real machine; Performing text recognition on the current screen images of the master cloud real machine and the slave cloud real machine respectively to obtain text recognition results; Determine the second control position information of each slave control cloud real machine according to the first control position information and the text recognition result; Generate a message body according to the first manipulation event, the first manipulation position information and the second manipulation position information; Send the message body to the server, so that the server synchronously controls the master cloud real machine and the slave cloud real machine according to the message body; The performing text recognition on the current screen images of the master cloud real machine and the slave cloud real machine respectively to obtain text recognition results includes: Performing text recognition on the current screen image of the master cloud real machine using the OCR text detection model to obtain all texts and the area and specific position of the text in the current screen image of the master cloud real machine as a first text recognition result; Performing text recognition on the current screen image of the slave control cloud real machine using the OCR text detection model to obtain all texts and the area and specific position of the text in the current screen image of the slave control cloud real machine as a second text recognition result; Determining the second manipulation position information of each slave control cloud real machine according to the first manipulation position information and the text recognition result includes: Acquire a first target text area corresponding to the first manipulation position information according to the first text recognition result; Acquire first target text content corresponding to the first target text area according to the first text recognition result; The second control position information of each of the slave control cloud real machines is determined according to the first target text content and the second text recognition result.

2. The multi-machine synchronous control method based on cloud real machine according to claim 1 is characterized in that: The server synchronously controls the master cloud real machine and the slave cloud real machine according to the message body, including: The server parses the message body to obtain a second control instruction for controlling the master cloud real machine and a third control instruction for controlling the slave cloud real machine, wherein the second control instruction includes a second control event and the first control position information, and the third control instruction includes a third control event and the second control position information; The server sends the second control instruction to the master cloud real machine, so that the master cloud real machine executes the second control instruction; The server sends the third control instruction to the corresponding slave cloud real machine, so that the slave cloud real machine executes the third control instruction.

3. The multi-machine synchronous control method based on cloud real machine according to claim 1 is characterized in that: The step of sending the message body to the server includes: The message body is sent to the server via the EventBus bus.

4. The multi-machine synchronous control method based on cloud real machine according to claim 1 is characterized in that: The OCR text detection model is a fine-tuned PaddleOCR model.

5. The multi-machine synchronous control method based on cloud real machine according to claim 1 is characterized in that: The step of determining the second control position information of each slave control cloud real machine according to the first target text content and the second text recognition result includes: Acquire second target text content matching the first target text content according to the second text recognition result; Acquire a second target text area corresponding to the second target text content according to the second text recognition result; The specific position of the second target text area is acquired according to the second text recognition result as the second manipulation position information.

6. A multi-machine synchronization control device based on cloud real machine, applied to the Web end of the cloud real machine testing platform, characterized in that: The device comprises: The command interception module is used to intercept the user's control command on the master cloud real machine, where the control command includes the control event and the first control position information; A picture acquisition module, used to acquire current screen pictures of the master cloud real machine and at least one slave cloud real machine; A text recognition module is used to perform text recognition on the current screen images of the master cloud real machine and the slave cloud real machine respectively to obtain text recognition results; A position acquisition module, used to determine the second control position information of each slave control cloud real machine according to the first control position information and the text recognition result; a message encapsulation module, configured to generate a message body according to the manipulation event, the first manipulation position information and the second manipulation position information; A message distribution module, used for sending the message body to the server, so that the server can synchronously control the master cloud real machine and the slave cloud real machine according to the message body; The performing text recognition on the current screen images of the master cloud real machine and the slave cloud real machine respectively to obtain text recognition results includes: Performing text recognition on the current screen image of the master cloud real machine using the OCR text detection model to obtain all texts and the area and specific position of the text in the current screen image of the master cloud real machine as a first text recognition result; Performing text recognition on the current screen image of the slave control cloud real machine using the OCR text detection model to obtain all texts and the area and specific position of the text in the current screen image of the slave control cloud real machine as a second text recognition result; Determining the second manipulation position information of each slave control cloud real machine according to the first manipulation position information and the text recognition result includes: Acquire a first target text area corresponding to the first manipulation position information according to the first text recognition result; Acquire first target text content corresponding to the first target text area according to the first text recognition result; The second control position information of each of the slave control cloud real machines is determined according to the first target text content and the second text recognition result.

7. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the multi-machine synchronous control method based on cloud real machines as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is enabled to execute the multi-machine synchronous control method based on a cloud real machine as described in any one of claims 1 to 5.

Images