Computer process control method, device and system, medium and equipment

Abstract

The invention provides a computer process control method, device and system, a medium and equipment, and relates to the technical field of computer process control. According to the method, firstly, a script sequence corresponding to a target operation process is obtained, the script sequence comprises a plurality of scripts, and the scripts comprise operation instructions corresponding to the scripts. And then, running a first script in the script sequence, and sending a corresponding operation instruction to the target computer to enable the target computer to execute a corresponding target operation. Then, a target image displayed after the target computer executes the target operation is obtained, an execution result corresponding to the target operation is obtained through recognition of the target detection model, and an expected result corresponding to the operation instruction is obtained; and under the condition that the execution result is consistent with the expected result, running the next script until the last script in the script sequence finishes running. Therefore, by combining the target detection algorithm with the man-machine interaction operation function, the monitoring and control of the computer automatic operation process are realized, and the safety of the computer automatic operation process is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer process control technology, and in particular to a computer process control method, device, system, medium and equipment. Background Art

[0002] In daily life and work in all walks of life, there are a lot of repetitive and mechanical computer operations, which consume a lot of human resources and reduce work and production efficiency. To solve these problems, a large number of solutions have emerged in the market, such as RPA (process robot), keystroke wizard and other software devices, which can imitate human operation of computers through software programs to automatically complete repetitive business processes.

[0003] Although these software can solve the above problems, they also have certain limitations. For example, RPA deployment costs are high, the deployment cycle is long, and the corresponding tools must be installed on the computer to be implemented; and the key wizard is not compatible with different operating systems and cannot continue to run after the computer is restarted. At the same time, it cannot detect the results of automatic operations, which poses a great risk during use. Summary of the invention

[0004] The present application provides a computer process control method, device, system, medium and equipment, which can monitor and control the computer automation operation process and ensure the security of the computer automation operation process.

[0005] A first aspect of an embodiment of the present application provides a computer process control method, which is applied to an edge computing device, and the edge computing device is connected to a target computer. The method includes:

[0006] Obtaining a script sequence corresponding to a target operation process; the script sequence includes a plurality of scripts, and the plurality of scripts include respective corresponding operation instructions;

[0007] Running the first script in the script sequence to send the operation instruction corresponding to the first script to the target computer, so that the target computer performs the target operation corresponding to the operation instruction;

[0008] Obtaining a target image displayed after the target computer executes the target operation, and inputting the target image into a pre-trained target detection model, and outputting an execution result corresponding to the target operation;

[0009] Get the expected results corresponding to the above operation instructions;

[0010] When it is determined that the execution result is consistent with the expected result, the next script of the first script is executed until the last script in the script sequence is executed.

[0011] Optionally, the above target detection model is trained by the following steps:

[0012] Build an improved YOLOV7 detection network based on the swin-Transformer structure;

[0013] Collecting a flow chart image displayed after the sample computer executes the test operation process, and adding a target label to the flow chart image to obtain a sample image;

[0014] Input the above sample image into the above improved YOLOV7 detection network for target detection to obtain the predicted label;

[0015] According to the above-mentioned predicted label and the above-mentioned target label of the above-mentioned sample image, the loss function of the above-mentioned improved YOLOV7 detection network is calculated, and the network parameters of the above-mentioned improved YOLOV7 detection network are updated according to the above-mentioned loss function until the above-mentioned improved YOLOV7 detection network meets the preset training cutoff condition, thereby obtaining the above-mentioned target detection model.

[0016] Optionally, the above step of constructing an improved YOLOV7 detection network based on the swin-Transformer structure includes:

[0017] Build an initial YOLOV7 detection network, which includes a backbone network, a reprocessing module, and a head network connected in sequence;

[0018] The above swin-Transformer structure is embedded in the backbone network and multi-branch stacking module of the head network of the above initial YOLOV7 detection network to obtain the above improved YOLOV7 detection network.

[0019] Optionally, the step of inputting the target image into a pre-trained target detection model and outputting an execution result corresponding to the target operation includes:

[0020] Input the target image into the target detection model, use the backbone network to perform multi-layer feature extraction on the target image, and output a feature map corresponding to each layer, so as to obtain a first feature map, a second feature map and a third feature map in sequence;

[0021] Using the reprocessing module to perform multiple pooling and fusion on the third feature map to obtain a fourth feature map;

[0022] Upsampling the fourth feature map by using the head network, and fusing the upsampled feature map with the second feature map to obtain a first fused feature map;

[0023] Upsampling the first fused feature map, and fusing the upsampled feature map with the first feature map to obtain a second fused feature map;

[0024] Downsampling the second fused feature map, and fusing the downsampled feature map with the first fused feature map to obtain a third fused feature map;

[0025] Downsampling the third fused feature map to obtain a fourth fused feature map;

[0026] Based on the second fused feature map, the third fused feature map and the fourth fused feature map, target recognition is performed respectively to obtain predicted targets at different scales, and the above execution results are determined based on the predicted targets at different scales.

[0027] Optionally, the step of obtaining the above target detection model includes:

[0028] After training the improved YOLOV7 detection network based on the sample images, a first target model is obtained;

[0029] The first target model is optimized and pruned to obtain the target detection model.

[0030] Optionally, the above method further includes:

[0031] When it is determined that the above execution result is inconsistent with the above expected result, the execution of the above script is stopped and an alarm message is generated.

[0032] Based on the same inventive concept, a second aspect of an embodiment of the present application provides a computer process control device, which is applied to an edge computing device, and the edge computing device is connected to a target computer, and the device includes:

[0033] A script acquisition module is used to acquire a script sequence corresponding to a target operation process; the script sequence includes a plurality of scripts, and the plurality of scripts include respective corresponding operation instructions;

[0034] An automatic control module, used for running the first script in the script sequence to send the operation instruction corresponding to the first script to the target computer, so that the target computer performs the target operation corresponding to the operation instruction;

[0035] A result detection module is used to obtain a target image displayed after the target computer executes the target operation, and input the target image into a pre-trained target detection model to output an execution result corresponding to the target operation;

[0036] An expected acquisition module is used to obtain the expected result corresponding to the operation instruction;

[0037] The automatic control supervision module is used to run the next script of the first script when it is determined that the execution result is consistent with the expected result, until the last script in the script sequence is completed.

[0038] Based on the same inventive concept, a third aspect of an embodiment of the present application provides a computer process control system, the system comprising an edge computing device and a target computer;

[0039] The edge computing device is used to obtain a script sequence corresponding to a target operation process, and run a first script in the script sequence to send an operation instruction corresponding to the first script to the target computer; wherein the script sequence includes a plurality of scripts, and the plurality of scripts include respective corresponding operation instructions;

[0040] The target computer is used to receive the operation instruction sent by the edge computing device, perform the target operation corresponding to the operation instruction, and display the target image after performing the target operation;

[0041] The above-mentioned edge computing device is also used to obtain the above-mentioned target image, and input the above-mentioned target image into a pre-trained target detection model, output the execution result corresponding to the above-mentioned target operation, and obtain the expected result corresponding to the operation instruction; when it is determined that the above-mentioned execution result is consistent with the above-mentioned expected result, run the next script of the above-mentioned first script until the last script in the above-mentioned script sequence is completed.

[0042] Based on the same inventive concept, the fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which is applied to an edge computing device. The storage medium stores a computer program. When the computer program is executed by a processor, the computer process control method proposed in the first aspect of the present application is implemented.

[0043] Based on the same inventive concept, a fifth aspect of an embodiment of the present application provides an edge computing device, the edge computing device comprising an execution unit, an image acquisition unit and a control unit;

[0044] The execution unit is used to obtain a script sequence corresponding to a target operation process, wherein the script sequence includes a plurality of scripts, and the plurality of scripts include respective corresponding operation instructions; and sequentially run the plurality of scripts according to the confirmation instruction of the control unit, so as to sequentially send the respective corresponding operation instructions of the plurality of scripts to a target computer, so that the target computer sequentially executes the target operation corresponding to the operation instructions;

[0045] The image acquisition unit is used to acquire the target image displayed after the target computer executes the target operation, and send the target image to the control unit;

[0046] The control unit uses a pre-trained target detection model to perform target recognition on the target image, outputs the execution result corresponding to the target operation, and obtains the expected result corresponding to the operation instruction; when it is determined that the execution result is consistent with the expected result, the confirmation instruction is sent to the execution unit, and the confirmation instruction is used to instruct the execution unit to run the next script in the script sequence.

[0047] Compared with the prior art, this application has the following advantages:

[0048] A computer process control method provided by an embodiment of the present application first obtains a script sequence corresponding to a target operation process, wherein the script sequence includes multiple scripts, and the multiple scripts include corresponding operation instructions. Then, the first script in the script sequence is run to send the operation instruction corresponding to the first script to the target computer, so that the target computer performs the target operation corresponding to the operation instruction. After that, the target image displayed after the target computer performs the target operation is obtained, and the target image is input into a pre-trained target detection model, and the execution result corresponding to the target operation is output. The expected result corresponding to the operation instruction is obtained, and when it is determined that the execution result is consistent with the expected result, the next script of the first script is run until the last script in the script sequence is completed. Thus, on the one hand, by directly running the script on the external edge computing device, the human-computer interaction operation process of the target computer is simulated, and the automated process operation of the target computer is realized. In this way, there is no need to install redundant drivers and software on the target computer, which ensures the independence of the computer. On the other hand, by combining the target detection algorithm with the human-computer interaction operation function, the monitoring and control of the computer automated operation process is realized, and the security of the computer automated operation process is ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of a computer process control method in one embodiment of the present application;

[0050] Figure 2 is a training flow chart of a target detection model in an embodiment of the present application;

[0051] Figure 3 It is a schematic diagram of the structure of the initial YOLOV7 detection network in an embodiment of the present application;

[0052] Figure 4 It is a schematic diagram of the structure of the improved YOLOV7 detection network in one embodiment of the present application;

[0053] Figure 5 It is a structural diagram of an edge computing device in one embodiment of the present application;

[0054] Figure 6It is a functional module diagram of a computer process control device in one embodiment of the present application;

[0055] Figure 7 It is a structural diagram of a computer process control system in one embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] In daily life and work in all walks of life, there are a lot of repetitive and mechanical computer operations, which consume a lot of human resources and reduce work and production efficiency. To solve these problems, a large number of solutions have emerged on the market, such as RPA (process robot), key wizard and other software devices. Among them, RPA imitates human operation of computers through software programs to automatically complete repetitive business processes, such as data processing, decision support, etc. Key wizard is usually used to simulate mouse and keyboard operations, focusing on performing repetitive, manual, computer tasks, such as data input, form filling, etc.

[0058] Although these software can solve the above problems, they also have certain limitations. For example, RPA deployment costs are high, the deployment cycle is long, and the corresponding tools must be installed on the computer to be implemented; and the key wizard is not compatible with different operating systems and cannot continue to run after the computer is restarted. At the same time, it cannot detect the results of automatic operations, which poses a great risk during use.

[0059] In view of this, the present application provides a computer process control method. On the one hand, by directly running the script on the external edge computing device, the human-computer interaction operation process of the target computer is simulated to realize the automated operation of the target computer process. In this way, there is no need to install redundant drivers and software on the target computer, which ensures the independence of the computer. On the other hand, by combining the target detection algorithm with the human-computer interaction operation function, the computer automation operation process can be monitored and controlled to ensure the security of the computer automation operation process.

[0060] Please refer to Figure 1 In a first aspect, an embodiment of the present application provides a computer process control method, which is applied to an edge computing device, and the edge computing device is connected to a target computer. Figure 1 As shown, the above method comprises the following steps:

[0061] S101: Obtain a script sequence corresponding to a target operation process; the script sequence includes a plurality of scripts, and the plurality of scripts include respective corresponding operation instructions.

[0062] A script is an executable file written in a certain format using a specific descriptive language. For example, scripts written in Python, javascript, Lua and other scripting languages ​​are not compiled into binary code like C\C++, but exist directly in the form of executable files. They do not need to be compiled and can be used directly. When used, the interpreter is responsible for interpreting them. When the script is executed, the computer will perform a series of operations. These operations may only involve image processing, or they may involve other applications such as word processing, spreadsheets, and data management programs.

[0063] It should be noted that this application does not directly run the script on the target computer that needs to be automatically controlled like the prior art, but runs the script on an edge computing device outside the target computer, and the edge computing device is connected to the target computer to be controlled via a USB interface. Compared with the method in the prior art that requires software to be deployed to the target computer, when the target computer is automatically controlled in this non-intrusive way, there is no need to install redundant drivers and software on the target computer, which ensures the independence of the computer. At the same time, there is no need to recompile the script for each computer operating system, which can shorten the script development cycle and save costs.

[0064] Specifically, in this embodiment, the target operation process refers to the computer operation process that needs to be implemented during design, such as opening or closing a file manager, creating a new blank document and entering corresponding text, making a spreadsheet, etc.

[0065] In this embodiment, firstly, an execution script is written according to the computer operation process to be implemented as needed, and a script sequence corresponding to the target operation process is obtained. Wherein, the script sequence includes multiple scripts, and the target operation processes executed between the multiple scripts can be associated or unassociated. For example, script 1 needs to execute the target operation process of creating a new blank document, and script 2 needs to execute the target operation process of entering the text "It's a great weather today" in the created new blank document. At this time, the target operation processes to be executed by script 1 and script 2 are associated. If script 1 needs to execute the target operation process of creating a new blank document, and script 2 needs to execute the target operation process of opening the chat software, then the target operation processes to be executed by script 1 and script 2 are unassociated.

[0066] Furthermore, each script includes corresponding operation instructions, and each script includes at least one operation instruction, so as to control the target computer to execute the corresponding target operation process through the corresponding operation instruction.

[0067] S102: Run the first script in the script sequence to send the operation instruction corresponding to the first script to the target computer, so that the target computer executes the target operation corresponding to the operation instruction.

[0068] In this embodiment, when the script sequence in the edge computing device is running, multiple scripts are executed in sequence, and if a script contains multiple operation instructions, then these multiple operation instructions are also executed in sequence.

[0069] In specific implementation, the operation instruction can be a command statement formulated according to the keyboard and mouse key value table of USB-HID (Universal Serial Bus-HumanInterface Device, human-machine interface device), so as to simulate the keyboard and mouse operation of the computer through the command statement, and realize the automatic control of the target computer. It should be noted that among various USB devices, as long as they meet the HID category specification, they are all HID devices, including USB mice, USB keyboards, and USB printers.

[0070] Exemplarily, the above keyboard and mouse key value table includes the key names and corresponding key values ​​of each function key on the keyboard, as well as the mouse key names and corresponding key values, that is, each key name corresponds to a key value, as shown in the following Table 1. For example, the key name corresponding to the left mouse button is VK_LBUTTON, and the key value is 0x01; the key name corresponding to the right mouse button is VK_RBUTTON, and the key value is 0x02; the key name corresponding to the ENTER key of the keyboard is VK_RETURN, and the key value is 0x0D; the key name corresponding to the SHIFT key of the keyboard is VK_SHIFT, and the key value is 0x10, and so on.

[0071] Table 1 Keyboard and mouse key value table

[0072] Key Name Key-value Corresponding button description VK_LBUTTON 0x01 Left mouse button VK_RBUTTON 0x02 Right mouse button VK_RETURN 0x0D ENTER key VK_SHIFT 0x10 SHIFT key VK_CONTROL 0x11 CTRL key …… …… ……

[0073] In this embodiment, when writing a script, an operation instruction is formed by encapsulating the corresponding key name into a standardized command statement. Then, the script is injected into the edge computing device, and after the edge computing device is connected to the target computer, the first script in the script sequence is started to run, and the operation instruction corresponding to the first script is sent to the target computer. After the target computer receives the operation instruction, the function key value corresponding to the key name in the operation instruction is called to execute the target operation corresponding to the operation instruction to realize the control of the target computer. After the target computer completes the target operation corresponding to the operation instruction, its display interface will also change accordingly.

[0074] S103: Acquire a target image displayed after the target computer executes the target operation, input the target image into a pre-trained target detection model, and output an execution result corresponding to the target operation.

[0075] In this embodiment, when acquiring the target image, a camera can be installed on the edge computing device, and the camera and the edge computing device can be connected through a USB interface, so that the camera can actively capture the operation interface of the target computer and obtain the target image displayed after the target computer executes the above target operation. In addition, the image return instruction can also be directly set in the script to control the target computer to return the currently displayed image to the edge computing device after executing the above target operation to obtain the target image.

[0076] After obtaining the target image displayed after the target computer performs the target operation, the target image is input into a pre-trained target detection model, and the target detection model is used to identify the key targets in the target image, and the execution result corresponding to the above target operation is output. Among them, the key targets include page titles, icons, etc. in the target image. The execution result is similar to the expected result in the above script, which is a keyword sequence or icon category obtained after identifying the key targets in the target image. Therefore, the target detection model is used to detect the result after the target computer performs the target operation by image detection.

[0077] It should be noted that the above-mentioned target detection model can adopt a one-stage (one-shot object detectors) type of target detection algorithm, which can directly predict the location and category of an object from a picture. It is characterized by one-step and high speed, thereby realizing rapid detection of the execution results of the target computer and facilitating the manipulation of the target computer process. For example, a series of convolutional neural networks such as YOLO and SSD can be used, and trained with sample images to obtain a target detection model, which is used to identify key targets in the target image to obtain the execution results corresponding to the target operation. The training process of the target detection model is detailed later and will not be described in detail here.

[0078] S104: Obtain the expected result corresponding to the above operation instruction.

[0079] In this embodiment, the expected result refers to the correct result that should be obtained after executing a certain operation instruction. The expected results corresponding to the operation instructions are pre-stored in the edge computing device, and each operation instruction corresponds to an expected result. For example, if the target operation process corresponding to the operation instruction 1 is to open a word document, the expected result obtained after executing the operation instruction 1 is an interface image of a word document, or a target keyword sequence containing keywords such as "start", "insert", "page", and "reference" in the interface image of the word document. This embodiment does not limit the specific form of the expected result. The expected result can be in the form of an image or in the form of data. As long as the expected result can distinguish different operation instructions and reflect the results corresponding to different operation instructions, it can be set accordingly according to actual needs during actual use.

[0080] S105: When it is determined that the execution result is consistent with the expected result, the next script of the first script is executed until the last script in the script sequence is executed.

[0081] In this embodiment, after the target detection model is used to identify the key targets in the target image and the execution result corresponding to the target operation is obtained, the execution result is compared with the expected result corresponding to the operation instruction in the first script. If the execution result is consistent with the expected result, it means that the target computer correctly executes the target operation corresponding to the operation instruction in the first script and successfully realizes the automated operation of the target computer; if the execution result is inconsistent with the expected result, it means that an error occurs in the process of the target computer executing the target operation, and effective control of the target computer operation process is not achieved by running the first script.

[0082] Furthermore, when it is determined that the execution result is consistent with the expected result, the next script of the first script, that is, the second script, is continued to run. The running process of the second script is similar to the running process of the first script, by sending the operation instruction corresponding to the second script to the target computer so that the target computer executes the target operation corresponding to the operation instruction, and performing target recognition on the target image displayed after the target computer executes the target operation through the target detection model to obtain the execution result. When the execution result is consistent with the expected result corresponding to the operation instruction of the second script, the third script is continued to run. In this way, the result after the target computer executes the target operation is detected by the target detection model, and only when the target operation corresponding to the previous operation instruction is successfully executed, the next script is continued to run until the last script in the script sequence is completed, thereby realizing the automated operation and monitoring of the computer process.

[0083] Optionally, the method further includes: when it is determined that the execution result is inconsistent with the expected result, stopping the execution of the script and generating an alarm message.

[0084] In this embodiment, when the execution result is inconsistent with the expected result, it means that the current operation control of the target computer is not successful, so the script is stopped and an alarm message is generated at the same time. For example, an alarm message is sent to the user's mobile terminal: Operation error, please try again. Thus, the alarm message reminds the user that an error has occurred in the current automatic control process of the target computer, and please check it in time.

[0085] On the one hand, the present application adopts a non-intrusive approach, by directly running the script on the external edge computing device, simulating the human-computer interaction operation process of the target computer, and realizing the automated process operation of the target computer. Compared with the method in the prior art where the control software needs to be deployed to the target computer, the present application does not need to install redundant drivers and software on the target computer, thereby ensuring the independence of the computer. At the same time, there is no need to recompile the script for each computer operating system, which can shorten the script development cycle and save costs. On the other hand, by combining the target detection algorithm with the human-computer interaction operation function, the target detection model is used to detect the result after each control of the target computer to perform the target operation. Moreover, only when the target operation corresponding to the previous operation instruction is successfully executed, the target computer will continue to be controlled to perform the next operation, thereby realizing the monitoring and control of the computer automation operation process and ensuring the security of the computer automation operation process.

[0086] Please refer to Figure 2 , Figure 2 : is a training flow chart of the target detection model in one embodiment of the present application. Figure 2 As shown in the figure, the above target detection model is trained through the following steps:

[0087] S201: Build an improved YOLOV7 detection network based on the swin-Transformer structure.

[0088] Specifically, this step mainly includes: building an initial YOLOV7 detection network, the initial YOLOV7 detection network includes a backbone network, a reprocessing module and a head network connected in sequence; embedding the swin-Transformer structure into the backbone network and the multi-branch stacking module of the head network of the initial YOLOV7 detection network to obtain the improved YOLOV7 detection network.

[0089] Among them, the initial YOLOV7 detection network is as follows Figure 3As shown in the figure, it mainly includes a backbone network (backbone), a reprocessing module (SPPCSPC) and a head network (Head) connected in sequence. First, the backbone network extracts the features of the input image, and returns images of three different feature layers to the head network. The image of the last feature layer is pooled and fused by the reprocessing module and then input into the head network. The head network is mainly a feature pyramid structure of FPN (up-sampling) + PAN (down-sampling). From bottom to top, the feature layer images of three different sizes, large, medium and small, are up-sampled and fused in sequence, and then down-sampled and layered, and three detection heads are output accordingly to perform target detection and recognition at different scales. Finally, the target detection results of the three branches are superimposed according to their corresponding weights to obtain the final target recognition result.

[0090] like Figure 4 As shown in the figure, when improving the initial YOLOV7 detection network, the swin-Transformer structure is first embedded into the multi-branch stacking module (Multi_Concat_Block module) at the top level of the YOLOv7 backbone network. This is because the backbone network performs global pixel operations on the extracted low-resolution feature maps, which can reduce the computational cost. In this way, the self-attention advantage of the swin-Transformer can be used to improve the feature representation ability of the backbone network. In addition, the swin-Transformer structure is integrated into the Multi_Concat_Block module in the feature pyramid structure of the head network. Combined with the advantage of the Multi_Concat_Block module in efficient feature extraction, the head network will further enhance the feature extraction capability and improve the detection capability for small targets. The specific added parts are as follows: Figure 4 As shown in "1", "2" and "3" in the figure.

[0091] The improved YOLOV7 detection network can fully utilize the advantages of the swin-Transformer structure, making the model have a larger receptive field, capturing the correlation and importance between objects of different scales and target areas, providing richer semantic information, and helping to improve the model's adaptability to objects of various sizes. The specific composition of the swin-Transformer structure can be found in the prior art and will not be repeated here.

[0092] S202: Collecting a flow chart image displayed after the sample computer executes the test operation process, and adding a target label to the flow chart image to obtain a sample image.

[0093] In this embodiment, by collecting the flowchart image displayed after the sample computer executes the test operation process, the key targets therein are marked to obtain a flowchart image with a target label, i.e., a sample image, for training the improved YOLOV7 detection network. Exemplarily, the key targets include page titles, icons, etc. in the flow image. For example, if the page title of a flow image contains "rice husk template", the target label of "WPS editing interface" is added to the flow image.

[0094] S203: Input the sample image into the improved YOLOV7 detection network to perform target detection and obtain a predicted label.

[0095] S204: According to the predicted label and the target label of the sample image, the loss function of the improved YOLOV7 detection network is calculated, and the network parameters of the improved YOLOV7 detection network are updated according to the loss function until the improved YOLOV7 detection network meets the preset training cutoff condition, thereby obtaining the target detection model.

[0096] In this embodiment, the sample images are sequentially input into the improved YOLOV7 detection network for target detection, and the predicted labels corresponding to the sample images are obtained for comparison with the annotated target labels to calculate the loss function, and the network model is reversely updated according to the loss function, so as to continuously update the network parameters of the YOLOV7 detection network in a cycle. When the preset training cutoff condition is met, the parameter update is stopped to obtain a trained target detection model.

[0097] Exemplarily, the loss function can use GIoU Loss, which has the advantage of being able to tolerate a certain difference between the predicted box and the true box, improving the accuracy of target detection, and reducing the root mean square error of the model, making the prediction results of the model more stable. The training cutoff condition can be set according to the number of training rounds or the accuracy you want to achieve. For example, by setting an upper limit on the number of rounds, when the number of training updates of the model reaches the upper limit of the number of rounds, the training is stopped. Alternatively, by setting a target accuracy, when the model reaches the target accuracy for target detection of sample images after multiple training updates, the training is stopped.

[0098] Optionally, the step of obtaining the above-mentioned target detection model also includes: training the above-mentioned improved YOLOV7 detection network based on the above-mentioned sample image to obtain a first target model; optimizing and pruning the above-mentioned first target model to obtain the above-mentioned target detection model.

[0099] In this embodiment, after the model is trained, it needs to be deployed to the edge computing device. However, considering that the performance of the edge computing device is not high, the improved YOLOV7 detection network is trained to obtain the first target model, and then the first target model is further optimized and pruned to remove redundant channels or feature layers in the network to obtain a lighter network for better deployment to the edge computing device. Exemplarily, the pruning method can adopt fine-grained pruning, vector pruning, convolution kernel pruning, filter pruning, channel pruning, layer pruning, etc., which are not specifically limited in this application.

[0100] This application uses the swin-Transformer structure to improve the YOLOV7 detection network, which helps to expand the receptive field of the model, extract more detailed information features, and improve the model's adaptability to targets of various sizes. At the same time, by streamlining the target detection model, it is convenient to deploy edge computing devices.

[0101] like Figure 4 As shown, the step of inputting the target image into a pre-trained target detection model and outputting an execution result corresponding to the target operation includes:

[0102] The target image is input into the target detection model, and the backbone network is used to perform multi-layer feature extraction on the target image, and each layer outputs a feature map, and the first feature map, the second feature map and the third feature map are obtained in sequence; the third feature map is pooled and fused multiple times by the reprocessing module to obtain a fourth feature map; the fourth feature map is upsampled by the head network, and the upsampled feature map is fused with the second feature map to obtain a first fused feature map; the first fused feature map is upsampled, and the upsampled feature map is fused with the first feature map to obtain a second fused feature map; the second fused feature map is downsampled, and the downsampled feature map is fused with the first fused feature map to obtain a third fused feature map; the third fused feature map is downsampled to obtain a fourth fused feature map; based on the second fused feature map, the third fused feature map and the fourth fused feature map, target recognition is performed respectively to obtain predicted targets at different scales, and the above execution results are determined based on the predicted targets at different scales.

[0103] In this embodiment, if Figure 3 As shown, the backbone network mainly includes CBS module (Conv 2D_BN_SiLU, convolution + batch normalization + activation function), MCB module (Multi_Concat-Block, multi-branch stacking module) and Transition_Block (transition block).

[0104] In this application, the target image displayed after the target computer performs the target operation is used as the input image of the target detection model to identify the key targets in the target image, and the execution result of the target computer is judged accordingly. Assume that the input image size is 640x640 and the number of channels is 3. After the input image is subjected to feature extraction by 4 CBS modules, the size of the input image becomes 160 x160x128, and the depth of the image is increased by the MCB module to obtain a feature map of size 160 x160x256. Then, feature extraction on different scales is further performed through the structure of 2 Transition_Block+MCB modules, and the first feature map of size 80 x80 and number of channels 512 and the second feature map of size 40 x40 and number of channels 1024 are obtained respectively, and sent to the head network. Next, the extracted low-resolution feature map is subjected to global pixel operations through the Transition_Block+swin-Transformer structure to improve computational efficiency. A third feature map with a size of 20 x20 and 1024 channels is obtained and sent to the reprocessing module.

[0105] The reprocessing module performs three Conv 2D_BN_SiLU operations on the last feature layer image output by the backbone network, and then performs three pooling operations and merges them separately. Then, it is fused with the feature map of the last feature layer image output by the backbone network after only one Conv2D_BN_SiLU operation to expand the receptive field of the model, and the fourth feature map is obtained and sent to the head network.

[0106] The head network first performs a convolution (Conv) on the fourth feature map output by the reprocessing module to adjust the number of channels, and then performs upsampling (UpSampling2D), and then combines it with the feature layer after a convolution of the second feature map, and extracts features through the MCB module to obtain the first fused feature map. Next, the first fused feature map is convolved (Conv) to adjust the number of channels, and then performs upsampling (UpSampling2D), and then combines it with the feature layer after a convolution of the first feature map, and extracts features through the swin-Transformer structure to obtain the second fused feature map, so as to capture the correlation and importance between different regions in the feature map, which helps to improve the model's adaptability to various sizes of targets and improve the accuracy of target detection. At this point, the upsampling operation is completed. After that, the second fused feature map is convoluted downsampled through the Transition_Block module, and combined with the first fused feature map, and the model's adaptability to small targets is adjusted again through the swin-Transformer structure to obtain the third fused feature map. The third fused feature map is then convoluted downsampled to obtain the fourth fused feature map. At this point, the downsampling operation is completed. Therefore, the features of different feature layers are fused through the feature pyramid structure of FPN (upsampling) + PAN (downsampling), which is conducive to extracting better features.

[0107] After obtaining the three enhanced feature maps of the second fused feature map, the third fused feature map, and the fourth fused feature map through the feature pyramid, the inference calculation is performed through the RepConv (model reparameterization) structure to identify the target in the feature map, thereby obtaining the predicted targets at different scales, and determining the execution result corresponding to the input target image based on the predicted targets at different scales. For example, when targets of the category "WPS editing interface" are detected in the feature maps at three scales, the execution result corresponding to the input target image is determined to be "WPS editing interface", which is used for subsequent comparison with the expected results to determine whether the target computer is successfully controlled to perform the corresponding target operation.

[0108] Please refer to Figure 5 In the second aspect, based on the same inventive concept, an embodiment of the present application provides an edge computing device, wherein the edge computing device 300 includes an execution unit 301, an image acquisition unit 302, and a control unit 303;

[0109] The execution unit 301 is used to obtain a script sequence corresponding to a target operation process, wherein the script sequence includes a plurality of scripts, and the plurality of scripts include respective corresponding operation instructions; and sequentially run the plurality of scripts according to the confirmation instruction of the control unit 303, so as to sequentially send the respective corresponding operation instructions of the plurality of scripts to the target computer, so that the target computer sequentially executes the target operation corresponding to the operation instructions;

[0110] The image acquisition unit 302 is used to acquire the target image displayed after the target computer executes the target operation, and send the target image to the control unit 303;

[0111] The control unit 303 uses a pre-trained target detection model to perform target recognition on the target image, outputs the execution result corresponding to the target operation, and obtains the expected result corresponding to the operation instruction; when it is determined that the execution result is consistent with the expected result, the confirmation instruction is sent to the execution unit 301, and the confirmation instruction is used to instruct the execution unit 301 to run the next script in the script sequence.

[0112] In this embodiment, a script file is written and run for different computer control processes, and then the script file is injected into the execution unit 301. The script file is a script sequence composed of multiple scripts, and each script includes corresponding operation instructions. The target computer is controlled to perform the target operation by running the script file in the execution unit 301, and the target image displayed after the target computer performs the target operation is acquired by the image acquisition unit 302. At the same time, the control unit 303 uses the target detection model to perform target recognition on the target image to detect the control result of the target computer, and compares the execution result obtained by the detection with the expected result to determine whether to continue running the script, thereby realizing the monitoring and control of the computer automation operation process.

[0113] In specific implementation, the execution unit 301 can be an STM32 single-chip computer system with USB-HID function, and one end of the execution unit 301 is connected to the target computer through a USB interface, and the other end communicates with the control unit 303 through a serial port or wirelessly. The image acquisition unit 302 can be a camera module, and sends the real-time acquired target image to the control unit 303 through USB, mipi (Mobile Industry Processor Interface, an open standard and a specification initiated by the MIPI Alliance for mobile application processors) or network protocol.

[0114] The above-mentioned control unit 303 can be an embedded device equipped with a deep learning model, for example, NVIDIA's Jetson AGX Xavier (NVIDIA's high-performance edge AI computing platform, suitable for various AI application scenarios requiring high-performance computing and low power consumption) can be used as an embedded device of the control unit 303. The target detection model can adopt the YOLOV7 detection network improved based on the swin-Transformer structure to improve the accuracy of small target detection. After the YOLOV7 detection network is trained with sample images, a first parameter model is obtained, and then when it is deployed, in order to meet the deployment requirements of embedded devices, the first parameter model needs to be converted into a more streamlined second parameter model. Specifically, the first parameter model is first converted into an onnx file, and then sent to Jetson AGX Xavier, and onnx is converted into an engine format file in Jetson AGX Xavier to obtain a second parameter model, that is, a target detection model for detecting and identifying target images. At the same time, tensorrt is used in Jetson AGXXavier to accelerate the model reasoning engineering, and deepstream is used to receive and process video images. It should be noted that the execution unit 301 and the control unit 303 may also be integrated together, that is, the STM32 single-chip microcomputer system is directly integrated into the above-mentioned embedded device.

[0115] During the working process, the STM32 microcontroller first runs the first script in the script file, uses the USB-HID function to simulate the keyboard and mouse, and controls the target computer to execute the target operation corresponding to the operation instruction in the first script. At the same time, the target image displayed on the display after the target computer executes the target operation is captured by the camera and sent to Jetson AGX Xavier. Jetson AGX Xavier performs target recognition on the target image through the target detection model, outputs the execution result corresponding to the above target operation, and sends a confirmation instruction to the STM32 microcontroller when it is determined that the execution result is consistent with the above expected result, instructing the STM32 microcontroller to continue running the next script of the first script. In this way, the STM32 microcontroller implements automatic operation on the target computer by executing the scripts in the script file in sequence. At the same time, Jetson AGX Xavier uses the target detection model to detect the results of each automatic operation of the target computer. Only when the previous script successfully controls the target computer to perform the target operation, a confirmation instruction is generated to instruct the STM32 microcontroller to continue running the next script. Otherwise, the script is stopped and an error is reported, thereby realizing the monitoring and control of the computer automation operation process and ensuring the safety of the computer automation operation process.

[0116] This application uses a combination of USB-HID and target detection algorithm to display images according to the identified target computer, simulate keyboard and mouse operations on the target computer, and realize process monitoring of the target computer. In addition, it has a non-intrusive feature, that is, there is no need to install redundant drivers and software on the target computer to be controlled, which ensures the independence of the computer. At the same time, the device and script are compatible with various operating systems such as Windows and Linux, and there is no need to re-compile scripts for each computer operating system, which can shorten the development cycle of the script and save costs. In addition, by adopting the YOLOV7 detection network improved based on the swin-Transformer structure, the accuracy of small target detection is improved, and the model is streamlined and optimized to facilitate deployment.

[0117] Please refer to Figure 6 Based on the same inventive concept, the third aspect of the embodiment of the present application provides a computer process control device, which is applied to an edge computing device 300, and the edge computing device 300 is connected to a target computer. The computer process control device 400 includes:

[0118] The script acquisition module 401 is used to acquire a script sequence corresponding to the target operation process; the script sequence includes multiple scripts, and the multiple scripts include respective corresponding operation instructions;

[0119] The automatic control module 402 is used to run the first script in the script sequence to send the operation instruction corresponding to the first script to the target computer, so that the target computer performs the target operation corresponding to the operation instruction;

[0120] The result detection module 403 is used to obtain the target image displayed after the target computer executes the target operation, and input the target image into a pre-trained target detection model to output the execution result corresponding to the target operation;

[0121] The expected acquisition module 404 is used to acquire the expected result corresponding to the operation instruction;

[0122] The automatic control supervision module 405 is used to run the next script of the first script when it is determined that the execution result is consistent with the expected result, until the last script in the script sequence is completed.

[0123] Optionally, the above device further includes:

[0124] Model building module, used to build an improved YOLOV7 detection network based on the swin-Transformer structure;

[0125] A sample acquisition module is used to collect the flow chart image displayed after the sample computer executes the test operation process, and add a target label to the above flow chart image to obtain a sample image;

[0126] The training module is used to input the above-mentioned sample image into the above-mentioned improved YOLOV7 detection network for target detection to obtain a predicted label; according to the above-mentioned predicted label of the above-mentioned sample image and the above-mentioned target label, calculate the loss function of the above-mentioned improved YOLOV7 detection network, and update the network parameters of the above-mentioned improved YOLOV7 detection network according to the above-mentioned loss function, until the above-mentioned improved YOLOV7 detection network meets the preset training cutoff condition, thereby obtaining the above-mentioned target detection model.

[0127] Optionally, the above model building module is specifically used for:

[0128] An initial YOLOV7 detection network is constructed, wherein the initial YOLOV7 detection network comprises a backbone network, a reprocessing module and a head network connected in sequence; the swin-Transformer structure is embedded in the backbone network and the multi-branch stacking module of the head network of the initial YOLOV7 detection network to obtain the improved YOLOV7 detection network.

[0129] Optionally, the result detection module 403 is specifically used for:

[0130] Input the target image into the target detection model, use the backbone network to perform multi-layer feature extraction on the target image, and output a feature map corresponding to each layer, so as to obtain a first feature map, a second feature map and a third feature map in sequence;

[0131] Using the reprocessing module to perform multiple pooling and fusion on the third feature map to obtain a fourth feature map;

[0132] Upsampling the fourth feature map by using the head network, and fusing the upsampled feature map with the second feature map to obtain a first fused feature map;

[0133] Upsampling the first fused feature map, and fusing the upsampled feature map with the first feature map to obtain a second fused feature map;

[0134] Downsampling the second fused feature map, and fusing the downsampled feature map with the first fused feature map to obtain a third fused feature map;

[0135] Downsampling the third fused feature map to obtain a fourth fused feature map;

[0136] Based on the second fused feature map, the third fused feature map and the fourth fused feature map, target recognition is performed respectively to obtain predicted targets at different scales, and the above execution results are determined based on the predicted targets at different scales.

[0137] Optionally, the above device further includes:

[0138] The model optimization module is used to train the improved YOLOV7 detection network based on the sample image to obtain a first target model; optimize and prune the first target model to obtain the target detection model.

[0139] Optionally, the above device further includes:

[0140] The alarm module is used to stop running the script and generate an alarm message when it is determined that the execution result is inconsistent with the expected result.

[0141] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0142] Please refer to Figure 7 , In a fourth aspect, based on the same inventive concept, an embodiment of the present application provides a computer process control system 500, wherein the computer process control system 500 includes an edge computing device 300 and a target computer 501;

[0143] The edge computing device 300 is used to obtain a script sequence corresponding to a target operation process, and run the first script in the script sequence to send an operation instruction corresponding to the first script to the target computer 501; wherein the script sequence includes a plurality of scripts, and the plurality of scripts include respective corresponding operation instructions;

[0144] The target computer 501 is used to receive the operation instruction sent by the edge computing device 300, execute the target operation corresponding to the operation instruction, and display the target image after executing the target operation;

[0145] The above-mentioned edge computing device 300 is also used to obtain the above-mentioned target image, and input the above-mentioned target image into a pre-trained target detection model, output the execution result corresponding to the above-mentioned target operation, and obtain the expected result corresponding to the operation instruction; when it is determined that the above-mentioned execution result is consistent with the above-mentioned expected result, run the next script of the above-mentioned first script until the last script in the above-mentioned script sequence is completed.

[0146] It should be noted that the specific implementation of the computer process control system 500 of the embodiment of the present application refers to the specific implementation of the computer process control method proposed in the first aspect of the embodiment of the present application, and will not be repeated here.

[0147] In the fifth aspect, based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which is applied to an edge computing device. The storage medium stores a computer program. When the computer program is executed by the processor, the computer process control method proposed in the first aspect of the present application is implemented.

[0148] It should be noted that the specific implementation of the readable storage medium of the embodiment of the present application refers to the specific implementation of the computer process control method proposed in the first aspect of the embodiment of the present application, and will not be repeated here.

[0149] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0150] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0152] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0154] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0155] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0156] The above is a detailed introduction to a computer process control method, device, system, medium and equipment provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A computer process control method, characterized in that: Applied to an edge computing device, and the edge computing device is connected to a target computer, the method includes: Obtaining a script sequence corresponding to a target operation process; the script sequence includes a plurality of scripts, and the plurality of scripts include respective corresponding operation instructions; Running the first script in the script sequence to send an operation instruction corresponding to the first script to the target computer, so that the target computer performs a target operation corresponding to the operation instruction; Acquire a target image displayed after the target computer executes the target operation, input the target image into a pre-trained target detection model, and output an execution result corresponding to the target operation; Obtaining the expected result corresponding to the operation instruction; When it is determined that the execution result is consistent with the expected result, the next script of the first script is executed until the last script in the script sequence is executed.

2. The method according to claim 1, characterized in that The target detection model is trained by the following steps: Build an improved YOLOV7 detection network based on the swin-Transformer structure; Collecting a flow chart image displayed after the sample computer executes the test operation process, and adding a target label to the flow chart image to obtain a sample image; Inputting the sample image into the improved YOLOV7 detection network to perform target detection and obtain a predicted label; According to the predicted label and the target label of the sample image, the loss function of the improved YOLOV7 detection network is calculated, and the network parameters of the improved YOLOV7 detection network are updated according to the loss function until the improved YOLOV7 detection network meets the preset training cutoff condition, thereby obtaining the target detection model.

3. The method according to claim 2, characterized in that The steps of constructing an improved YOLOV7 detection network based on the swin-Transformer structure include: Building an initial YOLOV7 detection network, wherein the initial YOLOV7 detection network includes a backbone network, a reprocessing module, and a head network connected in sequence; The swin-Transformer structure is embedded in the backbone network and the multi-branch stacking module of the head network of the initial YOLOV7 detection network to obtain the improved YOLOV7 detection network.

4. The method according to claim 3, characterized in that The step of inputting the target image into a pre-trained target detection model and outputting an execution result corresponding to the target operation comprises: Input the target image into the target detection model, use the backbone network to perform multi-layer feature extraction on the target image, and output a feature map corresponding to each layer, so as to obtain a first feature map, a second feature map and a third feature map in sequence; Using the reprocessing module to perform multiple pooling and fusion on the third feature map to obtain a fourth feature map; Upsampling the fourth feature map by using the head network, and fusing the upsampled feature map with the second feature map to obtain a first fused feature map; Upsampling the first fused feature map, and fusing the upsampled feature map with the first feature map to obtain a second fused feature map; Downsampling the second fused feature map, and fusing the downsampled feature map with the first fused feature map to obtain a third fused feature map; Downsampling the third fused feature map to obtain a fourth fused feature map; Based on the second fused feature map, the third fused feature map and the fourth fused feature map, target recognition is performed respectively to obtain predicted targets at different scales, and the execution result is determined based on the predicted targets at different scales.

5. The method according to claim 2, characterized in that: The step of obtaining the target detection model comprises: After training the improved YOLOV7 detection network based on the sample image, a first target model is obtained; The first target model is optimized and pruned to obtain the target detection model.

6. The method according to claim 1, characterized in that The method further comprises: When it is determined that the execution result is inconsistent with the expected result, the script is stopped and an alarm message is generated.

7. A computer process control device, characterized in that: Applied to an edge computing device, and the edge computing device is connected to a target computer, the device comprises: A script acquisition module, used to acquire a script sequence corresponding to a target operation process; the script sequence includes a plurality of scripts, and the plurality of scripts include respective corresponding operation instructions; An automatic control module, used for running the first script in the script sequence to send an operation instruction corresponding to the first script to the target computer, so that the target computer performs a target operation corresponding to the operation instruction; A result detection module is used to obtain a target image displayed after the target computer executes the target operation, and input the target image into a pre-trained target detection model to output an execution result corresponding to the target operation; An expected acquisition module, used to acquire the expected result corresponding to the operation instruction; The automatic control supervision module is used to run the next script of the first script when it is determined that the execution result is consistent with the expected result, until the last script in the script sequence is completed.

8. A computer process control system, characterized in that: including edge computing devices and target computers; The edge computing device is used to obtain a script sequence corresponding to a target operation process, and run a first script in the script sequence to send an operation instruction corresponding to the first script to the target computer; wherein the script sequence includes a plurality of scripts, and the plurality of scripts include respective corresponding operation instructions; The target computer is used to receive the operation instruction sent by the edge computing device, execute the target operation corresponding to the operation instruction, and display the target image after executing the target operation; The edge computing device is also used to obtain the target image, input the target image into a pre-trained target detection model, output an execution result corresponding to the target operation, and obtain an expected result corresponding to the operation instruction; when it is determined that the execution result is consistent with the expected result, run the next script of the first script until the last script in the script sequence is completed.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: Applied to an edge computing device, when the computer program is executed by a processor, the computer process control method as described in any one of claims 1 to 7 is implemented.

10. An edge computing device, characterized in that: The edge computing device includes an execution unit, an image acquisition unit and a control unit; The execution unit is used to obtain a script sequence corresponding to the target operation process, wherein the script sequence includes a plurality of scripts, and the plurality of scripts include respective corresponding operation instructions; and sequentially running the plurality of scripts according to the confirmation instruction of the control unit, so as to sequentially send the operation instructions corresponding to the plurality of scripts to the target computer, so that the target computer sequentially executes the target operations corresponding to the operation instructions; The image acquisition unit is used to acquire a target image displayed after the target computer executes the target operation, and send the target image to the control unit; The control unit uses a pre-trained target detection model to perform target recognition on the target image, outputs an execution result corresponding to the target operation, and obtains an expected result corresponding to the operation instruction; When it is determined that the execution result is consistent with the expected result, the confirmation instruction is sent to the execution unit, and the confirmation instruction is used to instruct the execution unit to run the next script in the script sequence.

Images