Thermal control unmanned test system and method based on large model game and RPA
By combining large-scale game theory with RPA, the thermal control testing system achieves automated execution and result feedback, solving the problem of excessive manual intervention in existing technologies and improving the intelligence level of the testing system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA ACADEMY OF SPACE TECHNOLOGY
- Filing Date
- 2025-12-11
- Publication Date
- 2026-05-05
AI Technical Summary
Existing automated thermal control testing systems have a low level of intelligence, requiring manual selection of test loops and editing of structured data. They cannot automatically complete the connection between processes, and testers need to manually start and terminate the process.
A thermal control unmanned testing system based on large-scale model game theory and RPA is adopted, including a process driving module, a process execution module, an RPA operation module, and an image recognition and positioning module. The system generates instructions by parsing test process prompts through a large-scale model and performs automated operations in combination with the image recognition and positioning module, thereby achieving adaptive driving and non-intrusive control of the test process.
It automates the execution of the testing process and generates reports, reduces manual intervention, improves the intelligence level of the testing system, and can automatically complete the testing process and provide test result feedback.
Smart Images
Figure CN121979706A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a thermal control unmanned testing system and method based on large model game theory and RPA, belonging to the field of thermal control testing technology. Background Technology
[0002] Currently, in spacecraft thermal control testing, a preliminary software testing method based on structured data and temperature inverse kinematics has been developed, and basic software has been used to achieve initial automation of testing. Through the remote control and telemetry interface with the current testing system, automatic command transmission and telemetry interpretation are achieved. However, the current operational automated thermal control testing system has a low level of intelligence and faces the following difficulties: 1) Manual screening of test circuits and editing of structured data as input are required; 2) The test process needs to be started and terminated manually. Although there is no need to manually send a large number of instructions, testers still need to pay attention to the test process and control the test process.
[0003] It can be seen that current thermal control automation software can only automate some processes and cannot meet the needs of connecting processes, requiring manual function combination and connection. Summary of the Invention
[0004] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a thermal control unmanned testing system and method based on large model game theory and RPA, which solves the problem that current thermal control automation software can only complete the automated execution of part of the process.
[0005] The technical solution of this invention is: Firstly, a thermal control unmanned testing system based on large-scale game theory and RPA, comprising: The process-driven module is used to parse test process prompts, combine them with the current test status to generate the next test instruction, and send it to the process execution module. The process execution module is semantically isolated from the process driving module. It receives the test instructions, calls the corresponding underlying test functions and generates execution instructions containing operation semantics, sends the execution instructions to the RPA operation module, restricts the test boundaries, and queries and merges test data in the database to form status feedback containing test results. The RPA operation module is connected to the process execution module and is used to perform simulated user operations on the front-end interface according to the execution instructions and the position coordinates and color information fed back by the image recognition and positioning module. An image recognition and positioning module, connected to the RPA operation module, is used to recognize the screen image captured by the RPA operation module, locate the position coordinates and color features of the target control, and feed back the position coordinates and color information to the RPA operation module. The scheduling module, connected to the process driving module and the process execution module, is used to store the semantic interaction records between the process driving module and the process execution module in natural language form, and to schedule the start, monitoring and termination of the test process according to the recognition results of the image recognition and positioning module or the execution status of the process execution module; the semantic interaction records include the status feedback.
[0006] Furthermore, the process-driven module and the process execution module are implemented by a driving agent and an execution agent respectively, which are built based on a large model; the driving agent is used to parse and understand the test process prompts and generate test instructions, and the execution agent is used to parse the test instructions and control the RPA operation; the two interact through natural language text.
[0007] Furthermore, the image recognition and localization module is implemented based on OpenCV and is used to perform target detection and feature extraction on buttons, dialog boxes, progress bars and feature icons.
[0008] Furthermore, the testing process includes functional testing, logic verification, or parameter tuning of the aerospace thermal control system loop.
[0009] Furthermore, the user operations include capturing screen images, clicking buttons, inputting text, reading data, performing adaptive operations based on the position coordinates fed back by the image recognition and positioning module, and monitoring the test progress based on the color features fed back by the image recognition and positioning module.
[0010] Furthermore, the RPA operation module runs in a non-intrusive manner at the operating system layer or application layer.
[0011] Secondly, a method for unmanned thermal control testing using the aforementioned large-scale game theory and RPA-based unmanned thermal control testing system includes: Analyze the test process prompts and generate the next test instruction based on the current test status; According to the test instructions, the corresponding underlying test functions are called and execution instructions containing operation semantics are generated to restrict the test boundaries, and test data in the database are queried and merged to form a status feedback containing test results; The screen image is captured according to the execution instruction, the position coordinates and color features of the target control are located, and the simulated user operation is performed on the front-end interface according to the position coordinates and color information of the target control. The system stores the semantic interaction records between the process-driven module and the process execution module in natural language, and schedules the start, monitoring, and termination of the test process based on the image recognition and positioning results or the execution status; the semantic interaction records include the status feedback.
[0012] Furthermore, the position coordinates and color features of the positioning target control are implemented based on OpenCV and are used for target detection and feature extraction of buttons, dialog boxes, progress bars and feature icons.
[0013] Furthermore, the testing process includes functional testing, logic verification, or parameter tuning of the aerospace thermal control system loop.
[0014] Furthermore, the user operations include capturing screen images, clicking buttons, inputting text, reading data, performing adaptive operations based on location coordinates, and monitoring test progress based on color features.
[0015] The advantages of this invention compared to the prior art are: (1) This invention achieves adaptive driving of test logic through large-scale interactive game, automatically completes the test process and generates test reports; (2) This invention realizes non-intrusive control of the system front-end controls through RPA technology and OpenCV image recognition technology, providing execution tools for semantic interaction. Attached Figure Description
[0016] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation
[0017] To better understand the above technical solutions, the technical solutions of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solutions of the present invention, rather than limitations on the technical solutions of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.
[0018] The following description, in conjunction with the accompanying drawings, provides a more detailed account of the unmanned thermal control testing system and method based on large-scale game theory and RPA provided in the embodiments of the present invention. Figure 1 Specific implementation methods may include: 1. Process self-driving based on large-scale game model The software system is based on a large model prompt word engineering approach. The large model is divided into roles, establishing a driving agent and an execution agent. The driving agent focuses on the thermal control test process, deeply understanding the description of the test process in the prompt words, and sending the next test instruction to the interactive front end according to the prompt words and the current state. The execution agent focuses on the execution of the thermal control test, establishing the connection between the backend underlying functions and test steps based on the prompt words, restricting relevant test boundaries, querying and merging relevant data in the database, and specifically executing each test step. The driving agent and execution agent are isolated from each other and interact semantically on the front end using language as a medium. Through driving and restricting, the thermal control test is continuously driven to execute automatically until all loop tests are completed.
[0019] 2. RPA-based front-end semantic interaction The core principle of RPA (Robotic Process Automation) is to simulate human actions on a computer, such as mouse clicks, keyboard input, and data reading and writing, and automatically execute various repetitive and highly routine tasks according to pre-defined rules. The software system, based on RPA, develops non-intrusive front-end operation software that completes button clicks and text input operations on relevant front-end pages according to instructions driven by a large model. It also monitors front-end image screenshots and initiates or terminates relevant test processes, effectively isolating the security risks associated with a single data source.
[0020] 3. Image-assisted localization based on OpenCV The core execution component of the software system, RPA technology, relies on structured rules during execution and has a relatively weak adaptability to the front-end UI. Therefore, the software system uses OpenCV to recognize the front-end images captured by RPA, detect the location and color of target buttons, dialog boxes, and feature graphics, and feeds back the output icon position coordinates and colors to the RPA component. Based on the position coordinates, the RPA component can achieve adaptive and precise operation, and based on the color feedback, it can accurately monitor specific data and realize the autonomous start and stop of the test process.
[0021] 4. Software System Components Based on the existing testing software, a large model interface module, an image recognition module, and a user interface operation module were added. The relevant underlying functions of the original system were updated and rewritten to enable large model calls. The automated thermal control test uses the model code as input. After the user inputs the model code, the background system initiates interaction with the large model, calling the Agent interface according to the test flow provided by the driver agent, further calling underlying functions, and starting the thermal control test.
[0022] The solution provided in the embodiments of the present invention specifically includes: 1. Process self-driving based on large-scale game model Based on the prompt word engineering, a driving agent and an execution agent are built separately. The driving agent is oriented towards the thermal control test process, deeply understands the description of the test process in the prompt words, and sends the next test instruction to the interactive front end according to the prompt words and the current state. The execution agent is oriented towards the execution of thermal control tests, establishes the connection between the backend underlying functions and test steps according to the prompt words, restricts the relevant test boundaries, queries and merges relevant data in the database, and specifically executes each test step.
[0023] The driver agent provides the next test instruction code based on the current test status and prompt words, and inputs text through the RPA module on the front end. The driver agent prompt words are as follows: 1) You are Xiao T, the spacecraft integrated test assistant. I need you to help me operate the test software to complete the automated thermal response test. The test software has dialog box buttons and dialog box areas. You only tell me the instruction code for one step at a time. 2) To start the test, first click the dialog box (command code: 'click_dialog'), second enter 'view model data' in the dialog box (command code: 'view_data'), third enter 'which loops haven't been tested' in the dialog box (command code: 'untested_data'), select the loops from the feedback from the test software, and fourth enter 'test thermal response' in the dialog box (command code: 'start_test') to start the software test. After the test is completed, you need to repeat steps three and four, but steps one and two do not need to be repeated.
[0024] 3) I will tell you the instruction code for one step at a time, and I will tell you when the test is over.
[0025] The agent executes database queries and data merging according to the test instructions, and outputs the required data structure according to the prompts. The agent interprets the test input, reports errors for invalid inputs, and sends feedback to the driver agent to re-enter the test input. The agent then calls underlying functions according to the test instructions, as follows:
[0026] The Agent prompt message is as follows: 1) Your name is Xiao T, and you are my testing assistant. I need you to help me complete the test. You need to ask me what kind of test I need to complete and wait for my reply. 2) If I need to view the model data, I will provide you with the model code (SCID), which will be used later. You provide me with the structure data, such as {'SCID':'YG999999','Tasktype':'GetdataFromDB'}; 3) If I need to complete the thermal response test, you need to confirm the model code (SCID) and loop number (HTnumber) with me. I will provide you with the model code and loop number. The model code usually consists of 8 characters, and the loop number usually starts with HT. You need to provide me with the model and loop number range information. If I say HTXXX loop to HTYYY loop, you provide me with the structure data such as {'SCID':'YG9999999','HTnumber':'HTXXX-HTYYYY','Tasktype':'TherThermalResponse'}. 4) Thermal response testing of all loops is not allowed. If I send a test request for all loops, I will receive the error message {'Tasktype':'Allerror'}. 5) If I haven't told you the model code, please look it up in the previous text; 6) If I need to import data, please provide me with the data structure, such as {'Tasktype':'WriteDB'}; 7) If I ask you which circuits haven't been tested yet, you should provide me with the model code (SCID) in the current conversation, along with the structure data such as {'SCID':'YG999999','Tasktype':'Asklast'}; 8) If I need to merge and summarize the number of loops, I will provide you with a list, similar to [HT001,HT002,HT003,HT019]. You will then merge the consecutive loop numbers and return the result to me as {'HTnumber':'HT001-HT003,HT019','Tasktype':'LastNumber'}. 2. RPA-based front-end semantic interaction The core principle of RPA (Robotic Process Automation) is to simulate human actions on a computer, such as mouse clicks, keyboard input, and data reading and writing, and automatically execute various repetitive and highly routine tasks according to pre-defined rules. The RPA module is the main execution module of the software system, primarily responsible for the interactive operations of the semantic model at the front end.
[0027] To enable semantic interaction-driven thermal response testing, a test assistant pop-up button, a chat window, a text input box, and a text send button were added to the existing software. The software system is based on the Pyautogui library. The `pyautogui.screenshot` function captures the screen; the `pyautogui.click` function handles button and input box clicks, with click locations derived from OpenCV's feature image location recognition; and the `pyautogui.hotkey` function handles text input, with the input text derived from the semantic model's interaction results.
[0028] For button click operations, after receiving the semantic interaction model instruction, the RPA module calls the pyautogui.screenshot function to complete the screen capture operation, outputs the image to the OpenCV module, obtains the button coordinates, and then calls the pyautogui.click function to complete the button click operation.
[0029] For text input operations, after receiving the semantic interaction model instruction, the RPA module calls the pyautogui.screenshot function to capture the screen, outputs the image to the OpenCV module, obtains the coordinates of the text input box and the text send button, calls the pyautogui.click function to click the input box, pastes the text result output by the semantic model into the text input box, and then calls the pyautogui.click function to complete the button click and send text operation.
[0030] 3. Image-assisted localization based on OpenCV The image-assisted localization module, developed based on the python.cv2 library, is divided into contour recognition of feature images and contour search of captured screens. First, it loads feature images stored in the database, processes them into grayscale using the cv2.cvtColor function, binarizes the grayscale image using cv2.threshold, and extracts the bounding box contours of the feature images using cv2.findContours, thus completing contour recognition. Next, it takes a screenshot taken by pyautogui.screenshot as input, processes the screenshot into grayscale using the cv2.cvtColor function, and uses the Canny operator cv2.Canny to identify all closed bounding boxes in the screenshot. Then, it iterates through all extracted bounding boxes and compares them sequentially with a template. The contour with the highest matching degree is obtained, and its center point coordinates are calculated using cv2.circle and output to other modules.
[0031] 4. Software System Components The newly added components of the software system consist of a semantic model interface module, an RPA operation module, and an image recognition module. The semantic interface module establishes a connection with the large model server via network and links with the Qwen3.0 model via chat. It sends prompts for the driving agent and the execution agent to the large model and displays the text returned by the large model on the software front end. The driving agent and the execution agent then independently begin interacting with the large model. The RPA operation module, based on the execution agent's calls, completes the clicking of buttons and related text input on the front end. During operation, it calls the image recognition module to perform screen capture and feature image recognition.
[0032] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
[0033] The contents not described in detail in this specification are common knowledge to those skilled in the art.
Claims
1. A thermal control unmanned testing system based on large-scale game theory and RPA, characterized in that, include: The process-driven module is used to parse test process prompts, combine them with the current test status to generate the next test instruction, and send it to the process execution module. The process execution module is semantically isolated from the process driving module. It receives the test instructions, calls the corresponding underlying test functions and generates execution instructions containing operation semantics, sends the execution instructions to the RPA operation module, restricts the test boundaries, and queries and merges test data in the database to form status feedback containing test results. The RPA operation module is connected to the process execution module and is used to perform simulated user operations on the front-end interface according to the execution instructions and the position coordinates and color information fed back by the image recognition and positioning module. An image recognition and positioning module, connected to the RPA operation module, is used to recognize the screen image captured by the RPA operation module, locate the position coordinates and color features of the target control, and feed back the position coordinates and color information to the RPA operation module. The scheduling module, connected to the process driving module and the process execution module, is used to store the semantic interaction records between the process driving module and the process execution module in natural language form, and to schedule the start, monitoring and termination of the test process according to the recognition results of the image recognition and positioning module or the execution status of the process execution module; the semantic interaction records include the status feedback.
2. The thermal control unmanned testing system based on large-scale game theory and RPA according to claim 1, characterized in that, The process-driven module and the process execution module are implemented by a driving agent and an execution agent respectively, which are built based on a large model. The driving agent is used to parse and understand the test process prompts and generate test instructions, while the execution agent is used to parse the test instructions and control the RPA operation. The two interact through natural language text.
3. The thermal control unmanned testing system based on large-scale game theory and RPA according to claim 1, characterized in that, The image recognition and localization module is implemented based on OpenCV and is used to perform target detection and feature extraction on buttons, dialog boxes, progress bars and feature icons.
4. The thermal control unmanned testing system based on large-scale game theory and RPA according to claim 1, characterized in that, The testing process includes functional testing, logic verification, or parameter tuning of the aerospace thermal control system circuit.
5. The thermal control unmanned testing system based on large-scale game theory and RPA according to claim 1, characterized in that, The user operations include capturing screen images, clicking buttons, inputting text, reading data, performing adaptive operations based on the position coordinates fed back by the image recognition and positioning module, and monitoring the test progress based on the color features fed back by the image recognition and positioning module.
6. The thermal control unmanned testing system based on large-scale game theory and RPA according to claim 1, characterized in that, The RPA operation module runs in a non-intrusive manner at the operating system layer or application layer.
7. A method for unmanned thermal control testing implemented using the large-scale game theory and RPA-based unmanned thermal control testing system according to any one of claims 1 to 6, characterized in that, include: Analyze the test process prompts and generate the next test instruction based on the current test status; According to the test instructions, the corresponding underlying test functions are called and execution instructions containing operation semantics are generated to restrict the test boundaries, and test data in the database are queried and merged to form a status feedback containing test results; The screen image is captured according to the execution instruction, the position coordinates and color features of the target control are located, and the simulated user operation is performed on the front-end interface according to the position coordinates and color information of the target control. The system stores the semantic interaction records between the process-driven module and the process execution module in natural language, and schedules the start, monitoring, and termination of the test process based on the image recognition and positioning results or the execution status; the semantic interaction records include the status feedback.
8. The unmanned thermal control testing method based on large-scale game theory and RPA according to claim 7, characterized in that, The position coordinates and color features of the positioning target control are implemented based on OpenCV and are used for target detection and feature extraction of buttons, dialog boxes, progress bars and feature icons.
9. The unmanned thermal control testing method based on large-scale game theory and RPA according to claim 7, characterized in that, The testing process includes functional testing, logic verification, or parameter tuning of the aerospace thermal control system circuit.
10. The method for unmanned thermal control testing based on large-scale game theory and RPA according to claim 7, characterized in that, The user operations include capturing screen images, clicking buttons, inputting text, reading data, performing adaptive operations based on location coordinates, and monitoring test progress based on color features.