Desktop system automatic test system and application method thereof

Abstract

The invention discloses a desktop system automatic test system and an application method thereof.The system comprises a test operation device and a device to be tested, and the test operation device comprises a video input port and a USB port; a video input port is connected with a display output port of the to-be-tested equipment so as to identify a control in a display output RGB signal of the desktop system output by the display output port and a click or input position and determine a simulation test operation required by an automatic test; and connecting the USB port with the USB port of the to-be-tested device so as to simulate the hid device to send the simulation test operation event to the to-be-tested device. The invention aims to solve the problem of interference of the test case in the system on the test in the desktop environment, improve the real degree of the actual test and improve the coverage degree of the test case.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and in particular to a desktop system automatic testing system and an application method thereof. Background Art

[0002] As Linux personal computer devices continue to account for an increasing share of the market, desktop environments developed based on Linux personal computer scenarios are also increasing. Improving automated testing methods based on desktop environments has become the basis for quality assurance. Under the current Linux technical conditions, general Linux desktop testing solutions are based on the local state for testing, lacking keyboard and mouse simulation capabilities, and the number of applicable software is basically focused on specific applications, lacking universal solutions. In short, traditional automated testing solutions have strong limitations and relatively poor user operation simulation. Summary of the invention

[0003] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a desktop system automatic testing system and an application method thereof are provided. The present invention aims to solve the problem of interference of test cases in the system on testing in a desktop environment, improve the authenticity of actual testing, and enhance the coverage of test cases.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: A desktop system automated testing system comprises a test operation device and a device to be tested, wherein the test operation device comprises a video input port and a USB port, and is connected to a display output port of the device to be tested via the video input port so as to identify controls and click or input positions in a display output RGB signal of the desktop system output by the display output port and determine simulated test operations required for automated testing, and is connected to a USB port of the device to be tested via a USB port so as to simulate a HID device to send simulated test operation events to the device to be tested, wherein the simulated test operation events include at least one of simulated mouse click operations and simulated keyboard key operations.

[0005] Optionally, the display output port of the device to be tested is an HDMI port, a VGA interface, a DP interface or a type-c port, and the video input port is an HDMI in port, a VGA interface, a DP interface or a type-c port.

[0006] Optionally, the test operation device is a computer terminal device with a built-in encoding module, and the built-in encoding module is used to convert a display output RGB signal output by a display output port of the device to be tested into an image.

[0007] Optionally, the test operation device is a computing device having an AI acceleration module, and the AI ​​acceleration module refers to a neural network processor NPU or a graphics processor GPU that supports AI acceleration.

[0008] A desktop system automated testing system comprises a testing operation device and a device to be tested, wherein the testing operation device comprises a video input port and a USB port, an encoder is further connected between the testing operation device and the display output port of the device to be tested, the encoder is used to convert a display output RGB signal output by the display output port of the device to be tested into an image, the video input port of the encoder is connected to the display output port of the device to be tested, the testing operation device has a USB OTG port, and the USB port of the encoder is connected to the USB OTG port on the testing operation device as an output end.

[0009] Optionally, an AI parsing device is also connected between the test operation device and the encoder. The AI ​​parsing device is a computing device with an AI acceleration module. The AI ​​acceleration module refers to a neural network processor NPU or a graphics processor GPU that supports AI acceleration. The AI ​​parsing device is used to identify controls and click or input positions in an image and send them to a computer terminal device. The computer terminal device and the AI ​​parsing device are connected via network communication. The AI ​​parsing device has a USB OTG port, and the USB port of the encoder is connected to the USB OTG port on the AI ​​parsing device as an output end.

[0010] In addition, the present invention also provides an application method of the desktop system automatic testing system, comprising the following steps: S1, connect the test operation device and the device to be tested; S2, starting the device under test to enable it to perform video output; S3, collect the display output RGB signal of the desktop system output by the display output port of the tested device through the video input port, and use the built-in encoding module to convert the display output RGB signal output by the display output port of the tested device into an image, perform video recording according to the converted image, and push the converted image stream to the AI ​​acceleration module; or collect the display output RGB signal of the desktop system output by the display output port of the tested device through the encoder, convert it into an image, send the converted image to the test operation device for video recording, and push the converted image stream to the AI ​​analysis device; S4, using an AI acceleration module or an AI analysis device to identify controls and click or input positions in the converted image; S5, determining the simulation test operation required for the automated test according to the controls and click or input positions in the image, and connecting to the USB port of the device to be tested via the USB port to simulate the HID device sending a simulation test operation event to the device to be tested.

[0011] Optionally, in step S5, when the USB port is connected to the USB port of the device to be tested to simulate the HID device to send a simulated test operation event to the device to be tested, sending a simulated mouse click operation includes: if a virtual machine is running on the device to be tested, sending the simulated mouse click operation is sending the global positioning data of the mouse, and the global positioning data of the mouse is the relative position of the mouse in the entire screen display resolution; if a physical machine is running on the device to be tested, sending the simulated mouse click operation is sending the single offset data of the mouse, and when sending the single offset data of the mouse, the speed of the mouse is inversely proportional to the distance between the mouse and the target position, so that the closer to the target position, the slower the speed of the mouse.

[0012] Optionally, when using the AI ​​acceleration module or AI analysis device to identify the controls and click or input positions in the converted image in step S4, it includes using the AI ​​acceleration module or AI analysis device to identify the controls and click or input positions in the converted image based on a pre-trained target recognition model, and the target recognition model is a deep learning neural network and is pre-trained to establish a mapping relationship between the graphical window interface image of the desktop system, the controls in the graphical window interface image, and the click or input positions.

[0013] Optionally, after step S5, the method further includes: obtaining a display output RGB signal after the tested device executes a simulated interaction event; identifying an execution result of the simulated interaction event based on the display output RGB signal after the simulated interaction event is executed; if the simulated interaction event is a click operation, then comparing the image of the display output RGB signal after the simulated interaction event is executed with a preset reference image to obtain the execution result of the simulated interaction event; if the image of the display output RGB signal after the simulated interaction event is executed is the same as the preset reference image, then determining that the execution of the simulated interaction event is successful; otherwise, determining that the execution of the simulated interaction event fails; if the simulated interaction event is a drag operation, then using the AI ​​acceleration module or the AI ​​analysis device to identify an execution result based on the execution of the simulated interaction event. The window position coordinates in the image obtained by converting the display output RGB signal after the event, if the obtained window position coordinates and the preset window position coordinates are the same, the simulation interaction event is determined to be executed successfully, otherwise the simulation interaction event is determined to be executed unsuccessfully; if the simulated interaction event is an input operation, the AI ​​acceleration module or the AI ​​analysis device is used to identify the rectangular blank area of ​​multiple input boxes or single input boxes in the image obtained based on the conversion of the display output RGB signal after executing the simulated interaction event, and the simulation interaction event is determined to be executed successfully, and a simulated interaction event of the mouse simulation program clicking the blank position is sent to the rectangular blank area; otherwise, the simulation interaction event is determined to be executed unsuccessfully; the simulated interaction events, videos and execution results of this test are logged and saved.

[0014] Compared with the prior art, the present invention mainly has the following advantages: 1. The system of the present invention comprises a test operation device and a device to be tested, wherein the test operation device comprises a video input port and a USB port, and is connected to the display output port of the device to be tested via the video input port so as to identify the controls and click or input positions in the display output RGB signal of the desktop system output by the display output port and determine the simulation test operation required for the automated test, and is connected to the USB port of the device to be tested via the USB port so as to simulate the HID device to send a simulation test operation event to the device to be tested. The present invention can realize full hardware simulation of human operation under non-human intervention conditions, without the need for test machine software adjustment, so as to realize a fully automatic and lightly intervened desktop automated testing solution, and can solve the problem of interference of test cases in the system with the test under the desktop environment, thereby improving the authenticity of the actual test and the coverage of the test cases.

[0015] 2. The present invention only relies on the common display output port in the desktop environment and the HID device protocol under the USB specification. Therefore, as long as the operating system supports the display output port and the HID device protocol under the USB specification, it can be supported. It is not only suitable for automated testing in the Linux desktop environment, but also can realize automated testing in the Windows desktop environment and other desktop environments. It has the advantage of good versatility and is particularly suitable for testing cross-platform programs in different desktop environments. In this way, automated testing can be performed directly across desktop environments without any modification. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The figure is a schematic diagram of the system topology structure of the first embodiment of the present invention.

[0017] Figure 2 The figure is a schematic diagram of the working principle of the test operation device in the first embodiment of the present invention.

[0018] Figure 3 It is a schematic diagram of the structure of the test operation device in the first embodiment of the present invention.

[0019] Figure 4 This is a schematic diagram of the system topology structure of the second embodiment of the present invention.

[0020] Figure 5 The figure is a schematic diagram of the working principle of the test operation device in the second embodiment of the present invention.

[0021] Figure 6 It is a schematic diagram of the structure of the test operation device in the second embodiment of the present invention.

[0022] Figure 7 This is a schematic diagram of the system topology structure of Embodiment 3 of the present invention.

[0023] Figure 8 The figure is a schematic diagram of the working principle of the test operation device in the third embodiment of the present invention.

[0024] Fig. 9 This is a schematic diagram of the structure of the test operation device in Embodiment 3 of the present invention.

[0025] Fig.10 Schematic diagram of the process of the method in the third embodiment of the present invention. DETAILED DESCRIPTION

[0026] Embodiment 1: like Figure 1As shown, this embodiment provides a desktop system automated testing system, including a test operation device and a device to be tested (specifically a personal computer PC in this embodiment, and can also be other hosts or servers installed with a desktop system), the test operation device includes a video input port and a USB port, and is connected to the display output port of the device to be tested through the video input port to identify the control and click or input position in the display output RGB signal of the desktop system output by the display output port and determine the simulated test operation required for the automated test, and is connected to the USB port of the device to be tested through the USB port to simulate the HID device to send a simulated test operation event to the device to be tested, and the simulated test operation event includes at least one of a simulated mouse click operation and a simulated keyboard key operation. It should be noted that the display output RGB signal (abbreviated as RGB signal) is an 8-bit display output signal in the RGB (Red, Green, Blue) color mode, and the RGB (Red, Green, Blue) color mode is a hardware-oriented color model, and the display system uses the RGB color model for image display. HID device (USB HID) is part of the USB specification for computer peripherals: it specifies the device class (hardware) for human-computer interface devices (such as keyboards, mice, touch screens, game controllers, and alphanumeric display devices). The test operation device is connected to the USB port of the device to be tested through the USB port to simulate human-computer interface devices such as keyboard and mouse devices.

[0027] In this embodiment, the display output port of the device to be tested is an HDMI port, the HDMI port of the device to be tested is connected to the test operation device through an HDMI cable, and the video input port is an HDMI in port. Figure 1 In addition, the display output port of the device to be tested may also adopt a VGA interface, a DP interface or a type-c port as required, and the video input port may also be a VGA interface, a DP interface or a type-c port as required.

[0028] It should be noted that identifying the control and click or input position in the display output RGB signal of the desktop system output by the display output port is a conventional implementation method of the artificial intelligence AI model. This embodiment only involves the application of this method, so its implementation details are not described here. The artificial intelligence AI model can be executed on the CPU or by using the AI ​​acceleration module to accelerate the execution to improve the execution efficiency. For example, as an optional implementation method, Figure 3As shown, in this embodiment, the test operation device is a computing device with an AI acceleration module. The computing device specifically uses an arm architecture Soc motherboard, and the AI ​​acceleration module refers to a neural network processor NPU (it can also be a graphics processor GPU that supports AI acceleration). If there is no AI acceleration module, the computing device needs to have computing power support. Figure 3 As shown, the video input port of the test operation device in this embodiment is an HDMI in port. In addition, the test operation device also has a USB OTG port.

[0029] As an optional implementation, Figure 2 and Figure 3 As shown, the test operation device in this embodiment is a computer terminal device with a built-in encoding module, and the built-in encoding module is used to convert the display output RGB signal output by the display output port of the device to be tested into an image. In this embodiment, the built-in encoding module is used to convert the display output RGB signal into a commonly used encoded video such as H264 / H265 and perform image recognition, and the real-time state of the image is retained through the video recording program, and the coordinates of the confirmation input box and the like are converted into HID data (simulating test operation events) through the conversion program. The working process of the desktop system automation test system in this embodiment is as follows: turn on the device to be tested, and output the RGB signal through the HDMI cable; complete the conversion of the RGB signal to the compressed video stream format through the built-in encoding module; use the video recording module to record the display situation; the video analysis conversion module (i.e., the AI ​​computing power module) performs video image analysis to obtain the cursor's intended click position; convert event expectations and position movements into HID events, and complete the simulation through the USB interface; the performance, display recording, and log of the device to be tested are handed over to the analysis device for overall analysis based on the same time axis.

[0030] Embodiment 2: This embodiment is a further extension of the first embodiment. Considering that the mainboard of the ARM architecture Soc in the first embodiment may not have an HDMI in port, this embodiment implements an external encoding method for it. Figure 4 As shown, an encoder is also connected between the test operation device and the display output port of the device to be tested in this embodiment. The encoder is used to convert the display output RGB signal output by the display output port of the device to be tested into an image. The video input port of the encoder is connected to the display output port of the device to be tested. The test operation device has a USB OTG port, and the USB port of the encoder is connected to the USB OTG port on the test operation device as an output terminal. Figure 6 As shown, in this embodiment, the test operation device also has a USB Host port.

[0031] like Figure 5As shown, the operating system of the device to be tested in this embodiment is the Kylin desktop operating system, and the key hardware of the device to be tested is the Feiteng D3000+ Moore Thread GPU; the encoder is a HiSilicon USB capture card, which supports 1080p / 60hz video input and outputs mjpeg format; the test operation device uses the Rk3588 development board, model is the orangepi 5 16G motherboard, and the system is suitable for the buildroot system. The working process of the desktop system automation test system is as follows: connect the device to be tested to the encoder through an HDMI cable; insert the encoder into the USB interface #1 of the test operation device; connect the USB interfaces #2 and #3 of the test operation device to the device to be tested through a USB cable, where USB interface #2 is used to simulate a keyboard device and USB interface #3 is used to simulate a mouse device; the test operation device completes the recording of the UVC input in the MJPEG format through the recording program; while recording, use the NPU capability of Rk3588 to perform real-time analysis of the UVC input video, such as the browser input box, or icons, etc.; after analysis, the AI ​​recognition program completes the output of the position, the HID conversion program locates the position, simulates the mouse event to move the cursor, and completes the text input or cursor click event through preset keywords or preset events; the complete recording situation and the performance data and logs during the test are handed over to high-performance equipment for tearing video analysis, event response analysis, and performance problem troubleshooting.

[0032] Embodiment three: This embodiment is a further extension of the second embodiment. Considering that the mainboard of the ARM architecture SoC in the second embodiment may not have an AI acceleration module, this embodiment provides a solution of an external AI acceleration module. Figure 7 As shown, an AI parsing device is also connected between the test operation device and the encoder. The AI ​​parsing device is a computing device with an AI acceleration module. The AI ​​acceleration module refers to a neural network processor NPU or a graphics processor GPU that supports AI acceleration. The AI ​​parsing device is used to identify controls and click or input positions in an image and send them to a computer terminal device. The computer terminal device and the AI ​​parsing device are connected via network communication. The AI ​​parsing device has a USB OTG port, and the USB port of the encoder is connected to the USB OTG port on the AI ​​parsing device as an output end.

[0033] like Figure 8As shown, the working process of the desktop system automation test system in this embodiment is as follows: turn on the device to be tested and output the RGB signal through the HDMI cable; complete the conversion of the RGB signal to the compressed video stream format through the encoder and send it to the AI ​​analysis device; use the AI ​​analysis device to analyze the video image, obtain the cursor's intended click position and send it together with the video to the test operation device; use the video recording module in the test operation device to record the display situation, and uniformly convert the event expectation and position movement into HID events, and complete the simulation through the USB interface; the performance, display recording, and log of the device to be tested are submitted to the analysis device for overall analysis based on the same timeline. Fig. 9 As shown, in this embodiment, the test operation device has a network interface for connecting to the AI ​​analysis device through network communication. The network interface can be a wired or wireless network interface, etc.

[0034] like Fig.10 As shown, this embodiment also provides an application method of the aforementioned desktop system automated testing system, comprising the following steps: S1, connect the test operation device and the device to be tested; S2, starting the device under test to enable it to perform video output; S3, collect the display output RGB signal of the desktop system output by the display output port of the tested device through the video input port, and use the built-in encoding module to convert the display output RGB signal output by the display output port of the tested device into an image, perform video recording according to the converted image, and push the converted image stream to the AI ​​acceleration module; or collect the display output RGB signal of the desktop system output by the display output port of the tested device through the encoder, and convert it into an image, send the converted image to the test operation device for video recording, and push the converted image stream to the AI ​​analysis device; S4, using an AI acceleration module or an AI analysis device to identify controls and click or input positions in the converted image; S5, determining the simulation test operation required for the automated test according to the controls and click or input positions in the image, and connecting to the USB port of the device to be tested via the USB port to simulate the HID device sending a simulation test operation event to the device to be tested.

[0035] In step S4 of this embodiment, when the AI ​​acceleration module or AI analysis device is used to identify the controls and click or input positions in the converted image, it includes using the AI ​​acceleration module or AI analysis device to identify the controls and click or input positions in the converted image based on a pre-trained target recognition model, and the target recognition model is a deep learning neural network and is pre-trained to establish a mapping relationship between the graphical window interface image of the desktop system, the controls in the graphical window interface image, and the click or input positions.

[0036] In this embodiment, according to the data analysis of the actual PC operation during mouse simulation, the mouse operation time can be roughly divided into two types. One can be understood as under the condition of a virtual machine, the mouse operation event is to send global positioning data, that is, the relative position of the entire screen display resolution. The other is under the condition of a physical machine, the mouse operation event is to send a single offset data. To ensure the simulation of human operation, this embodiment is implemented as a different speed embodiment, and the relative speed is weakened when approaching the target position. Specifically, in step S5 of this embodiment, when the USB port is connected to the USB port of the device to be tested to simulate the HID device to send a simulated test operation event to the device to be tested, sending a simulated mouse click operation includes: if the device to be tested is running a virtual machine, sending a simulated mouse click operation is to send the global positioning data of the mouse, and the global positioning data of the mouse is the relative position of the mouse in the entire screen display resolution; if the device to be tested is running a physical machine, sending a simulated mouse click operation is to send the single offset data of the mouse, and when sending the single offset data of the mouse, the speed of the mouse is inversely proportional to the distance between the mouse and the target position, so that the closer to the target position, the slower the speed of the mouse.

[0037] The behavior judgment of the operation logic is mainly preset according to the usage habits in this embodiment. The main window operations include clicking, dragging, and typing. The icon click operation is completed based on the recognition training of the preset icon. In this embodiment, the yolo model is mainly used for static recognition; the input box recognition is divided into multi-input box application and single input box reference. For example, word is a single input box, then the blank position is located through the visual algorithm, and the cursor can click the blank position through the mouse simulation program; multi-input box typing, such as browsers, is confirmed according to the pre-trained input box style and the input box changes generated by the keyboard tab; dragging selects the window operation button and the position positioning of different widths. After step S5 of this embodiment, the following further includes: obtaining the display output RGB signal after the tested device executes the simulated interaction event; identifying the execution result of the simulated interaction event based on the display output RGB signal after the simulated interaction event is executed; if the simulated interaction event is a click operation, then comparing the image of the display output RGB signal after the simulated interaction event is executed with a preset reference image to obtain the execution result of the simulated interaction event; if the image of the display output RGB signal after the simulated interaction event is executed is the same as the preset reference image, then determining that the execution of the simulated interaction event is successful; otherwise, determining that the execution of the simulated interaction event fails; if the simulated interaction event is a drag operation, then using the AI ​​acceleration module or the AI ​​parsing device (using the yolo target recognition model) to identify the execution result of the simulated interaction event based on the execution of the simulated interaction. The window position coordinates in the image obtained by converting the display output RGB signal after the event, if the obtained window position coordinates and the preset window position coordinates are the same, the simulated interaction event is judged to be executed successfully, otherwise the simulated interaction event is judged to fail to execute; if the simulated interaction event is an input operation, the AI ​​acceleration module or AI parsing device (using the yolo target recognition model) is used to identify the rectangular blank area of ​​multiple input boxes or single input boxes in the image obtained by converting the display output RGB signal after executing the simulated interaction event, and the simulated interaction event is judged to be executed successfully, and the simulated interaction event of the mouse simulation program clicking the blank position is sent to the rectangular blank area; otherwise, the simulated interaction event is judged to fail to execute; the simulated interaction event, video and execution results of this test are generated and saved. Among them, one way to identify the execution result of the simulated interaction event is to compare the graphic window interface image of the display output RGB signal and the preset graphic window interface image. If the two are consistent, the test is judged to pass, otherwise the test is judged to fail. In addition, the trained AI model can be further used to directly obtain the judgment result of whether the test is passed based on the graphic window interface image of the display output RGB signal. This method requires pre-training with samples. Since it is a conventional method of the AI ​​model, its implementation details are not described in detail here.

[0038] In summary, the desktop system automation test system of this embodiment is implemented by introducing an external hardware device (test operation device), processing the video output content through the external hardware, and realizing the simulation of the HID device, so as to realize the automation test scheme for the Linux desktop. The desktop system automation test system of this embodiment includes a test operation device and a device to be tested, wherein the test operation device includes a video input port and a USB port, and is connected to the display output port of the device to be tested through the video input port to identify the control and click or input position in the display output RGB signal of the desktop system output by the display output port and determine the simulation test operation required for the automation test, and is connected to the USB port of the device to be tested through the USB port to simulate the HID device to send a simulation test operation event to the device to be tested. The present invention can realize the full hardware simulation of human operation under non-human intervention conditions, without the need for test machine software adjustment, so as to realize the fully automatic and lightly intervened Linux desktop automation test scheme, which can solve the problem of interference of test cases in the system under the desktop environment with the test, improve the authenticity of the actual test, and improve the coverage of the test cases. This embodiment introduces external analysis and HID information simulation hardware, performs AI processing on video output content, simulates HID devices after analysis, and finally summarizes performance, display recording, and logs to implement an automated test solution for Linux desktops. This embodiment can achieve full hardware simulation of human operations under non-human intervention conditions, without the need for test machine software adjustment, to achieve a fully automatic, light-intervention Linux desktop automated test solution.

[0039] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A desktop system automated testing system, characterized in that: The invention comprises a test operation device and a device to be tested, wherein the test operation device comprises a video input port and a USB port, and is connected to a display output port of the device to be tested via the video input port so as to identify controls and click or input positions in a display output RGB signal of a desktop system output by the display output port and determine simulation test operations required for automated testing, and is connected to a USB port of the device to be tested via a USB port so as to simulate a HID device to send simulation test operation events to the device to be tested, wherein the simulation test operation events include at least one of simulation of mouse click operation and simulation of keyboard key operation.

2. The desktop system automated testing system according to claim 1, characterized in that: The display output port of the device to be tested is an HDMI port, a VGA interface, a DP interface or a type-c port, and the video input port is an HDMI in port, a VGA interface, a DP interface or a type-c port.

3. The desktop system automated testing system according to claim 2, characterized in that: The test operation device is a computer terminal device with a built-in encoding module, and the built-in encoding module is used to convert the display output RGB signal output by the display output port of the device to be tested into an image.

4. The desktop system automated testing system according to claim 3, characterized in that: The test operation device is a computing device with an AI acceleration module, and the AI ​​acceleration module refers to a neural network processor NPU or a graphics processor GPU that supports AI acceleration.

5. A desktop system automated testing system, characterized in that: The invention comprises a test operation device and a device to be tested, wherein the test operation device comprises a video input port and a USB port, an encoder is further connected between the test operation device and the display output port of the device to be tested, and the encoder is used to convert a display output RGB signal output by the display output port of the device to be tested into an image, the video input port of the encoder is connected to the display output port of the device to be tested, the test operation device has a USB OTG port, and the USB port of the encoder is connected to the USB OTG port on the test operation device as an output end.

6. The desktop system automated testing system according to claim 5, characterized in that: An AI parsing device is also connected between the test operation device and the encoder. The AI ​​parsing device is a computing device with an AI acceleration module. The AI ​​acceleration module refers to a neural network processor NPU or a graphics processor GPU that supports AI acceleration. The AI ​​parsing device is used to identify controls and click or input positions in an image and send them to a computer terminal device. The computer terminal device and the AI ​​parsing device are connected via network communication. The AI ​​parsing device has a USB OTG port, and the USB port of the encoder is connected to the USB OTG port on the AI ​​parsing device as an output end.

7. An application method of the desktop system automated testing system according to any one of claims 1 to 6, characterized in that: The steps include: S1, connect the test operation device and the device to be tested; S2, starting the device under test to enable it to perform video output; S3, collecting the display output RGB signal of the desktop system output by the display output port of the device under test through the video input port, and converting the display output RGB signal output by the display output port of the device under test into an image using the built-in encoding module, performing video recording according to the converted image, and pushing the converted image stream to the AI ​​acceleration module; Alternatively, the display output RGB signal of the desktop system output from the display output port of the device under test is collected through an encoder and converted into an image, and the converted image is sent to the test operation device for video recording, and the converted image stream is pushed to the AI ​​analysis device at the same time; S4, using an AI acceleration module or an AI analysis device to identify controls and click or input positions in the converted image; S5, determining the simulation test operation required for the automated test according to the controls and click or input positions in the image, and connecting to the USB port of the device to be tested via the USB port to simulate the HID device sending a simulation test operation event to the device to be tested.

8. The application method of the desktop system automated testing system according to claim 7, characterized in that: In step S5, when the USB port is connected to the USB port of the device to be tested to simulate the HID device to send a simulated test operation event to the device to be tested, sending a simulated mouse click operation includes: if a virtual machine is running on the device to be tested, sending the simulated mouse click operation is sending the global positioning data of the mouse, and the global positioning data of the mouse is the relative position of the mouse in the entire screen display resolution; if a physical machine is running on the device to be tested, sending the simulated mouse click operation is sending the single offset data of the mouse, and when sending the single offset data of the mouse, the speed of the mouse is inversely proportional to the distance between the mouse and the target position, so that the closer to the target position, the slower the speed of the mouse.

9. The application method of the desktop system automated testing system according to claim 8, characterized in that: In step S4, when the AI ​​acceleration module or AI analysis device is used to identify the controls and click or input positions in the converted image, it includes using the AI ​​acceleration module or AI analysis device to identify the controls and click or input positions in the converted image based on a pre-trained target recognition model, and the target recognition model is a deep learning neural network and is pre-trained to establish a mapping relationship between the graphical window interface image of the desktop system, the controls in the graphical window interface image, and the click or input positions.

10. The application method of the desktop system automated testing system according to claim 9, characterized in that: After step S5, it also includes: obtaining the display output RGB signal after the tested device executes the simulated interaction event; identifying the execution result of the simulated interaction event based on the display output RGB signal after the simulated interaction event is executed; if the simulated interaction event is a click operation, then comparing the image of the display output RGB signal after the simulated interaction event is executed with a preset reference image to obtain the execution result of the simulated interaction event; if the image of the display output RGB signal after the simulated interaction event is executed is the same as the preset reference image, then determining that the execution of the simulated interaction event is successful; otherwise, determining that the execution of the simulated interaction event fails; if the simulated interaction event is a drag operation, then using the AI ​​acceleration module or the AI ​​analysis device to identify the result based on the execution of the simulated interaction event. The window position coordinates in the image obtained by converting the display output RGB signal after executing the simulated interaction event are obtained. If the obtained window position coordinates and the preset window position coordinates are the same, the simulation interaction event is determined to be executed successfully, otherwise the simulation interaction event is determined to be executed unsuccessfully; if the simulated interaction event is an input operation, the AI ​​acceleration module or the AI ​​analysis device is used to identify the rectangular blank area of ​​multiple input boxes or single input boxes in the image obtained by converting the display output RGB signal after executing the simulated interaction event, and the simulation interaction event is determined to be executed successfully, and a simulated interaction event of the mouse simulation program clicking the blank position is sent to the rectangular blank area; otherwise, the simulation interaction event is determined to be executed unsuccessfully; the simulated interaction events, videos and execution results of this test are logged and saved.

Images