Computer system control method, script file generation method and computer system controller

Abstract

The invention provides a computer system controller, a script file generation method and a computer system control method. The computer system controller is used for controlling a computer system and comprises an image acquisition circuit and a control circuit. The image acquisition circuit is coupled to the computer system and is used for transmitting a current picture image received from the computer system. The control circuit is coupled to the computer system and the image acquisition circuit and is used for reading the picture operation record in the script file; receiving a current picture image from the image acquisition circuit; comparing the current picture image with a storage picture image indicated by the picture operation record; and when the current picture image is substantially the same as the storage picture image, controlling the computer system according to at least one operation task and at least one task time interval corresponding to each other in the picture operation record.

Description

Technical Field

[0001] The present disclosure relates to a method and apparatus for controlling a computer system, and more particularly to a method and apparatus for controlling a computer system using a script file. Background Art

[0002] Many computer operations are repetitive tasks, such as format adjustment, data filling, software installation, etc. To automate the above computer operations, related technologies often write and execute script content through a script control software, thereby triggering events that simulate keyboard and / or mouse operations (e.g., moving the mouse, pressing a key, releasing a key, etc.) in sequence according to a set waiting time.

[0003] However, these related technologies using script control software still encounter many situations. For example, for different operating systems, different versions of the script control software need to be installed on the computer system. Another example is that the loading time of each computer system's screen is different, making it easy for the situation where the waiting time has passed but the screen has not finished loading to occur, resulting in errors in the execution of the script content. Therefore, users usually need to spend a lot of time repeating operations to test the waiting time that can make the script content less error-prone during execution. In short, these related technologies usually have problems such as the need to manually install a specific version of the script control software on the computer system, high operation thresholds (possibly requiring users to have basic programming knowledge), and time-consuming script content writing. Summary of the Invention

[0004] An aspect of the present disclosure is a computer system control method. The computer system control method is used to control a computer system and includes: reading, by a computer system controller, at least one screen operation record in a script file; obtaining, by the computer system controller, a current screen image of the computer system; comparing, by the computer system controller, the current screen image with a stored screen image indicated by the at least one screen operation record; and when the current screen image and the stored screen image are substantially the same, controlling, by the computer system controller, the computer system according to at least one operation task and at least one task time interval corresponding to each other in the at least one screen operation record.

[0005] Another aspect of the present disclosure is a method for generating a script file. The method for generating a script file is applicable to a computer system controller and includes: receiving a current screen image from the computer system; recording at least one operation event occurring on the computer system according to at least one input signal received from the computer system; determining whether a screen update event has occurred on the computer system according to the current screen image and a previously stored previous screen image; storing the current screen image when it is determined that the screen update event has occurred on the computer system; and generating a script file according to the previous screen image and the at least one operation event occurring between two time points corresponding to the previous screen image and the current screen image.

[0006] Another aspect of the present disclosure is a computer system controller. The computer system controller is used to control a computer system, where the computer system includes an input device, a host, and a display, and the computer system controller includes an image acquisition circuit and a control circuit. The image acquisition circuit is coupled to the host and the display and is used to transmit the current screen image output by the host to the display. The control circuit is coupled to the input device, the host, and the image acquisition circuit and is used to: read at least one screen operation record in the script file; receive the current screen image from the image acquisition circuit; compare the current screen image with the stored screen image indicated by the at least one screen operation record; and control the computer system according to at least one operation task and at least one task time interval corresponding to each other in the at least one screen operation record when the current screen image is substantially the same as the stored screen image.

[0007] By generating a screen operation record according to the previous screen image stored when a screen update event occurs and at least one operation event recorded after that and before the screen update event occurs, generating a script file according to the at least one screen operation record, and using the script file to reproduce the operation purpose of the user, the computer system controller, the method for generating a script file, and the method for controlling a computer system of the present disclosure have advantages such as plug-and-play, easy operation, quick generation of a script file, and high success rate of reproducing the operation purpose of the user. Description of the Drawings

[0008] Figure 1 It is a schematic diagram of a computer system and a computer system controller illustrated according to some embodiments of the present disclosure.

[0009] Figure 2 It is a block diagram of a computer system and a computer system controller illustrated according to some embodiments of the present disclosure.

[0010] Figure 3 It is a flowchart of a method for generating a script file illustrated according to some embodiments of the present disclosure.

[0011] Figure 4 The flowchart of a script file generation method illustrated according to some embodiments of the present disclosure.

[0012] Figure 5 The schematic diagram of a timing sequence of a screen update event and an operation event and a script file generated based on the timing sequence illustrated according to some embodiments of the present disclosure.

[0013] Figure 6 The flowchart of a computer system control method illustrated according to some embodiments of the present disclosure.

[0014] Explanation of reference numerals:

[0015] 10: Computer system

[0016] 20: Input device

[0017] 21: Mouse

[0018] 23: Keyboard

[0019] 30: Host

[0020] 40: Monitor

[0021] 100: Computer system controller

[0022] 110: Control circuit

[0023] 111: First control circuit

[0024] 113: Second control circuit

[0025] 120: Image acquisition circuit

[0026] 300: Script file generation method

[0027] 600: Computer system control method

[0028] JF: Script file

[0029] PN[T1], PN[T5], PN[T9]: File name

[0030] PR[T2], PR[T3], PR[T6], PR[T7], PR[T10]: Operation parameter

[0031] RSO[1]~RSO[3]: Screen operation record

[0032] S301~S311, S401~S403, S601~S605: Operation

[0033] SC: Control signal

[0034] SII: Input signal

[0035] SIO: Output signal

[0036] SVC: Current screen image

[0037] SVP: Previous screen image

[0038] T1 to T11: Time

[0039] TI[T2], TI[T3], TI[T6], TI[T7], TI[T10]: Task time interval

[0040] TK[T2], TK[T3], TK[T6], TK[T7], TK[T10]: Operation task

[0041] TN[T2], TN[T3], TN[T6], TN[T7], TN[T10]: Task name Detailed implementation manners

[0042] The following are detailed descriptions with reference to embodiments and accompanying drawings. However, the specific embodiments described are only used to explain the present case and not to limit the present case. The description of the structure and operation is not used to limit the execution order. Any structure formed by recombining elements and generating a device with equivalent functions is within the scope covered by the present disclosure. The same reference numerals will be used to represent the same or similar elements in all the drawings.

[0043] The terms used throughout the specification and the claims of the patent application, unless otherwise specified, generally have the ordinary meanings of each term in this field, in the content disclosed herein, and in the specific context.

[0044] Regarding the use of "coupled" or "connected" in this article, it can refer to two or more elements making direct physical or electrical contact with each other, or making indirect physical or electrical contact with each other, and can also refer to two or more elements operating or acting on each other.

[0045] Please refer to Figure 1 , Figure 1 which is a schematic diagram of a computer system 10 and a computer system controller 100 illustrated according to some embodiments of the present disclosure. As Figure 1 shown, the computer system 10 includes an input device 20, a host 30, and a display 40. The input device 20 includes a mouse 21 and a keyboard 23, and the host 30 can be composed of components such as a motherboard, a memory, a radiator, a power supply, a hard disk, and a graphics card. In addition, the computer system controller 100 is electrically coupled to the mouse 21 and the keyboard 23 in the input device 20, the host 30, and the display 40.

[0046] In some embodiments, a user (not shown) operates the input device 20 to perform operations on the computer system 10. For example, the user can move the mouse 21, click the left and / or right buttons on the mouse 21, and / or press keys on the keyboard 23 to allow the host 30 to control the display 40 to display a specific web page.

[0047] Continuing with the above description, the computer system controller 100 can record at least one operation event (e.g., a key pressed on the keyboard 23, a mouse moved, etc.) occurring on the computer system 10 due to a user operating the input device 20, and can also record at least one screen update event occurring on the display 40 due to the at least one operation event. By using these previously recorded events, the computer system controller 100 can simulate the user's operation on the input device 20 in real time based on changes in the screen displayed on the display 40, even when the user is not operating the computer system 10, thereby reproducing the user's operation purpose on the computer system 10. Thus, the computer system controller 100 can be used to control the computer system 10.

[0048] See also Figure 2 , Figure 2 FIG1 is a block diagram of a computer system 10 and a computer system controller 100 according to some embodiments of the present disclosure. In some embodiments, the computer system controller 100 includes a control circuit 110 and an image acquisition circuit 120. Figure 2 As shown, the control circuit 110 may also be composed of a first control circuit 111 and a second control circuit 113. Specifically, the first control circuit 111 may be implemented by a combination of programmable hardware circuits and software (i.e., firmware), the second control circuit 113 may be implemented by software stored in a storage circuit (e.g., memory) within the computer system controller 100, and the image acquisition circuit 120 may be implemented by an electronic circuit having an image acquisition function.

[0049] In some embodiments, such as Figure 2 As shown, the first control circuit 111 is electrically or communicatively coupled to the input device 20 , the host 30 and the second control circuit 113 , and the image acquisition circuit 120 is electrically or communicatively coupled to the host 30 , the display 40 and the second control circuit 113 .

[0050] Then match Figure 3 Detailed description The computer system controller 100 records various events that occur on the computer system 10 due to user operations on the computer system 10. Figure 3 , Figure 3 FIG. 3 is a flow chart of a script file generation method 300 according to some embodiments of the present disclosure. In some embodiments, Figure 3As shown, the script file generation method 300 can be executed by the computer system controller 100 and includes multiple operations S301 to S311, but the present disclosure is not limited thereto. However, those skilled in the art of the present invention can understand that the script file generation method 300 of the embodiments of the present invention is not limited to being applied to Figure 1 , Figure 2 the computer system controller 100, nor is it limited to Figure 3 the operation sequence of each item in the flowchart.

[0051] In operation S301, the computer system controller 100 receives the current screen image SVC from the computer system 10. In some embodiments, as Figure 2 shown, the host 30 can generate the current screen image SVC to the computer system controller 100 by receiving and processing signals, data, and / or information. The computer system controller 100 can directly output the current screen image SVC to the display 40 through the image acquisition circuit 120 for the display 40 to display. Also, the image acquisition circuit 120 can output the current screen image SVC to the second control circuit 113. It should be understood that the current screen image SVC can be transmitted between the host 30, the display 40, the image acquisition circuit 120, and the second control circuit 113 in the form of signals, data, information, etc.

[0052] It is worth noting that by using the image acquisition circuit 120 instead of the second control circuit 113 to directly output the current screen image SVC from the host 30 to the display 40, the computer system controller 100 can avoid affecting the display of the display 40 (for example, causing the display 40 to display the current screen image SVC with a delay), enabling the display 40 to display smoothly.

[0053] In operation S302, the computer system controller 100 determines whether a screen update event has occurred on the computer system 10 based on the current screen image SVC and the previously stored previous screen image SVP. In some embodiments, as Figure 2 shown, after receiving the current screen image SVC, the second control circuit 113 compares the current screen image SVC with the previous screen image SVP and calculates the difference value of the current screen image SVC compared to the previous screen image SVP to determine whether a screen update event has occurred on the computer system 10 based on the difference value.

[0054] In some embodiments, both the current frame image SVC and the previous frame image SVP are black-and-white images. Accordingly, the second control circuit 113 can calculate the difference value according to the following formula (1), where diff is the difference value, wid and hei are respectively the pixel width and height of the screen of the display 40, Im(x, y) is the pixel value at a certain position in the current frame image SVC, Im0(x, y) is the pixel value at a certain position in the previous frame image SVP, and x and y are respectively used to represent the aforementioned certain position.

[0055]

[0056] Continuing with the above description, when the difference value is greater than a preset threshold (for example: 5% of the screen resolution of the display 40), the second control circuit 113 determines that the current frame image SVC is significantly different from the previous frame image SVP. In other words, the second control circuit 113 determines that a frame update event has occurred on the computer system 10. Accordingly, operation S303 will then be executed. When the difference value is not greater than the preset threshold, the second control circuit 113 determines that the current frame image SVC is substantially equivalent to the previous frame image SVP. In other words, the second control circuit 113 determines that no frame update event has occurred on the computer system 10. Accordingly, operation S301 will be executed again.

[0057] In operation S303, the computer system controller 100 stores the current frame image SVC. In some embodiments, the computer system controller 100 can store the current frame image SVC in the storage circuit in the computer system controller 100 through the second control circuit 113, and the second control circuit 113 can use this current frame image SVC as another previous frame image SVP required when operation S302 is executed next time.

[0058] From the descriptions of the above operation S302 and operation S303, it can be seen that the previous frame image SVP can be the frame image stored by the computer system controller 100 when it previously determined that a frame update event occurred on the computer system 10, but the present disclosure is not limited thereto. In some embodiments, the previous frame image SVP can be the current frame image SVC initially stored by the computer system controller 100 when triggered by the user to start executing the script file generation method 300.

[0059] In operation S304, the computer system controller 100 receives at least one input signal SII from the computer system 10. In some embodiments, such as Figure 2As shown, the input device 20 generates an input signal SII to the computer system controller 100 in response to a user operation (e.g., moving the mouse 21, clicking the left and / or right buttons on the mouse 21, or tapping a key on the keyboard 23). The computer system controller 100 receives the input signal SII via the first control circuit 111, which then transmits the input signal SII to the second control circuit 113. Accordingly, operation S305 is then executed. In some embodiments, the input device 20 does not generate the input signal SII due to no user operation. Accordingly, operation S304 is executed again.

[0060] In operation S305 , the computer system controller 100 records at least one operation event occurring in the computer system 10 according to at least one input signal SII.

[0061] In some embodiments of operation S305, the input signal SII is generated in response to the user's operation of moving the mouse 21, and the input signal SII includes relative displacement information corresponding to the user's operation of moving the mouse 21 (e.g., movement trajectory, movement distance, etc.). In this case, the second control circuit 113 converts the relative displacement information in the input signal SII into absolute position information (e.g., the absolute coordinate position of the cursor displayed on the display 40 after the mouse 21 is moved), and connects the operation of moving the mouse 21 and the corresponding absolute position information as an operation event. Specifically, the second control circuit 113 can convert the relative displacement information into absolute position information according to the following formula (2), where para x and para y For absolute location information, location x and location y is the absolute position information calculated last time (ie, the absolute coordinate position of the cursor displayed on the display 40 before the mouse 21 moves), mouse x and mouse y is the relative displacement information, wid and hei are the pixel width and height of the screen of the display 40 respectively, and 32767 is the maximum value of the vertical / horizontal axis of the absolute coordinate system adopted by the first control circuit 111.

[0062]

[0063] In some embodiments of operation S305, the input signal SII is generated in response to the user's operation of moving the mouse 21, and the input signal SII includes first absolute position information corresponding to the user's operation of moving the mouse 21, wherein the first absolute position information is applicable to the absolute coordinate system used by the mouse 21 (for example, the maximum value of the vertical / horizontal axis is 65535). In this case, the second control circuit 113 converts the first absolute position information in the input signal SII into second absolute position information (which is applicable to the absolute coordinate system used by the first control circuit 111) and connects the operation of moving the mouse 21 and the corresponding second absolute position information as an operation event. Specifically, the second control circuit 113 can convert the first absolute position information into the second absolute position information according to the following formula (3), where para x ' and para y ' is the second absolute position information, mouse x ' and mouse I ′ is the first absolute position information, and 0.5 is the ratio of the maximum vertical / horizontal axis value of the absolute coordinate system adopted by the first control circuit 111 to the maximum vertical / horizontal axis value of the absolute coordinate system adopted by the mouse 21 .

[0064]

[0065] As can be seen from the above description, formulas (2) and (3) are not limited to those shown above. For example, 32767 in formula (2) will change as the absolute coordinate system used by the first control circuit 111 changes. For another example, 0.5 in formula (3) will change as the absolute coordinate system used by the first control circuit 111 and the absolute coordinate system used by the mouse 21 change.

[0066] In some embodiments of operation S305, input signal SII is generated in response to a user scrolling the scroll wheel on mouse 21, and input signal SII includes scroll information corresponding to the user scrolling the scroll wheel on mouse 21 (e.g., scroll count, scroll direction, etc.). Specifically, the scroll count is typically represented by a number, and the scroll direction is typically represented by a symbol such as "+" or "-." For example, "+3" indicates that the scroll wheel has scrolled forward or upward by three unit scale increments, while "-4" indicates that the scroll wheel has scrolled backward or downward by four unit scale increments. In this case, second control circuit 113 combines the scrolling operation of the scroll wheel on mouse 21 and the corresponding scroll information as a single operation event.

[0067] In some embodiments of operation S305, the input signal SII is generated in response to a user tapping a key on the keyboard 23, and the input signal SII includes key information corresponding to the user tapping a key on the keyboard 23 (e.g., the key pressed, the key released, etc.). Specifically, the key information can be represented by a key code (Key Code) publicly defined by Microsoft. For example, "91" represents the left Win key. In this case, the second control circuit 113 connects the tapping operation on the keyboard 23 and the corresponding key information as a single operation event.

[0068] As can be seen from the description of the above-mentioned operation S305 embodiment, in some embodiments, the computer system controller 100 obtains the user's operation on the input device 20 and the corresponding operation information (i.e., absolute position information, scroll information, key information, etc.) from the input signal SII through the second control circuit 113, and connects the user's operation on the input device 20 and the corresponding operation information to record them as an operation event.

[0069] Further explanation is given above. During the execution of operation S305, if Figure 2 As shown, the second control circuit 113 generates an output signal SIO according to the operation information obtained from the input signal SII, and transmits the output signal SIO to the host 30 through the first control circuit 111, so that the host 30 can take corresponding actions in response to the user's operation on the input device 20 (for example, generate a corresponding current screen image SVC).

[0070] In operation S306, the computer system controller 100 records the time point of storing the screen image. In some embodiments, during the execution of the script file generation method 300, each time the second control circuit 113 stores the current screen image SVC, the second control circuit 113 also records the timestamp corresponding to the action of storing the current screen image SVC as the time point of storing the screen image.

[0071] In operation S307, the computer system controller 100 records the time when the operation event occurred. In some embodiments, during the execution of the script file generation method 300, each time the second control circuit 113 receives the input signal SII and records the operation event, the second control circuit 113 also records the timestamp corresponding to the action of receiving the input signal SII as the time when the operation event occurred.

[0072] In some embodiments, the computer system controller 100 determines that a screen update event has occurred on the computer system 10 and stores the current screen image SVC (this corresponds to operations S302 and S303), and records the time point corresponding to the action of storing the current screen image SVC (this corresponds to operation S306), so that operation S308 is then executed.

[0073] In operation S308, the computer system controller 100 detects whether there is at least one operation event recorded between the two time points corresponding to the previous screen image SVP and the current screen image SVC. Based on the execution of operations S306 and S307, in some embodiments, the second control circuit 113 can determine whether there is at least one operation event recorded between the time point of storing the previous screen image SVP and the time point of storing the current screen image SVC according to the time stamp corresponding to the recorded operation event, the time stamp corresponding to the previous screen image SVP, and the time stamp corresponding to the current screen image SVC.

[0074] Continuing from the above description, when the computer system controller 100 detects at least one operation event recorded between the two time points corresponding to the previous screen image SVP and the current screen image SVC, operations S309, S310, and S311 will be executed sequentially. When the computer system controller 100 does not detect at least one operation event recorded between the two time points corresponding to the previous screen image SVP and the current screen image SVC, operation S301 will be executed again. It should be understood that in some embodiments, regardless of the result of the determination in operation S308 being yes or no, operations S304, S305, and S307 will be continuously and repeatedly executed. That is, during the execution of the script file generation method 300, the computer system controller 100 will continuously record the operation events occurring on the computer system 10.

[0075] In operation S309, the computer system controller 100 uses at least one operation event recorded between the two time points corresponding to the previous screen image SVP and the current screen image SVC as at least one operation task. In some embodiments, the second control circuit 113 defines the task name according to the operation of the user on the input device 20 indicated by at least one operation event detected in operation S308, defines the operation parameters according to the operation information indicated by at least one operation event detected in operation S308, and connects the task name and the operation parameters as an operation task. In some embodiments, the operation information (such as the aforementioned absolute position information, scrolling information, key information, etc.) indicated by at least one operation event detected in operation S308 is directly used as the operation parameter.

[0076] In some practical applications, the task names can be, for example, "LeftButtonDown" representing the left button of the mouse 21 being pressed, "LeftButtonUp" representing the left button of the mouse 21 being released, "RightButtonDown" representing the right button of the mouse 21 being pressed, "RightButtonUp" representing the right button of the mouse 21 being released, "MiddleButtonDown" representing the middle button of the mouse 21 being pressed, "MiddleButtonUp" representing the middle button of the mouse 21 being released, "MouseWheel" representing the mouse wheel of the mouse 21 being scrolled, "MouseMove" representing the mouse 21 being moved, "keyboardDown" representing a key on the keyboard 23 being pressed, "keyboardUp" representing a key on the keyboard 23 being released, etc.

[0077] In operation S310, the computer system controller 100 generates at least one task time interval based on the time point corresponding to the previous frame image SVP and at least one time point corresponding to at least one operation task. Based on the execution of operations S306 and S307, in some embodiments, the second control circuit 113 can calculate at least one task time interval according to the timestamp corresponding to the previous frame image SVP and at least one timestamp corresponding to at least one operation event detected in operation S308 (which is used as at least one operation task in operation S309). Assuming two operation events detected by the second control circuit 113 in S308 (which are represented as the first operation task and the second operation task respectively in the following description), the second control circuit 113 subtracts the timestamp corresponding to the previous frame image SVP from the timestamp corresponding to the first operation task to calculate the first task time interval corresponding to the first operation task, and subtracts the timestamp corresponding to the first operation task from the timestamp corresponding to the second operation task to calculate the second task time interval corresponding to the second operation task. Other cases can be understood by analogy with this example, so they will not be elaborated here.

[0078] In operation S311, the computer system controller 100 integrates the file name of the previous screen image SVP, at least one task time interval, and at least one operation task into at least one screen operation record to generate a script file JF. In some embodiments, the second control circuit 113 may list the file name of the previous screen image SVP (i.e., the name used by the computer system controller 100 when storing the previous screen image SVP) at the top of the array, and then, below the file name of the previous screen image SVP, arrange at least one operation task according to the timing sequence of at least one operation task. At least one task time interval is arranged corresponding to at least one operation task. Thereafter, the second control circuit 113 may use this array as at least one screen operation record of the script file JF. The script file JF and its screen operation record will be further described in the following paragraphs in conjunction with Figure 5 further illustration.

[0079] From the descriptions of operations S309 to S311 above, in some embodiments, the computer system controller 100 is used to generate a script file JF based on the previous screen image SVP and at least one operation event occurring between two time points corresponding to the previous screen image SVP and the current screen image SVC.

[0080] It should be understood that the script file generation method 300 of the present disclosure is not limited to Figure 3 the multiple operations and sequences shown. Please refer to Figure 4 , Figure 4 which is a flowchart of the script file generation method 300 illustrated according to some embodiments of the present disclosure. In some embodiments, as Figure 4 shown, the script file generation method 300 further includes multiple operations S401 to S403. Specifically, operation S401 can be executed after any one of the multiple operations S301 to S311 in Figure 3 .

[0081] In operation S401, the computer system controller 100 receives an end command. In some embodiments, the end command can be generated in response to a user operating a human-machine interface (not shown in the figure) on the computer system controller 100. For example, the computer system controller 100 is provided with a switch (i.e., the human-machine interface) electrically coupled to the second control circuit 113, and this switch can be toggled by the user to indicate whether the computer system controller 100 starts or ends the execution of the script file generation method 300. When the user toggles the switch to indicate that the computer system controller 100 ends the execution of the script file generation method 300, this switch will send an end command to the second control circuit 113, causing operation S402 to be executed next.

[0082] In operation S402, the computer system controller 100 detects whether at least one operation event is recorded after the time point corresponding to the last stored last frame image (e.g., the current frame image SVC last stored in operation S303). Similarly, based on the execution of operations S306 and S307, in some embodiments, the second control circuit 113 can determine whether at least one operation event is recorded after the time point when the last frame image was stored based on the timestamp corresponding to the recorded operation event and the timestamp corresponding to the last frame image.

[0083] As described above, when the computer system controller 100 detects at least one operation event recorded after the time point when the last frame image was stored (i.e., the result of the determination in operation S402 is yes), operation S403 is executed. When the computer system controller 100 does not detect at least one operation event recorded after the time point when the last frame image was stored (i.e., the result of the determination in operation S402 is no), the script file generation method 300 terminates.

[0084] In operation S403, the computer system controller 100 generates a last screen operation record of the script file JF according to the last screen image and at least one operation event recorded after the time point corresponding to the last screen image. Operation S403 can be based on Figure 3 The descriptions of operations S309 to S311 can be understood by analogy, so their descriptions are omitted here.

[0085] Then match Figure 5 The generation of the script file JF is described by taking some practical operations of the user on the computer system 10 and the computer system controller 100 as an example. Figure 5 , Figure 5 FIG2 is a schematic diagram illustrating a timing sequence of a screen update event and an operation event and a script file JF generated based on the timing sequence according to some embodiments of the present disclosure.

[0086] like Figure 5 As shown, at time T1, the display 40 displays a desktop image. Specifically, the desktop image may be the screen image typically displayed on the display 40 after the host 30 completes the boot process using the Microsoft Windows operating system. Simultaneously, the user operates the computer system controller 100 to enter the learning mode (i.e., the computer system controller 100 begins executing the script file generation method 300).

[0087] As described in the previous description of the screen image SVP, in some embodiments, after the computer system controller 100 is triggered to start executing the script file generation method 300, it will first receive and store the current screen image SVC (that is, the desktop image displayed by the display 40 at time T1).

[0088] At time T2, the user presses the Win key on the keyboard 23. At time T3, the user releases the Win key on the keyboard 23. As described in the previous operations S304 and S305, the computer system controller 100 will record the operation event of the Win key being pressed and the operation event of the Win key being released.

[0089] Since the Win key on the keyboard 23 is tapped by the user, at time T4, the display 40 loads the start menu. Specifically, when the display 40 loads the start menu, the start menu pops up from the lower left corner of the desktop image displayed by the display 40 and gradually enlarges. It should be understood that at this time, the size of the start menu has not reached the preset size and covers a part of the desktop image. As described in the previous operations S302 and S303, the computer system controller 100 will determine that the current screen image SVC (that is, the desktop image covered by the start menu with a smaller size stored at time T4) is significantly different from the previous screen image SVP (that is, the desktop image without the start menu covering it stored at time T1), and then store the current screen image SVC.

[0090] Again, as described in the previous operations S308 to S311, the computer system controller 100 detects the operation events of the Win key being pressed and the Win key being released recorded between time T1 when the desktop image without the start menu covering it is stored and time T4 when the desktop image with a smaller size start menu covering it is stored, and then generates, according to the desktop image without the start menu covering it, the operation event of the Win key being pressed, and the operation event of the Win key being released, Figure 5 the screen operation record RSO[1] in the script file JF as shown.

[0091] In Figure 5 the screen operation record RSO[1] includes the file name PN[T1], the operation tasks TK[T2] and TK[T3] used by the computer system controller 100 when storing the desktop image without the start menu covering it, where the operation task TK[T2] includes the task name TN[T2] representing that a key has been pressed and the key being pressed is the Win key (corresponding to Figure 5The operation parameter PR[T2] of the key code “91”) defined by Microsoft, and the operation task TK[T3] includes the task name TN[T3] representing that the key has been released and the operation parameter PR[T3] representing that the released key is the Win key. Also, the screen operation record RSO[1] further includes the task time interval TI[T2] corresponding to the operation task TK[T2] (which is equivalent to the time difference obtained by subtracting time T1 from time T2) and the task time interval TI[T3] corresponding to the operation task TK[T3] (which is equivalent to the time difference obtained by subtracting time T2 from time T3).

[0092] At time T5, the display 40 finishes loading the start menu. Specifically, when the display 40 finishes loading the start menu, the size of the start menu reaches a preset size and covers a larger part of the desktop image. As described in the foregoing operations S302 and S303, the computer system controller 100 will determine that the current screen image SVC (that is, the desktop image covered by the start menu with the preset size stored at time T5) is significantly different from the previous screen image SVP (that is, the desktop image covered by the start menu with a smaller size stored at time T4), and then stores the current screen image SVC.

[0093] Also, as described in the foregoing operation S308, the computer system controller 100 does not detect at least one operation event recorded between time T4 when the desktop image covered by the start menu with a smaller size is stored and time T5 when the desktop image covered by the start menu with the preset size is stored, and then executes operation S301 again. In some further embodiments, the computer system controller 100 will also discard the desktop image covered by the start menu with a smaller size stored at time T4 to avoid occupying the storage resources of the computer system controller 100.

[0094] At time T6, the user presses the Win key on the keyboard 23. At time T7, the user releases the Win key on the keyboard 23. As described in the foregoing operations S304 and S305, the computer system controller 100 will record again the operation event that the Win key has been pressed and the operation event that the Win key has been released.

[0095] Since the Win key on the keyboard 23 is tapped again by the user, at time T8, the display 40 loads the desktop image. Specifically, when the display 40 loads the desktop image, the currently existing start menu gradually shrinks. It should be understood that at this time, the size of the start menu will again be smaller than the preset size and cover a part of the desktop image. As described in the foregoing operations S302 and S303, the computer system controller 100 will determine that the current screen image SVC (that is, the desktop image covered by the start menu with a smaller size stored at time T8) is significantly different from the previous screen image SVP (that is, the desktop image covered by the start menu with a preset size stored at time T5), and then store the current screen image SVC.

[0096] Also as described in the foregoing operations S308 to S311, the computer system controller 100 detects the operation events of the Win key being pressed and the operation events of the Win key being released recorded between time T5 when the desktop image covered by the start menu with a preset size is stored and time T8 when the desktop image covered by the start menu with a smaller size is stored, and then generates, based on the desktop image covered by the start menu with a preset size, the operation events of the Win key being pressed, and the operation events of the Win key being released, the screen operation record RSO[2] in the script file JF as Figure 5 shown.

[0097] In Figure 5 it, the screen operation record RSO[2] includes the file name PN[T5], the operation tasks TK[T6] and TK[T7] used by the computer system controller 100 when storing the desktop image covered by the start menu with a preset size. Among them, the operation task TK[T6] includes the task name TN[T6] representing the Win key being pressed and the operation parameter PR[T6], and the operation task TK[T7] includes the task name TN[T7] representing the Win key being released and the operation parameter PR[T7]. Also, the screen operation record RSO[2] further includes the task time interval TI[T6] corresponding to the operation task TK[T6] (this is equivalent to the time difference obtained by subtracting time T5 from time T6) and the task time interval TI[T7] corresponding to the operation task TK[T7] (this is equivalent to the time difference obtained by subtracting time T6 from time T7).

[0098] At time T9, the display 40 finishes loading the desktop image. Specifically, when the display 40 finishes loading the desktop image, the start menu disappears (that is, the desktop image is not covered by the start menu). As described in the foregoing operations S302 and S303, the computer system controller 100 will determine that the current screen image SVC (that is, the desktop image without the start menu covering it stored at time T9) is significantly different from the previous screen image SVP (that is, the desktop image with a smaller-sized start menu covering it stored at time T8), and then store the current screen image SVC.

[0099] Also as described in the foregoing operation S308, the computer system controller 100 does not detect at least one operation event recorded between time T8 when the desktop image with a smaller-sized start menu covering it is stored and time T9 when the desktop image without the start menu covering it is stored, and then executes operation S301 again. In some further embodiments, the computer system controller 100 will also discard the desktop image with a smaller-sized start menu covering it stored at time T8 to avoid occupying the storage resources of the computer system controller 100.

[0100] At time T10, the user moves the mouse 21 to control the cursor to move to the upper left corner of the screen (the corresponding absolute coordinate position is "(0,0)"). As described in the foregoing operations S304 and S305, the computer system controller 100 will record the operation event of the movement of the mouse 21.

[0101] At time T11, the user operates the computer system controller 100 to end the learning mode (that is, the computer system controller 100 is instructed to stop the execution of the script file generation method 300). As described in operations S401 to S402, the computer system controller 100 will receive an end command and then detect an operation event of the movement of the mouse 21 recorded after the time point corresponding to the last stored last screen image (that is, the desktop image without the start menu covering it stored at time T9). Accordingly, as described in operation S403, the computer system controller 100 generates the screen operation record RSO[3] in the script file JF as shown in Figure 5 (that is, the last screen operation record described in operation S403).

[0102] At Figure 5Among them, the screen operation record RSO[3] includes the file name PN[T9] and the operation task TK[T10] used when the computer system controller 100 stores the desktop image without starting the function table overlay. The operation task TK[T10] includes the task name TN[T10] representing the movement of the mouse 21 and the operation parameter PR[T10] (where the operation parameter PR[T10] includes the absolute position information representing the absolute coordinate position "(0,0)" where the cursor should be after the movement of the mouse 21 ends). Also, the screen operation record RSO[3] further includes the task time interval TI[T10] corresponding to the operation task TK[T10] (this is equivalent to the time difference obtained by subtracting time T9 from time T10).

[0103] Next, it will be paired with Figure 6 to illustrate how the computer system controller 100 reproduces the user's operation purpose through the script file JF. Please refer to Figure 6 , Figure 6 which is a flowchart of a computer system control method 600 illustrated according to some embodiments of the present disclosure. In some embodiments, as Figure 6 shown, the computer system control method 600 can be executed by the computer system controller 100 and includes multiple operations S601 to S605, but the present disclosure is not limited thereto. The description of the multiple operations S601 to S605 will be carried out in conjunction with Figure 5 the script file JF shown

[0104] In operation S601, the computer system controller 100 reads at least one screen operation record in the script file JF. In some embodiments, as Figure 2 shown, the second control circuit 113 reads the first screen operation record RSO[1] in the script file JF as Figure 5 shown. When the second control circuit 113 reads the file name PN[T1] listed at the top of the screen operation record RSO[1], operations S602 and S603 are sequentially executed

[0105] In operation S602, the computer system controller 100 obtains the current screen image SVC of the computer system 10. The description of operation S602 is similar to that of operation S301, so it will not be elaborated here

[0106] In operation S603, the computer system controller 100 compares the current screen image SVC with the stored screen images indicated by at least one screen operation record. In some embodiments, the second control circuit 113 compares the current screen image SVC with the stored screen image corresponding to the file name PN[T1] of the first screen operation record RSO[1] in the script file JF (i.e., the desktop image stored at time T1 without the start menu overlay), and calculates the difference value between the current screen image SVC and the stored screen image, so as to determine whether the current screen image SVC is substantially equivalent to the stored screen image based on the difference value. In some embodiments where both the current screen image SVC and the stored screen image are black and white images, the second control circuit 113 can calculate the difference value according to the above formula (1).

[0107] Continuing the above description, when the difference value is less than or equal to another preset threshold (e.g., 5%, 10%, or 16.67% of the screen resolution of the display 40), the second control circuit 113 determines that the current screen image SVC is substantially equivalent to the stored screen image. Accordingly, operation S604 will then be executed. When the difference value is greater than the other preset threshold, the second control circuit 113 determines that the current screen image SVC is significantly different from the stored screen image. Accordingly, operation S602 will be executed again.

[0108] In operation S604, the computer system controller 100 waits for at least a period of time according to at least one task time interval in at least one screen operation record. In some embodiments, when the second control circuit 113 reads the task time interval TI[T2] corresponding to the first operation task TK[T2] listed below the file name PN[T1] as shown in Figure 5 shown, the second control circuit 113 starts timing from 0 milliseconds through the timing circuit (not shown in the figure) in the computer system controller 100 to 1109 milliseconds indicated by the task time interval TI[T2], and stands by until the timing circuit times to 1109 milliseconds. After the timing circuit times to 1109 milliseconds, operation S605 is then executed.

[0109] In operation S605, the computer system controller 100 sends a control signal SC for the analog input device 20 to execute at least one operation task in at least one screen operation record to the computer system 10. In some embodiments, as shown in Figure 2 shown, the second control circuit 113 generates a control signal SC for the operation of pressing the Win key on the analog keyboard 23 according to the task name TN[T2] and the operation parameter PR[T2] of the operation task TK[T2] as shown in Figure 5 shown, and transmits the control signal SC to the host 30 of the computer system 10 through the first control circuit 111.

[0110] After that, since there is still a second operation task TK[T3] in the screen operation record RSO[1] that has not been executed, as Figure 6 shown, the computer system controller 100 executes operation S604 again. The second control circuit 113 starts timing from 0 milliseconds through the timing circuit until 94 milliseconds indicated by the task time interval TI[T3], and stands by until the timing circuit counts up to 94 milliseconds. After the timing circuit counts up to 94 milliseconds, the computer system controller 100 executes operation S605 again. The second control circuit 113 generates a control signal SC simulating the operation of releasing the Win key on the virtual keyboard 23 according to the task name TN[T3] and operation parameters PR[T3] of the operation task TK[T3] as Figure 5 shown, and transmits the control signal SC to the host 30 of the computer system 10 through the first control circuit 111.

[0111] As described above, in some embodiments, after sequentially receiving the control signal SC for the operation of pressing the Win key on the virtual keyboard 23 and the control signal SC for the operation of releasing the Win key on the virtual keyboard 23, the host 30 generates a corresponding screen image (for example: a desktop image covered by a start menu with a smaller size or a preset size), and then transmits the corresponding screen image to the display 40 through the image acquisition circuit 120 for the display 40 to complete the loading of the start menu (that is, display a desktop image covered by a start menu with a preset size).

[0112] From the descriptions of the above operation S604 and operation S605, it can be seen that in some embodiments, the computer system controller 100 controls the computer system 10 according to at least one operation task and at least one task time interval corresponding to each other in at least one screen operation record.

[0113] In some embodiments, since the two operation tasks TK[T2] and TK[T3] in the screen operation record RSO[1] are all completed, as Figure 6 shown, the computer system controller 100 executes operation S601 again. Specifically, the second control circuit 113 reads the second screen operation record RSO[2] in the script file JF as Figure 5 shown. The subsequent operations can be analogized according to the description of the screen operation record RSO[1], so the description thereof is omitted here. It should be understood that when the two operation tasks TK[T6] and TK[T7] in the screen operation record RSO[2] are all completed, the display 40 will display a desktop image without a start menu covering it.

[0114] In some embodiments, after both operation tasks TK[T6] and TK[T7] in the screen operation record RSO[2] are completely executed, the computer system controller 100 executes operation S601 again, that is, the second control circuit 113 reads the last screen operation record RSO[3] in the script file JF as shown in Figure 5 . The subsequent operations can be analogized according to the description of the screen operation record RSO[1], so the description thereof is omitted here. It should be understood that when the operation task TK[T10] in the screen operation record RSO[3] is executed, the cursor displayed on the display 40 will be located at the upper left corner of the desktop image (that is, the absolute position information indicated by the operation parameter PR[T10] corresponding to the operation task TK[T10]).

[0115] In some embodiments, when the operation task TK[T10] in the screen operation record RSO[3] is executed, it means that Figure 5 all three screen operation records RSO[1] to RSO[3] of the script file JF in are completely read. Accordingly, as shown in Figure 6 , the computer system controller 100 ends the computer system control method 600.

[0116] It should be understood that during the execution of the computer system control method 600, the input device 20 may also generate an input signal SII to the computer system controller 100 in response to a user operation. In some embodiments, the computer system controller 100 will block the input signal SII received from the input device 20 from being transmitted to the host 30, or block the output signal SIO generated based on the input signal SII from being transmitted to the host 30. In this way, it is possible to prevent the operation of the computer system controller 100 from reproducing the user operation purpose through the script file JF from being affected or interrupted by the user operation.

[0117] As can be seen from the above embodiments of the present disclosure, by generating a screen operation record based on the previously stored screen image (i.e., the previous screen image SVP) at the time of the screen update event and at least one operation event recorded after that and before the screen update event, generating a script file JF based on at least one screen operation record, and using the script file JF to reproduce the operation purpose of the user, the computer system controller 100, the script file generation method 300, and the computer system control method 600 are applicable to various operating systems, do not require additional software to be installed on the computer system, and do not require the user to have basic programming knowledge. The computer system controller 100, the script file generation method 300, and the computer system control method 600 can avoid the problem in the related art that the user needs to spend a lot of time repeating operations to test the waiting time that can make the script file not prone to errors during execution, and can avoid the problem in the related art that the operation purpose of the user fails to be reproduced because the computer system screen has not been fully loaded but the waiting time has passed.

[0118] In addition, as long as it is to control the movement of the cursor displayed on the display 40, the computer system controller 100 will generate an output signal SIO or a control signal SC to the host 30 based on the converted absolute position information (i.e., the absolute coordinate position), which can avoid the problem in the related art that errors are easily generated when the screen ratio changes due to the use of relative displacement information, resulting in the cursor acting incorrectly.

[0119] In summary, the computer system controller 100, the script file generation method 300, and the computer system control method 600 of the present disclosure have the advantages of plug-and-play, easy operation, rapid generation of script files, and high success rate of reproducing the operation purpose of the user.

[0120] Although the present disclosure has been disclosed as above in embodiments, it is not intended to limit the present disclosure. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the scope defined by the appended claims.

Claims

1. A computer system control method for controlling a computer system, characterized in that, Comprising: Reading, by a computer system controller, at least one screen operation record in a script file; Obtaining, by the computer system controller, a current screen image of the computer system; Comparing, by the computer system controller, the current screen image with a stored screen image indicated by the at least one screen operation record; And When the current screen image is substantially the same as the stored screen image, controlling, by the computer system controller, the computer system according to at least one operation task and at least one task time interval corresponding to each other in the at least one screen operation record.

2. The computer system control method according to claim 1, wherein, Controlling the operation of the computer system according to the at least one operation task and the at least one task time interval corresponding to each other includes: Waiting for at least a period of time according to the at least one task time interval; And Sending a control signal simulating the input device of the computer system to execute the at least one operation task to the host of the computer system.

3. A method for generating a script file, applicable to a computer system controller, characterized in that, Comprising: Receiving a current screen image from a computer system; Recording, according to at least one input signal received from the computer system, at least one operation event occurring on the computer system; Judging, according to the current screen image and a previously stored previous screen image, whether a screen update event occurs on the computer system; When it is judged that the screen update event occurs on the computer system, storing the current screen image; and Generating a script file according to the previous screen image and the at least one operation event occurring between two time points corresponding to the previous screen image and the current screen image.

4. The method for generating a script file according to claim 3, wherein The two time points include a first time point for storing the previous screen image and a second time point for storing the current screen image, the at least one operation event occurring between the first time point and the second time point corresponds to at least one third time point, and the operation of generating the script file according to the previous screen image and the at least one operation event occurring between the two time points includes: Taking the at least one operation event recorded between the first time point and the second time point as at least one operation task; Generating, according to the first time point and the at least one third time point, at least one task time interval corresponding to the at least one operation task; And Integrating the file name of the previous screen image, the at least one task time interval and the at least one operation task into at least one screen operation record to generate the script file.

5. The method for generating a script file according to claim 3, characterized in that, Further comprising: Detecting whether there is at least one operation event recorded between the two time points corresponding to the previous screen image and the current screen image.

6. The method for generating a script file according to claim 3, wherein Further comprising: In response to receiving an end command, detecting whether there is at least one operation event recorded after a time point corresponding to the last screen image finally stored by the computer system controller; And When detecting at least one operation event recorded after the time point, generating a last screen operation record of the script file according to the last screen image and the at least one operation event recorded after the time point.

7. A computer system controller for controlling a computer system, characterized in that, The computer system includes an input device, a host and a display, and the computer system controller includes: An image acquisition circuit, coupled to the host and the display, and configured to transmit the current screen image output by the host to the display; and A control circuit, coupled to the input device, the host, and the image acquisition circuit, and configured to: Read at least one screen operation record in a script file; Receive the current screen image from the image acquisition circuit; Compare the current screen image with the stored screen images indicated by the at least one screen operation record; and When the current screen image is substantially the same as the stored screen image, control the computer system according to at least one operation task and at least one task time interval corresponding to each other in the at least one screen operation record.

8. The computer system controller according to claim 7, characterized in that, The control circuit is configured to: Wait for at least a period of time according to the at least one task time interval; and Send a control signal simulating the input device to execute the at least one operation task to the host.

9. The computer system controller according to claim 7, wherein, The control circuit is further configured to: Record at least one operation event occurring on the computer system according to at least one input signal received from the input device; Judge whether a screen update event occurs on the computer system according to the current screen image and the previously stored previous screen image; When it is judged that the computer system has the screen update event, store the current screen image; and Generate the script file according to the previous screen image and the at least one operation event occurring between two time points corresponding to the previous screen image and the current screen image.

10. The computer system controller according to claim 9, wherein, The two time points include a first time point for storing the previous screen image and a second time point for storing the current screen image. The at least one operation event occurring between the first time point and the second time point corresponds to at least one third time point, and the control circuit is configured to: Use the at least one operation event recorded between the first time point and the second time point as the at least one operation task; Generate the at least one task time interval corresponding to the at least one operation task according to the first time point and the at least one third time point; and Integrate the file name of the previous screen image, the at least one task time interval, and the at least one operation task into the at least one screen operation record to generate the script file.

11. The computer system controller according to claim 9, wherein, The control circuit is further configured to: Detect whether there is at least one operation event recorded between the two time points corresponding to the previous screen image and the current screen image.

12. The computer system controller according to claim 9, wherein, The control circuit is further configured to: In response to receiving an end command, detect whether there is at least one operation event recorded after the time point corresponding to the last stored last screen image; and When detecting at least one operation event recorded after the time point, generate the last screen operation record of the script file according to the last screen image and the at least one operation event recorded after the time point.

13. The computer system controller according to claim 9, wherein The control circuit is further configured to: Calculate the difference value between the current screen image and the previous screen image; and When the difference value is greater than a preset threshold, judge that the screen update event occurs on the computer system.

Images