Unattended computer operating system and method based on machine vision

Abstract

The invention discloses an unattended computer operating system and method based on machine vision, and relates to the technical field of electronic data processing. The unattended operation computer operating system and method based on machine vision comprise the following steps: S1, collecting interface state data before and after operation and preprocessing the interface state data; s2, performing control precision judgment on the interface state data; s3, executing feedback consistency comparison on the interface state data with the qualified control precision judgment result; s4, maintenance failure records and trial times are limited, and infinite retry is avoided; and S5, integrating the interface state data, the control precision judgment result and the execution feedback consistency comparison result, and carrying out a task to complete stability evaluation. The problem that whether the operation is actually effective or not cannot be judged due to lack of an effective feedback confirmation mechanism after the operation is executed is solved.

Description

Technical Field

[0001] The present invention relates to the field of electronic data processing, and particularly to an unattended computer operating system and method based on machine vision. Background Art

[0002] With the development of artificial intelligence and automation, machine vision, as an important means of image perception and understanding, is widely used in fields such as industrial inspection and intelligent driving, and is gradually extended to the recognition and interactive control of complex interfaces. At the same time, unattended operating systems have shown great potential in remote control and automated operation and maintenance. However, existing solutions generally lack the ability to make real-time judgments on operation feedback, making it difficult to handle interface changes or recognition errors, which restricts stability and accuracy.

[0003] For example, the invention patent with the publication number CN103235750B discloses a computer unattended control system, including an intelligent control chip, a storage module, an external power supply module, an internal power supply module, and a computer operation feedback module. The intelligent control chip includes a detection module, a power control module, and a storage read-write module respectively connected to the central processing unit. The detection module is connected to the external power supply module, and judges the computer power access state and computer operation state through level detection technology, the external power supply module, and the computer operation feedback module, and filters interference signals. The power control module is connected to the internal power supply module. The storage read-write module is connected to the storage module. It also discloses a computer unattended control method, which realizes that the computer can automatically start and run when power is restored, and there is no need for manual operation to start the computer when power is restored, greatly reducing labor costs, expanding the application scope of the computer, and being convenient to use.

[0004] For example, the invention patent with the publication number CN117076036A discloses an RPA process unattended method and system for a remote computer. The method includes: S1, starting the RPA robot, preventing the remote computer from entering the sleep state by setting system functions, and performing a running environment detection before the RPA robot starts to execute the RPA process. The running environment detection includes resolution detection, session detection, system clock detection, interface availability detection, user setting detection, and power mode detection. S2, after completing the running environment detection, when the RPA robot recognizes that the session of the remote computer has exited or the interface has stopped rendering, the RPA robot immediately enables unattended operation to make the remote desktop active, ensuring the normal operation of the RPA process on the remote computer.

[0005] Existing unattended computer operations mostly rely on methods such as script playback, template matching, and coordinate clicking to achieve automatic operations. Some solutions are supplemented by OCR recognition for control confirmation. However, such methods are prone to failure in the case of dynamic interface changes, control offset, and abnormal feedback, making it difficult to achieve flexible response and high-precision control, which affects the operation success rate and task completion quality.

[0006] Therefore, in view of the above problems, there is an urgent need for an unattended computer operating system and method based on machine vision. Summary of the Invention

[0007] Technical Problems to be Solved Aiming at the deficiencies of the prior art, the present invention provides an unattended computer operating system and method based on machine vision, which solves the problem that there is no effective feedback confirmation mechanism after operation execution and it is impossible to judge whether the operation actually takes effect.

[0008] Technical Solution: To achieve the above objectives, the present invention is realized through the following technical solutions: An unattended computer operating system and method based on machine vision, including the following steps: S1: Collect the interface state data before and after the operation, and preprocess the interface state data; S2: Determine the control accuracy of the interface state data, and judge the spatial accuracy, behavior stability, and reliability of system response of an operation according to the control accuracy determination result; S3: Perform a feedback consistency comparison on the interface state data with qualified control accuracy determination results, and judge whether the interface state is consistent with the expectation according to the execution feedback consistency comparison result; S4: Maintain the failure record and the limit of the number of attempts, avoid infinite retries, and adaptively match the processing method according to the failure type to achieve basic exception tolerance control; S5: Comprehensive interface state data, control accuracy determination results, and execution feedback consistency comparison results, conduct a stable evaluation of task completion, and judge whether a complete unattended task is stable and reliable according to the task completion stable evaluation result.

[0009] Further, the specific steps for collecting the interface state data before and after the collection operation and preprocessing the interface state data are as follows: The interface state data includes: response delay time, recognized text data set, and number of failed retries; Collect the current screen image and save it as a screenshot for subsequent structural similarity comparison; Obtain the state attribute information of the control, including auxiliary judgment information such as color change, focus state, and display and hide flags; Obtain the response delay time by subtracting the click timestamp recorded by the system from the completion timestamp of the interface state change; Extract the text content of the specified control area and use OCR recognition technology to form a recognized text set; Write the retry record after each operation failure into the control log and record it as the number of failed retries; Preprocess the interface state data, and the preprocessed data includes: click deviation distance, click speed variance, target text set, image structural similarity, and OCR recognition confidence; Perform size normalization and resolution correction on the screenshot image to ensure the structural consistency of the images collected by different terminals; Perform image enhancement and edge smoothing processing on the control area image; Perform unified language encoding, special character cleaning, and space merging on the recognized text to form structured text for subsequent comparison; Calculate the click deviation distance from the Euclidean distance between the center coordinates of the control and the actual click position coordinates; Calculate the average speed by combining the time series and displacement series of multiple click events, and then calculate the click speed variance through the sliding window variance formula; Obtain the target text set based on the preset field matching rules in the task template; Input the images before and after the operation obtained from the screenshot into the image structure comparison function, and use the local image analysis algorithm to calculate the image structural similarity; By comparing and analyzing the character differences between the recognized text and the target text and normalizing the matching degree, obtain the OCR recognition confidence; Record the calculated interface control accuracy value, execution feedback consistency value, and task completion stability value in real time.

[0010] Further, the specific steps for determining the control accuracy of the interface state data are as follows: Obtain the click deviation distance, response delay time, and click speed variance; Calculate the ratio of the click deviation distance to the response delay time, square the ratio, multiply the result by the behavior quality factor, and then multiply the result by a constant one-half to obtain the deviation speed suppression term; Calculate the click speed variance, take the natural logarithm after adding one to it, and multiply it by the instability adjustment coefficient to obtain the speed fluctuation suppression term; Subsequently, add the deviation speed suppression term and the speed fluctuation suppression term as the negative exponent part of the exponential function to calculate the interface control accuracy value.

[0011] Further, the specific steps for judging the spatial accuracy, behavior stability, and system response reliability of an operation based on the control accuracy determination result are as follows: Compare the interface control accuracy value with the accuracy threshold in real time. The accuracy threshold includes a primary accuracy threshold and a secondary accuracy threshold. When the interface control accuracy value is greater than or equal to the primary accuracy threshold, directly enter the next task process, mark the operation sample as a high-quality template, include it in the behavior imitation training set, and at the same time reduce the control recognition error tolerance range and keep the current rhythm parameter unchanged. When the interface control accuracy value is greater than the secondary accuracy threshold and less than the primary accuracy threshold, there are minor deviations and unstable responses. Perform operations such as recognition trigger delay, center perturbation click on the control, and confidence marking. At the same time, record the control as a low-confidence interaction target for subsequent model iteration and optimization. When the interface control accuracy value is less than or equal to the secondary accuracy threshold, trigger an abnormal remedy mechanism, abandon the original click path, switch to an alternative recognition template to reposition the control, and at the same time perform multi-point probing clicks within five pixels above, below, left, and right of the control center. If the accuracy value is still lower than the threshold for two consecutive times, mark the control as a high-risk interaction area and transfer it to the manual verification path.

[0012] Further, the specific steps for performing feedback consistency comparison on the interface state data with qualified control accuracy determination results are as follows: Obtain the image structure similarity, the recognized text set, and the target text set. Subtract the image structure similarity from one to obtain the difference degree of the image feedback. Square the difference value and multiply it by the visual feedback weight coefficient to obtain the image feedback error term. Calculate the number of intersection elements between the recognized text set after the operation and the expected target text set, divide it by the total number of the target text set to obtain the text matching ratio, and then take the absolute value of this ratio. Subtract the absolute value of the text matching ratio from one to obtain the difference degree of the text feedback. Square the difference value and multiply it by the text feedback weight coefficient to obtain the text feedback error term. Add the image feedback error term and the text feedback error term, and take the square root of the sum result to obtain the execution feedback consistency value.

[0013] Further, the specific steps for determining whether the interface state is consistent with the expectation according to the execution feedback consistency comparison result are as follows: Compare the execution feedback consistency value with the feedback threshold in real time. The feedback threshold includes a primary feedback threshold and a secondary feedback threshold. When the execution feedback consistency value is less than or equal to the primary feedback threshold, both the interface image structure and the text feedback meet the expectation. Then continue to execute the subsequent operation process and mark the current recognition template as a valid template. When the execution feedback consistency value is greater than the primary feedback threshold and less than or equal to the secondary feedback threshold, it is determined that there is a certain deviation. Then execute strategies such as recognition delay adjustment, relaxation of the image comparison range, and reduction of the comparison threshold. At the same time, enable the auxiliary recognition mechanism and archive the images, texts, and difference information of this round. When the execution feedback consistency value is greater than the secondary feedback threshold, the operation is ineffective. Automatically execute the operation retry process. If the retry exceeds the limit number of times, roll back to the previous task step, activate the image fault tolerance mechanism to try to confirm the control state again, and at the same time package and archive the recognition image, text results, and control information as error samples for subsequent analysis and model training.

[0014] Further, the specific steps for maintaining the failure record and the limit of the number of attempts, avoiding infinite retries, and adaptively matching the processing method according to the failure type to achieve basic exception fault tolerance control are as follows: After the operation fails, first record the failure type, control identification, and failure timestamp of the current operation, and update the corresponding failure count counter. When the failure count of a certain operation control does not exceed the preset maximum attempt threshold, execute the corresponding regulation strategies according to the failure reason type, including fine-tuning of the click position, adjustment of the input rhythm, switching of the alternative template, and jumping of the alternative path. When the failure count reaches the attempt limit, stop further retries, mark the control as a high-risk object, and at the same time archive the relevant operation data, failure logs, and tried strategies. Adaptively select the adjustment method according to different failure types, construct an abnormal handling diversion path, and achieve the goal of uninterrupted control flow and basic operation fault tolerance.

[0015] Further, the specific steps for conducting a stable assessment of task completion by integrating the interface state data, the control accuracy determination result, and the execution feedback consistency comparison result are as follows: Obtain the interface control accuracy value, the number of failure retries, and the OCR recognition confidence. Take the reciprocal of the interface control accuracy value, take the logarithm after adding one to the number of failure retries, and then take the reciprocal after adding the OCR recognition confidence and a very small correction term. Add the above three items and then add one, and finally take the reciprocal of the overall result to obtain the task completion stability value.

[0016] Further, the specific steps for determining whether a complete unattended task is stable and reliable based on the stable evaluation result of task completion are as follows: Compare the task completion stability value with the stability threshold in real time. The stability threshold includes a primary stability threshold and a secondary stability threshold. When the task completion stability value is greater than or equal to the primary stability threshold, the task execution accuracy is high, the feedback is accurate, and the recognition result is credible. Mark the current process as a high-quality path, and incorporate the execution record into the sample library for subsequent task imitation and optimization reference. When the task completion stability value is greater than the secondary stability threshold and less than the primary stability threshold, the task has minor fluctuations, and the recognition parameter optimization, path annotation, and task identification process are automatically triggered. If the recognition confidence is lower than the threshold, it is marked as unstable text recognition. If the number of retries reaches the threshold, it is marked as a structural improvement path and fed back to the task flow editing module. When the task completion stability value is less than or equal to the secondary stability threshold, the task is overall abnormal and there is a risk of interruption. Automatically re-initialize the current process and re-execute it. If it still fails, trigger the emergency exit mechanism and generate an operation failure report. At the same time, record the recognition image, result features, and error samples as high-risk path samples for subsequent training and key optimization.

[0017] The second aspect of the present invention provides an unattended computer operating system based on machine vision, which is applied to the above-mentioned unattended computer operation method based on machine vision. It includes an interface data acquisition module, an automatic behavior execution module, a result feedback judgment module, an abnormal regulation and remedy module, and a process record and optimization module. The interface data acquisition module is used to collect the interface state data before and after the operation and preprocess the interface state data. The automatic behavior execution module is used to determine the control accuracy of the interface state data and judge the spatial accuracy, behavior stability, and reliability of system response of an operation according to the control accuracy determination result. The result feedback judgment module is used to perform a feedback consistency comparison on the interface state data with qualified control accuracy determination results and judge whether the interface state is consistent with the expectation according to the execution feedback consistency comparison result. The abnormal regulation and remedy module is used to maintain the failure record and the limit of the number of attempts, avoid infinite retries, and adaptively match the processing method according to the failure type to achieve basic abnormal fault tolerance control. The process record and optimization module is used to comprehensively carry out the stable evaluation of task completion based on the interface state data, the control accuracy determination result, and the execution feedback consistency comparison result, and judge whether a complete unattended task is stable and reliable according to the stable evaluation result of task completion.

[0018] Beneficial effects The present invention has the following beneficial effects: (1) By collecting the image and text status information before and after the operation and combining with the multi-dimensional data preprocessing method, the present invention makes the interface status expression more structured and standardized, provides a unified and reliable feature basis for subsequent control accuracy determination and feedback consistency comparison, and significantly improves the adaptability and versatility of the system in multiple scenarios.

[0019] (2) The present invention introduces a control accuracy evaluation mechanism, comprehensively considers click deviation, response delay and behavior stability, and uses an exponential function to construct a non-linear accuracy scoring model to achieve accurate quantification of the execution quality of a single operation, improving the judgment reliability and adjustability of operation behavior.

[0020] (3) In the feedback judgment process, the present invention combines the dual information of image structure change and text recognition result, introduces a feedback consistency value as the judgment basis, effectively solves the problem of single-dimensional judgment error of traditional systems, and enhances the intelligent recognition ability of the system to whether the operation really takes effect.

[0021] (4) The present invention constructs a task completion stability value evaluation model, unifies operation accuracy, feedback accuracy and retry cost into a comprehensive evaluation system, realizes the dynamic judgment of the operation quality of a complete unattended task, and helps to construct a stable and reliable task execution closed loop.

[0022] Of course, any product implementing the present invention does not necessarily need to achieve all the above-mentioned advantages at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flowchart of an unattended computer operation method based on machine vision according to the present invention; Figure 2 is a structural diagram of an unattended computer operation system based on machine vision according to the present invention; Figure 3 is a bar chart of the interface control accuracy value according to the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0024] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0025] Please refer to Figures 1 - 3, an embodiment of the present invention provides a technical solution: an unattended computer operating system based on machine vision, which is applied to the above-mentioned unattended computer operation method based on machine vision, and includes S1: collecting interface state data before and after an operation, and preprocessing the interface state data; S2: determining the control accuracy of the interface state data, and judging the spatial accuracy, behavior stability and system response reliability of an operation according to the control accuracy determination result; S3: performing a feedback consistency comparison on the interface state data with qualified control accuracy determination results, and judging whether the interface state is consistent with the expectation according to the execution feedback consistency comparison result; S4: maintaining a failure record and a limit on the number of attempts, avoiding infinite retries, and adaptively matching a processing method according to the failure type to achieve basic exception tolerance control; S5: comprehensively considering the interface state data, the control accuracy determination result and the execution feedback consistency comparison result, carrying out a stable evaluation of task completion, and judging whether a complete unattended task is stable and reliable according to the task completion stable evaluation result.

[0026] Specifically, the interface state data before and after the acquisition operation is collected, and the specific steps for preprocessing the interface state data are as follows: The interface state data includes: response delay time, recognized text dataset, and failure retry count; The system first captures the current screen image and saves it as the basic data for subsequent image structure similarity comparison. At the same time, the system timestamp is recorded synchronously during the screenshot for time series comparison; The state attribute information of the control is obtained, including visual auxiliary signals such as color change, focus state, and display / hide flag, to assist in judging whether the operation behavior is effectively perceived by the system; By subtracting the click timestamp recorded by the system from the timestamp when the interface state change is completed, the operation response delay time is obtained, which reflects the timeliness of the interface's feedback to user input; The text content of the specified control area is extracted, and the OCR recognition technology is used to form a recognized text set, supporting multi-language recognition and character cleaning to enhance adaptability; The retry record after each operation failure is written into the control log, and the failure count is automatically incremented and bound to the ID of the operation target control, recorded as the failure retry count; The interface state data is preprocessed. The preprocessed data includes: click deviation distance, click speed variance, target text set, image structure similarity, and OCR recognition confidence; In image processing, the screenshot image is subjected to size normalization and resolution correction to ensure the comparability and structural consistency of images from devices with different resolutions; The control area image is subjected to image enhancement and edge smoothing processing to improve the boundary clarity and text recognition quality; In text data processing, the recognized text is subjected to operations such as unified language encoding, special character cleaning, and space merging to generate structured text for subsequent comparison; The click deviation distance related to the click behavior is calculated by the Euclidean distance between the center coordinates of the control and the actual click position coordinates, which is used to measure the operation accuracy; At the same time, the system analyzes the time series and displacement series of multiple click behaviors within a set time window, calculates the average click speed, and then applies the sliding window variance formula to calculate the click speed variance, which reflects the stability of the operation process; The target text set is dynamically generated according to the field matching rules in the task template to ensure the clarity and consistency of the operation target; The image structure similarity is calculated by inputting the images before and after the operation obtained from the screenshot into the structure comparison function through the local image analysis algorithm; The OCR recognition confidence is obtained by comparing and analyzing the character differences between the recognized text and the target text, using the edit distance algorithm and performing normalization processing; Finally, the calculated interface control accuracy value, execution feedback consistency value, and task completion stability value are recorded in real time to provide high-quality quantitative basis for subsequent evaluation, control, and optimization.

[0027] In this implementation scheme, through multi-dimensional acquisition and structured preprocessing of the interface state data before and after operations, not only the standardized expression of images, texts, and state attributes is realized, but also high-quality input features in a unified format are provided for subsequent accuracy determination and feedback consistency analysis. This step effectively improves the system's perception ability of operation behaviors and the clarity of data expression, ensures the consistency and accuracy of the calculation basis for subsequent modules in aspects such as image comparison, text matching, and stability evaluation, and significantly enhances the reliability and adaptability of the unattended operation process.

[0028] Specifically, the specific steps for determining the control accuracy of the interface state data are as follows: Obtain the click deviation distance, response delay time, and click speed variance, which respectively reflect the spatial accuracy of the operation, the response timeliness of the system feedback, and the stability of the user behavior; The system performs a ratio operation on the click deviation distance and the response delay time. This ratio reflects the deviation trend of the operation position per unit time, and then squares this ratio to magnify the influence degree of the deviation by weighting; Subsequently, multiply this squared value by the behavior quality factor, which is automatically adjusted by Bayesian optimization according to the fitting relationship between the deviation speed term and the success rate in a large number of operation samples, and the value range is 1.3 - 3.0; The obtained result is then multiplied by the constant one-half to form the deviation speed suppression term, which is used to measure the negative impact on the operation control quality under the combined action of the position deviation and the response delay; Then the system obtains the click speed variance, which represents the stability of the speed change during continuous clicks. Take the natural logarithm after adding one to it to suppress the influence of extreme values, and then multiply by the instability adjustment coefficient to form the speed fluctuation suppression term. The instability adjustment coefficient is calculated through the covariance matrix between the speed variance and the interface response abnormality rate, and is adaptively converged and generated during the minimization process of the loss function, and the value range is 0.6 - 2.0; Finally, add the deviation speed suppression term and the speed fluctuation suppression term as the negative exponential part of the exponential function, which is used to depict the overall attenuation degree of the operation control quality, and further calculate the interface control accuracy value through the exponential function, reflecting the comprehensive performance of the operation behavior in aspects such as spatial accuracy, time responsiveness, and behavior stability.

[0029] The specific calculation formula for the interface control accuracy value is: ; In the formula, represents the interface control accuracy value, represents the behavior quality factor, represents the click deviation distance, represents the response delay time, represents the instability adjustment coefficient, represents the click speed variance.

[0030] As shown in Table 1, it is a data table of interface control accuracy values provided by the embodiments of the present application. In this embodiment, the response delay time of Sample 1 is set to 0.95, the click deviation distance is set to 2.8, and the click speed variance is set to 0.50; the response delay time of Sample 2 is set to 0.85, the click deviation distance is set to 3.2, and the click speed variance is set to 0.30; the response delay time of Sample 3 is set to 0.50, the click deviation distance is set to 4.5, and the click speed variance is set to 0.15; the response delay time of Sample 4 is set to 0.60, the click deviation distance is set to 3.8, and the click speed variance is set to 0.25; the response delay time of Sample 5 is set to 0.20, the click deviation distance is set to 6.5, and the click speed variance is set to 0.01; the response delay time of Sample 6 is set to 0.35, the click deviation distance is set to 5.5, and the click speed variance is set to 0.05; the response delay time of Sample 7 is set to 0.75, the click deviation distance is set to 2.5, and the click speed variance is set to 0.40; the response delay time of Sample 8 is set to 0.25, the click deviation distance is set to 7.0, and the click speed variance is set to 0.02; the response delay time of Sample 9 is set to 0.40, the click deviation distance is set to 3.0, and the click speed variance is set to 0.35; the response delay time of Sample 10 is set to 0.18, the click deviation distance is set to 6.5, and the click speed variance is set to 0.03.

[0031] Table 1 Data Table of Interface Control Accuracy Values

[0032] As Figure 2 shown, it is a distribution diagram of interface control accuracy values provided by the embodiments of the present application. According to Table 1 and the bar chart in the figure, the interface control accuracy value of Sample 5 is the highest, reaching 0.9872, indicating that under the conditions of the shortest response delay time, a relatively large click deviation but stable speed, the optimal operation accuracy is achieved; the first-level accuracy threshold set in the figure is 0.80, and the second-level accuracy threshold is 0.55. The interface control accuracy value of Sample 1 is 0.5479, slightly higher than the second-level accuracy threshold of 0.55, indicating that there are problems of response lag and unstable operation, and a remedial strategy needs to be executed. From the overall trend, the accuracy values of Sample 3, Sample 5, Sample 6, Sample 8, and Sample 10 all exceed the first-level accuracy threshold and belong to the high-precision interval; Sample 2, Sample 4, and Sample 9 are in the medium-precision interval; Sample 1 and Sample 7 are close to or slightly lower than the second-level threshold, with obvious operation errors, and the system needs to execute auxiliary recognition and path correction measures. The higher the interface control accuracy value, the better the performance of the current operation in terms of spatial click accuracy, behavior stability, and system response consistency, and the higher the confidence level of the system's behavior determination.

[0033] In this implementation, by jointly modeling key behavioral characteristics such as click deviation distance, response delay time, and click speed variance, the system can quantify the spatial accuracy, response efficiency, and behavioral stability during operation and output the interface control accuracy value in the form of an exponential function. This indicator effectively integrates the multi-dimensional performance of operation behaviors, improves the discrimination ability of the quality of a single interaction, provides a scientific and quantifiable basis for subsequent feedback comparison and anomaly adjustment, and significantly enhances the intelligent judgment accuracy and adaptive regulation ability of the unattended operation process.

[0034] Specifically, the specific steps for judging the spatial accuracy, behavioral stability, and reliability of system response of an operation according to the control accuracy determination result are as follows: First, the system compares the calculated interface control accuracy value with the set accuracy thresholds in real time. The accuracy thresholds include a primary accuracy threshold and a secondary accuracy threshold, which are used to classify and identify the operation quality level. When the interface control accuracy value is greater than or equal to the primary accuracy threshold, it indicates that the current operation meets the ideal standards in terms of position accuracy, behavioral stability, and feedback response. The system directly enters the next task process without any correction operations. At the same time, the high-quality operation sample is marked into the behavior imitation training set for subsequent model reinforcement learning, and the control recognition error tolerance range is reduced synchronously to improve the refined matching ability, while keeping the current input and output rhythm parameters unchanged to ensure the consistency and continuity of subsequent operations. When the interface control accuracy value is greater than the secondary accuracy threshold but lower than the primary accuracy threshold, it means that there are slight deviations in the operation and the feedback response is slightly unstable. The system will perform a series of fine-tuning operations, including delaying the recognition trigger, performing a slight perturbation click on the center coordinates of the control, and marking the control as a "low-confidence interaction target". At the same time, the sample data is incorporated into the model optimization training set to improve the system's adaptability to boundary-blurred scenarios. When the interface control accuracy value is less than or equal to the secondary accuracy threshold, the system determines that there are serious deviations and unstable risks in the current operation and will immediately trigger an anomaly remediation mechanism, abandon the original click path, switch to an alternative recognition template for re-control positioning, and perform multi-point probing clicks within five pixels above, below, left, and right of the control center to try to restore the effectiveness of the control operation. If the accuracy value is still lower than the threshold for two consecutive times, the system marks the control as a "high-risk interaction area", and the subsequent process node will transfer to the manual verification path to ensure the stable execution of the overall task through human-machine collaboration.

[0035] In this implementation scheme, by comparing the interface control precision value with the multi-level thresholds in real time, the system can accurately identify the reliability level of the operation behavior, and dynamically match different processing strategies according to the judgment result, realizing a full-process response mechanism from high-quality sample archiving, mild deviation correction to abnormal behavior remedy. This step not only improves the system's automatic judgment ability for the quality of a single operation, but also significantly enhances the stability, self-adaptability and fault tolerance of unattended tasks in complex environments, providing a key guarantee for the continuity of subsequent operations and the overall intelligent level of the system.

[0036] Specifically, the specific steps for performing the feedback consistency comparison on the interface state data with qualified control precision judgment results are as follows: The system first obtains the image structure similarity value between the interface screenshots before and after the operation, and the set of recognized texts extracted by OCR recognition and the set of target texts defined in the task template; Subsequently, subtract the image structure similarity from one to obtain the degree of difference reflected by the image, and this difference value reflects the deviation degree between the current interface image and the expected state; Square this difference value to amplify the influence of the difference, and then multiply it by the visual feedback weight coefficient to obtain the image feedback error term. The visual feedback weight coefficient is dynamically allocated in the feature importance ranking according to the information gain of the image structure similarity to the operation success rate, and the value range is 0.6–0.9; At the same time, the system performs an intersection operation on the set of recognized texts and the set of target texts, counts the number of intersection elements and divides it by the total number of elements in the set of target texts to calculate the text matching ratio, which reflects the content consistency at the text level. Then, subtract the ratio after taking the absolute value from one to obtain the degree of difference in text feedback; Square this difference value and multiply it by the text feedback weight coefficient to obtain the text feedback error term. The text feedback weight coefficient is extracted based on the gradient response interval of the OCR text similarity to the task completion situation in the actual task, and the value range is 0.1–0.4; Finally, the system adds the image feedback error term and the text feedback error term, and performs a square root operation on the sum result, and finally outputs the execution feedback consistency value as a unified quantitative index for comprehensively evaluating the feedback consistency in the two dimensions of image and text, which is used to judge whether the current operation really takes effect in the future.

[0037] The specific calculation formula for the execution feedback consistency value is: ; In the formula, represents the execution feedback consistency value, represents the visual feedback weight coefficient, represents the image structure similarity, represents the text feedback weight coefficient, represents the set of recognized texts, represents the set of target texts.

[0038] In this implementation, this step constructs a feedback consistency evaluation model by jointly analyzing the image structure similarity and text matching degree, which can accurately judge whether the interface state meets the expectations from both visual and semantic dimensions. Through the weighted fusion of image differences and text errors, the system quantitatively outputs the execution feedback consistency value, which serves as the key basis for judging whether the operation takes effect, significantly improving the accuracy and reliability of automated feedback confirmation in the unattended process.

[0039] Specifically, the specific steps for judging whether the interface state is consistent with the expectations according to the execution feedback consistency comparison result are as follows: The system compares the execution feedback consistency value with the feedback thresholds in real time. The feedback thresholds include the primary feedback threshold and the secondary feedback threshold, which are used to divide the high, medium, and low levels of feedback effects respectively; When the execution feedback consistency value is less than or equal to the primary feedback threshold, it indicates that the current interface image structure is highly matched, and the text recognition result is basically consistent with the expected content. The system determines that the operation has taken effect successfully, continues to execute the subsequent task process, and marks the current recognition template as a valid template for subsequent operation reuse and model reinforcement training; When the execution feedback consistency value is greater than the primary feedback threshold and less than or equal to the secondary feedback threshold, it indicates that there is a medium degree of deviation in the feedback, which may be caused by factors such as delayed interface image refresh or incomplete OCR recognition. The system automatically triggers a series of fault-tolerant strategies, including appropriately delaying the recognition start time, expanding the boundary of the image comparison area, reducing the image structure similarity judgment threshold, and enabling auxiliary recognition mechanisms such as keyword-assisted extraction and layout correction mechanisms. At the same time, the image screenshots, text sets, and difference indicators related to this round of operation are archived for subsequent error analysis and model optimization; When the execution feedback consistency value is greater than the secondary feedback threshold, the system determines that the operation has not taken effect and immediately automatically executes the retry operation process, and continues to try the operation before the preset retry times are exceeded; If the retry times reach the upper limit, the system will roll back this process step to the previous task node and activate the image fault-tolerant mechanism, and try to reconfirm the control state using methods such as strong template matching and contour enhancement; Finally, the relevant recognition images, extracted texts, control information, and failure labels are packaged and archived as high-risk error samples for model training and strategy adjustment.

[0040] In this implementation, this step can accurately judge whether the interface state is consistent with the expectations by setting feedback consistency thresholds and performing multi-dimensional comparisons on the interface image structure and text recognition results after the operation. The system automatically triggers the recognition fault-tolerant, retry rollback, and error sample archiving mechanisms according to the level of the feedback consistency value, realizing the adaptive response and intelligent repair of abnormal feedback, and effectively improving the operation success rate and overall robustness of unattended tasks in complex scenarios.

[0041] Specifically, maintain failure records and attempt count limits to avoid infinite retries, and adaptively match processing methods according to failure types to achieve basic exception tolerance control. The specific steps are as follows: After an operation fails, the system immediately calls the log recording component to record in detail the failure type corresponding to the current operation, the unique identifier of the control, the failure timestamp, and the current task context information, and synchronously updates the failure count counter of the corresponding control to ensure that historical failure data is immediately available for query; When the cumulative failure count of a certain operation control does not exceed the maximum attempt count threshold set by the system, the system automatically matches and executes predefined regulation strategies according to the failure reason, including performing slight position perturbation clicks within the central coordinate range of the control to cover possible offset errors, dynamically extending the time interval between the input action and the response determination to alleviate interface latency problems, switching to an alternative image template to re-identify the control graphic features, or directly starting an alternative task path to bypass high-failure nodes; When the failure count reaches the attempt upper limit threshold, the system immediately stops further operation attempts for this control and marks it as a high-risk interaction target. At the same time, it archives the operation data, failure logs, attempt counts, exception contexts, and executed regulation strategies related to this control into the system failure database; Subsequently, the system dynamically constructs an exception handling diversion path according to the failure type. For example, it transfers the interface recognition exception task to the enhanced recognition module, pushes the operation response sluggish task to the task scheduling module for rhythm reconstruction, or incorporates the error-prone control recognition samples into the model training data pool to achieve a robust operation system with uninterrupted processes and basic fault tolerance capabilities.

[0042] In this implementation plan, this step achieves intelligent fault tolerance and dynamic diversion control of abnormal tasks by accurately recording and limiting the type, count, and time of operation failures, and adaptively matching regulation strategies according to the failure reason. On the one hand, it avoids resource waste and system blockage caused by infinite retries. On the other hand, through high-risk marking and failure sample archiving, it provides data support for subsequent model optimization and process reconstruction, significantly improving the stability and fault response ability of the unattended operation process.

[0043] Specifically, by comprehensively integrating the interface status data, the control precision determination result, and the execution feedback consistency comparison result, the specific steps for conducting the stable assessment of task completion are as follows: First, obtain the interface control precision value, the number of failure retries, and the OCR recognition confidence corresponding to the operation. The interface control precision value is used to quantify the operation precision and response stability, the number of failure retries reflects the operation fault tolerance requirement, and the OCR recognition confidence measures the reliability of the text recognition result. Then, take the reciprocal of the interface control precision value to represent its uncertainty; take the logarithm after adding one to the number of failure retries to represent the impact of repeated attempts on the system load; add the OCR recognition confidence to a very small correction term and then take the reciprocal. The very small correction term is used to avoid division-by-zero errors when the OCR matching confidence is zero. The system automatically sets it after statistical distribution analysis of the error fluctuations in the historical minimum value samples during the training phase, and its value range is , avoiding mathematical anomalies caused by a zero recognition confidence and reflecting the risk level of the recognition quality; then add these three results together, and then add one to form the evaluation denominator. Finally, take the reciprocal of the entire sum result to obtain the task completion stability value, which is used as a unified quantitative indicator for judging the execution quality of the current unattended task.

[0044] The specific calculation formula for the task completion stability value is: ; In the formula, represents the task completion stability value, represents the interface control precision value, represents the number of failure retries, represents the recognition confidence, represents the very small correction term.

[0045] In this implementation plan, this step constructs the task completion stability value index by integrating three types of key data: interface control precision, operation retry frequency, and OCR recognition confidence, comprehensively reflecting the execution quality and process reliability of an unattended task. This index not only quantifies the stability of operation behavior and the effectiveness of feedback, but also combines the failure risk and recognition accuracy to form a comprehensive evaluation result, providing a unified basis for system decision-making. With the help of this evaluation result, the system can automatically judge whether the task is reliably completed, and accordingly optimize path recording, sample annotation, and strategy adjustment to achieve dynamic monitoring and continuous improvement of the execution quality of unattended tasks.

[0046] Specifically, the specific steps to determine whether a complete unattended task is stable and reliable based on the task completion stability evaluation result are as follows: Compare the task completion stability value with the stability threshold in real time. The stability threshold includes a primary stability threshold and a secondary stability threshold. When the task completion stability value is greater than or equal to the primary stability threshold, the system determines that the current task execution is stable and reliable, indicating high operation behavior accuracy, accurate feedback results, and good OCR recognition confidence. The system marks the task process as a high-quality path and incorporates its complete operation record into the system sample library as a key data source for subsequent task imitation, recognition template optimization, and parameter training. When the task completion stability value is greater than the secondary stability threshold and less than the primary stability threshold, the system believes that there are slight fluctuations in the task execution, which may affect the overall quality due to unstable recognition or operation retries. The system automatically triggers the parameter optimization process of the OCR recognition model and annotates and classifies the current process path. If the recognition confidence is lower than the set threshold, the path is marked as "unstable text recognition". If the number of operation retries reaches the limit value, it is marked as "structural improvement path" and feedback to the task flow editing module for structural rule revision and process update. When the task completion stability value is less than or equal to the secondary stability threshold, the system determines that there is a risk of abnormal execution or logical interruption of the task, and will automatically re-initialize the current process and execute a complete task again. If it still fails after re-execution, the emergency exit mechanism will be triggered, a structured operation failure report will be generated, and the relevant recognition images, feedback feature values, and error sample data will be packaged and uploaded as high-risk path samples for subsequent model training and key scenario optimization, so as to enhance the adaptability and robustness of the system to complex tasks.

[0047] In this implementation plan, this step aims to evaluate the stability of the entire process of task execution to determine whether the current unattended task is reliable and sustainable. After the system obtains the interface control accuracy value, the number of failure retries, and the OCR recognition confidence, it constructs a stability calculation expression, normalizes and comprehensively processes each index, and compares the calculation result with the set threshold to dynamically divide the task stability level. Through result determination, high-quality execution paths can be effectively screened, potential abnormal processes can be identified, and decision-making basis can be provided for subsequent task template optimization and fault handling mechanisms, thereby improving the stable operation ability of the system in multi-task and multi-scenario environments.

[0048] Such as Figure 2As shown in the figure, it is a schematic structural diagram of an unattended computer operating system based on machine vision provided by an embodiment of the present application. This embodiment provides an unattended computer operating system based on machine vision, which is applied to the above-mentioned unattended computer operation method based on machine vision. The overall system is constructed with a closed-loop control logic as the core to ensure the capabilities of state perception, dynamic response, and performance evaluation during the task execution process. The system includes an interface data acquisition module, an automatic behavior execution module, a result feedback judgment module, an abnormal regulation and remedy module, and a process record optimization module; The interface data acquisition module is used to collect the interface state data before and after the operation, including image screenshots, control states, and text recognition content, and preprocess the interface state data in a unified format to provide standardized input features for subsequent analysis; The automatic behavior execution module is used to determine the control accuracy of the interface state data. By calculating indicators such as click deviation, response delay, and behavior stability, the interface control accuracy value is obtained. According to the control accuracy determination result, the spatial accuracy, behavior stability, and reliability of the system response of an operation are judged, and the corresponding task operation is executed; The result feedback judgment module is used to perform a feedback consistency comparison on the interface state data with qualified control accuracy determination results. The feedback consistency score is obtained by fusing the structural similarity and the OCR text matching results. According to the result of the feedback consistency comparison, it is judged whether the interface state is consistent with the expectation, and the subsequent operation strategy is assisted in adjustment; The abnormal regulation and remedy module is used to maintain the failure records and the limit of the number of attempts. When an operation anomaly is detected, it automatically executes remedy mechanisms such as click fine-tuning, input rhythm change, template replacement, etc., to avoid infinite retries, and adaptively matches the processing method according to the failure type to achieve basic anomaly fault tolerance control and process sustainability; The process record optimization module is used to comprehensively consider the interface state data, the control accuracy determination result, and the result of the feedback consistency comparison. Based on a unified index, the task completion stability value is calculated, and the task completion stability evaluation is carried out. According to the task completion stability evaluation result, it is judged whether a complete unattended task is stable and reliable, providing a key reference basis for system model iteration and process optimization.

[0049] In this implementation plan, the core role of this step is to comprehensively evaluate the overall performance of the unattended task operation behavior through multi-source information fusion and module collaborative processing, and finally output a comprehensive index reflecting the task execution stability. Its effect is that it can dynamically reflect the comprehensive performance of key factors such as operation accuracy, feedback consistency, and abnormal recovery, so as to accurately judge the reliability and execution success rate of the entire automated task process, provide a clear basis for system optimization, adaptive adjustment, and high-risk process identification, and improve the overall task stability and automation level.

[0050] It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variation thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device.

[0051] The preferred embodiments of the present invention disclosed above are only used to assist in the description of the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the present invention, so that those skilled in the art can well understand and utilize the present invention. The present invention is only limited by the claims and their full scope and equivalents.

Claims

1. An unattended computer operation method based on machine vision, characterized in that, It includes the following steps: S1: Collect the interface state data before and after the operation, and preprocess the interface state data; S2: Determine the control accuracy of the interface state data, and judge the spatial accuracy, behavior stability and system response reliability of an operation according to the control accuracy determination result; S3: Perform a feedback consistency comparison on the interface state data with qualified control accuracy determination results, and judge whether the interface state is consistent with the expectation according to the execution feedback consistency comparison result; S4: Maintain the failure record and the attempt number limit, avoid infinite retries, and adaptively match the processing method according to the failure type to achieve basic exception tolerance control; S5: Comprehensive interface state data, control accuracy determination results and execution feedback consistency comparison results, carry out a stable evaluation of task completion, and judge whether a complete unattended task is stable and reliable according to the task completion stable evaluation result.

2. The unattended computer operation method based on machine vision according to claim 1, characterized in that: The specific steps of collecting the interface state data before and after the operation and preprocessing the interface state data are as follows: The interface state data includes: response delay time, recognized text dataset, and failure retry times; Collect the current screen image and save the screenshot for subsequent structural similarity comparison; Obtain the state attribute information of the control, including auxiliary judgment information such as color change, focus state, and display and hide flags; Obtain the response delay time by subtracting the click timestamp recorded by the system from the time timestamp of the interface state change; Extract the text content of the specified control area, and use the OCR recognition technology to form a recognized text set; Write the retry record after each operation failure into the regulation log and record it as the failure retry times; Preprocess the interface state data, and the preprocessed data includes: click deviation distance, click speed variance, target text set, image structure similarity, and OCR recognition confidence; Perform size normalization and resolution correction on the screenshot image to ensure the structural consistency of the images collected by different terminals; Perform image enhancement and edge smoothing processing on the control area image; Perform unified language encoding, special character cleaning and space merging on the recognized text to form a structured text for subsequent comparison; Calculate the click deviation distance from the Euclidean distance between the center coordinates of the control and the actual click position coordinates; Calculate the average speed by combining the time series and displacement series of multiple click events, and then calculate the click speed variance through the sliding window variance formula; Obtain the target text set based on the preset field matching rules in the task template; Input the pre-operation and post-operation images obtained by screenshot into the image structure comparison function, and calculate the image structure similarity using the local image analysis algorithm; Compare and analyze the character differences between the recognized text and the target text, and perform normalization processing on the matching degree to obtain the OCR recognition confidence; Record the calculated interface control accuracy value, execution feedback consistency value and task completion stability value in real time.

3. The unattended computer operation method based on machine vision according to claim 1, characterized in that: The specific steps of determining the control accuracy of the interface state data are as follows: Obtain the click deviation distance, response delay time and click speed variance; Calculate the ratio of the click deviation distance to the response delay time, square the ratio, multiply the result by the behavior quality factor, and then multiply the obtained result by the constant one-half to get the deviation velocity suppression term; calculate the click velocity variance, take the natural logarithm after adding one to it, and multiply by the instability adjustment coefficient to get the velocity fluctuation suppression term; Subsequently, add the deviation velocity suppression term and the velocity fluctuation suppression term as the negative exponent part of the exponential function to calculate the interface control accuracy value.

4. A machine vision-based unattended computer operation method according to claim 1, characterized in that: The specific steps for judging the spatial accuracy, behavior stability, and system response reliability of an operation according to the control accuracy determination result are as follows: Compare the interface control accuracy value with the accuracy thresholds in real time. The accuracy thresholds include a first-level accuracy threshold and a second-level accuracy threshold; When the interface control accuracy value is greater than or equal to the first-level accuracy threshold, directly enter the next task process, mark the operation sample as a high-quality template, include it in the behavior imitation training set, and at the same time reduce the control recognition error tolerance range and keep the current rhythm parameters unchanged; When the interface control accuracy value is greater than the second-level accuracy threshold and less than the first-level accuracy threshold, there are slight deviations and unstable responses. Perform recognition trigger delay, control center perturbation click, and confidence marking operations, and at the same time record the control as a low-confidence interaction target for subsequent model iteration and optimization; When the interface control accuracy value is less than or equal to the second-level accuracy threshold, trigger the abnormal remedy mechanism, abandon the original click path, switch to the backup recognition template to reposition the control, and at the same time perform multi-point probing clicks within five pixels above, below, left, and right of the control center. If the accuracy value is still lower than the threshold for two consecutive times, mark the control as a high-risk interaction area and transfer it to the manual verification path.

5. A machine vision-based unattended computer operation method according to claim 1, characterized in that: The specific steps for performing the feedback consistency comparison on the interface state data with qualified control accuracy determination results are as follows: Obtain the image structure similarity, the recognized text set, and the target text set; Subtract the image structure similarity from one to get the difference degree of the image feedback; square the difference value and multiply by the visual feedback weight coefficient to get the image feedback error term; calculate the number of intersection elements between the recognized text set after the operation and the expected target text set, divide it by the total number of the target text set, get the text matching ratio, and then take the absolute value of the ratio; subtract the absolute value of the text matching ratio from one to get the difference degree of the text feedback; square the difference value and multiply by the text feedback weight coefficient to get the text feedback error term; add the image feedback error term and the text feedback error term, and take the square root of the sum result to get the execution feedback consistency value.

6. A machine vision-based unattended computer operation method according to claim 1, characterized in that: The specific steps for judging whether the interface state is consistent with the expectation according to the execution feedback consistency comparison result are as follows: Compare the execution feedback consistency value with the feedback thresholds in real time. The feedback thresholds include a first-level feedback threshold and a second-level feedback threshold; When the execution feedback consistency value is less than or equal to the first-level feedback threshold, both the interface image structure and the text feedback meet the expectation, continue to execute the subsequent operation process, and mark the current recognition template as an effective template; When the execution feedback consistency value is greater than the first-level feedback threshold and less than or equal to the second-level feedback threshold, it is determined that there is a certain deviation. Strategies such as recognition delay adjustment, relaxation of the image comparison range, and downward adjustment of the comparison threshold are executed. At the same time, the auxiliary recognition mechanism is enabled, and the images, texts, and difference information of this round are archived. When the execution feedback consistency value is greater than the second-level feedback threshold, the operation is ineffective, and the operation retry process is automatically executed. If the number of retries exceeds the limit, it will roll back to the previous task step, and the image fault tolerance mechanism is activated to try to confirm the control state again. At the same time, the recognized image, text results, and control information are packaged and archived as error samples for subsequent analysis and model training.

7. A machine vision-based unattended computer operation method according to claim 1, characterized in that: The maintenance of failure records and the limit of the number of attempts are used to avoid infinite retries, and the processing method is adaptively matched according to the failure type to achieve basic exception fault tolerance control. The specific steps are as follows: After the operation fails, first record the failure type, control identifier, and failure timestamp of the current operation, and update the corresponding failure count counter. When the number of failures of a certain operation control does not exceed the preset maximum attempt threshold, corresponding regulation strategies are executed according to the failure reason type, including fine-tuning of the click position, adjustment of the input rhythm, switching of the backup template, and jumping of the backup path. When the number of failures reaches the attempt limit, further retries are stopped, and the control is marked as a high-risk object. At the same time, the relevant operation data, failure logs, and recorded attempted strategies are archived. According to different failure types, the adjustment method is adaptively selected to construct an abnormal handling diversion path to achieve the goal of uninterrupted control flow and basic operation fault tolerance.

8. A machine vision-based unattended computer operation method according to claim 1, characterized in that: The comprehensive interface state data, control accuracy determination result, and execution feedback consistency comparison result are used to carry out the specific steps of the stable evaluation of task completion as follows: Obtain the interface control accuracy value, the number of failure retries, and the OCR recognition confidence. Take the reciprocal of the interface control accuracy value, take the logarithm after adding one to the number of failure retries, and then take the reciprocal after adding the OCR recognition confidence to the minimum correction term; add the above three items and then add one, and finally take the reciprocal of the overall result to obtain the task completion stability value.

9. A machine vision-based unattended computer operation method according to claim 1, characterized in that: The specific steps for judging whether a complete unattended task is stable and reliable according to the task completion stability evaluation result are as follows: Compare the task completion stability value with the stability threshold in real time. The stability threshold includes the first-level stability threshold and the second-level stability threshold. When the task completion stability value is greater than or equal to the first-level stability threshold, the task execution accuracy is high, the feedback is accurate, and the recognition result is credible. The current process is marked as a high-quality path, and the execution record is incorporated into the sample library for subsequent task imitation and optimization reference. When the task completion stability value is greater than the second-level stability threshold and less than the first-level stability threshold, the task has slight fluctuations, and the recognition parameter optimization, path marking, and task identification processes are automatically triggered. If the recognition confidence is lower than the threshold, it is marked as unstable text recognition. If the number of retries reaches the threshold, it is marked as a structural improvement path and feedback to the task flow editing module. When the task completion stability value is less than or equal to the secondary stability threshold, the overall task is abnormal and there is a risk of interruption. The current process is automatically re-initialized and re-executed. If it still fails, the emergency exit mechanism is triggered and an operation failure report is generated. At the same time, the recognized image, result features, and error samples are recorded as high-risk path samples for subsequent training and key optimization.

10. A machine vision-based unattended computer operating system according to claim 1, characterized in that: It includes an interface data acquisition module, an automatic behavior execution module, a result feedback judgment module, an abnormal regulation and remedy module, and a process record and optimization module; The interface data acquisition module is used to collect the interface state data before and after the operation and preprocess the interface state data; The automatic behavior execution module is used to determine the control accuracy of the interface state data, and judge the spatial accuracy, behavior stability, and reliability of system response of an operation according to the control accuracy determination result; The result feedback judgment module is used to perform a feedback consistency comparison on the interface state data with qualified control accuracy determination results, and judge whether the interface state is consistent with the expectation according to the execution feedback consistency comparison result; The abnormal regulation and remedy module is used to maintain the failure record and the limit of the number of attempts, avoid infinite retries, and adaptively match the processing method according to the failure type to achieve basic abnormal fault tolerance control; The process record and optimization module is used to comprehensively evaluate the task completion stability based on the interface state data, the control accuracy determination result, and the execution feedback consistency comparison result, and judge whether a complete unattended task is stable and reliable according to the task completion stability evaluation result.

Images