An unattended computer operating system and method based on machine vision

Abstract

The present invention discloses an unattended computer operating system and method based on machine vision, and relates to the field of electronic data processing technology. The unattended computer operating system and method based on machine vision include the following steps: S1: collecting interface status data before and after an operation and performing preprocessing; S2: performing control accuracy determination on the interface status data; S3: performing feedback consistency comparison on the interface status data with qualified control accuracy determination results; S4: maintaining failure records and attempt limit to avoid infinite retries; S5: comprehensively comparing the interface status data, control accuracy determination results, and execution feedback consistency comparison results to conduct a task completion stability assessment. This solves the problem of lacking an effective feedback confirmation mechanism after an operation is executed and being unable to determine whether the operation has actually taken effect.

Description

Technical Field

[0001] The present invention relates to the field of electronic data processing, and in particular to an unattended computer operating system and method based on machine vision. Background Art

[0002] With the development of artificial intelligence and automation, machine vision, as a key means of image perception and understanding, has been widely used in fields such as industrial inspection and intelligent driving, and has gradually expanded to the recognition and interactive control of complex interfaces. At the same time, unattended operating systems (OSs) have shown great potential in remote control and automated operations and maintenance. However, existing solutions generally lack the ability to interpret operational feedback in real time, making it difficult to cope with interface changes or recognition errors, which limits stability and accuracy.

[0003] For example, the invention patent with announcement number CN103235750B discloses a computer unattended control system, including an intelligent control chip, a storage module, an external power supply module, an internal power supply module, and a computer work feedback module. The intelligent control chip includes a detection module, a power control module and a storage read-write module respectively connected to the central processing unit. The detection module is connected to the external power supply module, and judges the computer power access status and computer working status through level detection technology and the external power supply module and the computer work feedback module, and filters interference signals; the power control module is connected to the internal power supply module; the storage read-write module is connected to the storage module; a computer unattended control method is also disclosed, which realizes that the computer can automatically start up and run when a call comes in, and no manual operation is required to start up when a call comes in, which greatly reduces labor costs, expands the application range of computers, and is easy to use.

[0004] For example, the invention patent with publication number CN117076036A discloses a method and system for unattended RPA processes on remote computers. The method includes: S1, starting the RPA robot, preventing the remote computer from entering a dormant state by setting a system function, and performing an operating environment test before the RPA robot starts executing the RPA process; the operating environment test includes resolution detection, session detection, system clock detection, interface availability detection, user settings detection, and power mode detection; S2, after completing the operating environment test, when the RPA robot recognizes that the remote computer session has been exited or the interface has stopped rendering, the RPA robot immediately enables unattended operation, making the remote desktop active, and ensuring the normal operation of the RPA process on the remote computer.

[0005] Existing unattended computer operations often rely on automated methods such as script playback, template matching, and coordinate clicking. Some solutions also utilize optical character recognition (OCR) for control confirmation. However, these methods are prone to failure when the interface changes dynamically, control offsets occur, and feedback anomalies occur. This makes it difficult to achieve flexible response and high-precision control, impacting both the success rate of operations and the quality of task completion.

[0006] Therefore, in response to the above problems, there is an urgent need for an unattended computer operating system and method based on machine vision. Summary of the Invention

[0007] Technical problems solved

[0008] In response to the shortcomings of the existing technology, the present invention provides an unattended computer operating system and method based on machine vision, which solves the problem of lack of an effective feedback confirmation mechanism after the operation is executed and the inability to determine whether the operation is actually effective.

[0009] Technical solution. To achieve the above objectives, the present invention is implemented through the following technical solution: an unattended computer operating system and method based on machine vision, comprising the following steps: S1: collecting interface status data before and after the operation, and preprocessing the interface status data; S2: performing control accuracy judgment on the interface status data, and judging the spatial accuracy, behavioral stability and reliability of the system response of an operation based on the control accuracy judgment result; S3: performing feedback consistency comparison on the interface status data with qualified control accuracy judgment results, and judging whether the interface status is consistent with expectations based on the execution feedback consistency comparison result; S4: maintaining failure records and attempt number limits to avoid infinite retries, and adaptively matching processing methods according to failure types to achieve basic abnormal fault-tolerant control; S5: comprehensively analyzing the interface status data, control accuracy judgment results and execution feedback consistency comparison results, conducting task completion stability assessment, and judging whether a complete unattended task is stable and reliable based on the task completion stability assessment result.

[0010] Furthermore, the interface state data before and after the operation is collected, and the specific steps of pre-processing the interface state data are as follows: the interface state data includes: response delay time, recognition text data set and number of failed retries; the current screen image is collected and a screenshot is saved for subsequent structural similarity comparison; the state attribute information of the control is obtained, including auxiliary judgment information such as color change, focus state and display and hide flags; the response delay time is obtained by subtracting the click timestamp recorded by the system from the interface state change completion timestamp; the text content of the specified control area is extracted, and the recognition text set is formed using OCR recognition technology; the retry record after each operation failure is written into the control log and recorded as the number of failed retries; the interface state data is pre-processed, and the pre-processed data includes: click deviation distance, click speed variance, target text set, image structure similarity and OCR recognition confidence; The screenshot images are normalized in size and corrected in resolution to ensure structural consistency of images captured by different terminals; image enhancement and edge smoothing are performed on the control area images; the recognized text is unified in language encoding, special characters are cleaned, and spaces are merged to form structured text for subsequent comparison; the click deviation distance is calculated by the Euclidean distance between the control center coordinates and the actual click position coordinates; the average speed is calculated by combining the time series and displacement series of multiple click events, and the click speed variance is calculated using the sliding window variance formula; the target text set is obtained based on the preset field matching rules in the task template; the images before and after the operation obtained from the screenshots are input into the image structure comparison function, and the image structure similarity is calculated using the local image analysis algorithm; the OCR recognition confidence is obtained by comparing and analyzing the character differences between the recognized text and the target text, and the matching degree is normalized; the calculated interface control accuracy value, execution feedback consistency value, and task completion stability value are recorded in real time.

[0011] Furthermore, the specific steps for determining the control accuracy of the interface state data are as follows: obtaining the click deviation distance, response delay time and click speed variance; calculating the ratio of the click deviation distance to the response delay time, squaring the ratio and multiplying it by the behavior quality factor, and then multiplying the result by half of the constant to obtain the deviation speed suppression term; calculating the click speed variance, adding one to it and taking the natural logarithm, and multiplying it by the instability adjustment coefficient to obtain the speed fluctuation suppression term; then adding the deviation speed suppression term and the speed fluctuation suppression term as the negative exponential part of the exponential function to calculate the interface control accuracy value.

[0012] Furthermore, the specific steps for determining the spatial accuracy, behavioral stability, and reliability of system response of an operation based on the control accuracy determination result are as follows: comparing the interface control accuracy value with the accuracy threshold in real time, wherein the accuracy threshold includes a first-level accuracy threshold and a second-level accuracy threshold; when the interface control accuracy value is greater than or equal to the first-level accuracy threshold, directly proceeding to the next task process, marking the operation sample as a high-quality template and incorporating it into the behavior imitation training set, while reducing the control recognition tolerance range and maintaining the current rhythm parameters unchanged; when the interface control accuracy value is greater than the second-level accuracy threshold and less than the first-level accuracy threshold, indicating a slight deviation and unstable response, performing recognition trigger delay, perturbation click on the control center, and confidence marking operations, and recording the control as a low-confidence interaction target for subsequent model iterative optimization; when the interface control accuracy value is less than or equal to the second-level accuracy threshold, triggering the abnormality remediation mechanism, abandoning the original click path, switching to a backup recognition template to reposition the control, and performing multiple tentative clicks within a range of five pixels above, below, left, and right of the control center. If the accuracy value remains below the threshold for two consecutive times, the control is marked as a high-risk interaction area and transferred to the manual verification path.

[0013] Furthermore, the specific steps of performing feedback consistency comparison on the interface state data with qualified control accuracy judgment results are as follows: obtaining image structure similarity, recognition text set and target text set; subtracting the image structure similarity from one to obtain the degree of difference in image feedback; squaring the difference value and multiplying it by the visual feedback weight coefficient to obtain the image feedback error term; calculating the number of intersection elements between the recognition text set and the expected target text set after the operation, and dividing it by the total number of the target text set to obtain the text matching ratio, and then taking the absolute value of the ratio; subtracting the absolute value of the text matching ratio from one to obtain the degree of difference in text feedback; squaring the difference value and multiplying it by the text feedback weight coefficient to obtain the text feedback error term; adding the image feedback error term and the text feedback error term, and taking the square root of the sum to obtain the execution feedback consistency value.

[0014] Furthermore, the specific steps of judging whether the interface state is consistent with expectations based on the execution feedback consistency comparison result are as follows: real-time comparison of the execution feedback consistency value and the feedback threshold, the feedback threshold including the first-level feedback threshold and the second-level feedback threshold; when the execution feedback consistency value is less than or equal to the first-level feedback threshold, the interface image structure and text feedback meet expectations, and the subsequent operation process continues to be executed, and the current recognition template is marked as a valid template; when the execution feedback consistency value is greater than the first-level feedback threshold and less than or equal to the second-level feedback threshold, it is judged that there is a certain deviation, and strategies such as recognition delay adjustment, image comparison range relaxation and comparison threshold lowering are executed, and the auxiliary recognition mechanism is enabled at the same time, and the image, text and difference information of this round are archived; when the execution feedback consistency value is greater than the second-level feedback threshold, the operation does not take effect, and the operation retry process is automatically executed. If the retry exceeds the limit, it will fall back to the previous task step, and activate the image fault tolerance mechanism to try to confirm the control state again. At the same time, the recognition image, text results and control information are packaged and archived as error samples for subsequent analysis and model training.

[0015] Furthermore, the maintenance failure records and attempt limit are avoided to avoid infinite retries, and the processing method is adaptively matched according to the failure type to achieve basic abnormal fault-tolerant control. The specific steps are as follows: when the operation fails, first record the failure type, control identifier and failure timestamp of the current operation, and update the corresponding failure counter; when the number of failures of a certain operation control does not exceed the preset maximum attempt threshold, execute the corresponding control strategy according to the type of failure cause, including fine-tuning of click position, input rhythm adjustment, backup template switching and backup path jump; when the number of failures reaches the attempt upper limit, stop further retries, and mark the control as a high-risk object, and archive the relevant operation data, failure logs and attempted strategy records; adaptively select the adjustment method according to different failure types, build an abnormal processing diversion path, and achieve the goal of uninterrupted control process and basic operation fault tolerance.

[0016] Furthermore, the comprehensive interface status data, control accuracy judgment results and execution feedback consistency comparison results are used to carry out task completion stability assessment. The specific steps are as follows: obtain the interface control accuracy value, the number of failed retries and the OCR recognition confidence; take the inverse of the interface control accuracy value, add one to the number of failed retries and take the logarithm, then add the OCR recognition confidence and the minimum correction term and take the inverse; add the above three items and then add one, and finally take the inverse of the overall result to obtain the task completion stability value.

[0017] Furthermore, the specific steps of judging whether a complete unattended task is stable and reliable based on the task completion stability assessment result are as follows: real-time comparison of the task completion stability value and the stability threshold, the stability threshold includes a first-level stability threshold and a second-level stability threshold; when the task completion stability value is greater than or equal to the first-level stability threshold, the task execution accuracy is high, the feedback is accurate, and the recognition result is credible, the current process is marked as a high-quality path, and the execution record is included in the sample library as a reference for subsequent task imitation and optimization; when the task completion stability value is greater than the second-level stability threshold and less than the first-level stability threshold, the task has slight fluctuations, and the recognition parameter optimization, path labeling and task identification process are automatically triggered. If the recognition confidence is lower than the threshold, it is marked as text recognition instability. If the number of retries reaches the threshold, it is marked as a structural improvement path and fed back to the task flow editing module; when the task completion stability value is less than or equal to the second-level stability threshold, the task is abnormal as a whole and there is a risk of interruption. The current process is automatically reinitialized and re-executed. If it still fails, the emergency exit mechanism is triggered and an operation failure report is generated. At the same time, the recognition image, result features and error samples are recorded as high-risk path samples for subsequent training and key optimization.

[0018] A second aspect of the present invention provides an unattended computer operating system based on machine vision, which is applied to the above-mentioned unattended computer operation method based on machine vision, and includes an interface data acquisition module, an automatic behavior execution module, a result feedback judgment module, an abnormality control and remediation module, and a process record optimization module. The interface data acquisition module is used to collect interface state data before and after an operation and preprocess the interface state data. The automatic behavior execution module is used to perform control accuracy determination on the interface state data and, based on the control accuracy determination result, determine the spatial accuracy, behavioral stability, and reliability of the system response of an operation. The result feedback judgment module is used to perform feedback consistency comparison on the interface state data that meets the control accuracy determination result and, based on the execution feedback consistency comparison result, determine whether the interface state is consistent with the expectation. The abnormality control and remediation module is used to maintain failure records and attempt number limits to avoid infinite retries and adaptively match processing methods based on failure types to achieve basic abnormality fault-tolerant control. The process record optimization module is used to comprehensively analyze the interface state data, control accuracy determination results, and execution feedback consistency comparison results to conduct a task completion stability assessment and, based on the task completion stability assessment result, determine whether a complete unattended task is stable and reliable.

[0019] Beneficial effects

[0020] The present invention has the following beneficial effects:

[0021] (1) The present invention collects image and text status information before and after the operation and combines it with a multi-dimensional data preprocessing method to make the interface state expression more structured and standardized, providing a unified and reliable feature basis for subsequent control accuracy judgment and feedback consistency comparison, and significantly improving the adaptability and versatility of the system in multiple scenarios.

[0022] (2) The present invention introduces a control accuracy evaluation mechanism, comprehensively considers click deviation, response delay and behavior stability, and uses an exponential function to construct a nonlinear accuracy scoring model to achieve accurate quantification of the execution quality of a single operation, thereby improving the reliability and adjustability of the judgment of the operation behavior.

[0023] (3) In the feedback judgment process, the present invention combines the dual information of image structure changes and text recognition results, and introduces the feedback consistency value as the judgment basis, which effectively solves the problem of single-dimensional judgment errors in traditional systems and enhances the system's intelligent recognition ability of whether the operation is truly effective.

[0024] (4) The present invention constructs a task completion stability value evaluation model, which integrates operation accuracy, feedback accuracy and retry cost into a comprehensive evaluation system, realizes dynamic judgment of the operation quality of complete unattended tasks, and helps to build a stable and reliable task execution closed loop.

[0025] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of an unattended computer operation method based on machine vision of the present invention;

[0027] Figure 2 This is a structural diagram of an unattended computer operating system based on machine vision of the present invention;

[0028] Figure 3 A histogram of the interface control accuracy values of the present invention; DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0030] See also Figure 1-Figure 3The embodiment of the present invention provides a technical solution: an unattended computer operating system based on machine vision, which is applied to the above-mentioned unattended computer operation method based on machine vision, including S1: collecting interface status data before and after the operation, and preprocessing the interface status data; S2: performing control accuracy judgment on the interface status data, and judging the spatial accuracy, behavioral stability and reliability of the system response of an operation according to the control accuracy judgment result; S3: performing feedback consistency comparison on the interface status data with qualified control accuracy judgment results, and judging whether the interface status is consistent with expectations according to the execution feedback consistency comparison result; S4: maintaining failure records and attempt number limits to avoid infinite retries, and adaptively matching processing methods according to failure types to achieve basic abnormal fault-tolerant control; S5: comprehensively comparing the interface status data, control accuracy judgment results and execution feedback consistency comparison results, conducting task completion stability assessment, and judging whether a complete unattended task is stable and reliable according to the task completion stability assessment result.

[0031] Specifically, the interface status data before and after the operation is collected, and the interface status data is preprocessed. The specific steps are as follows: the interface status data includes: response delay time, recognition text data set and number of failed retries; the system first collects the current screen image and saves it as the basic data for subsequent image structure similarity comparison, and simultaneously records the system timestamp when taking the screenshot for use in time series comparison; obtains the status attribute information of the control, including visual auxiliary signals such as color change, focus state and show and hide signs, to assist in judging whether the operation behavior is effectively perceived by the system; completes the timestamp subtraction by the click timestamp recorded by the system and the interface status change, Obtain the operation response delay time, which reflects the timeliness of the interface's feedback to user input; extract the text content of the specified control area, use OCR recognition technology to form a recognized text set, support multi-language recognition and character cleaning, and enhance adaptability; write the retry record after each operation failure into the control log, automatically accumulate the number of failures and bind the operation target control ID, and record it as the number of failed retries; preprocess the interface status data, and the preprocessed data includes: click deviation distance, click speed variance, target text set, image structure similarity and OCR recognition confidence; in terms of image processing, perform size normalization and resolution correction on the screenshot image, Ensure that images from devices with different resolutions are comparable and structurally consistent; perform image enhancement and edge smoothing on the control area image to improve boundary clarity and text recognition quality; in text data processing, perform language encoding unification, special character cleaning and space merging operations on the recognized text to generate structured text for subsequent comparison; the click behavior related data is calculated by the Euclidean distance between the control center coordinates and the actual click position coordinates to obtain the click deviation distance, which is used to measure the operation accuracy; at the same time, the system analyzes the time series and displacement series of multiple click behaviors within the set time window, calculates the average click speed, and then applies the sliding window method The difference formula is used to calculate the click speed variance, reflecting the stability of the operation process; the target text set is dynamically generated based on the field matching rules in the task template to ensure the clarity and consistency of the operation goals; the image structure similarity is input into the structure comparison function by the screenshot of the images before and after the operation, and the calculation is completed by the local image analysis algorithm; the OCR recognition confidence is obtained by comparing and analyzing the character differences between the recognized text and the target text, using the edit distance algorithm and normalization processing; finally, the calculated interface control accuracy value, execution feedback consistency value and task completion stability value are recorded in real time to provide high-quality quantitative basis for subsequent evaluation, regulation and optimization.

[0032] In this implementation, multi-dimensional acquisition and structured preprocessing of interface state data before and after an operation not only achieves standardized representation of images, text, and state attributes, but also provides high-quality, uniformly formatted input features for subsequent accuracy assessment and feedback consistency analysis. This step effectively improves the system's perception of operational behavior and the clarity of data representation, ensuring the consistency and accuracy of the computational foundations used by subsequent modules in areas such as image comparison, text matching, and stability assessment, significantly enhancing the reliability and adaptability of the unattended operation process.

[0033] Specifically, the specific steps for determining the control accuracy of interface state data are as follows: obtain the click deviation distance, response delay time, and click speed variance, which respectively reflect the spatial accuracy of the operation, the response time of system feedback, and the stability of user behavior; the system performs a ratio operation on the click deviation distance and the response delay time, and the ratio reflects the offset trend of the operation position per unit time, and then squares the ratio to weightedly amplify the impact of the deviation; then, the square value is multiplied by the behavior quality factor, which is automatically adjusted by Bayesian optimization based on the fitting relationship between the deviation speed term and the success rate in a large number of operation samples, and has a value range of 1.3-3.0; the result is then multiplied by half of the constant to form a deviation speed suppression term, which is used to measure the combined effect of position deviation and response delay. The system then obtains the click speed variance, which represents the stability of speed changes in continuous clicks, and takes the natural logarithm after adding one to it to suppress the influence of extreme values. It is then multiplied by the instability adjustment coefficient to form a speed fluctuation suppression term. The instability adjustment coefficient is calculated through the covariance matrix between the speed variance and the interface response abnormality rate, and is adaptively converged and generated in the process of minimizing the loss function, with a value range of 0.6–2.0. Finally, the deviation speed suppression term and the speed fluctuation suppression term are added as the negative exponential part of the exponential function to characterize the overall attenuation degree of the operation control quality. The interface control accuracy value is further calculated through the exponential function to reflect the comprehensive performance of the operation behavior in terms of spatial accuracy, temporal responsiveness and behavioral stability.

[0034] The specific calculation formula for the interface control accuracy value is:

[0035] ;

[0036] Where, Indicates the interface control accuracy value, represents the behavior quality factor, Indicates click deviation distance. Indicates the response delay time, represents the instability adjustment coefficient, represents the click speed variance.

[0037] As shown in Table 1, this is a data table of interface control accuracy values provided in an embodiment of the present application. In this embodiment, the response delay time of sample 1 is set to 0.95, the click deviation distance is set to 2.8, and the click speed variance is set to 0.50; the response delay time of sample 2 is set to 0.85, the click deviation distance is set to 3.2, and the click speed variance is set to 0.30; the response delay time of sample 3 is set to 0.50, the click deviation distance is set to 4.5, and the click speed variance is set to 0.15; the response delay time of sample 4 is set to 0.60, the click deviation distance is set to 3.8, and the click speed variance is set to 0.25; the response delay time of sample 5 is set to 0.20, the click deviation distance is set to 6 .5, and the click speed variance is set to 0.01; the response delay time of sample 6 is set to 0.35, the click deviation distance is set to 5.5, and the click speed variance is set to 0.05; the response delay time of sample 7 is set to 0.75, the click deviation distance is set to 2.5, and the click speed variance is set to 0.40; the response delay time of sample 8 is set to 0.25, the click deviation distance is set to 7.0, and the click speed variance is set to 0.02; the response delay time of sample 9 is set to 0.40, the click deviation distance is set to 3.0, and the click speed variance is set to 0.35; the response delay time of sample 10 is set to 0.18, the click deviation distance is set to 6.5, and the click speed variance is set to 0.03.

[0038] Table 1 Interface control accuracy data table

[0039]

[0040] like Figure 2 The figure shows the distribution of interface control accuracy values provided by the embodiments of the present application. According to Table 1 and the bar chart in the figure, Sample 5 has the highest interface control accuracy value, reaching 0.9872. This indicates that this sample achieves optimal operational accuracy under the conditions of minimal response delay, large click deviation, but stable speed. The first-level accuracy threshold set in the figure is 0.80, and the second-level accuracy threshold is 0.55. Sample 1 has an interface control accuracy value of 0.5479, slightly higher than the second-level accuracy threshold of 0.55, indicating that it has response lag and unstable operation issues, and requires the implementation of remedial strategies. From the overall trend, the accuracy values of Samples 3, 5, 6, 8, and 10 all exceed the first-level accuracy threshold and belong to the high-precision range; Samples 2, 4, and 9 are in the medium-precision range; Samples 1 and 7 are close to or slightly below the second-level threshold, indicating significant operational errors, and the system needs to implement auxiliary identification and path correction measures. The higher the interface control accuracy value, the better the current operation performs in terms of spatial click accuracy, behavioral stability, and system response consistency, and the higher the credibility of the system's judgment of its behavior.

[0041] In this implementation, by jointly modeling key behavioral characteristics such as click deviation distance, response delay time, and click speed variance, the system quantifies spatial accuracy, response efficiency, and behavioral stability during operation, outputting the interface control accuracy value as an exponential function. This metric effectively integrates the multidimensional manifestations of operational behavior, improving the ability to discern the quality of individual interactions. This provides a scientific and quantifiable foundation for subsequent feedback comparison and anomaly adjustment, significantly enhancing the intelligent judgment accuracy and adaptive control capabilities of unattended operation processes.

[0042] Specifically, the specific steps for judging the spatial accuracy, behavioral stability and reliability of system response of an operation based on the control accuracy judgment result are as follows: the system first compares the calculated interface control accuracy value with the set accuracy threshold in real time. The accuracy threshold includes the first-level accuracy threshold and the second-level accuracy threshold, which are used to grade and identify the operation quality level; when the interface control accuracy value is greater than or equal to the first-level accuracy threshold, it means that the current operation has reached the ideal standard in terms of position accuracy, behavioral stability and feedback response. The system directly enters the next task process without any correction operation, and at the same time marks the high-quality operation sample into the behavior imitation training set for subsequent model reinforcement learning, and simultaneously reduces the control recognition fault tolerance range to improve the refined matching capability, while keeping the current input and output rhythm parameters unchanged to ensure the consistency and continuity of subsequent operations; when the interface control accuracy value is greater than the second-level accuracy threshold but lower than the first-level accuracy threshold, It indicates that there is a slight deviation in the operation and the feedback response is slightly unstable. The system will perform a series of fine-tuning operations, including delaying the recognition trigger, performing a slight perturbation click on the center coordinates of the control, and marking the control as a "low-confidence interaction target". At the same time, the sample data will be included in the model optimization training set to improve the system's adaptability to boundary fuzzy scenarios; when the interface control accuracy value is less than or equal to the secondary accuracy threshold, the system determines that the current operation has serious deviation and instability risks, and will immediately trigger the abnormal remediation mechanism, abandon the original click path, switch to the backup recognition template to re-position the control, and perform multi-point tentative click operations within five pixels above, below, left and right of the center of the control to try to restore the effectiveness of the control operation. If the accuracy value is still lower than the threshold for two consecutive times, the system will mark the control as a "high-risk interaction area". Subsequently, this process node will be transferred to the manual verification path, and human-machine collaborative processing will ensure the overall stable execution of the task.

[0043] In this implementation, by comparing interface control accuracy values against multi-level thresholds in real time, the system accurately identifies the reliability level of operational behaviors and dynamically matches different processing strategies based on the determination results, implementing a comprehensive response mechanism from archiving high-quality samples, correcting minor deviations, to remediating abnormal behavior. This step not only improves the system's ability to automatically determine the quality of individual operations, but also significantly enhances the stability, adaptability, and fault tolerance of unattended tasks in complex environments, providing a key guarantee for the continuity of subsequent operations and the overall intelligence level of the system.

[0044] Specifically, the specific steps for performing feedback consistency comparison on the interface status data with qualified control accuracy judgment results are as follows: the system first obtains the image structure similarity value between the interface screenshots before and after the operation, as well as the recognition text set obtained by OCR recognition and extraction and the target text set defined in the task template; then, the image structure similarity is subtracted from one to obtain the degree of difference in image feedback, and the difference value reflects the degree of deviation between the current interface image and the expected state; the difference value is squared to amplify the difference effect, and then multiplied by the visual feedback weight coefficient to obtain the image feedback error term, which is dynamically allocated in the feature importance ranking based on the information gain of the image structure similarity on the operation success rate, and the value range is 0.6-0.9; at the same time, the system compares the recognition text set with the target An intersection operation is performed on the text sets, and the number of intersection elements is counted and divided by the total number of elements in the target text set. The text matching ratio is calculated to reflect the degree of content consistency at the text level. The absolute value of the ratio is then taken and subtracted from one to obtain the degree of difference in text feedback; the difference value is squared and multiplied by the text feedback weight coefficient to obtain the text feedback error term. The text feedback weight coefficient is extracted based on the gradient response interval of OCR text similarity to task completion in the actual task, and the value range is 0.1-0.4; finally, the system adds the image feedback error term and the text feedback error term, and performs a square root operation on the sum, and finally outputs the execution feedback consistency value, which serves as a unified quantitative indicator for comprehensively evaluating the consistency of feedback in the two dimensions of image and text, and is used to subsequently determine whether the current operation is truly effective.

[0045] The specific calculation formula for the execution feedback consistency value is:

[0046] ;

[0047] Where, Indicates the execution feedback consistency value, represents the visual feedback weight coefficient, Represents the image structure similarity, represents the text feedback weight coefficient, Represents a set of recognized texts, Represents the target text collection.

[0048] In this implementation, this step constructs a feedback consistency assessment model by jointly analyzing image structural similarity and text matching. This model accurately determines whether the interface state meets expectations from both visual and semantic perspectives. By weightedly integrating image differences and text errors, the system quantifies and outputs an execution feedback consistency value, which serves as the key basis for determining whether an operation has been effective. This significantly improves the accuracy and reliability of automated feedback confirmation in unattended processes.

[0049] Specifically, the specific steps for judging whether the interface status is consistent with expectations based on the execution feedback consistency comparison results are as follows: the system compares the execution feedback consistency value with the feedback threshold in real time. The feedback threshold includes the first-level feedback threshold and the second-level feedback threshold, which are respectively used to divide the feedback effect into high, medium and low levels; when the execution feedback consistency value is less than or equal to the first-level feedback threshold, it means that the current interface image structure is highly matched, and the text recognition result is basically consistent with the expected content. The system determines that the operation has been successfully implemented and continues to execute subsequent task processes, and marks the current recognition template as a valid template for subsequent operation reuse and model reinforcement training; when the execution feedback consistency value is greater than the first-level feedback threshold and less than or equal to the second-level feedback threshold, it means that there is a moderate degree of deviation in the feedback, which may be caused by factors such as delayed refresh of the interface image or incomplete OCR recognition. The system automatically triggers a series of fault tolerance The strategy includes appropriately delaying the recognition start time, expanding the image comparison area boundary, lowering the image structure similarity judgment threshold, and enabling auxiliary recognition mechanisms such as keyword-assisted extraction and layout correction mechanisms. At the same time, the image screenshots, text collections and difference indicators related to this round of operations are archived and stored for subsequent error analysis and model optimization. When the execution feedback consistency value is greater than the secondary feedback threshold, the system determines that the operation has not taken effect, and immediately and automatically executes the retry operation process, and continues to try the operation before the preset number of retries is exceeded. If the number of retries reaches the upper limit, the system will roll back the process step to the previous task node and activate the image fault tolerance mechanism, trying to use strong template matching, contour enhancement and other methods to reconfirm the control status. Finally, the relevant recognition images, extracted text, control information and failure labels are packaged and archived as high-risk error samples for model training and strategy adjustment.

[0050] In this implementation, this step sets a feedback consistency threshold and performs a multi-dimensional comparison of the interface image structure and text recognition results after the operation is executed, accurately determining whether the interface state is consistent with expectations. Based on the level of feedback consistency, the system automatically triggers recognition fault tolerance, retry fallback, and error sample archiving mechanisms, enabling adaptive response and intelligent repair of abnormal feedback, effectively improving the success rate and overall robustness of unattended tasks in complex scenarios.

[0051] Specifically, failure records and attempt limits are maintained to avoid infinite retries, and processing methods are adaptively matched according to the failure type to achieve basic abnormal fault-tolerant control. The specific steps are as follows: When an operation fails, the system immediately calls the logging component to record in detail the failure type, control unique identifier, failure timestamp, and current task context information corresponding to the current operation, and synchronously updates the failure counter of the corresponding control to ensure that historical failure data is available in real time; when the cumulative failure count of a certain operation control does not exceed the maximum attempt threshold set by the system, the system automatically matches and executes predefined control strategies based on the cause of failure, including implementing slight position perturbation clicks within the control center coordinate range to cover possible offset errors, and dynamically extending the time interval between input actions and response judgments. To alleviate interface delay problems, switch to backup image templates to re-identify control graphic features, or directly start backup task paths to bypass high failure rate nodes; when the number of failures reaches the upper limit of attempts, the system immediately stops further operation attempts on the control and marks it as a high-risk interaction target. At the same time, the control-related operation data, failure logs, number of attempts, exception context and executed adjustment strategies are integrated and archived into the system fault database; then the system dynamically constructs exception handling diversion paths according to the failure type, such as transferring the interface recognition exception task to the enhanced recognition module, pushing the slow operation response task to the task scheduling module for rhythm reconstruction, or incorporating control recognition error-prone samples into the model training data pool to achieve a robust operation system with uninterrupted processes and basic fault-tolerant operations.

[0052] In this implementation, this step achieves intelligent fault tolerance and dynamic offload control for abnormal tasks by accurately recording and limiting the type, number, and duration of operation failures, combined with an adaptive matching and control strategy based on the failure cause. This avoids the resource waste and system congestion caused by infinite retries. Furthermore, by marking high-risk instances and archiving failed samples, it provides data support for subsequent model optimization and process refactoring, significantly improving the stability and troubleshooting capabilities of unattended operations.

[0053] Specifically, the steps for conducting task completion stability assessment are as follows: First, obtain the interface control accuracy value, number of failed retries and OCR recognition confidence corresponding to the operation, where the interface control accuracy value is used to quantify the operation accuracy and response stability, the number of failed retries reflects the operation fault tolerance requirements, and the OCR recognition confidence measures the reliability of the text recognition results; then take the inverse of the interface control accuracy value to represent its uncertainty; add one to the number of failed retries and take the logarithm to represent the impact of repeated attempts on the system load; add the OCR recognition confidence and the minimum correction term and take the inverse; the minimum correction term is used to avoid division by zero errors when the OCR matching confidence is zero. The system automatically sets it after performing a statistical distribution analysis on the error fluctuations in the historical minimum value samples during the training phase, and the value range is , avoiding the mathematical anomaly caused by zero recognition confidence and reflecting the risk level of recognition quality; then add these three results, add one to form the evaluation denominator, and finally take the inverse of the entire sum to obtain the task completion stability value, which serves as a unified quantitative indicator for judging the current unattended task execution quality.

[0054] The specific calculation formula for the task completion stability value is:

[0055] ;

[0056] Where, Indicates the stable value of task completion, Indicates the interface control accuracy value, Indicates the number of failed retries. express Recognition confidence, Indicates a very small correction term.

[0057] In this implementation, this step integrates three key data types: interface control accuracy, operation retry frequency, and OCR recognition confidence, to construct a task completion stability value indicator, comprehensively reflecting the execution quality and process reliability of an unattended task. This indicator not only quantifies the stability of operational behavior and the effectiveness of feedback, but also combines failure risk and recognition accuracy to form a comprehensive evaluation result, providing a unified basis for system decision-making. With this evaluation result, the system can automatically determine whether the task has been completed reliably and optimize path recording, sample annotation, and strategy adjustment accordingly, achieving dynamic monitoring and continuous improvement of the quality of unmanned task execution.

[0058] Specifically, the specific steps for judging whether a complete unattended task is stable and reliable based on the task completion stability assessment results are as follows: real-time comparison of the task completion stability value and the stability threshold, the stability threshold includes the first-level stability threshold and the second-level stability threshold; when the task completion stability value is greater than or equal to the first-level stability threshold, the system determines that the current task execution is stable and reliable, indicating that the operation behavior is highly accurate, the feedback results are accurate, and the OCR recognition confidence is good. The system marks the task process as a high-quality path, and at the same time includes its complete operation records in the system sample library as a key data source for subsequent task imitation, recognition template optimization and parameter training; when the task completion stability value is greater than the second-level stability threshold and less than the first-level stability threshold, the system believes that there are slight fluctuations in task execution, which may be due to unstable recognition or operation retries affecting the overall quality, and the system automatically triggers OCR recognition. The parameter optimization process of the model is carried out, and the current process path is labeled and classified. If the recognition confidence is lower than the set threshold, the path is marked as "unstable text recognition". If the number of operation retries reaches the limit value, it is marked as "structural special improvement path" and fed back to the task flow editing module for structural rule revision and process update; when the task completion stability value is less than or equal to the secondary stability threshold, the system determines that the task has the risk of execution abnormality or logical interruption, and will automatically reinitialize the current process and re-execute a complete round of tasks. If it still fails after re-execution, the emergency exit mechanism is triggered, and a structured operation failure report is generated. The relevant recognition images, feedback feature values and error sample data are packaged and uploaded as high-risk path samples for subsequent model training and key scenario optimization, thereby enhancing the system's adaptability and robustness to complex tasks.

[0059] In this implementation plan, this step aims to determine whether the current unattended task is reliable and sustainable by evaluating the stability of the entire task execution process. After obtaining the interface control accuracy value, the number of failed retries, and the OCR recognition confidence, the system constructs a stability calculation expression, normalizes and comprehensively processes each indicator, and dynamically divides the task stability level based on the comparison of the calculation results with the set threshold. The result judgment can effectively screen high-quality execution paths, identify potential abnormal processes, and provide a decision-making basis for subsequent task template optimization and fault handling mechanisms, thereby improving the system's stable operation capabilities in multiple tasks and scenarios.

[0060] like Figure 2The figure shows a schematic diagram of the structure of an unattended computer operating system based on machine vision provided by an embodiment of the present application. This embodiment provides an unattended computer operating system based on machine vision, which is applied to the above-mentioned unattended computer operation method based on machine vision. The overall system construction is based on closed-loop control logic to ensure that the task execution process has state perception, dynamic response and performance evaluation capabilities. The system includes an interface data acquisition module, an automatic behavior execution module, a result feedback judgment module, an abnormal control and remediation module and a process record optimization module; the interface data acquisition module is used to collect interface state data before and after the operation, including image screenshots, control state and text recognition content, and pre-process the interface state data in a unified format to provide standardized input features for subsequent analysis; the automatic behavior execution module is used to determine the control accuracy of the interface state data, and obtain the interface control accuracy value by calculating indicators such as click deviation, response delay and behavior stability. According to the control accuracy determination result, the spatial accuracy, behavior stability and reliability of the system response of an operation are determined, and the corresponding task operation is executed; the result feedback judgment module is used to perform feedback consistency comparison on the interface state data with qualified control accuracy determination results, and compare the interface state data with the structure similarity and O The CR text matching results are integrated to obtain a feedback consistency score. The interface status is judged to be consistent with expectations based on the execution feedback consistency comparison results, and the subsequent operation strategies are adjusted to assist. The abnormal control and remediation module is used to maintain failure records and attempt limit. When an operation abnormality is detected, remediation mechanisms such as click fine-tuning, input rhythm changes, and template replacement are automatically executed to avoid infinite retries. The matching processing method is adaptively matched according to the failure type to achieve basic abnormal fault-tolerant control and process sustainability. The process record optimization module is used to integrate interface status data, control accuracy judgment results, and execution feedback consistency comparison results, calculate the task completion stability value based on unified indicators, conduct task completion stability assessment, and judge whether a complete unattended task is stable and reliable based on the task completion stability assessment results, providing a key reference basis for system model iteration and process optimization.

[0061] In this implementation, the core function of this step is to comprehensively evaluate the entire process of unattended task operations through multi-source information fusion and module collaborative processing, ultimately outputting a comprehensive indicator reflecting the stability of task execution. This effectively reflects the comprehensive performance of key factors such as operational accuracy, feedback consistency, and exception recovery, thereby accurately determining the reliability and execution success rate of the entire automated task process. This provides a clear basis for system optimization, adaptive adjustment, and identification of high-risk processes, thereby improving the overall stability and automation level of the task.

[0062] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0063] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. An unattended computer operation method based on machine vision, characterized in that: The following steps are involved: S1: Collect interface status data before and after the operation and pre-process the interface status data; S2: Determine the control accuracy of the interface state data, and judge the spatial accuracy, behavioral stability and reliability of the system response of an operation based on the control accuracy determination results; The specific steps of determining the control accuracy of the interface status data are as follows: Obtain click deviation distance, response delay time and click speed variance; Calculate the ratio of the click deviation distance to the response delay time, square the ratio, multiply it by the behavior quality factor, and then multiply the result by half of the constant to obtain the deviation speed suppression term. Calculate the click speed variance, add one to it, take the natural logarithm, and multiply it by the instability adjustment coefficient to obtain the speed fluctuation suppression term. Then the deviation speed suppression term and the speed fluctuation suppression term are added together as the negative exponential part of the exponential function to calculate the interface control accuracy value; S3: Performing feedback consistency comparison on the interface state data with qualified control accuracy judgment results, and judging whether the interface state is consistent with expectations based on the feedback consistency comparison results; S4: Maintain failure records and attempt limits to avoid infinite retries, and adaptively match processing methods based on failure types to achieve basic exception fault tolerance control; S5: Conduct a task completion stability assessment based on the comprehensive interface status data, control accuracy determination results, and execution feedback consistency comparison results. Determine whether a complete unattended task is stable and reliable based on the task completion stability assessment results.

2. The unattended computer operation method based on machine vision according to claim 1, characterized in that: The specific steps of collecting the interface status data before and after the operation and pre-processing the interface status data are as follows: Interface status data includes: response delay time, recognized text dataset, and number of failed retries; Capture the current screen image and save it as a screenshot for subsequent structural similarity comparison; obtain the state attribute information of the control, including color changes, focus status, and display and hide flags and other auxiliary judgment information; The response delay time is obtained by subtracting the click timestamp recorded by the system from the timestamp of the completion of the interface state change; the text content of the specified control area is extracted and the OCR recognition technology is used to form a recognized text set; the retry record after each operation failure is written into the control log and recorded as the number of failed retries; Preprocessing the interface state data, wherein the preprocessed data includes: click deviation distance, click speed variance, target text set, image structure similarity and OCR recognition confidence; Screenshot images are normalized and resolution corrected to ensure structural consistency across different terminals. Image enhancement and edge smoothing are performed on control area images. Language encoding is unified, special characters are removed, and spaces are merged for recognized text to form structured text for subsequent comparison. The click deviation distance is calculated by the Euclidean distance between the control center coordinates and the actual click position coordinates; the average speed is calculated by combining the time series and displacement series of multiple click events, and the click speed variance is calculated using the sliding window variance formula; the target text set is obtained based on the preset field matching rules in the task template; the before and after images obtained by the screenshot are input into the image structure comparison function, and the image structure similarity is calculated using the local image analysis algorithm; the OCR recognition confidence is obtained by comparing and analyzing the character differences between the recognized text and the target text, and the matching degree is normalized; the calculated interface control accuracy value, execution feedback consistency value and task completion stability value are recorded in real time.

3. The unattended computer operation method based on machine vision according to claim 1, characterized in that: The specific steps of judging the spatial accuracy, behavioral stability and reliability of system response of an operation based on the control accuracy judgment result are as follows: The real-time comparison interface controls the accuracy value and accuracy threshold, wherein the accuracy threshold includes the first-level accuracy threshold and the second-level accuracy threshold; When the interface control accuracy value is greater than or equal to the first-level accuracy threshold, the system directly enters the next task process and marks a set of interface state data generated by the interactive operation as a high-quality template as an operation sample and includes it in the behavior imitation training set. At the same time, the control recognition error tolerance range is reduced and the current rhythm parameters remain unchanged. When the interface control accuracy value is greater than the second-level accuracy threshold and less than the first-level accuracy threshold, there is a slight deviation and unstable response. The recognition trigger delay, control center disturbance click and confidence marking operations are performed. At the same time, the control is recorded as a low-confidence interaction target for subsequent model iteration and optimization; When the interface control accuracy value is less than or equal to the secondary accuracy threshold, the abnormal remediation mechanism is triggered, the original click path is abandoned, the backup recognition template is switched to reposition the control, and multiple tentative clicks are performed within a range of five pixels above, below, left and right of the center of the control. If the accuracy value is still lower than the threshold for two consecutive times, the control is marked as a high-risk interaction area and switched to the manual verification path.

4. The unattended computer operation method based on machine vision according to claim 1, characterized in that: The specific steps of performing feedback consistency comparison on the interface status data with qualified control accuracy determination results are as follows: Obtain image structure similarity, recognition text set and target text set; Subtract the image structure similarity from one to obtain the image feedback difference value; square the image feedback difference value and multiply it by the visual feedback weight coefficient to obtain the image feedback error term; after the calculation operation, identify the number of intersection elements between the text set and the expected target text set, and divide it by the total number of the target text set to obtain the text matching ratio, and then take the absolute value of the text matching ratio; subtract the absolute value of the text matching ratio from one to obtain the text feedback difference value; square the text feedback difference value and multiply it by the text feedback weight coefficient to obtain the text feedback error term; add the image feedback error term and the text feedback error term, and take the square root of the sum to obtain the execution feedback consistency value.

5. The unattended computer operation method based on machine vision according to claim 1, characterized in that: The specific steps of judging whether the interface state is consistent with the expected state according to the execution feedback consistency comparison result are as follows: Comparing the execution feedback consistency value with the feedback threshold in real time, wherein the feedback threshold includes a first-level feedback threshold and a second-level feedback threshold; When the execution feedback consistency value is less than or equal to the first-level feedback threshold, the interface image structure and text feedback meet expectations, and the subsequent operation process continues, and the current recognition template is marked as a valid template; When the execution feedback consistency value is greater than the first-level feedback threshold and less than or equal to the second-level feedback threshold, it is determined that there is a certain deviation. Strategies such as adjusting the recognition delay, widening the image comparison range, and lowering the comparison threshold are implemented. At the same time, the auxiliary recognition mechanism is enabled, and the images, texts, and difference information of this round are archived. When the execution feedback consistency value is greater than the secondary feedback threshold, the operation does not take effect and the operation retry process is automatically executed. If the retry limit is exceeded, the system will roll back to the previous task step and activate the image fault tolerance mechanism to try to confirm the control status again. At the same time, the recognition image, text results and control information are packaged and archived as error samples for subsequent analysis and model training.

6. The unattended computer operation method based on machine vision according to claim 1, characterized in that: The maintenance failure record and attempt number limit are used to avoid infinite retries, and the processing method is adaptively matched according to the failure type to achieve basic abnormal fault tolerance control. The specific steps are as follows: When an operation fails, first record the failure type, control identifier, and failure timestamp of the current operation, and update the corresponding failure count counter; When the number of failures for a certain control does not exceed the preset maximum number of attempts, the corresponding control strategy is implemented according to the type of failure reason, including fine-tuning the click position, adjusting the input rhythm, switching to an alternative template, and jumping to an alternative path. When the number of failures reaches the upper limit, further retries are stopped and the control is marked as a high-risk object. At the same time, relevant operation data, failure logs, and attempted policy records are archived; Adaptively select adjustment methods based on different failure types, build exception handling diversion paths, and achieve the goals of uninterrupted control processes and fault tolerance for basic operations.

7. The unattended computer operation method based on machine vision according to claim 1, characterized in that: The specific steps for conducting task completion stability assessment by comparing the comprehensive interface status data, control accuracy determination results, and execution feedback consistency results are as follows: Obtain interface control accuracy, number of failed retries, and OCR recognition confidence; Take the inverse of the interface control accuracy value, add one to the number of failed retries and take the logarithm, then add the OCR recognition confidence and the minimum correction term and take the inverse; add the above three items and then add one, and finally take the inverse of the overall result to obtain the task completion stability value.

8. The unattended computer operation method based on machine vision according to claim 1, characterized in that: The specific steps for judging whether a complete unattended task is stable and reliable according to the task completion stability evaluation result are as follows: Comparing the task completion stability value with the stability threshold in real time, wherein the stability threshold includes a first-level stability threshold and a second-level stability threshold; When the task completion stability value is greater than or equal to the first-level stability threshold, the task execution accuracy is high, the feedback is accurate, and the recognition result is reliable. The current process is marked as a high-quality path, and the execution record is included in the sample library as a reference for subsequent task imitation and optimization; When the task completion stability value is greater than the second-level stability threshold and less than the first-level stability threshold, the task has slight fluctuations, and the recognition parameter optimization, path labeling and task identification process are automatically triggered. If the recognition confidence is lower than the threshold, it is marked as unstable text recognition. If the number of retries reaches the threshold, it is marked as a structural improvement path and fed back to the task flow editing module; When the task completion stability value is less than or equal to the secondary stability threshold, the task as a whole is abnormal and there is a risk of interruption. The current process is automatically reinitialized and re-executed. If it still fails, the emergency exit mechanism is triggered and an operation failure report is generated. At the same time, the recognition image, result features and error samples are recorded as high-risk path samples for subsequent training and key optimization.

9. An unattended computer operating system based on machine vision, applying the unattended computer operating method based on machine vision according to any one of claims 1 to 8, comprising: The interface data acquisition module, automatic behavior execution module, result feedback judgment module, abnormality control and remediation module and process record optimization module are characterized by: The interface data acquisition module is used to collect interface status data before and after the operation and pre-process the interface status data; The automatic behavior execution module is used to determine the control accuracy of the interface state data, and to determine the spatial accuracy, behavior stability and reliability of the system response of an operation based on the control accuracy determination result; The result feedback judgment module is used to perform feedback consistency comparison on the interface state data with qualified control accuracy judgment results, and judge whether the interface state is consistent with expectations based on the feedback consistency comparison results; The exception control and remediation module is used to maintain failure records and attempt number limits to avoid infinite retries, and adaptively match processing methods according to failure types to achieve basic exception fault tolerance control; The process record optimization module is used to integrate interface status data, control accuracy determination results and execution feedback consistency comparison results to carry out task completion stability assessment, and judge whether a complete unattended task is stable and reliable based on the task completion stability assessment results.

Images