An artificial intelligence autonomous decision-based measurement instrument unmanned operation system and a control method thereof

Abstract

The application discloses a kind of measurement instrument unmanned operation system and control method based on artificial intelligence autonomous decision-making, the system includes cross slide rail movement mechanism and installs on mobile module eye in hand vision-execution composite module.System uses template matching and deep learning weighted fusion strategy, realizes the adaptive identification of interface state;Based on multi-modal decision model, fusion vision, text and historical operation sequence, use cross-modal attention mechanism to autonomously infer task intent and generate action instruction;Combined with AprilTag pose solution and homogeneous coordinate transformation chain, realize high-precision mapping from pixel space to physical space.The application can autonomously cope with abnormal pop-up window and realize fault self-healing.The application solves the problem that traditional automated test relies on fixed script, cannot handle dynamic interface and environmental offset.

Description

Technical Field

[0001] This invention belongs to the field of measuring instrument technology, specifically relating to an unmanned operating system for measuring instruments based on artificial intelligence autonomous decision-making. Background Technology

[0002] With the rapid development of Industry 4.0, power electronics, and precision manufacturing, automated testing and remote unmanned operation of measuring instruments have become core means to improve production efficiency and reduce labor costs. Currently, automated operating systems for measuring instruments mainly rely on industrial robots or cross-rail mechanisms, combined with pre-set automated scripts. However, in complex industrial applications, existing solutions still have the following significant drawbacks: First, they have poor environmental adaptability and rely on manual teaching. Most traditional automated equipment adopts an "eye-outside-hand" or fixed-coordinate click mode, which requires the instrument under test to be placed precisely and fixedly in a preset position. Once the instrument position is slightly offset by millimeters due to mechanical tolerances or improper manual placement, the traditional fixed script, lacking real-time visual feedback and dynamic pose compensation, will cause the click position to shift, resulting in test task failure or even damage to the instrument screen due to misoperation.

[0003] Secondly, the operational logic is rigid and unable to handle dynamic interfaces. Existing technologies are mainly based on preset linear process scripts, which have a weak ability to recognize interface content. When unexpected pop-ups appear on the measuring instrument interface (such as system warnings, connection timeout prompts, permission requests) or when the interface layout changes due to firmware upgrades, traditional scripts cannot autonomously recognize the current interface semantic logic, and usually get stuck in a logical dead loop or stop due to errors, making it difficult to achieve truly unmanned and continuous operation.

[0004] Finally, coordinate mapping accuracy is low and the calibration process is cumbersome. Traditional measurement and operation equipment often requires a complex manual calibration process to establish the correspondence between the camera, mechanical actuator, and the operation screen. For instruments with multi-level menus and complex UI layouts, the mapping error from pixel coordinates to physical space coordinates is often difficult to control effectively. In addition, existing systems generally lack a closed-loop verification mechanism after action execution, and cannot perform autonomous correction and fault self-healing after a click failure.

[0005] Therefore, there is an urgent need to develop an unmanned operating system for measuring instruments that can adapt to different interface layouts, has autonomous decision-making capabilities, and can achieve high-precision, closed-loop control. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing automated measurement instruments, such as poor environmental adaptability, rigid logic inability to handle dynamic interfaces, cumbersome coordinate calibration, and lack of error correction mechanisms. It aims to provide an unmanned operating system for measurement instruments based on artificial intelligence-driven autonomous decision-making. This system, through a "hands-on" hardware configuration combined with multimodal decision-making algorithms, achieves autonomous recognition, precise alignment, and self-healing execution of heterogeneous interfaces and dynamic environments.

[0007] To achieve the above objectives, the present invention provides the following technical solution: The present invention discloses an unmanned operating system for measuring instruments based on artificial intelligence autonomous decision-making, comprising a transverse slide rail base, a longitudinal slide rail slidably mounted on the transverse slide rail base, a moving module mounted on the longitudinal slide rail, an end-effector vision-execution composite module mounted on the moving module, and a control device electrically connected to the end-effector vision-execution composite module and the moving module; the axial direction of the longitudinal slide rail is parallel to the operation panel of the instrument under test, so that the moving module can perform two-dimensional movement in a plane parallel to the operation panel; the end-effector vision-execution composite module includes An industrial camera and a click actuator are mounted side-by-side on the mobile module via a rigid bracket, forming a visual configuration resembling an eye on a hand. The lens of the industrial camera faces the operation panel, and the end of the click actuator is equipped with a flexible click head for touching the operation panel. A control device is electrically connected to the industrial camera, a drive element for driving the longitudinal slide rail and the displacement of the mobile module, and the click actuator. The control device performs interface state recognition and decision-making based on the images captured by the industrial camera, drives the mobile module to the target physical coordinates, and controls the click actuator to perform a clicking action.

[0008] Preferably, the control device identifies the interface state through an adaptive weighted hybrid recognition strategy; the strategy includes: dynamic proportion calculation: using the three-frame difference method and background modeling algorithm to extract moving pixel regions in consecutive video frames, calculating the ratio of the total number of pixels in the moving region to the total number of pixels on the interface, to obtain the dynamic element pixel proportion. Weight function allocation: Preset piecewise linear function allocation template matching weights. With deep learning weights ,in Based on the pixel characteristics of industrial instrument interfaces and practical experience, a first threshold is set. Set the second threshold to 4%. It is 20%. When Below the first threshold At that time, the interface is determined to be in a quasi-static state. The system trusts the template matching results; when Above the second threshold At that time, the judgment interface contains a real-time refreshed video stream, which can easily cause false drift in template matching. Decision-making is taken over by deep learning models; when In When the interval is, Follow The increase is linear, and according to the practical value formula, it is expressed as follows: Coordinate weighted fusion: Obtain the candidate coordinates output by template matching respectively. and its confidence level And candidate coordinates output by deep learning and its confidence level ; pixel coordinates of the final target control Through formula The calculation yielded the result.

[0009] Preferably, the control device runs a multimodal decision model, which receives visual features captured by the industrial camera, interface text features extracted by OCR, and historical operation state sequences as inputs. It uses a linear projection matrix to map the patch sequence (block sequence) of the visual feature map and the text vector sequence to a unified latent feature space (hidden feature vector space) of equal dimensions, and superimposes modal encoding to preserve modality specificity.

[0010] Preferably, the multimodal decision model acquires task intent recognition capability through supervised training on a labeled expert operation sequence dataset; the dataset contains a causal chain of "interface state - expert action - jump feedback", and the model learns to perform fault self-healing actions when a pop-up containing specific abnormal semantic tags is detected by minimizing the action prediction loss based on cross-entropy loss.

[0011] Preferably, the target physical coordinates are calculated through a homogeneous coordinate transformation chain. The specific process includes: using the AprilTag tag cluster preset on the panel of the instrument under test, the rotation matrix R and translation vector T of the industrial camera relative to the panel reference system are solved by the PnP algorithm; and combining the camera intrinsic parameter matrix, the rigid offset vector between the camera and the click actuator, and the real-time pose feedback of the slide rail, a mapping equation from pixel coordinates to the reference coordinate system of the motion actuator is constructed.

[0012] Preferably, the system further includes closed-loop verification logic after the action. After the click actuator performs the action, the control device controls the industrial camera to re-acquire images. By comparing the grayscale distribution Shannon entropy of the interface images before and after the action, the change in interface complexity is quantified. When the relative change rate of feature entropy before and after the action exceeds... When the jump is completed, it is determined to be in accordance with the expected jump.

[0013] Preferably, the control device is equipped with a multi-level error correction and recovery mechanism, including a first-level error correction: performing an initial radius of [missing information] around the target's physical coordinates. Radial step size is The Archimedes spiral search click method physically compensates for recognition errors by increasing the click radius at equal intervals.

[0014] Preferably, it also includes secondary error correction: if the compensation click is invalid, the pose calibration based on AprilTag is retried and the recognition strategy weights are adjusted; Preferably, it also includes three levels of error correction: if the number of consecutive failures reaches a preset threshold, a safety reset is performed and an alarm is triggered.

[0015] This invention also discloses an unmanned operation control method for measuring instruments based on artificial intelligence autonomous decision-making. The method is characterized by the application of an operating system and includes the following steps: Step S1: The system starts and drives the mobile module to identify feature markers on the panel of the instrument under test using an industrial camera, completing automated hand-eye calibration; Step S2: Real-time capture of interface images, determination of the current task stage and location of the target control pixel coordinates using a multimodal decision model; Step S3: Calculation of the target physical coordinates through homogeneous transformation mapping, driving the motion actuator to move to the designated position; Step S4: Control of the click actuator to perform the operation; Step S5: Closed-loop verification is performed by acquiring interface images after the action, and the decision to proceed to the next task stage or trigger an error correction process is made based on the verification results.

[0016] The beneficial effects of this invention are as follows: First, by employing an "eye-on-hand" configuration in conjunction with a homogeneous coordinate transformation chain, this invention achieves real-time, dynamic pose compensation. Unlike the "dead-point click" mode in traditional solutions that require precise instrument positioning, this system can automatically identify millimeter-level offsets caused by mechanical tolerances or manual placement, and control the physical mapping error within 0.5mm. This gives the device a "plug-and-play" characteristic, completely eliminating the tedious manual teaching process and greatly improving its adaptability in complex industrial environments. Second, this invention utilizes a multimodal decision model and an adaptive weighted recognition strategy to endow the system with "semantic understanding" capabilities similar to those of a human operator. Through an adaptive weighted mechanism, the system can accurately identify target controls under heterogeneous layouts or complex UI interference, solving the technical problem of traditional scripts failing after software upgrades or interface tweaks. Furthermore, through the Transformer cross-modal attention mechanism, the system can autonomously identify and handle unexpected error pop-ups, connection timeouts, and other abnormal logic, rather than getting stuck in an infinite loop or stopping due to errors, achieving truly unmanned, intelligent, and continuous operation. This invention achieves automated and precise calibration of hand-eye relationships by integrating the AprilTag tag family and the PnP algorithm, replacing the traditional, cumbersome manual calibration work. Simultaneously, this invention constructs a complete closed-loop control system from "execution" to "verification" and then to "multi-level error correction": 1. The closed-loop verification mechanism ensures the validity of every action command, solving the drawback of traditional open-loop systems that cannot perceive whether an action has taken effect; 2. Multi-level error correction logic (spiral search, pose recalibration, strategy reconstruction) endows the system with significant fault self-healing capabilities, significantly reducing the interruption rate during unmanned operation and ensuring high reliability and high success rate of measurement tasks. Finally, based on the generalization capability of a multimodal large model, this invention does not require writing dedicated scripts for specific brands or models of measuring instruments. A single system can quickly adapt to various heterogeneous operating interfaces, significantly reducing the later maintenance costs and secondary development cycle for enterprises. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments disclosed in this invention, the accompanying drawings of the embodiments will be briefly described below. These drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0018] Figure 1 This is a schematic diagram of the hardware structure of the unmanned operating system provided in an embodiment of the present invention; Figure 2 A functional block diagram of a software system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the principle of coordinate mapping and hand-eye offset compensation provided in an embodiment of the present invention. Figure 4 The autonomous decision-making algorithm and closed-loop error correction flowchart provided in the embodiments of the present invention.

[0019] The labels in the attached figures are explained as follows: 1—Horizontal slide rail base; 2—Longitudinal slide rail; 3—Servo motor (drive element); 4—Movement module; 5—Industrial camera; 6—Actuator; 7—Industrial computer (control device); —Origin of the industrial camera coordinate system; —Click the origin of the actuator coordinate system; —Hand-eye offset vector; — Rotation matrix and translation vector. Detailed Implementation

[0020] The technical solutions (including preferred technical solutions) of the present invention will be further described in detail below with reference to the accompanying drawings and by way of listing some optional embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0021] refer to Figure 1 and Figure 3 This embodiment provides an unmanned operating system for measuring instruments based on artificial intelligence autonomous decision-making. Its hardware architecture mainly consists of three parts: a motion actuator, an end vision-execution composite module, and a control device.

[0022] In terms of spatial layout and mechanical connection, the physical support base of the system is a horizontal slide rail base 1, on which a vertical slide rail 2 is slidably mounted via a precision guide rail slider mechanism. The axis of the vertical slide rail 2 is perpendicular to the axis of the horizontal slide rail base 1 and remains parallel to the operation panel of the instrument under test. A driving element, namely a high-precision servo motor 3, is mounted on both the horizontal slide rail base 1 and the vertical slide rail 2. The servo motor 3 converts rotational motion into linear displacement of the vertical slide rail 2 or the moving module 4 via a lead screw transmission mechanism. The moving module 4 is slidably mounted on the vertical slide rail 2, and an industrial camera 5 and a click actuator 6 are fixedly mounted side-by-side on it via a rigid bracket, thus forming a compact vision-execution composite configuration of "Eye-in-Hand". In this configuration, the optical axis of the lens of the industrial camera 5 and the downward pressing direction of the click actuator 6 are both perpendicular to the operation panel, and their relative spatial positions are locked by the rigid bracket. The click actuator 6 is preferably a cylinder finger or an electromagnetic push rod, with a click head made of flexible silicone material installed at its end to simulate the clicking action of a human fingertip.

[0023] In terms of electrical connections, the control device uses an industrial computer 7 as the core control hub. The industrial computer 7 is electrically connected to the industrial camera 5 via an image transmission interface such as a GigE interface or a USB 3.0 interface to acquire interface image data in real time. The industrial computer 7 is also electrically connected to the drivers of two sets of servo motors 3 via a motion control bus such as an EtherCAT bus to issue pulse displacement commands. Simultaneously, the industrial computer 7 is electrically connected to the solenoid valve control circuit of the actuator 6, controlling the extension and retraction of the actuator head by outputting high and low level signals, thus forming a complete perception-decision-execution hardware closed loop.

[0024] like Figure 3 As shown, the specific implementation steps of the present invention are as follows: S1. Definition of Spatial Coordinate Systems: To achieve precise alignment, the system predefines the following four core coordinate systems: Image Pixel Coordinate System The origin is the top left corner of the image captured by the camera. Industrial camera coordinate system. With the camera's optical center as the origin and the optical axis direction as the Z-axis, click on the actuator coordinate system. The origin is the center point of contact (TCP) of the cylinder finger. Motion actuator base coordinate system. The origin is the zero point of intersection between the transverse slide rail base (1) and the longitudinal slide rail (2).

[0025] S2. Real-time pose calculation based on the marker cluster: The system uses the preset AprilTag marker cluster on the instrument panel to perform spatial positioning. Feature point extraction: After the industrial camera (5) captures the interface image, the industrial control computer (7) identifies the pixel coordinates of each feature point in the marker cluster. PnP algorithm analysis: Combining the known physical layout information of each marker on the instrument panel (such as the relative coordinates defined in the YAML configuration file), the system uses the PnP (Perspective-n-Point) algorithm to calculate the camera coordinate system in real time. Rotation matrix relative to the instrument panel reference frame With translation vector Reprojection optimization: Further, camera intrinsic parameter distortion is corrected through reprojection error minimization algorithms (such as Levenberg-Marquardt optimization) to ensure the analytical accuracy of spatial attitude.

[0026] S3. Hand-eye offset compensation for "eye on hand": Since the industrial camera (5) and the click actuator (6) are installed side by side, there is a physical deviation between their centers. Rigid transformation matrix: The system pre-measures the rigid offset vector of the click actuator center relative to the camera optical center through offline calibration. (For example, 0.057m and 0.0115m configured in YAML). Compensation logic: When the multimodal decision module outputs the target button... Pixel centroid At this time, the system first back-projects it onto the camera coordinate system. The physical plane below, then superimposed The offset vector is used to obtain the target alignment point of the actuator in the camera's field of view.

[0027] S4. Closed-loop coordinate chain transformation and execution: Finally, the industrial control computer (7) combines the encoder feedback value of the current cross coordinate motion module. The target position is uniformly mapped to the base coordinate system using the following formula: Calculated physical space coordinates The pulse control commands are directly converted into those of the servo motor (3). Due to the use of dynamic pose calculation, even if there is a slight deviation in the placement angle of the instrument under test, the system can still automatically complete the compensation through the homogeneous transformation chain.

[0028] This invention achieves high reliability for unmanned operation by constructing a closed-loop control system based on task intent. For example... Figure 4 As shown, the specific execution logic is as follows: The closed-loop verification logic compares the pixel entropy values ​​of key areas before and after the click operation using an industrial control computer. If the interface changes from a static configuration page to a dynamic progress bar page, the entropy value... Significant drift may occur; the system will detect this statistical change and confirm the command completion. If the error correction process begins, the servo motor will drive the click actuator to follow a spiral trajectory. times (total) The system employs a comprehensive approach (covering multiple points) to achieve self-healing without relying on high-precision external calibration. Task intent parsing: The model fuses the visual features of the current interface, the textual semantics extracted by OCR (such as "Confirm," "Cancel," "System Error"), and historical operation sequences across modalities. Action priority determination: A cross-attention mechanism is introduced. When an unexpected error dialog box appears on the interface, the system automatically assigns a very high attention weight to that area, prioritizing the generation of "Close Pop-up" or "Handle Error" instructions instead of continuing to execute the main task, thus achieving autonomous error avoidance in dynamic environments.

[0029] The closed-loop execution and visual verification process of this invention includes the following steps: after the system drives the click actuator (6) to complete the action, it immediately enters the visual feedback verification stage, forming a control closed loop: based on the physical coordinates calculated by the third module, the system drives the servo motor (3) to move to the target position, controls the click actuator (6) to press down to execute the actuator retraction, and then controls the industrial camera (5) to re-capture the current interface image. The system compares the interface state changes before and after the click. If it is determined that the interface has undergone the expected jump (such as the progress bar starting to move or the pop-up disappearing), the action is confirmed to be valid, and the system enters the next task stage.

[0030] If the visual verification judgment action fails (e.g., due to slight deviation in recognition caused by reflective interference or instrument response delay), the system will trigger the following multi-level error correction logic: Level 1 error correction (repositioning): The system re-triggers pose calculation based on AprilTag (see Section 3) to eliminate the positioning deviation caused by minor mechanical vibrations, and performs repeated "spiral search" clicks within a small range in the target pixel area. Level 2 error correction (strategy reconstruction): If the repositioning click is still ineffective, the system will automatically reduce the template matching weight, increase the sensitivity of deep learning feature recognition, attempt to identify whether there are hidden obstructive elements (such as transparent prompt boxes covered by the top layer), and update the decision instructions. Level 3 error correction (safety self-protection): If all M consecutive attempts fail, the system will record a snapshot of the current abnormal interface, reset the horizontal slide rail (1) and the vertical slide rail (2) to the safe zero position, and trigger an alarm instruction.

[0031] It is understood that, in order to achieve a precise mapping from image pixel coordinates to the physical coordinates of the mechanical execution space, this embodiment of the invention constructs a spatial mapping model based on a homogeneous transformation chain. First, the system predefines four core coordinate systems, namely the pixel coordinate system with the top-left corner of the image as the origin. Camera coordinate system with the camera optical center as the origin The actuator coordinate system with the center point of the click head as the origin. and the base coordinate system with the zero point of the intersection of the two sets of slide rails as the origin. .

[0032] In the dynamic pose calculation process of this invention, the industrial control computer 7 captures the image of the preset AprilTag marker cluster on the panel of the instrument under test through the industrial camera 5, identifies the pixel coordinates of the feature points, and combines them with the preset physical coordinates of each marker in the YAML configuration file. The camera coordinate system is then calculated in real time using the PnP (Perspective-n-Point) algorithm. Rotation matrix relative to the panel reference frame With translation vector To improve resolution, the system further employs the Levenberg-Marquardt algorithm to minimize reprojection error and correct camera intrinsic parameter distortion.

[0033] In the hand-eye offset compensation stage, the system utilizes the rigid offset vector of the click actuator center relative to the camera optical center obtained through offline calibration. Real-time compensation is performed. In this embodiment, the offset vector is configured with values ​​of 0.057m and 0.0115m in the X and Y directions, respectively. The decision model outputs the centroid of the target control in the pixel coordinate system. At this time, the system first back-projects it onto the camera coordinate system. The physical plane below, and superimposed Offset vector. The target physical coordinates are calculated using a cascaded homogeneous transformation matrix chain. The system acquires encoder feedback values ​​from the moving guide rail in real time. and Construct the dynamic compensation matrix for the sliding rail Meanwhile, the rotation matrix of the current camera relative to the panel is extracted from image features using the PnP algorithm. and depth distance Combined with the pre-calculated camera intrinsic parameter matrix Through formula The system realizes the real-time conversion of pixel centroids in the dynamic interface into absolute pulse commands for the motion mechanism, thereby achieving sub-millimeter level precise alignment and clicking without the need for fixed instrument placement.

[0034] refer to Figure 2 and Figure 4 This embodiment does not rely on traditional linear process scripts, but instead achieves intelligent decision-making through multimodal fusion logic. For interface recognition, the system employs an adaptive weighted hybrid recognition strategy, using a normalized cross-correlation coefficient (NCC) template matching algorithm to identify basic fixed buttons, and deep learning models such as YOLOv8 or lightweight neural networks to identify dynamic UI controls. The industrial computer 7 calculates the pixel percentage of dynamic elements such as progress bars and waveforms in the interface image in real time. The system dynamically adjusts the recognition confidence weights accordingly: when the interface is detected to be highly dynamic, the system automatically reduces the template matching weights and increases the recognition sensitivity of deep learning features.

[0035] In terms of image feature processing, this embodiment divides the high-resolution interface images captured by the industrial camera into fixed-size image blocks, i.e., patches, and projects them into a patch sequence. This allows the Transformer architecture to extract local fine-grained features and their global spatial correlations. In terms of cross-modal feature alignment, the system constructs a unified-dimensional latent feature space (hidden feature vector space), mapping heterogeneous visual patch vectors and text tag vectors to the same high-dimensional vector set. By calculating the cosine similarity between vectors, precise semantic matching and fusion are achieved. In terms of modal information preservation, the system additionally overlays modal encoding (Modal Embedding) after entering the hidden feature vector space. This is used to mark and preserve the source attributes of the original information during feature interaction, ensuring that the model can take into account the modal specificity of both visual composition and text semantics when making deep fusion decisions.

[0036] In terms of dataset construction, the system collects expert operation sequences containing a causal chain of "interface state - expert action - jump feedback," providing the model with supervised samples with temporal logic, enabling it to learn the complete mapping from perception to decision-making. Regarding the selection of optimization algorithms, the model introduces Cross-Entropy Loss as the core loss function. By quantifying the difference between the predicted action probability distribution and the expert-annotated actions, it uses a backpropagation mechanism to guide the model parameters to converge towards high-confidence decision-making. In terms of the logical learning mechanism, this loss function-based training process allows the model to deeply establish a strong correlation between "abnormal semantic tags" and "self-healing compensation actions." Thus, when unexpected pop-ups are detected in real-time operation, the model can autonomously trigger higher-priority error correction strategies, ensuring the task continuity of the unmanned operating system in complex dynamic environments.

[0037] In terms of task decision-making, the multimodal decision model receives visual features, textual semantic features extracted by an OCR engine, and a sequence of historical operation states sorted by timestamps as input. The system introduces a cross-modal cross-attention mechanism to align visual graphs with textual semantics such as "OK," "Abnormal," and "Connection Failure." When an unexpected error pop-up appears on the interface, the cross-attention mechanism assigns high attention weights to the pop-up area, driving the lightweight Transformer model to parse the current task intent and prioritize generating action instructions such as "Close Pop-up" or "Handle Abnormality," thereby achieving autonomous error avoidance and logical transitions in complex and dynamic environments.

[0038] To ensure high operational reliability, this invention establishes a closed-loop system of "execution-verification-error correction." After the drive click actuator 6 performs a pressing action and retracts, the industrial camera 5 immediately re-captures the current interface image. The industrial control computer 7 determines whether the current interface has undergone the expected state transition, such as the pop-up window disappearing, entering the next menu level, or the progress bar starting to move, by comparing the changes in interface feature entropy or key pixel areas before and after the click. If the action is determined to be as expected, it is confirmed as valid and proceeds to the next task stage; if the interface state is determined not to have changed or an unexpected jump occurs, multi-level error correction logic is triggered.

[0039] The first-level error correction involves micro-compensation clicks based on relocation. The system re-triggers AprilTag pose calculation to eliminate deviations caused by mechanical micro-vibrations and performs repeated "search-style" clicks along a spiral path within the original target pixel area. If the first-level error correction is still ineffective, the system proceeds to the second-level error correction stage. The system automatically reduces the recognition weight of template matching through strategy reconstruction, attempts to identify hidden obstruction elements such as transparent tooltips, and updates the decision instructions. If these attempts continue... This embodiment takes If all attempts fail, the system will enter the third-level error correction stage, which is the safety self-protection stage. The system will record the current abnormal interface snapshot, drive the motion slide rail to reset to the safe zero position, and trigger an audible and visual alarm.

[0040] In actual operation, the system first drives the mobile module 4 to the calibration station, and uses the industrial camera 5 to identify panel features to complete automatic hand-eye calibration. Subsequently, the system monitors the image of the instrument under test in real time, and the multimodal decision model identifies that it is currently in the "main configuration interface" and accurately locates the "Execute Test" button. The industrial control computer 7 calculates the physical space coordinates of the button through homogeneous transformation mapping, drives the servo motor to move, and controls the actuator to press down. If an unexpected "Connection Timeout" warning pops up after clicking, the system will recognize the text semantics through OCR, and the Transformer decision model will generate a "Click to Close" action to handle the abnormal pop-up. The main task will continue to execute after the interface returns to normal. This fully closed-loop operation mode greatly improves the efficiency of unmanned operation of measurement and testing tasks.

[0041] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An unmanned operating system for measuring instruments based on artificial intelligence autonomous decision-making, characterized in that: The instrument includes a transverse slide rail base (1), a longitudinal slide rail (2) slidably mounted on the transverse slide rail base (1), a moving module (4) mounted on the longitudinal slide rail (2), an end-effector vision-execution composite module mounted on the moving module (4), and a control device electrically connected to the end-effector vision-execution composite module and the moving module; the axial direction of the longitudinal slide rail (2) is parallel to the operation panel of the instrument under test, so that the moving module (4) can move in two dimensions in a plane parallel to the operation panel; the end-effector vision-execution composite module includes an industrial camera (5) and a click actuator (6), the industrial camera (5) and the click actuator... The actuator (6) is fixed in parallel to the moving module (4) by a rigid bracket, forming a visual configuration of an eye on the hand. The lens of the industrial camera (5) is set facing the operation panel. The end of the click actuator (6) is equipped with a flexible click head for touching the operation panel. The control device is electrically connected to the industrial camera (5), the driving element for driving the longitudinal slide rail and the displacement of the moving module, and the click actuator (6). The control device performs interface state recognition and decision based on the image captured by the industrial camera (5), drives the moving module (4) to the target physical coordinates, and controls the click actuator (6) to perform the click action.

2. The unmanned operating system for measuring instruments based on artificial intelligence autonomous decision-making as described in claim 1, characterized in that: The control device identifies the interface state through an adaptive weighted hybrid recognition strategy; the strategy includes: dynamic proportion calculation: using the three-frame difference method and background modeling algorithm to extract moving pixel regions in continuous video frames, calculating the ratio of the total number of pixels in the moving region to the total number of pixels on the interface, and obtaining the dynamic element pixel proportion. Weight function allocation: Preset piecewise linear function allocation template matching weights. With deep learning weights ,in ;when Below the first threshold hour, ;when Above the second threshold hour, ;when In When the interval is, Follow The increase is linear; coordinate weighted fusion: obtain the candidate coordinates of the template matching output respectively. and its confidence level And candidate coordinates output by deep learning and its confidence level ; pixel coordinates of the final target control Through formula The calculation yielded the result.

3. The unmanned operating system for measuring instruments based on artificial intelligence autonomous decision-making as described in claim 1, characterized in that: The control device operates a multimodal decision model, which receives visual features captured by the industrial camera (5), interface text features extracted by OCR, and historical operation state sequences as inputs. It uses a linear projection matrix to map the patch sequence of the visual feature map and the text vector sequence to a unified latent feature space of equal dimensions, and superimposes modal coding to preserve modal specificity.

4. The unmanned operating system for measuring instruments based on artificial intelligence autonomous decision-making as described in claim 3, characterized in that: The multimodal decision model acquires task intent recognition capability through supervised training on a labeled expert operation sequence dataset. The dataset contains a causal chain of "interface state - expert action - jump feedback". The model learns to perform fault self-healing actions when it detects a pop-up containing specific abnormal semantic tags by minimizing the action prediction loss.

5. The unmanned operating system for measuring instruments based on artificial intelligence autonomous decision-making as described in claim 1, characterized in that: The target physical coordinates are calculated through a homogeneous coordinate transformation chain. The specific process includes: using the AprilTag flag cluster preset on the panel of the instrument under test, the rotation matrix R and translation vector T of the industrial camera (5) relative to the panel reference system are solved by the PnP algorithm; combined with the camera intrinsic parameter matrix, the rigid offset vector between the camera and the click actuator, and the real-time pose feedback of the slide rail, a mapping equation from pixel coordinates to the reference coordinate system of the motion actuator is constructed.

6. The unmanned operating system for measuring instruments based on artificial intelligence autonomous decision-making as described in claim 1, characterized in that: The system also includes closed-loop verification logic after the action. After the click actuator (6) performs the action, the control device controls the industrial camera (5) to re-acquire images. By comparing the grayscale distribution Shannon entropy of the interface images before and after the action, the change in interface complexity is quantified. When the relative change rate of feature entropy before and after the action exceeds When the jump is completed, it is determined to be in accordance with the expected jump.

7. The unmanned operating system for measuring instruments based on artificial intelligence autonomous decision-making as described in claim 6, characterized in that: The control device is equipped with a multi-level anomaly correction and recovery mechanism, including a first-level error correction: performing an initial radius of [missing information] around the target's physical coordinates. Radial step size is The Archimedes spiral search click method physically compensates for recognition errors by increasing the click radius at equal intervals.

8. The unmanned operating system for measuring instruments based on artificial intelligence autonomous decision-making according to claim 7, characterized in that: It also includes secondary error correction: if the compensation click is invalid, the pose calibration based on AprilTag is retried and the recognition strategy weights are adjusted.

9. The unmanned operating system for measuring instruments based on artificial intelligence autonomous decision-making according to claim 8, characterized in that: It also includes three levels of error correction: if the number of consecutive failures reaches a preset threshold, a safety reset is performed and an alarm is triggered.

10. A method for unmanned operation control of measuring instruments based on artificial intelligence autonomous decision-making, characterized in that, The application of the operating system as described in any one of claims 1-9 includes the following steps: Step S1: The system starts and drives the mobile module (4) to identify the feature markers on the panel of the instrument under test through the industrial camera (5) to complete the hand-eye automatic calibration; Step S2: Capture the interface image in real time, determine the current task stage and locate the pixel coordinates of the target control through a multimodal decision model; Step S3: Calculate the target physical coordinates through homogeneous transformation mapping, and drive the motion actuator to move to the specified position; Step S4: Control the click actuator (6) to press down and perform the operation; Step S5: Perform closed-loop verification by collecting the interface image after the action, and decide whether to proceed to the next task stage or trigger the error correction process based on the verification result.

Images