A medical system automatic write method based on semantic anchor points and dynamic templates

Abstract

The application provides a medical system automatic writing method based on semantic anchor points and dynamic templates, and relates to the technical field of automatic writing, which comprises the following steps: pre-configuring semantic fingerprints of business fields and spatial correlation rules; acquiring a DPI scaling factor and a window displacement vector of a current operating system; screening a text block with the highest matching degree with the semantic fingerprints as an absolute semantic anchor point; mapping a preset relative search range to an absolute search field under a current physical screen; determining a comprehensive energy cost of each candidate input box in the absolute search field; determining a candidate box with the minimum comprehensive energy cost as a final writing target, and performing simulated clicking and keyboard input; and the application can improve the writing accuracy.

Description

Technical Field

[0001] This invention relates to the field of automatic value writing technology, and in particular to an automatic value writing method for medical systems based on semantic anchors and dynamic templates. Background Technology

[0002] During the digital transformation of the healthcare industry, core business systems such as HIS (Hospital Information System) and EMR (Electronic Medical Record System) generally suffer from issues such as fragmented interfaces, version separation, and closed interfaces. Systems from different hospitals, and even different departments within the same hospital, exhibit significant differences in field label text (e.g., "chief complaint" versus "chief complaint"), input box layout, and DPI scaling adaptation. Existing automated data entry solutions primarily rely on absolute coordinate recording, pixel matching, or obtaining underlying control handles. These methods are prone to failure when faced with window displacement, resolution changes, or UI fine-tuning, resulting in extremely high script maintenance costs. Furthermore, in complex forms with multiple similar input boxes, tools lacking semantic awareness often experience sequential or misaligned data entry and lack closed-loop verification mechanisms, failing to meet the stringent accuracy requirements of medical scenarios. Summary of the Invention

[0003] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0004] According to a first aspect of this application, an automatic value writing method for a medical system based on semantic anchors and dynamic templates is provided, comprising the following steps:

[0005] S100, pre-configure dynamic templates for business fields, the dynamic templates including: semantic fingerprints, spatial association rules, and adaptation parameters; the semantic fingerprints include regular expressions for matching field labels; the spatial association rules include the preset orientation of the input box relative to the label and the relative search radius range; the adaptation parameters include the template reference resolution; the dynamic templates are decoupled from physical coordinates;

[0006] S200 captures the real-time image of the target window, extracts the full-screen text and corresponding pixel-level bounding boxes through the OCR engine, and obtains the DPI scaling factor and window displacement vector of the current operating system.

[0007] S300, Traverse the text blocks extracted by OCR, run a confidence model based on weighted multi-factor fusion, and select the text block with the highest matching degree with the semantic fingerprint as the absolute semantic anchor point; the factors of the confidence model include text semantic similarity, spatial prior probability and context environment matching degree.

[0008] S400, using the DPI scaling factor and window displacement vector to construct an affine transformation matrix, mapping the relative search radius range and the relative offset vector corresponding to the preset orientation in the dynamic template to the absolute search field under the current physical screen; the mapping relationship simultaneously compensates for DPI scaling and window displacement.

[0009] S500: Generate candidate input boxes using a preset method; within the absolute search field, determine the comprehensive energy cost of each candidate input box; the comprehensive energy cost is determined based on the distance between the candidate box and the anchor point, the angular deviation relative to the preset orientation, and the morphological characteristics of the candidate box; the candidate box with the minimum comprehensive energy cost is determined as the final write target, and simulated clicks and keyboard input are executed.

[0010] The present invention has at least the following beneficial effects:

[0011] This invention presents an automatic data entry method for medical systems based on semantic anchors and dynamic templates. By introducing semantic anchors to replace traditional dead-coordinate positioning, it utilizes a weighted multi-factor fusion confidence model to accurately identify field labels from OCR text. Combined with affine transformation matrices to dynamically compensate for DPI scaling and window displacement, it achieves zero-code adaptive migration of automated scripts across different hospitals, scaling ratios, and UI styles. Simultaneously, it employs a comprehensive energy cost function incorporating distance, direction, and morphological features to optimize candidate input boxes, effectively avoiding the problem of incorrect serialization in dense forms. Compared to existing technologies, this invention significantly reduces the operational costs of cross-hospital deployments, shortens single-system deployment time from several days to several hours, and greatly improves data entry accuracy, providing a highly reliable, low-maintenance, and non-intrusive solution for automated medical data entry. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating the automatic value writing method for a medical system based on semantic anchors and dynamic templates provided in this embodiment of the invention;

[0014] Figure 2 This is a schematic diagram of the decoupling template and anchor point matching provided in an embodiment of the present invention;

[0015] Figure 3 This is a comparison chart of configuration-based execution optimizations provided in embodiments of the present invention;

[0016] Figure 4This is a system architecture design diagram based on YAML configuration-driven architecture provided in an embodiment of the present invention;

[0017] Figure 5 A schematic diagram of the principle of affine transformation and adaptive mapping of spatial field provided in the embodiments of the present invention;

[0018] Figure 6 The flow diagram for verifying the dual-modal closed-loop state machine provided in this embodiment of the invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that, based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Furthermore, this device and / or practice the method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0021] The following will refer to Figure 1 The flowchart shown is a method for automatically writing values ​​in a medical system based on semantic anchors and dynamic templates. This paper introduces a method for automatically writing values ​​in a medical system based on semantic anchors and dynamic templates.

[0022] The automatic value writing method for the medical system based on semantic anchors and dynamic templates may include the following steps:

[0023] S100, pre-configure dynamic templates for business fields. The dynamic templates include: semantic fingerprints, spatial association rules, and adaptation parameters. The semantic fingerprints include regular expressions for matching field labels. The spatial association rules include the preset orientation of the input box relative to the label and the relative search radius range. The adaptation parameters include the template reference resolution. The dynamic templates are decoupled from physical coordinates.

[0024] In this embodiment, as Figure 2 As shown, this step is completed offline before the automation task starts, and the configuration data is stored in a structured file (such as YAML or JSON). The configuration consists of four parts:

[0025] Semantic fingerprint: Define one or more regular expressions for each business field that needs to be automatically filled in, to match field labels in the HIS / EMR interface.

[0026] Example: Define a regular expression for the "Present Illness" field. This can match variations such as "present illness history", "present illness history", "present illness history:", and "present illness history:". For "chief complaint", a regular expression is defined. Additionally, a list of display names ["Chief Complaint", "Chief Complaint", "ChiefComplaint"] can be attached for alternative matches.

[0027] Spatial association rules: Define the orientation of the target input box relative to the field label and the search range.

[0028] Preset orientation: Enumerated values ​​such as right (right side), below (below), left (left side), above (above), or a composite orientation right_or_below.

[0029] Search radius range: the minimum and maximum pixel distance at a reference resolution (e.g., 1920×1080), such as radius_min:30, radius_max:300.

[0030] Once configured, deploy the files to a local directory accessible to the automation program or a remote configuration center. The system loads the configuration upon startup and resolves it into in-memory rule objects.

[0031] By pre-defining semantic fingerprints and spatial association rules, business knowledge is decoupled from hard-coded data. Operations personnel do not need to write code; they can adapt to UI differences between different hospitals (such as variations in label text and changes in the relative positions of input boxes) simply by modifying configuration files. This enables zero-code cross-hospital deployment, significantly reducing implementation and maintenance costs.

[0032] Furthermore, the semantic fingerprint and spatial association rules in step S100 are stored in the remote configuration center in YAML format. The rule entries include the following fields: field name, regular expression pattern, priority, preset orientation, minimum search radius, and maximum search radius. The system requests the rule version number from the configuration center every preset period. When the local version number is lower than the remote version number, an incremental update is performed, and only the changed rule entries are downloaded. After the update, it is automatically loaded into memory without restarting the automation process.

[0033] In this embodiment, the step may include the following sub-steps:

[0034] S110: Define the rule data structure and store it in YAML format in a remote configuration center.

[0035] Figure 4 This embodiment presents a YAML-based configuration-driven system architecture design diagram. For each business field requiring automatic filling, a rule entry is defined in YAML format, including the following required fields:

[0036] field_name: Field identifier (e.g., present_illness).

[0037] regex_pattern: A regular expression used to match field labels in the interface. .

[0038] priority: A positive integer. The smaller the value, the higher the matching priority. It is used to resolve conflicts when multiple rules match the same text block.

[0039] direction: Preset direction, enumerated values ​​are right, below, left, above, or right_or_below.

[0040] radius_min, radius_max: The minimum and maximum search radius (in pixels) at the reference resolution (e.g., 1920×1080).

[0041] YAML configuration example (saved as rules.yaml):

[0042] version:1.3

[0043] hospital_group:"general"

[0044] fields:

[0045] - field_name:"present_illness"

[0046] regex_pattern:" "

[0047] priority:1

[0048] direction:"right"

[0049] radius_min:30

[0050] radius_max:250

[0051] - field_name:"past_history"

[0052] regex_pattern:" "

[0053] priority:2

[0054] direction: "below"

[0055] radius_min:20

[0056] radius_max:180

[0057] The remote configuration center can use components such as etcd, Nacos, or Redis to store the aforementioned YAML content under a fixed key, such as / automation / rules / current. Simultaneously, the configuration center maintains a version number for this key (which can be an integer or a content hash value), incrementing the version number and storing the new content each time a rule is updated.

[0058] Rules are defined using a structured YAML format, completely separating business knowledge from automation code. Rules are centrally stored in a remote configuration center, laying the foundation for unified management and dynamic deployment across multiple clients.

[0059] S120: Deploy the remote configuration center and initialize rule version management.

[0060] Configuration center deployment: Deploy an etcd cluster on an internal server (or use a cloud-hosted service), and open the HTTP / gRPC access port. Configure access authentication (such as username / password or TLS certificate).

[0061] Rule initialization: The operations and maintenance personnel upload the written rules.yaml to the configuration center. The system automatically calculates the MD5 value of the file as the initial version number and records the update timestamp.

[0062] Client configuration: The configuration file of each automated client (i.e., the medical system terminal running the method of this invention) contains the access address of the configuration center, authentication information, rule key name and polling interval (default 30 seconds).

[0063] Local caching: When the client starts for the first time, it pulls the complete rule file from the configuration center and saves it to the local disk (such as cached_rules.yaml), while also recording the version number.

[0064] This step establishes a centralized rule management and multi-terminal synchronization architecture. Operations personnel only need to modify the rules once in the configuration center, and all terminals will receive the update, avoiding the tedious and error-prone process of manually copying configuration files to each machine. Local caching ensures that clients can continue running using the last valid rule even if the configuration center's network connection is interrupted.

[0065] S130: Periodically requests the rule version number from the configuration center to determine whether an update is needed.

[0066] After the client starts, it creates a separate background thread (or timer) that performs the following operations every preset period (e.g., 30 seconds): It sends a request to the configuration center to retrieve only the version number metadata of the rule key, without fetching the complete content. In etcd, the current modifiedIndex can be obtained via the `?consistent=true` parameter. In Nacos, the content hash can be obtained via the listener interface or an HTTP HEAD request. It compares the remote version number with the version number cached locally on the client. If the remote version number is greater than the local version number (or the content hash is inconsistent), it proceeds to the update process (step S140); otherwise, it continues to sleep for 30 seconds.

[0067] S140: Perform incremental updates, download only the changed rule entries, and hot-load them into memory.

[0068] When a remote version number update is detected, the client performs the following steps:

[0069] Retrieve old and new rule content: Pull the complete rules.yaml content from the configuration center, and load the old version of the rules cached locally.

[0070] Incremental calculation of differences: Compare the field lists in the old and new rules to identify newly added, deleted, or modified rule entries. Modification can be determined by comparing the complete content of the entry (or its hash value).

[0071] Furthermore, only the changed parts can be downloaded: if the configuration center supports partial key-value queries (such as the etcd directory structure), each rule can be stored as an independent subkey, for example: / automation / rules / fields / present_illness, / automation / rules / fields / past_history. In this way, the client only needs to GET the subkeys whose version numbers have changed, instead of the entire rule set.

[0072] Hot reloading to memory: The client uses atomic operations (such as read-write locks or double buffering) to replace the rule objects in memory with the new rules. Specifically, a new rule map is created, built entirely on the new rules, then the client's rule pointers are atomically pointed to the new map, and finally the memory of the old map is released. This process does not require restarting the automation process.

[0073] Update local cache: Write the new rule content to the local disk cached_rules.yaml and update the local version number to the remote version number.

[0074] Example: In the old rule, the radius_max of present_illness was 250; in the new rule, it is changed to 300. The client only downloads the new value for that subkey, updates the corresponding field in memory, and then rebuilds the rule object for that field. The entire operation takes less than 10ms and does not affect ongoing write tasks.

[0075] Incremental update mechanism significantly reduces network traffic (especially when there are many rule entries), while hot reloading technology ensures that rule updates do not interrupt ongoing automated tasks. Hospital operations and maintenance personnel can adjust field rules in the configuration center at any time (for example, the search radius of a hospital's special needs outpatient department needs to be expanded), and all terminals will automatically take effect within 30 seconds, without queuing for restarts or manual intervention, greatly improving the operational efficiency and system availability under large-scale deployment.

[0076] S200 captures a real-time image of the target window, extracts full-screen text and corresponding pixel-level bounding boxes through an OCR engine, and obtains the DPI scaling factor and window displacement vector of the current operating system.

[0077] In this embodiment, the handle of the target HIS / EMR window and its current position on the screen (top-left corner coordinates (win_x, win_y), width win_w, height win_h) can be obtained by calling operating system APIs (such as Windows' FindWindow and GetWindowRect). Then, a screenshot function (such as PrintWindow or BitBlt) can be used to capture the pixel image of the window.

[0078] The captured image is fed into an OCR engine (such as Tesseract or PaddleOCR). The engine returns a list of recognition results, each containing: the recognized text string "text"; and the bounding box coordinates (x_min, y_min, x_max, y_max) in the image (in pixels, relative to the top-left corner of the captured image). For subsequent localization, the bounding box coordinates need to be converted to absolute screen coordinates: adding the coordinates of the top-left corner of the window (win_x, win_y).

[0079] DPI scaling factor (s) x ,s y ): Obtain the current screen scaling ratio via system APIs (such as GetDpiForWindow or reading the registry). For example, at 125% scaling, s x =s y =1.25.

[0080] Window displacement vector (Δx, Δy): that is, the coordinates of the top left corner of the window relative to the screen origin (0, 0), i.e. (win_x, win_y) mentioned above.

[0081] Example: Assuming the top left corner of the window is at screen coordinates (100, 200), the DPI is scaled to 125%, and the OCR recognizes the bounding box of "present medical history" at coordinates (50, 80, 150, 110) within the window, then the absolute screen coordinates are (150, 280, 250, 310).

[0082] By capturing window images and environmental parameters in real time, all subsequent calculations are based on the current physical screen state, rather than pre-recorded fixed coordinates. Even if the user moves the window or the operating system's zoom level changes, the system can still obtain accurate visual information and spatial references, laying the foundation for adaptive positioning.

[0083] S300, traverse the text blocks extracted by OCR, run a confidence model based on weighted multi-factor fusion, and select the text block with the highest matching degree with the semantic fingerprint as the absolute semantic anchor point; the factors of the confidence model include text semantic similarity, spatial prior probability and context environment matching degree.

[0084] In this embodiment, the system scores each text block recognized by OCR based on three dimensions: whether the text content matches the target label (e.g., "present medical history"), whether the text position is in a common area (e.g., the center of the screen), and whether there are related words around it (e.g., "onset time"). The text block with the highest score is selected as the "semantic anchor point," which is the reference point for locating the input box in the future.

[0085] Furthermore, the confidence model in step S300 is expressed as follows:

[0086] ;

[0087] in, For the first OCR text blocks Confidence level; This is a preset semantic fingerprint template; For spatial prior information; For contextual environment templates; This is a text semantic similarity function; Let be the spatial prior probability function; This is a context matching function; These are the preset weighting coefficients.

[0088] In this embodiment, the system provides each OCR text block with... Calculate a confidence score The score is a weighted sum of three independent factors (textual semantic similarity, spatial prior probability, and contextual matching). : No. Each OCR text block contains a string. and its center coordinates (x) i y i ). : A semantic fingerprint template preset for the current business field, including a regular expression and one or more standard names (such as "present medical history"). Spatial prior information, usually the center point of a Gaussian distribution (such as the center of the screen) and bandwidth parameters. : Contextual template, which is a list of related terms that frequently co-occur with this field (such as "onset time" and "cause"). , , : Three factor functions, each taking values ​​in the range [0,1]. The default value is the experience value, which can be set to... , , .

[0089] Furthermore, the system can record historical matching success rates and make slight adjustments when there are consecutive failures. (Step size ±0.05), select the combination that minimizes the mean squared error of the confidence level of the nearest match.

[0090] By employing weighted multi-factor fusion, the system can comprehensively evaluate the reliability of text blocks as semantic anchors, overcoming the limitations of a single dimension (text only or location only). The weight allocation reflects the core position of semantic matching while taking into account the spatial layout habits and semantic context of medical forms, significantly improving robustness against variant labels, OCR recognition errors, and interface layout interference.

[0091] Furthermore, the text semantic similarity The following relationship must be satisfied:

[0092] ;

[0093] in: For regular expression matching indicator functions: when a complete match is achieved. Pick When matching regular expressions Pick ;otherwise, Pick ; To edit distance; The string length is specified; max() is the default function for finding the maximum value.

[0094] The spatial prior probability Using a Gaussian kernel function:

[0095] ;

[0096] This is the normalized distance from the center of the text box to the center of the screen. For bandwidth parameters;

[0097] ;

[0098] In order to The number of preset related words hit within a 150-pixel radius centered on the target; The total number of pre-defined related terms for the corresponding field; For context matching degree.

[0099] Used for measurement The text content and the preset semantic fingerprint template The degree of matching. The regular expression matching indicator function has three possible values:

[0100] Exact match: OCR text strings and templates The standard names in the text are exactly the same (e.g., "present illness history" and "present illness history"). .

[0101] Regular expression matching: OCR text cannot perfectly match standard names, but it can be matched by preset regular expression patterns (e.g., "present medical history" matches the regular expression ^present\s*disease\s*history$, "present medical history:" matches ^present\s*disease\s*history[::]?$). .

[0102] No match: The regular expression also cannot match, in this case .

[0103] Edit distance, coming soon The minimum number of single-character editing operations (insert, delete, replace) required to convert to a template standard name.

[0104] : The maximum length of both the OCR text and the template standard name, used to normalize the edit distance to the [0,1] interval.

[0105] Normalized edit distance similarity. When two strings are completely identical, the edit distance is 0, and this item is 1; as the difference increases, this item decreases linearly to 0.

[0106] Weights 0.6 and 0.4: Give a higher weight (0.6) to the regular matching result because regular matching is a strong signal; the edit distance similarity is used as a supplement (0.4) to handle cases where regular matching fails but the texts are highly similar (for example, "现病吏" recognized by OCR has an edit distance of 1 and a similarity of about 0.8). The specific values of the above weights can be used as preferred values, and the above weights and thresholds can be adjusted according to different medical system interfaces, field types, and OCR quality.

[0107] In this embodiment, a perfect match gets a full score: In the most ideal case, it is given 1.0.

[0108] Regular matching is given 0.9: Although the string is not exactly equal to the standard name (such as containing spaces or colons), regular matching confirms that it is a legal variant of the target label and should be given a reward close to the full score without relying on the edit distance (the edit distance may be non-zero. For example, "现 病 史" has two more spaces than "现病史", and the edit distance is 2. If only using the edit distance similarity, it will be severely underestimated).

[0109] Rely on the edit distance when regular matching fails: When there are real recognition errors in OCR (such as "吏" instead of "史"), regular matching fails, but the edit distance is small, and a certain score (such as 0.267) can still be given, so as not to completely discard the candidate.

[0110] Weight assignment: The weight of regular matching (0.6) is higher than that of the edit distance (0.4), which reflects the principle of "strong signal first" and avoids unreasonable score drops due to minor format differences.

[0111] Regular matching ensures lossless recognition of the standard format, while the edit distance provides fault tolerance, allowing for a small number of extra characters or typos during OCR recognition (such as the edit distance of "现病吏" is 1 and the score is 0.8). The combination of the two enables the system to both precisely match and smoothly handle minor errors, thus maintaining a high recall rate in real medical interfaces (which often have non-standard labels).

[0112] Based on the layout habits of medical forms: Field labels usually do not appear at the screen edge but are mostly concentrated in the upper half or the central area. Use a two-dimensional Gaussian kernel function to calculate the probability density of the center point (x i , y i ) of the text block relative to the center of the screen:

[0113] ;

[0114] Among them, is the Euclidean distance (after normalization) from the center of the text block to the center of the screen:

[0115] ;

[0116] These are the screen width and height, respectively. . This is a bandwidth parameter that controls the degree of concentration of spatial priors. (Default) .when hour, ;when hour, ;when hour, .

[0117] In practice, the screen center coordinates can be pre-calculated in the code, and the normalized distance can be quickly calculated for each text block.

[0118] Furthermore, step S300 also includes the following steps:

[0119] S310, when the regular expression matching in step S300 fails completely, calculate the i-th OCR text block respectively. The string is compared with the minimum edit distance of each display name in the current field in the rule base, and the minimum value is selected; if the minimum value is less than or equal to the preset minimum edit distance threshold and If the string length is greater than a preset string length threshold, it is considered an approximate match, and... confidence level Multiply by the preset discount factor Then it will be re-sorted.

[0120] When a regular expression fails to match the OCR text, the system calculates the minimum edit distance between the text and all displayed names (such as "present medical history", "present medical history:") in the rule base for the current field. If this distance is small (e.g., ≤2), and the text itself is not too short (e.g., length >3, to avoid accidental matching of a single character), it is considered an approximate match. In this case, the original confidence score of the text block is multiplied by a discount factor (e.g., 0.85), and then sorted together with other text blocks. This allows OCR to retain text even when a small number of character errors occur, while appropriately lowering its priority to prevent false matches.

[0121] Furthermore, for the preset key fields, the system calls two OCR engines, Tesseract and PaddleOCR, in parallel to recognize the same image region; the recognition result with higher character-level confidence and completely consistent output text from both engines is taken as the final OCR text; if the outputs of the two engines are inconsistent, the result with higher average confidence is selected.

[0122] For high-value fields such as allergy history and diagnosis conclusions, the system simultaneously activates two different OCR engines (Tesseract and PaddleOCR) to recognize the same image area. Compare the output text and character-level confidence of the two: If the recognition results of the two engines are exactly the same and the confidence levels are both high, then adopt this result; if the results are inconsistent, then select the one with a higher average confidence level and record "low consensus" in the log. This strategy significantly reduces the risk of misrecognition of a single OCR engine through cross-validation between engines.

[0123] S320, when the confidence level of a single character output by the OCR engine is lower than the preset threshold, query the preset glyph similarity mapping table for candidate characters that are glyph-similar to the current character and conform to the context; if there is a unique match, automatically replace it and increase the confidence level of the character; if there are multiple candidates, retain the original recognition result and mark it for manual review; the glyph similarity mapping table is used to record pairs of easily confused characters.

[0124] When the confidence level of a certain character by the OCR engine is lower than the preset threshold (such as 0.7), the system queries the built-in glyph similarity mapping table (such as "已" and "己", "未" and "末"). If, with the assistance of the context of the character (such as the adjacent characters before and after), there is a unique matching candidate character in the mapping table, automatically replace the character and increase its confidence level to 0.9; if there are multiple possible candidates, retain the original recognition result and mark it for manual review. This step performs targeted error correction based on common glyph confusions in medical terms to improve recognition accuracy.

[0125] The above steps significantly improve the robustness and accuracy of semantic anchor point positioning through approximate matching tolerance, parallel verification of multiple OCR engines, and correction of the glyph similarity of low-confidence characters. This mechanism can effectively handle common OCR recognition errors (such as character omissions, glyph confusions) and single-engine systematic biases in medical interfaces. When regular matching fails completely, it can still retain candidate anchor points through the edit distance, and further reduce the misrecognition rate through dual-engine cross-validation and glyph error correction. The combined effect of the three significantly improves the tolerance of the confidence model to noisy interfaces, and the anchor point recall rate of key fields is close to the ideal level. At the same time, it avoids incorrect value writing caused by mismatching, providing a more reliable input for subsequent dynamic field mapping and target optimization, thus ensuring the stable operation of the entire automated value writing process in complex real environments.

[0126] S400, use the DPI scaling factor and window displacement vector to construct an affine transformation matrix, and map the relative search radius range and the relative offset vector corresponding to the preset orientation in the dynamic template to the absolute search field under the current physical screen; the mapping relationship compensates for DPI scaling and window displacement at the same time.

[0127] Furthermore, the affine transformation matrix M in step S400 is:

[0128] ;

[0129] in, These are the DPI scaling factors for the current operating system in the horizontal and vertical directions, respectively; These represent the horizontal and vertical displacements of the top-left corner of the target window relative to the screen origin, respectively.

[0130] The absolute search field is mapped by the following formula:

[0131] ;

[0132] To preset the relative search vector, superscript Indicates matrix transpose; These represent the relative horizontal and vertical offsets of the target input box relative to the semantic anchor point, respectively, at the template reference resolution. Based on template resolution settings; This represents the absolute search field coordinate range under the current physical screen.

[0133] like Figure 5 As shown, step S400 involves an affine transformation matrix. Set the preset relative search vector Mapped to the absolute search field under the current screen Specifically, matrix DPI scaling factor and window offset This configuration can simultaneously compensate for coordinate drift caused by system scaling and window movement. Preset relative search vector. In This is the offset of the target input box relative to the semantic anchor point (e.g., 140 pixels to the right), defined at the template resolution (e.g., 1920×1080). It is achieved through matrix multiplication. The system converts this relative offset into an absolute coordinate range on the current physical screen in real time, thus accurately defining the search area at any resolution and window position. This mapping ensures the cross-environment adaptability of the automation script, eliminating the need to individually calibrate coordinates for each terminal.

[0134] S500: Generate candidate input boxes using a preset method; within the absolute search field, determine the comprehensive energy cost of each candidate input box; the comprehensive energy cost is determined based on the distance between the candidate box and the anchor point, the angular deviation relative to the preset orientation, and the morphological characteristics of the candidate box; the candidate box with the minimum comprehensive energy cost is determined as the final write target, and simulated clicks and keyboard input are executed.

[0135] In this embodiment, a comprehensive energy cost is calculated for each candidate input box within the absolute search field. This cost comprehensively considers the distance between the candidate box and the semantic anchor point, the angular deviation relative to a preset orientation, and the morphological features of the candidate box itself (such as aspect ratio and background texture). The system selects the candidate box with the lowest energy cost as the final write target, and then simulates a mouse click to input the content to be written. This mechanism can intelligently select the best-matching target from multiple neighboring input boxes, avoiding misfilling.

[0136] Furthermore, the preset methods include: edge detection and contour analysis, color segmentation and blank area detection, editable region SVM classifier prediction, and OCR text block neighborhood blank area recognition.

[0137] Edge detection and contour analysis: Perform Canny edge detection on the image within the absolute search field to find closed rectangular contours, and filter out regions similar to the input box based on aspect ratio, area, etc.

[0138] Color segmentation and blank area detection: Based on the characteristic that the input box is usually a white / light-colored background, the blank area is segmented by color thresholding, and then the candidate box is located by combining shape analysis.

[0139] Editable region SVM classifier prediction: A pre-trained SVM model (based on features such as color histogram and LBP texture) is used to classify the sliding window in the search field and output the probability of belonging to the editable input box. High-probability regions are selected as candidates.

[0140] OCR text block neighborhood blank area recognition: In the vicinity of the field label recognized by OCR (such as to the right or below), find the adjacent blank rectangular area (without text or icon) and use it as a candidate input box.

[0141] Furthermore, the overall energy cost in step S500 satisfies the following relationship:

[0142] ;

[0143] in, For the first The overall energy cost of each candidate input box; For the first The center coordinates of the candidate input boxes; The coordinates of the semantic anchor point; This is the Euclidean distance between the candidate box and the anchor point; The angle between the candidate box and the preset orientation vector; For the first The shape dissimilarity of the candidate input boxes; These are the weighting coefficients;

[0144] The following relationship must be satisfied:

[0145] ;

[0146] Penalty for aspect ratio; Penalize the background texture; , The aspect ratio of the candidate bounding box; , The probability that the candidate bounding box output by the SVM classifier belongs to an editable region; These are the weighting coefficients.

[0147] Comprehensive energy cost in step S500 This formula is used to select the most likely target from multiple candidate input boxes. The three terms in the formula correspond to:

[0148] Distance Term The Euclidean distance between the candidate box center and the semantic anchor point is used. The smaller the distance, the lower the cost, and it encourages getting closer to the label.

[0149] Directional item : It is the angle between the candidate box and the preset orientation vector (such as the vector (1,0) corresponding to "right side"). Increase the size when deviating from the preset direction to ensure that the candidate box appears in the correct position (e.g., to the right of the label, not above it).

[0150] Morphological items : This represents the shape dissimilarity; a larger value indicates that the candidate box is less similar to the input box. , A typical input box has an aspect ratio of about 3; the greater the deviation, the greater the penalty. , It is the probability that the SVM classifier (trained based on features such as color histogram and LBP texture) predicts that the region is an editable input box, and the more the background is unlike an input box, the higher the penalty. Balance the weight of aspect ratio and background texture (default 0.5).

[0151] By weighted summation (default) (The above values ​​are empirical values). The candidate box with the lowest energy is the best write target.

[0152] The integrated energy cost function combines spatial distance, orientation logic, and visual morphology, enabling it to intelligently select the most likely target from multiple similar-looking input boxes, avoiding misplacement caused by simple nearest neighbor (e.g., accidentally filling in another input box above). This decision-making mechanism endows automated programs with human-like "common sense reasoning" capabilities, significantly improving the accuracy of writing values ​​in complex forms.

[0153] Furthermore, the integrated energy cost function in this embodiment fully reflects the analysis of topological orientation constraints, color / border features, and control editability, which distinguishes it from ordinary nearest neighbor search or the fixed coordinate scheme of traditional RPA.

[0154] Topological orientation constraints: the direction term in the energy cost In addition to penalizing angle deviations, different angle tolerance ranges can be set according to preset orientations (such as "right side", "below" or "right side or below") to ensure that the candidate box must be located within the reasonable topological direction of the label, and to avoid misjudging the input box on the other side as the target.

[0155] Color / Grayscale / Border Features: In terms of shape dissimilarity In addition to aspect ratio penalties, the average color of the candidate boxes (e.g., white or light gray background) and border gradient features (e.g., whether there is an inner shadow or border line at the edge) are also introduced. Specifically, this is achieved by calculating the cosine similarity between the hue histogram of pixels within the candidate boxes and a typical input box template, and by detecting border strength using the Sobel operator, thus forming a color-border penalty term. It is weighted and blended with aspect ratio penalty and background texture penalty.

[0156] Editability features of the control: An offline-trained SVM classifier is used (inputs include HOG features of candidate boxes, LBP texture, color moments, etc.), and the output is the probability that the region belongs to an editable input box. Background texture penalty This reflects editability. Furthermore, the control type can be further confirmed by detecting cursor blinking characteristics (changes over multiple frames) or system API calls.

[0157] By combining the above multi-dimensional features, the energy cost function can accurately distinguish similar-looking controls such as input boxes, buttons, and static text in dense forms, significantly improving the robustness of positioning.

[0158] Furthermore, after step S500, the method further includes the following steps:

[0159] S600: After writing the value, capture the window image again. If the state of the focus area and the target area meets the preset state conditions, the image difference meets the first preset condition, and the text similarity meets the second preset condition, then the judgment is successful; otherwise, trigger the retry mechanism.

[0160] Furthermore, the preset state conditions are: the center distance between the focal region and the target region is less than a preset distance threshold and the area overlap ratio is not less than a preset overlap threshold; wherein, the focal region is obtained through system API or visual cursor detection, and the target region is the final write target region determined in step S500; the first preset condition corresponding to the image difference is that the difference is not less than the first preset threshold; the second preset condition corresponding to the text similarity is that the text similarity is not less than the second preset threshold.

[0161] After simulating clicks and keyboard input, the system recaptures the target window image and verifies the following three conditions in sequence. The write operation is considered successful only if all three conditions are met; otherwise, a retry mechanism is triggered.

[0162] 1. Preset state conditions: The state of the focus area and the target area are matched.

[0163] Target area: The range of pixel coordinates of the final write target input box determined in step S500 (e.g., a rectangle (x_min, y_min, x_max, y_max)).

[0164] Focus area: The UI element area where the current operating system's keyboard focus is located, usually manifested as a blinking cursor within an input box or a highlighted border. It can be obtained in two ways:

[0165] System APIs (such as Windows' GetFocus to get the handle of the focused control, and then GetWindowRect to get its screen coordinates).

[0166] Visual cursor detection: Perform multi-frame differential analysis on the screenshot after writing the value, locate the cursor blinking position, and then expand it into a candidate focus area.

[0167] Specific conditions: **Center distance less than a preset distance threshold:** The Euclidean distance between the center of the focal area and the center of the target area should be less than the threshold (e.g., 10 pixels). This condition ensures the focal point roughly falls within the target area. **Area overlap greater than a preset overlap threshold:** The ratio of the intersection area of ​​the focal area and the target area to the area of ​​the target area (or the union area) should be greater than the threshold (e.g., 80%). This condition ensures a high degree of overlap between the focal area and the target area.

[0168] 2. Image difference condition: Changes in the image before and after writing values.

[0169] Compare the images of the target area in the screenshots taken before and after the value was written, and calculate the image difference (the larger the value, the more significant the change). You can choose any of the following metrics:

[0170] Pixel change rate: The proportion of the number of pixels that have changed (grayscale difference or RGB difference exceeding the threshold) to the total number of pixels.

[0171] Mean Square Error (MSE): The mean of the squared difference in gray levels of the corresponding pixels.

[0172] 1-Structural Similarity Index (1-SSIM): more in line with human visual perception, with a value range of [0,1], where 1 indicates completely different.

[0173] 1-Image overlap (1-IoU): If the image is binarized (foreground / background), calculate the IoU between the foreground and background regions, and the difference = 1-IoU.

[0174] First preset condition: The difference is not lower than the first preset threshold (e.g., 0.15). That is, at least 15% of the pixels must change (or the structural similarity must decrease by more than 0.15).

[0175] 3. Text similarity condition: The written content matches the expected content.

[0176] OCR recognition is performed on the target area in the screenshot after writing the value to obtain the recognized text. , and the string to be written Calculate similarity. Use the mean of weighted edit distance and Jaro-Winkler similarity:

[0177] ;

[0178] in, The string to be written; This refers to the text recognized by OCR after writing the value. To be Convert to Minimum number of single-character editing operations required; They are respectively and The string length; max() is the function to find the maximum value; for and Jaro-Winkler similarity between them.

[0179] Second precondition: Text similarity Not lower than the second threshold (e.g., 0.92).

[0180] like Figure 6 As shown, step S600 ensures that the write operation is accurate and effective through double verification.

[0181] As a preferred embodiment, the text similarity threshold The requirement is that the similarity between the text recognized by OCR after writing the value and the expected string should not be less than 0.92, to ensure the content is correct.

[0182] Text similarity Using an equal-weighted average of two distances:

[0183] Normalized edit distance: This reflects global character differences (insertion, deletion, replacement). A value of 1 indicates identical characters, and the value decreases as the difference increases.

[0184] Jaro-Winkler similarity: It gives higher weight to string prefix matching, and is suitable for common prefixes in medical terms (such as "hypertension" and "hypertensive disease"), and can more sensitively capture correct entries.

[0185] The average of both metrics can comprehensively assess character-level differences and prefix consistency, avoiding bias from a single indicator. A threshold of 0.92 is relatively strict and suitable for high accuracy requirements in medical settings. A retry is triggered if image difference does not meet the first preset condition, text similarity does not meet the second preset condition, or focus state does not meet the preset conditions.

[0186] In this embodiment, geometric overlap verifies whether the click action is effective (whether the focus falls on the input box), and text similarity verifies whether the written content is accurate. The combination of the two can detect abnormal situations such as interface lag, occlusion, and dynamic rearrangement, and automatically retry, further improving the reliability of writing values.

[0187] Combined with appendix Figure 6 The dual-modal closed-loop state machine verification flow diagram shown indicates that step S600 actually corresponds to the "dual-modal closed-loop feedback verification" stage in the diagram. In this stage, the system utilizes real-time screenshot analysis to achieve dual judgment: firstly, "calculate IoU and..." All are greater than the safety threshold.

[0188] Here, IoU corresponds to the aforementioned preset state conditions, that is, by comparing the degree of spatial overlap (center distance and area overlap ratio) between the current focus area after writing the value and the target area determined in step S500, it is ensured that the mouse click correctly reaches the target input box; Then, corresponding to the aforementioned text similarity conditions, the formula for calculating the average of the weighted edit distance and Jaro-Winkler distance defined in step S600 is used to ensure that the written content is accurate.

[0189] If IoU and If all values ​​exceed the preset security threshold, the system determines the current write operation is successful and the process ends; if verification fails (e.g., due to window obstruction causing focus shift or OCR recognition anomalies), then a trigger is initiated. Figure 6 The system employs a "retry strategy." It enters anti-interference processing logic and re-initiates the write operation, looping within the retry limit until verification passes or an exception is thrown if the maximum number of retries is exceeded. This bimodal closed-loop mechanism, through joint verification of spatial layout and semantic content, effectively eliminates interference from UI lag, obstruction, or abnormal reordering, significantly improving the stability and accuracy of automated write operations in the medical system.

[0190] like Figure 3 As shown, compared with the prior art, the solution of the present invention has the advantages of low UI preemption overhead and low maintenance cost.

[0191] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0192] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention.

Claims

1. A method for automatically writing values ​​in a medical system based on semantic anchors and dynamic templates, characterized in that, Includes the following steps: S100, pre-configure dynamic templates for business fields, the dynamic templates including: semantic fingerprints, spatial association rules, and adaptation parameters; the semantic fingerprints include regular expressions for matching field labels; the spatial association rules include the preset orientation of the input box relative to the label and the relative search radius range; the adaptation parameters include the template reference resolution; the dynamic templates are decoupled from physical coordinates; S200 captures the real-time image of the target window, extracts the full-screen text and corresponding pixel-level bounding boxes through the OCR engine, and obtains the DPI scaling factor and window displacement vector of the current operating system. S300, Traverse the text blocks extracted by OCR, run a confidence model based on weighted multi-factor fusion, and select the text block with the highest matching degree with the semantic fingerprint as the absolute semantic anchor point; the factors of the confidence model include text semantic similarity, spatial prior probability and context environment matching degree. S400, using the DPI scaling factor and window displacement vector to construct an affine transformation matrix, mapping the relative search radius range and the relative offset vector corresponding to the preset orientation in the dynamic template to the absolute search field under the current physical screen; the mapping relationship simultaneously compensates for DPI scaling and window displacement. S500: Generate candidate input boxes using a preset method; within the absolute search field, determine the comprehensive energy cost of each candidate input box; the comprehensive energy cost is determined based on the distance between the candidate box and the anchor point, the angular deviation relative to the preset orientation, and the morphological characteristics of the candidate box; the candidate box with the minimum comprehensive energy cost is determined as the final write target, and simulated clicks and keyboard input are executed.

2. The automatic value writing method for medical systems based on semantic anchors and dynamic templates according to claim 1, characterized in that, The confidence model in step S300 is expressed as follows: ； in, For the first OCR text blocks Confidence level; This is a preset semantic fingerprint template; For spatial prior information; For contextual environment templates; This is a text semantic similarity function; Let be the spatial prior probability function; This is a context matching function; These are the preset weighting coefficients.

3. The automatic value writing method for medical systems based on semantic anchors and dynamic templates according to claim 2, characterized in that, The text semantic similarity The following relationship must be satisfied: ； in: For regular expression matching indicator functions: when a complete match is achieved. Pick When matching regular expressions Pick ;otherwise, Pick ; To edit distance; The string length is specified; max() is the default function for finding the maximum value. The spatial prior probability Using a Gaussian kernel function: ； This is the normalized distance from the center of the text box to the center of the screen. For bandwidth parameters; ； In order to The number of preset related words hit within a 150-pixel radius centered on the target; The total number of pre-defined related terms for the corresponding field; For context matching degree.

4. The automatic value writing method for medical systems based on semantic anchors and dynamic templates according to claim 1, characterized in that, The affine transformation matrix M in step S400 is: ； in, These are the DPI scaling factors for the current operating system in the horizontal and vertical directions, respectively; These represent the horizontal and vertical displacements of the top-left corner of the target window relative to the screen origin, respectively. The absolute search field is mapped by the following formula: ； To preset the relative search vector, superscript Indicates matrix transpose; These represent the relative horizontal and vertical offsets of the target input box relative to the semantic anchor point, respectively, at the template reference resolution. Based on template resolution settings; This represents the absolute search field coordinate range under the current physical screen.

5. The automatic value writing method for medical systems based on semantic anchors and dynamic templates according to claim 1, characterized in that, The overall energy cost in step S500 satisfies the following relationship: ； in, For the first The overall energy cost of each candidate input box; For the first The center coordinates of the candidate input boxes; The coordinates of the semantic anchor point; This is the Euclidean distance between the candidate box and the anchor point; The angle between the candidate box and the preset orientation vector; For the first The shape dissimilarity of the candidate input boxes; These are the weighting coefficients; The following relationship must be satisfied: ； Penalty for aspect ratio; Penalize the background texture; , The aspect ratio of the candidate bounding box; , The probability that the candidate bounding box output by the SVM classifier belongs to an editable region; These are the weighting coefficients.

6. The automatic value writing method for medical systems based on semantic anchors and dynamic templates according to claim 1, characterized in that, The semantic fingerprint and spatial association rules in step S100 are stored in the remote configuration center in YAML format. The rule entries include the following fields: field name, regular expression pattern, priority, preset orientation, minimum search radius, and maximum search radius. The system requests the rule version number from the configuration center every preset period. When the local version number is lower than the remote version number, an incremental update is performed, and only the changed rule entries are downloaded. After the update, it is automatically loaded into memory without restarting the automation process.

7. The automatic value writing method for medical systems based on semantic anchors and dynamic templates according to claim 1, characterized in that, Step S300 also includes the following steps: S310, when the regular expression matching in step S300 fails completely, calculate the i-th OCR text block respectively. The string is compared with the minimum edit distance of each display name in the current field in the rule base, and the minimum value is selected; if the minimum value is less than or equal to the preset minimum edit distance threshold and If the string length is greater than a preset string length threshold, it is considered an approximate match, and... confidence level Multiply by the preset discount factor Then it will re-enter the sorting process; S320: When the confidence level of a single character output by the OCR engine is lower than the preset threshold, the system queries the preset glyph similarity mapping table for candidate characters that are similar in glyph to the current character and conform to the context. If a unique match is found, the character is automatically replaced and its confidence level is increased. If multiple candidates exist, the original recognition result is retained and marked for manual review. The glyph similarity mapping table is used to record easily confused character pairs.

8. The automatic value writing method for medical systems based on semantic anchors and dynamic templates according to claim 1, characterized in that, The preset methods include: edge detection and contour analysis, color segmentation and blank area detection, editable region SVM classifier prediction, and OCR text block neighborhood blank area recognition.

9. The automatic value writing method for medical systems based on semantic anchors and dynamic templates according to claim 1, characterized in that, Following step S500, the method further includes the following steps: S600: After writing the value, capture the window image again. If the state of the focus area and the target area meets the preset state conditions, the image difference meets the first preset condition, and the text similarity meets the second preset condition, then the judgment is successful; otherwise, trigger the retry mechanism.

10. The automatic value writing method for medical systems based on semantic anchors and dynamic templates according to claim 9, characterized in that, In step S600: The preset state conditions are: the center distance between the focal region and the target region is less than a preset distance threshold and the area overlap ratio is not less than a preset overlap threshold; wherein, the focal region is obtained through system API or visual cursor detection, and the target region is the final write target region determined in step S500; the first preset condition corresponding to the image difference is that the difference is not less than the first preset threshold; the second preset condition corresponding to the text similarity is that the text similarity is not less than the second preset threshold. The text similarity The mean of the weighted edit distance and the Jaro-Winkler distance is used: ； in, The string to be written; This refers to the text recognized by OCR after writing the value. To be Convert to Minimum number of single-character editing operations required; They are respectively and The string length; max() is the function to find the maximum value; for and Jaro-Winkler similarity between them.

Images