An assisted reproduction report intelligent analysis and mapping device based on multi-type image recognition and a method thereof

Abstract

The application discloses an assisted reproduction report intelligent analysis and mapping device and method based on multi-type image recognition, relates to the technical field of medical image processing and assisted reproduction medical data processing, and comprises a medical image OCR recognition and structured reconstruction module, a dynamic threshold checking module, a self-adaptive multi-target matching module, an intelligent automatic scheduling module, a man-machine cooperation and exception processing module and a privacy calculation and security module. Through template-driven extraction, dynamic threshold self-adaptive checking, multi-target adaptation based on a mapping library and parallel automatic scheduling, the application realizes intelligent and high-consistency processing of a whole process from an assisted reproduction medical report image to multiple heterogeneous target systems, significantly improves data entry efficiency and accuracy, has a continuous self-optimization capability, and meets medical data safety and compliance requirements.

Description

Technical Field

[0001] This invention relates to the field of medical image processing and assisted reproductive medical data processing technology, specifically to an intelligent analysis and mapping device and method for assisted reproductive reports based on multi-type image recognition. Background Technology

[0002] Assisted reproductive technology (ART) is an important means of treating infertility, including in vitro fertilization-embryo transfer (IVF-ET) and intracytoplasmic sperm injection (ICSI). During ART treatment, patients need to undergo regular hormone testing, semen analysis, genetic screening, and other examinations, generating a large number of medical reports. The accurate interpretation and efficient entry of these reports are crucial to ensuring the correctness of clinical decisions and the continuity of treatment.

[0003] However, the current processing of assisted reproductive medicine reports mainly relies on manual operation or general image processing tools, which presents the following technical problems.

[0004] In terms of image recognition, existing technologies rely on general OCR tools and lack template-driven structured extraction mechanisms. When faced with changes in report layout, new parsing logic needs to be developed, resulting in poor adaptability.

[0005] In terms of numerical verification, existing technologies rely on fixed reference ranges and cannot dynamically adapt to changes in the normal range of data caused by the replacement of testing equipment or the introduction of new reagents, requiring frequent manual adjustment of thresholds.

[0006] Regarding multi-system integration, existing technologies struggle to handle the heterogeneity of multiple target systems, resulting in low efficiency and a high risk of errors during manual data entry. Reproductive centers need to simultaneously input data into the Electronic Medical Record (EMR) system, the Embryo Laboratory Information System (LIS), and the National Health Commission's regulatory platform. Each system has different field definitions, formatting requirements, and page structures, leading to inefficient manual repetitive data entry and potential inconsistencies across systems.

[0007] In terms of process integration, the existing processing workflow is fragmented and lacks closed-loop optimization. Tools such as OCR and RPA operate independently, failing to form an intelligent closed loop from image input to multi-system data entry. When recognition or matching errors occur, they cannot be corrected immediately through manual intervention, nor can the correction feedback be used for continuous model optimization.

[0008] Therefore, there is an urgent need in this field for an intelligent processing device that can perform templated OCR recognition, dynamic threshold verification, multi-objective adaptive scheduling, and human-machine collaborative closed-loop optimization for assisted reproductive medicine reports. Summary of the Invention

[0009] To address the aforementioned shortcomings of the prior art, this invention provides an intelligent analysis and mapping device and method for assisted reproductive reports based on multi-type image recognition, in order to solve the problems mentioned in the background art.

[0010] To achieve the above objectives, the present invention provides the following technical solution: an intelligent analysis and mapping device for assisted reproductive reports based on multi-type image recognition, comprising: The medical image OCR recognition and structured reconstruction module is used to receive at least one assisted reproductive medicine report image, perform type recognition on the image, and perform corresponding image enhancement and content extraction based on the recognized type to generate structured data and / or image feature vectors. The dynamic threshold verification module is connected to the medical image OCR recognition and structured reconstruction module. It is used to receive structured data, perform unit normalization, automatic threshold generation and anomaly detection, and update the threshold or correct the data according to the manual verification results to generate verified structured data. An adaptive multi-target matching module, connected to a dynamic threshold verification module, is used to receive the verified structured data, adapt it to multiple heterogeneous target systems, and generate standardized data specific to each target system. The intelligent automated scheduling module, connected to the adaptive multi-target matching module, is used to receive standardized data specific to each system and simulate manual operation to automatically input data into multiple heterogeneous target systems in parallel. The human-machine collaboration and anomaly handling module is connected to the medical image OCR recognition and structured reconstruction module, the dynamic threshold verification module, the adaptive multi-target matching module, and the intelligent automated scheduling module, respectively. It provides a visual interface for manually adding new templates, manually verifying values ​​exceeding the threshold, verifying and correcting templates, and verifying and correcting mappings. It also uses human feedback signals to optimize the corresponding modules. The privacy computing and security module connects to all the above modules and is used to perform encryption, de-identification, and access control throughout the entire data lifecycle to ensure that sensitive data in assisted reproduction is secure and controllable and meets medical data compliance requirements.

[0011] Furthermore, the medical image OCR recognition and structured reconstruction module includes: A multi-format receiving unit is used to receive report images in various formats. The image enhancement unit is used to perform general preprocessing on the input image; An OCR recognition engine is used to perform domain fine-tuning on reproductive medicine report images and employs data augmentation strategies to simulate image quality degradation, outputting text blocks and their coordinate positions. The template matching unit receives the text block and its coordinates output by the OCR recognition engine and matches them with features in the template library. The structured extraction unit receives the template and coordinate information from the template matching unit and directly extracts the project name, value, and unit according to the predefined extraction path of the template. The extraction result verification unit performs post-verification on the results output by the structured extraction unit.

[0012] Furthermore, the dynamic threshold verification module includes: The data reading and parsing unit is used to read a list of key-value pairs and extract items, values, and units. Unit normalization and automatic conversion unit, used to unify numerical units; An automatic threshold generation unit is used to dynamically generate the maximum and minimum values ​​for each numerical item; The data validation and labeling unit compares the unit-normalized values ​​with a dynamic threshold. The result generation unit is used to generate the validated structured data and output it to the adaptive multi-objective matching module. The correct value receiving unit is used to receive the correct values ​​manually entered by the human-machine collaboration and exception handling module, and output them to the result generation unit; The threshold update unit is used to receive values ​​that exceed the threshold but are correctly identified, and to update the threshold range.

[0013] Furthermore, the adaptive multi-objective matching module includes: Multi-target field recognition unit, used to automatically crawl and recognize field metadata of each target system; The mapping library is used to store verified field mapping relationships and to perform query operations on the mapping library. The data conversion and verification unit is used to receive valid mapping relationships matched by the mapping library and convert the source values ​​into the format required by the target system according to the conversion rules in the mapping records.

[0014] Furthermore, the intelligent automated scheduling module includes: A multi-path parallel scheduling unit is used to drive multiple browser containers to execute automated tasks in parallel. The iframe penetration and complex control processing unit is used to automatically identify and switch to the target iframe to perform element operations. Anomaly detection and transaction coordinator are used to monitor the status of all automated tasks; The rollback execution unit performs reverse operations on each target system that has been successfully entered. The logging unit records detailed logs for each step of the automated operation, with failure details and rollback operations stored in encrypted form.

[0015] Furthermore, the human-machine collaboration and anomaly handling module includes: When a template matching unit fails to match, the engineer creates a new template, which is then automatically added to the template library after being saved. The threshold manual verification unit receives threshold abnormality records sent by the data verification and marking unit, and displays a portion of the original image, the identified value, and the current dynamic threshold range on the interface.

[0016] The template verification and correction unit is used to receive reports of abnormal template extraction results and manually determine the cause. When a manual verification unit is first mapped, if no valid mapping record is found in the mapping library, a verification interface pops up. The physician selects the correct target field from the candidate fields and confirms whether unit conversion or other conversion rules need to be applied. After confirmation, the mapping relationship and conversion rules are stored in the mapping library, and the verifier and timestamp are recorded. The mapping verification and correction unit performs cross-system consistency verification on the mapping relationships and conversion rules stored in the mapping library by the initial mapping manual verification unit. When the key field values ​​output by each target system are found to be inconsistent, the system pops up a comparison interface to display the specific values ​​output by each system. After the physician checks, the target field is reselected or the conversion rule is modified for the inconsistent field. After the correction is completed, the system automatically re-executes the data conversion and verification.

[0017] Furthermore, the privacy computing and security module includes a data desensitization unit, an encrypted storage and transmission unit, and an access control and auditing unit, which are used for data desensitization, encrypted transmission, and access and export, respectively.

[0018] A method for intelligent parsing and mapping of assisted reproductive reports based on multi-type image recognition, applied to an intelligent parsing and mapping device for assisted reproductive reports based on multi-type image recognition, includes the following steps: a. Using the medical image OCR recognition and structured reconstruction module, the input assisted reproductive medicine report image is subjected to OCR recognition and template extraction to generate structured data; b. Through the dynamic threshold verification module, the structured data is normalized, automatically generated, and anomaly detected. When the value exceeds the dynamic threshold, manual verification is triggered. The threshold is updated or the data is corrected based on manual feedback, and the structured data after threshold verification is generated. c. Through the adaptive multi-target matching module, based on the mapping library query and the initial mapping manual verification mechanism, the structured data after threshold verification is adapted in parallel to multiple heterogeneous target systems to generate their own dedicated standardized data; d. Through an intelligent automated scheduling module, the standardized data is automatically entered into multiple target systems in parallel, simulating manual operation; e. Through the human-machine collaboration and anomaly handling module, an interface is provided for adding templates, manually verifying exceeding thresholds, and verifying and correcting templates.

[0019] Furthermore, step d also includes: when data is entered into multiple target systems in parallel, if the entry operation of any target system fails, a cross-system transaction rollback is automatically triggered to cancel the data that has been successfully entered into other target systems.

[0020] This invention provides an intelligent analysis and mapping device and method for assisted reproductive reports based on multi-type image recognition, which has the following beneficial effects: 1. This invention significantly improves the parsing efficiency and adaptability of reproductive report images. Through template-driven structured extraction and human-machine collaborative closed-loop processing, it achieves unified parsing and rapid adaptation of multiple types of assisted reproductive reports. Through continuous accumulation of the template library and human closed-loop feedback, it effectively addresses the clinical needs of diverse report formats and frequent changes, significantly shortens the adaptation time for new report types, and significantly improves processing efficiency compared to manual operation. The system's usability and maintainability are also significantly enhanced. 2. This invention achieves adaptive and self-repairing numerical verification. Through a closed loop of automatic dynamic threshold generation and manual verification exceeding thresholds, it realizes adaptive optimization and OCR self-repairing capabilities for numerical verification. Dynamic statistics based on historical data eliminate the need for manually preset reference ranges. Abnormal thresholds automatically trigger manual review; once confirmed correct, the threshold adaptively expands; and errors are automatically corrected by feeding back feedback to adjust the OCR model. This allows for continuous adaptation to dynamic environments such as equipment replacement and the introduction of new reagents, significantly reducing maintenance costs and ensuring the accuracy and robustness of numerical verification. 3. This invention achieves cross-system adaptive matching and parallel automated scheduling. Through mapping library reuse and initial mapping manual verification mechanisms, combined with a Playwright-based multi-core browser parallel scheduling framework, it achieves "parsing once, inputting in multiple places." By guiding manual verification to update mapping rules, it can adapt to multiple heterogeneous systems without interface development. Simultaneously, anomaly detection and transaction rollback mechanisms ensure cross-system data consistency; if input fails in any system, operations on other successful systems can be automatically rolled back. 4. This invention establishes a closed-loop continuous optimization mechanism for human-machine collaboration. By providing interfaces for template addition, manual verification of threshold values, and template verification and correction, physicians can instantly handle reports without templates, confirm threshold values, correct template rules, verify and correct mappings, and handle scheduling anomalies. Furthermore, the localized deployment solution ensures that sensitive data does not leave the domain, and the entire process operation log is encrypted, stored, and traceable, meeting medical compliance requirements. 5. This invention integrates medical image OCR recognition, template-driven structured extraction, dynamic threshold verification, adaptive multi-target matching, and automated scheduling technologies to achieve intelligent processing of the entire process from image-based medical reports to multiple reproductive target systems. This significantly improves the efficiency, accuracy, and consistency of data processing. It eliminates the need for interface development with each target system and directly simulates manual operation to complete data entry, which can significantly save interface development costs and manual entry costs, demonstrating outstanding economic benefits and clinical promotion value. Attached Figure Description

[0021] Figure 1 This is a structural block diagram of the entire invention; Figure 2 This is a flowchart of the medical image OCR recognition and structured reconstruction module of the present invention; Figure 3 This is a flowchart of the dynamic threshold verification module (including some human-machine collaboration and exception handling modules) of the present invention. Figure 4 This is a flowchart of the adaptive multi-target matching module of the present invention; Figure 5 This is a flowchart of the intelligent automated scheduling module of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] The present invention will now be described in detail through specific embodiments, as follows: This invention provides a hardware platform for an intelligent analysis and mapping device for assisted reproductive reports based on multi-type image recognition. This platform provides computing, storage, communication and interaction support for the above modules.

[0024] The hardware platform includes the following components: The main frame is made of medical-grade metal shell, with dimensions of 450mm×350mm×150mm, IP54 protection rating, and antibacterial coating on the surface, making it suitable for the clean environment requirements of reproductive centers.

[0025] The core processor uses an Intel Xeon W-10885M (8 cores, 16 threads, base frequency 2.4GHz, turbo frequency 5.3GHz) as the central processing unit; dual NVIDIA Jetson Orin edge AI computing modules, each providing 275 TOPS of computing power for deep learning model inference.

[0026] The storage system includes a 1TB NVMe SSD (operating system and real-time data cache) and a 4TB HDD (RAID 1 mirroring for long-term data storage).

[0027] Communication interfaces include dual gigabit Ethernet ports (supporting physical isolation, connecting to the hospital network and external network respectively), WiFi 6E, Bluetooth 5.3, optional 5G module, USB 3.2 Gen2×6, HDMI 2.0, dedicated scanner interface (supporting TWAIN protocol), and printer interface.

[0028] For display and interaction, it features a 15.6-inch medical-grade touchscreen with a resolution of 1920×1080, a brightness of 400cd / m², and an anti-glare coating; physical shortcut keys (start / pause / emergency stop); and a voice interaction array microphone.

[0029] Security hardware, including TPM 2.0 hardware encryption chip, fingerprint recognition module, and physical isolation gateway (optional).

[0030] This invention discloses an intelligent parsing and mapping device for assisted reproductive reports based on multi-type image recognition. It includes a medical image OCR recognition and structured reconstruction module, a dynamic threshold verification module, an adaptive multi-target matching module, an intelligent automated scheduling module, a human-machine collaboration and anomaly handling module, and a privacy computing and security module. External input is directed to the medical image OCR recognition and structured reconstruction module. The results from this module are output to the dynamic threshold verification module, which is then sequentially connected to the adaptive multi-target matching module and the intelligent automated scheduling module, ultimately outputting to an external target system. The human-machine collaboration and anomaly handling module interacts with all modules, while the privacy computing and security module permeates all data pathways, covering the entire process.

[0031] Medical Image OCR Recognition and Structured Reconstruction Module The medical image OCR recognition and structured reconstruction module receives at least one assisted reproductive medicine report image, performs type recognition on the image, and performs corresponding image enhancement and content extraction based on the recognized type to generate structured data and / or image feature vectors. The medical image OCR recognition and structured reconstruction module includes the following units: The multi-format receiving unit is used to receive report images in various formats, including JPG, PNG, TIFF, and image-based PDFs. This unit connects to a high-speed scanner interface, a mobile terminal docking unit, and a PDF import interface.

[0032] The high-speed scanner interface connects to a medical scanner to receive scanned images of paper reports, supporting automatic paper feeding and duplex scanning; the mobile terminal docking unit receives photos taken by mobile phones / tablets via Bluetooth / WiFi, supporting image preprocessing; the PDF import interface supports the import and parsing of image-type PDF files, extracting embedded image data; in addition, the image processing acceleration unit provides hardware acceleration for the medical image OCR recognition and structured reconstruction module.

[0033] The image enhancement unit is used to perform general preprocessing on the input image, including tilt correction, noise reduction, and contrast enhancement, in order to improve the accuracy of OCR recognition.

[0034] The OCR recognition engine, based on a CNN+Transformer architecture (TrOCR), underwent domain-specific fine-tuning on 100,000 reproductive medicine report images and employed data augmentation strategies to simulate image quality degradation (blurring, noise, partial occlusion, and stamp coverage), enabling the model to withstand interference. It supports the recognition of mixed Chinese and English text, handwritten text, printed text, and text within tables, outputting text blocks and their coordinate positions.

[0035] The template matching unit has a built-in, expandable template library. Each template defines a report type's layout structure, key area coordinates, keywords, and numerical extraction rules. This unit receives text blocks and their coordinates output by the OCR recognition engine and matches them against features in the template library (e.g., through title keywords). If a match is successful, the matched template identifier and aligned coordinate information are passed to the structured extraction unit; if a match fails, the image and OCR results are sent to the template adding unit of the human-machine collaboration and anomaly handling module to request a manual addition of a new template. Manually added templates are verified and stored in the template library for use in subsequent reports of the same type.

[0036] The structured extraction unit receives the template and coordinate information from the template matching unit, and directly extracts the item name, value, and unit according to the predefined extraction path of the template (such as the second text block to the right of the keyword, or a specific coordinate area). After extraction, the extraction results (a list of key-value pairs) along with the original image information are sent to the extraction result verification unit.

[0037] The extraction result verification unit performs post-verification on the output of the structured extraction unit, including but not limited to: checking whether the values ​​are numbers, whether the units are valid, and whether key fields are missing. If the verification passes, the structured data is sent to the dynamic threshold verification module; if the verification fails, the abnormal information (including the original image, extraction result, and reason for failure) is sent to the template verification and correction unit of the human-machine collaboration and abnormality handling module, triggering the template verification and correction process. The physician determines whether the error is due to OCR recognition or template rule error, and corrects the template or fills in the correct value accordingly.

[0038] The working process is as follows: the multi-format receiving unit receives the image; the image enhancement unit enhances the image and sends it to the OCR recognition engine; the OCR outputs the text block and coordinates; the template matching unit queries the template library: if there is a matching template, the structured extraction unit extracts data according to the template, the extraction result verification unit verifies the result, if normal, it is output to the dynamic threshold verification module, if abnormal, it is sent to the human-machine collaboration and exception handling module; if there is no matching template, it requests manual addition of the template.

[0039] Dynamic threshold verification module like Figure 3 As shown, the dynamic threshold verification module, connected to the medical image OCR recognition and structured reconstruction module, receives structured data, performs unit normalization, automatic threshold generation, and anomaly detection, and updates the threshold or corrects the data based on manual verification results to generate verified structured data. The dynamic threshold verification module includes the following units: The data reading and parsing unit is used to read the list of key-value pairs and extract items, values, units, etc.

[0040] The unit normalization and automatic conversion unit is used to standardize numerical units. Based on the built-in unit conversion library, it unifies the units of numerical values ​​to the units required for threshold verification. For example, the unit conversion relationship for AMH is: 1 ng / mL = 7.14 pmol / L.

[0041] An automatic threshold generation unit is used to dynamically generate the maximum and minimum values ​​for each numerical item, based on accumulated historical report data.

[0042] The data verification and labeling unit compares the unit-normalized value with a dynamic threshold. If the value is within the threshold range, it is output normally to the result generation unit; if it exceeds the threshold, the abnormal record (including the original image cropped area, the identified value, and the current threshold) is packaged and sent to the "Exceed Threshold Manual Verification Unit" of the human-machine collaboration and anomaly handling module. After receiving manual feedback: if the manual confirmation is correct, the threshold range for this item is updated (the maximum value is increased or the minimum value is decreased); if the manual confirmation is incorrect and the correct value is entered, the correct value is received.

[0043] The result generation unit is used to generate the validated structured data and output it to the adaptive multi-objective matching module.

[0044] The correct value receiving unit is used to receive the correct values ​​manually entered by the human-machine collaboration and exception handling module and output them to the result generation unit.

[0045] The threshold update unit is used to receive values ​​that exceed the threshold but are correctly identified, and to update the threshold range.

[0046] The working process is as follows: The data reading and parsing unit reads the structured key-value pair list from the extraction result verification unit of the medical image OCR recognition and structured reconstruction module, extracting the item name, value, and unit, etc.; the unit normalization and automatic conversion unit unifies the numerical units to the format required by the automatic threshold generation unit; the automatic threshold generation unit dynamically calculates the maximum and minimum values ​​of each test item based on the accumulated historical report data and updates the threshold parameters in real time. The data verification and labeling unit compares the normalized value with the current dynamic threshold: if the value is within the threshold range, it is judged as normal, and the result generation unit directly generates the verified structured data and outputs it to the adaptive multi-target matching module; if the value exceeds the threshold range, it is judged as abnormal, and the abnormal record (including the original image cropping area, recognition value, item name, and current threshold) is packaged and sent to the over-threshold manual verification unit of the human-machine collaboration and abnormality handling module, and the subsequent processing of the current data is suspended. Upon receiving human feedback: if the feedback is "Value correct, update threshold", the minimum or maximum value range of the item will be automatically expanded through the threshold update unit and fed back to the automatic threshold generation unit; if the feedback is "Value incorrect, fill in manually", the correct value received by the physician will be received through the correct value receiving unit and then fed back to the result generation unit to generate structured data.

[0047] Adaptive multi-target matching module like Figure 4 As shown, the adaptive multi-target matching module, connected to the dynamic threshold verification module, receives the verified structured data, adapts it to multiple heterogeneous target systems, and generates standardized data specific to each target system. This module employs an intelligent learning mechanism of "initial manual verification, subsequent mapping library reuse".

[0048] The multi-target field recognition unit automatically acquires field structure information from various target systems without requiring manual documentation or configuration. For web systems (such as the web versions of EMR and LIS), the unit simulates a browser access to the target system's data entry page, parses the page's DOM tree structure, and, combined with visual layout analysis, automatically extracts all input field names, types (text boxes, dropdown lists, date pickers, radio buttons, etc.), required attributes, value ranges (such as the option list in a dropdown list), and field display labels from the form. Simultaneously, the unit can detect dynamically loaded fields (such as those populated via AJAX).

[0049] The mapping library stores verified field mapping relationships. Each mapping record includes: source field (the item name in the semantic parsing result, such as "estradiol"), target system identifier, target field name (such as "E2"), conversion rules (unit conversion formula, data type conversion, default value filling, etc.), creation time, last usage time, and verifier identifier. The mapping library uses a key-value pair storage structure (with "source field + target system" as the key) and supports millisecond-level queries. Data in the mapping library is persistently stored and loaded into memory when the system starts. The mapping library also performs query operations to receive verified structured data and page information extracted by the multi-target field recognition unit. For each target system, each source field is processed sequentially. First, the mapping library is queried using "source field + target system" as the key. If a match is found, the valid mapping relationship is output to the data conversion and verification unit. If no match is found, the initial mapping manual verification unit of the human-machine collaboration and exception handling module is called. After manual confirmation, the mapping relationship and conversion rules are obtained, and then the conversion is executed.

[0050] The data conversion and verification unit receives valid mapping relationships matched by the mapping library and converts the source values ​​to the format required by the target system according to the conversion rules in the mapping records (e.g., unit conversion: pmol / L to pg / mL, multiplied by 0.136); if no conversion rules are available, values ​​are directly assigned. After all fields are converted, a standardized data package specific to the target system (e.g., JSON for EMR, HL7 v2.x for LIS, XML for the monitoring platform) is generated. Subsequently, the unit performs cross-system consistency verification: comparing whether key fields (e.g., patient ID, cycle number, and FSH value of major tests) in the data packages of each target system are consistent. If consistent, the multi-channel standardized data is output to the intelligent automated scheduling module; if inconsistent, an error log is recorded, and the mapping verification and correction unit of the human-machine collaboration module prompts the physician to check the relevant mapping rules in the mapping library, and manually corrects them if necessary.

[0051] The workflow is as follows: The mapping library receives structured data output from the dynamic threshold verification module and page information extracted by the multi-target field recognition unit. For each target system, the mapping library processes each source field sequentially: if a valid mapping exists, it outputs the data to the data conversion and verification unit for data conversion; if no mapping exists, it calls the initial mapping manual verification unit, where the mapping relationship and conversion rules are manually confirmed before being stored in the mapping library. After all fields are converted, a dedicated data packet for that system is generated. Subsequently, a cross-system consistency check is performed. If consistent, multiple data streams are output to the intelligent automated scheduling module; if inconsistent, a log is recorded and a manual check is prompted.

[0052] Intelligent automated scheduling module like Figure 5 As shown, the intelligent automated scheduling module, connected to the adaptive multi-target matching module, receives standardized data specific to each system and simulates manual operation to automatically input data into multiple heterogeneous target systems in parallel. This module is built on the Playwright automation framework, leveraging its cross-browser compatibility, automatic waiting, network interception, and iframe penetration features to achieve highly reliable automated data input. The intelligent automated scheduling module uses a multi-core browser runtime environment. Based on Playwright's browser containerization capabilities, it allocates an independent browser context to each target system. Each context has isolated cookies, caches, and login sessions to prevent mutual interference. It supports multiple kernels, including Chromium, Firefox, and WebKit, allowing selection based on the compatibility requirements of the target system.

[0053] The multi-threaded scheduling unit utilizes Playwright's asynchronous API (async / await) to manage automation tasks across multiple browser contexts simultaneously. It dynamically allocates resources based on the complexity and priority of each task, supporting simultaneous automated data entry to multiple target systems. The multi-threaded scheduling unit connects to an anomaly detection and transaction coordinator, providing real-time reports on the execution status of each task.

[0054] The iframe penetration and complex control handling unit, based on Playwright's built-in frame positioning method (page.frameLocator()), automatically identifies and switches to the target iframe for element manipulation. For complex controls such as date pickers, dropdowns, and dynamic tables, stable operation is achieved by using Playwright's locator in conjunction with JavaScript injection (evaluate) or simulating keyboard / mouse events (click, fill, selectOption). For example, for date pickers, locator.fill(dateString) is used to directly assign a value and trigger the change event; for dropdowns, locator.click() is used to expand the iframe, and then page.keyboard.press('ArrowDown') and press('Enter') are used for selection.

[0055] The anomaly detection and transaction coordinator monitors the status (success, failure, timeout) of all Playwright tasks. If data entry fails in a system (e.g., page timeout, element not found, submission error), the coordinator first uses Playwright's automatic retry mechanism (expect configuration) to retry up to 3 times, with a 2-second interval. If it still fails, the task is paused, and subsequent actions are executed according to the preset transaction consistency strategy: if the current task is an independent task, only a failure log is recorded and the human-machine collaboration module is notified; if the task has the "cross-system transaction consistency" flag enabled, the coordinator immediately sends a rollback instruction to the rollback execution unit to undo the data that has been successfully entered in other systems.

[0056] The rollback execution unit, upon receiving instructions from the coordinator, performs reverse operations for each successfully entered data on the target system. These reverse operations may include: calling the target system's API (if any) to delete the newly created record; or re-entering the details page of the entered data via Playwright and simulating clicking the "Delete" or "Void" button; if deletion fails, a "Manual Rollback Required" alert is recorded. After rollback completion, the unit notifies the anomaly detection and transaction coordinator of the task status update.

[0057] The logging unit records detailed logs for each automated operation (timestamp, operation screenshot, input data, return result), and failure details and rollback operations are stored in encrypted form.

[0058] The workflow is as follows: First, after receiving the multi-channel standardized data output from the adaptive multi-target matching module, the multi-channel parallel scheduling unit allocates an independent browser context for each target system and loads the corresponding page recognition model. Playwright drives the browser to navigate to the login page, automatically filling in and logging in using credentials encrypted and stored by TPM. After entering the data entry page, the iframe penetration and complex control processing unit automatically handles cross-domain frames and fills in the form fields sequentially (using field mapping provided by the mapping library, through methods such as locator.fill() and selectOption()). The form submission operation uses Playwright's page.click('button[type=submit]') and waits for navigation or network idle time. The exception detection and transaction coordinator monitors the status of each submission operation in real time: if all are successful, a success log is recorded; if any fails, a retry strategy is applied (maximum 3 times, 2-second interval), triggering a rollback if necessary. All operation screenshots and execution results are stored in encrypted storage by the log recording unit for subsequent auditing.

[0059] Through the above design, the device of the present invention achieves highly reliable, cross-browser parallel automated input based on Playwright, and supports robust operation and transaction consistency rollback for complex pages.

[0060] Human-machine collaboration and exception handling module The human-machine collaboration and anomaly handling module connects to the medical image OCR recognition and structured reconstruction module, the dynamic threshold verification module, the adaptive multi-target matching module, and the intelligent automated scheduling module, respectively. It provides a visual interface for manually adding new templates, manually verifying values ​​exceeding thresholds, verifying and correcting templates, and verifying and correcting mappings. Manual feedback signals are used to optimize the corresponding modules. This module includes the following units: When a template matching unit fails to match a template, the interface displays the report image and OCR text. Engineers can create new templates by selecting areas, defining field names, and setting extraction rules (keywords, relative positions, regular expressions). After saving, the template is automatically added to the template library and triggers the medical image OCR recognition and structured reconstruction module to re-parse the report.

[0061] The threshold manual verification unit receives threshold abnormality records sent by the data verification and marking unit. It displays a portion of the original image (highlighting the abnormal value area), the identified value, and the current dynamic threshold range on the interface, providing two options: "Value correct, update threshold" - the system automatically expands the threshold range for this item (updates the maximum or minimum value); "Value incorrect, manually enter" - the physician enters the correct value.

[0062] The template validation and correction unit is used when the structured extraction unit fails to validate the results (e.g., incorrect numerical type, missing key fields) after extraction using the template. The interface displays the original image and the template extraction results, allowing the physician to determine the cause: if it's an OCR recognition error, the correct value is entered; if it's a template rule error (e.g., keyword offset, inaccurate coordinates), the physician is taken to the template editing interface to modify the extraction rules and save.

[0063] The initial mapping verification unit is triggered when the mapping library fails to find a valid mapping record while processing a source field. This unit displays a verification interface via the human-machine collaboration and anomaly handling module, showing: the name and example value of the current source field, the original parsing result output by the dynamic threshold verification module, and a list of all candidate fields extracted by the multi-target field identification unit in the target system (sorted by similarity, calculated based on field name string matching and field type consistency). The physician selects the correct target field from the candidate fields and confirms whether unit conversion (e.g., automatically recommended conversion formulas by the system) or other conversion rules are required. After manual confirmation, this unit stores the mapping relationship and conversion rules in the mapping library and records the verifier and timestamp. This mapping relationship can then be used directly for subsequent identical source fields.

[0064] The mapping verification and correction unit performs cross-system consistency verification on the mapping relationships and conversion rules stored in the mapping library by the initial mapping manual verification unit. When the key field values ​​output by each target system are found to be inconsistent, the system pops up a comparison interface to display the specific values ​​output by each system. After the physician checks, the target field is reselected or the conversion rule is modified for the inconsistent field. After the correction is completed, the system automatically re-executes the data conversion and verification.

[0065] Privacy Computing and Security Module The privacy computing and security module, connected to all the above modules, performs encryption, anonymization, and access control throughout the data lifecycle to ensure the security and controllability of sensitive assisted reproductive data and meet medical data compliance requirements. The privacy computing and security module includes the following hardware and software units: Hardware unit: The TPM 2.0 hardware encryption chip is used to store keys and certificates and supports Chinese national cryptographic algorithms.

[0066] Physical isolation gateway (optional) is used for data transfer in environments where internal and external networks are physically isolated.

[0067] Software unit: The data anonymization unit automatically hides the patient's name, ID number, and contact information before the data enters the processing flow, and generates an anonymous ID for internal processing. The anonymized ID is only restored according to permissions when the data is output to the target system.

[0068] The encrypted storage and transmission unit uses the national standard SM4 algorithm for encryption of the local database and TLS 1.3+ national standard suite for network communication. The key is protected by TPM.

[0069] The access control and auditing unit enables multi-level permission management (administrator, physician, technician), encrypted storage of operation logs, and supports audit export.

[0070] The workflow is as follows: During image input, the data anonymization unit automatically hides the patient's identity information and generates an anonymous ID; all data is encrypted using SM4 by the encrypted storage and transmission unit before storage, and network transmission uses TLS 1.3+ national cryptographic suite; before each operation, the access control and audit unit verifies the user's identity and permissions, and records encrypted logs after the operation; all AI inference is completed on local hardware, with no data leaving the domain. The entire process ensures compliance with the Personal Information Protection Law and relevant regulations on medical data security.

[0071] Overall working process of the device The entire working process of this invention is divided into five stages, with user interaction and automatic processing closely integrated.

[0072] Phase 1: Medical Image Input, OCR Recognition, and Template Extraction The user uploads a report image; the image enhancement unit enhances the image; the OCR recognition engine outputs text blocks and coordinates; the template matching unit queries the template library: if a matching template exists, the structured extraction unit extracts items, values, units, etc., according to the template, and the extraction result verification unit verifies the result (if normal, it outputs a list of key-value pairs; if abnormal, it triggers template verification and correction); if no matching template exists, it requests the human-machine collaboration module to add a new template.

[0073] Phase Two: Dynamic Threshold Generation and Verification After the data is read, the units are normalized; the automatic threshold generation unit calculates the dynamic threshold based on historical data; the data verification and labeling unit compares the value with the threshold: if it is within the threshold, the structured data after verification is directly generated; if it exceeds the threshold, it is sent to the human-machine collaboration module, where the physician judges: if correct, the threshold is updated; if incorrect, the correct value is entered.

[0074] Phase 3: Adaptive Multi-Objective Matching After receiving the structured data output by the dynamic threshold verification module, the adaptive multi-target matching module starts working.

[0075] First, for each target system (such as EMR, LIS, or regulatory platform), the module processes each source field in the data packet sequentially (such as "estradiol," "FSH," etc.). For the current source field, the system queries the mapping library: if a valid mapping record exists from the source field to the current target system (including the target field name, conversion rules, etc.), the data conversion is performed directly according to the record; if not, the "Initial Mapping Manual Verification Unit" is triggered, and a verification interface pops up through the human-machine collaboration and anomaly handling module, displaying a list of candidate fields for the source field and the target system (pre-crawled and cached by the multi-target field identification unit). The physician selects the correct target field and confirms the conversion rules (such as unit conversion). After manual confirmation, the mapping relationship is stored in the mapping library for direct use by subsequent identical source fields.

[0076] After all fields have been transformed, a standardized data package specific to the current target system (such as JSON for EMR, HL7 v2.x for LIS, and XML for the regulatory platform) is generated. Subsequently, the data transformation and verification unit performs cross-system consistency checks.

[0077] Phase Four: Multi-Objective Intelligent Automated Scheduling The intelligent automated scheduling module receives multi-channel standardized data output from the adaptive multi-target matching module and simulates manual input to multiple target systems in parallel based on the Playwright automation framework.

[0078] The multi-path parallel scheduling unit allocates an independent Playwright browser context to each target system. Each context has isolated cookies, caches, and login sessions. The system automatically logs into each target system using credentials encrypted and stored in the TPM and navigates to the data entry page.

[0079] For each target system, the scheduling unit processes each field in its standardized data packet sequentially. It uses Playwright's native selectors (based on text, CSS, or XPath) to locate form input fields, dropdowns, date pickers, and other controls on the page, and populates the data using methods such as `locator.fill()`, `selectOption()`, and `evaluate()`. For pages with embedded iframes, `frameLocator()` is used for automatic penetration; for complex controls (such as cascading dropdowns and dynamic tables), stable operation sequences (such as simulated clicks and keyboard selections) are encapsulated. After all fields are populated, the form is submitted.

[0080] The anomaly detection and transaction coordinator monitors the status of each commit operation in real time: if all are successful, a success log is recorded and the entry status is updated; if any system entry fails (e.g., page timeout, element not found, submission error), it automatically retryes up to 3 times (with 2-second intervals), re-executing from the form filling and submission steps during retry. If the retry still fails, a rollback is executed according to the preset transaction consistency strategy: if the current task has the "cross-system transaction consistency" flag enabled, the rollback execution unit is triggered to perform reverse operations on other systems that have successfully entered data (e.g., calling the delete API or simulating clicking the delete button) to ensure data consistency.

[0081] Phase 5: Human-Machine Collaboration and Anomaly Handling Closed Loop When no template is matched in the template matching unit, the system automatically pops up the template addition interface, displaying the original image and OCR text blocks. Administrators can create new templates by selecting key areas, defining field names, and setting extraction rules (keywords, coordinates, or relative positions). After the new template is verified, it is stored in the template library. The system immediately uses the new template to re-parse the current report, and continues the subsequent process after successful parsing.

[0082] When the dynamic threshold verification module detects a value exceeding the dynamic threshold range, the system displays an over-threshold verification interface, showing a portion of the original image (highlighting the outlier area), the identified value, and the current threshold range. The physician, based on clinical judgment, determines whether the value is correct (e.g., due to a new range resulting from equipment replacement) and clicks "Value Correct, Update Threshold." The system automatically expands the threshold range for that item (updating the maximum or minimum value) and re-verifies. If the value is indeed an OCR recognition error, the physician enters the correct value, and the system uses the correct value in subsequent processes. Simultaneously, the system marks the erroneous sample and sends it to the OCR recognition engine for incremental learning, enabling continuous correction of the OCR model.

[0083] When the extraction result verification unit detects an anomaly in the extraction result (such as incorrect numerical type or missing key fields), the system pops up a template verification and correction interface, displaying the original image and the template extraction result. The physician determines the cause: if it is an OCR recognition error, the correct value is entered; if it is a template rule error (such as keyword offset or inaccurate coordinates), the physician enters the template editing interface, modifies the extraction rules, and saves the changes. The system then automatically uses the corrected template to re-parse the current report.

[0084] All manual operations are recorded in audit logs, which are traceable to the specific operator, time, and data content. Through these three closed loops, the system can continuously adapt to changes in report format and replacement of inspection equipment.

[0085] The device of this invention fully realizes the automation and intelligence of the entire process from multi-source report input to multi-target system entry, from the hardware platform to each software module and the overall workflow, and has self-optimization capabilities.

[0086] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An assisted reproductive report intelligent analysis and mapping device based on multi-type image recognition, characterized in that, include: The medical image OCR recognition and structured reconstruction module is used to receive at least one assisted reproductive medicine report image, perform type recognition on the image, and perform corresponding image enhancement and content extraction based on the recognized type to generate structured data and / or image feature vectors. The dynamic threshold verification module is connected to the medical image OCR recognition and structured reconstruction module. It is used to receive structured data, perform unit normalization, automatic threshold generation and anomaly detection, and update the threshold or correct the data according to the manual verification results to generate verified structured data. An adaptive multi-target matching module, connected to a dynamic threshold verification module, is used to receive the verified structured data, adapt it to multiple heterogeneous target systems, and generate standardized data specific to each target system. The intelligent automated scheduling module, connected to the adaptive multi-target matching module, is used to receive standardized data specific to each system and simulate manual operation to automatically input data into multiple heterogeneous target systems in parallel. The human-machine collaboration and anomaly handling module is connected to the medical image OCR recognition and structured reconstruction module, the dynamic threshold verification module, the adaptive multi-target matching module, and the intelligent automated scheduling module, respectively. It provides a visual interface for manually adding new templates, manually verifying values ​​exceeding the threshold, verifying and correcting templates, and verifying and correcting mappings. It also uses human feedback signals to optimize the corresponding modules. The privacy computing and security module connects to all the above modules and is used to perform encryption, de-identification, and access control throughout the entire data lifecycle to ensure that sensitive data in assisted reproduction is secure and controllable and meets medical data compliance requirements. 2.The assisted reproductive report intelligent analysis and mapping device based on multi-type image recognition of claim 1, wherein, The medical image OCR recognition and structured reconstruction module includes: A multi-format receiving unit is used to receive report images in various formats. The image enhancement unit is used to perform general preprocessing on the input image; An OCR recognition engine is used to perform domain fine-tuning on reproductive medicine report images and employs data augmentation strategies to simulate image quality degradation, outputting text blocks and their coordinate positions. The template matching unit receives the text block and its coordinates output by the OCR recognition engine and matches them with features in the template library. The structured extraction unit receives the template and coordinate information from the template matching unit and directly extracts the project name, value, and unit according to the predefined extraction path of the template. The extraction result verification unit performs post-verification on the results output by the structured extraction unit. 3.The assisted reproductive report intelligent analysis and mapping device based on multi-type image recognition of claim 1, wherein, The dynamic threshold verification module includes: The data reading and parsing unit is used to read a list of key-value pairs and extract items, values, and units. Unit normalization and automatic conversion unit, used to standardize numerical units; An automatic threshold generation unit is used to dynamically generate the maximum and minimum values ​​for each numerical item; The data validation and labeling unit compares the unit-normalized values ​​with a dynamic threshold. The result generation unit is used to generate the validated structured data and output it to the adaptive multi-objective matching module. The correct value receiving unit is used to receive the correct values ​​manually entered by the human-machine collaboration and exception handling module, and output them to the result generation unit; The threshold update unit is used to receive values ​​that exceed the threshold but are correctly identified, and to update the threshold range. 4.The assisted reproductive report intelligent analysis and mapping device based on multi-type image recognition of claim 1, wherein, The adaptive multi-objective matching module includes: A multi-target field recognition unit is used to automatically crawl and recognize the field metadata of each target system; The mapping library is used to store verified field mapping relationships and to perform query operations on the mapping library. The data conversion and verification unit is used to receive valid mapping relationships matched by the mapping library and convert the source values ​​into the format required by the target system according to the conversion rules in the mapping records. 5.The assisted reproductive report intelligent analysis and mapping device based on multi-type image recognition of claim 1, wherein, The intelligent automated scheduling module includes: A multi-path parallel scheduling unit is used to drive multiple browser containers to execute automated tasks in parallel. The iframe penetration and complex control processing unit is used to automatically identify and switch to the target iframe to perform element operations. Anomaly detection and transaction coordinator are used to monitor the status of all automated tasks; The rollback execution unit performs reverse operations on each target system that has been successfully entered. The logging unit records detailed logs for each step of the automated operation, with failure details and rollback operations stored in encrypted form. 6.The assisted reproductive report intelligent analysis and mapping device based on multi-type image recognition of claim 1, wherein, The human-machine collaboration and anomaly handling module includes: When a template matching unit fails to match, the engineer creates a new template, which is then automatically added to the template library after being saved. The threshold manual verification unit receives threshold abnormality records sent by the data verification and marking unit, and displays a portion of the original image, the identified value, and the current dynamic threshold range on the interface. The template verification and correction unit is used to receive reports of abnormal template extraction results and manually determine the cause. When a manual verification unit is first mapped, if no valid mapping record is found in the mapping library, a verification interface pops up. The physician selects the correct target field from the candidate fields and confirms whether unit conversion or other conversion rules need to be applied. After confirmation, the mapping relationship and conversion rules are stored in the mapping library, and the verifier and timestamp are recorded. The mapping verification and correction unit performs cross-system consistency verification on the mapping relationships and conversion rules stored in the mapping library by the initial mapping manual verification unit. When the key field values ​​output by each target system are found to be inconsistent, the system pops up a comparison interface to display the specific values ​​output by each system. After the physician checks, the target field is reselected or the conversion rule is modified for the inconsistent field. After the correction is completed, the system automatically re-executes the data conversion and verification. 7.The assisted reproductive report intelligent analysis and mapping device based on multi-type image recognition of claim 1, wherein, The privacy computing and security module includes a data desensitization unit, an encrypted storage and transmission unit, and an access control and auditing unit, which are used for data desensitization, encrypted transmission, and access and export, respectively.

8. An assisted reproductive report intelligent analysis and mapping method based on multi-type image recognition, applied to an assisted reproductive report intelligent analysis and mapping device based on multi-type image recognition according to any one of claims 1 to 7, characterized in that, Includes the following steps: a. Using the medical image OCR recognition and structured reconstruction module, the input assisted reproductive medicine report image is subjected to OCR recognition and template extraction to generate structured data; b. Through the dynamic threshold verification module, the structured data is normalized, automatically generated, and anomaly detected. When the value exceeds the dynamic threshold, manual verification is triggered. The threshold is updated or the data is corrected based on manual feedback, and the structured data after threshold verification is generated. c. Through the adaptive multi-target matching module, based on the mapping library query and the initial mapping manual verification mechanism, the structured data after threshold verification is adapted in parallel to multiple heterogeneous target systems to generate their own dedicated standardized data; d. Through an intelligent automated scheduling module, the standardized data is automatically entered into multiple target systems in parallel, simulating manual operation; e. Through the human-machine collaboration and anomaly handling module, an interface is provided for adding templates, manually verifying exceeding thresholds, and verifying and correcting templates.

9. The intelligent parsing and mapping method for assisted reproductive reports based on multi-type image recognition according to claim 8, characterized in that, Step d further includes: when data is entered into multiple target systems in parallel, if the entry operation of any target system fails, a cross-system transaction rollback is automatically triggered to cancel the data that has been successfully entered into other target systems.

Images