Dynamic form automatic mapping method and system for multi-modal data

Abstract

The application discloses a kind of dynamic form automatic mapping methods and systems for multi-modal data.The method comprises: obtaining multi-modal original data and target dynamic form unit information;The multi-modal original data is preprocessed to obtain structured intermediate data;For form field dynamically generating structured prompt word template;Structured intermediate data and prompt word template are input into large language model, and structured data segment containing field value, confidence and source position is obtained;The output JSON Schema matched with form structure is constructed, and the data segment is organized as standard JSON data package;After the field value in JSON data package is matched by semantics, type checking and logical constraint, it is automatically filled into the corresponding control of form, and the automatic mapping of multi-modal data to dynamic form is completed.By the present application, multi-modal original data can be automatically mapped, filled and configured to any dynamic form with high precision, which greatly improves data entry efficiency and reduces error rate.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, medical informatics and multimodal data processing technology, specifically to a method and system for automatically mapping multimodal data to dynamic forms based on dynamic prompt word generation, adaptive JSON schema construction and semantic matching, which is particularly suitable for scenarios such as clinical research (IIT / RWS), real-world data acquisition, electronic medical record structuring, and automatic filling of electronic medical record reports (eCRF). Background Technology

[0002] In clinical trials, real-world research, and medical data structured acquisition scenarios, researchers and medical professionals need to extract key fields from multimodal unstructured / semi-structured data sources such as text, test results images, voice recordings, and PDF reports, and manually enter them into dynamic forms such as eCRF. Existing technologies suffer from the following core shortcomings:

[0003] Structure mismatch barrier: LLM output is a flat list of entities, which cannot be directly mapped to the nested structure or repeating group structure of eCRF. It requires manual secondary sorting, which is time-consuming, labor-intensive, and prone to omissions and errors, and cannot meet the needs of large-scale data collection.

[0004] Poor form adaptability: The field naming, control types, validation rules, and nesting structures of eCRF dynamic forms vary significantly across different projects and organizations. Traditional rule engines need to reconfigure the mapping rules for each set of forms, making it impossible to achieve zero-configuration adaptive behavior.

[0005] Fragmented multimodal data processing: Existing systems only support single-modal data processing, lacking unified semantic understanding and cross-modal fusion capabilities for text, images, audio, and PDF, resulting in low data utilization.

[0006] Semantic drift barrier: Relying solely on exact field name matching cannot handle synonyms, abbreviations, or semantic differences in context (such as "blood sugar" versus "GLU" and "fasting blood sugar"). General LLM lacks domain constraints and is prone to misjudgment.

[0007] Output structure mismatch: The model output is a list of general entities or free text, which cannot be aligned with the nested, repeating groups, and other structures of dynamic forms, requiring secondary manual correction.

[0008] In summary, existing technologies cannot achieve automated form filling that requires no human intervention, adapts to any dynamic form, and is compatible with all modal data, which severely restricts the efficiency of clinical research and medical data structuring. Summary of the Invention

[0009] This invention aims to overcome the shortcomings of existing technologies and provide a method and system for automatic mapping of dynamic forms for multimodal data based on dynamic prompt word generation and structured template matching. Through dynamic prompt word generation driven by table cell information, adaptive JSON Schema construction, multimodal unified semantic fusion, hierarchical semantic matching, and dual verification mechanism, it achieves zero-configuration, high-precision, and automated mapping and filling of multimodal raw data to any dynamic form, significantly improving data entry efficiency, reducing error rate, and meeting the compliance and practicality requirements of medical clinical research.

[0010] To achieve the above objectives, this application adopts the following technical solution: In a first aspect, this application provides a method for automatic dynamic form mapping for multimodal data, comprising the following steps: Obtain the metadata of multimodal raw data and the target dynamic form. The multimodal raw data includes at least one of text data, image data, audio data, or PDF data. The metadata of the target dynamic form includes the identifier, label, control type, and validation rules of each field. The multimodal raw data is preprocessed to obtain structured intermediate data. The preprocessing includes OCR recognition of image data to extract text and coordinate information, ASR transcription of audio data to extract text and timestamp information, and entity recognition of text data. For each field of the target dynamic form, based on the control type, tag semantics and validation rules of that field, and combined with synonyms and semantic constraints in the domain knowledge base, a structured prompt word template corresponding to that field is dynamically generated; The structured intermediate data and the structured prompt word template are input into the large language model to obtain the structured data fragment output by the large language model. The structured data fragment includes field values, confidence scores and source location information. Based on the field structure of the target dynamic form, dynamically construct an output JSON Schema that matches the field structure of the target dynamic form, and organize the structured data fragments into a standard JSON data packet according to the output JSON Schema; The field values ​​in the standard JSON data packet are automatically populated into the corresponding controls of the target dynamic form according to semantic matching, type validation, and logical constraints, thus completing the automatic mapping of multimodal data to dynamic forms.

[0011] Optionally, the specific steps for dynamically generating structured prompt word templates include: Based on the control type of the field, select the corresponding basic template from the predefined prompt word template library; Fill the field's label, unit, and value range into the corresponding positions in the basic template; Call the domain knowledge base interface to calculate the semantic similarity between the field tags and the domain knowledge base, and obtain the set of synonyms and semantic constraint rules related to the field tags. The set of synonyms includes abbreviations, aliases and multilingual expressions of standard terms. A weighted semantic vector is constructed based on a set of synonyms, and constraints are generated by integrating field validation rules. The constraints are encoded into JSON format and integrated with the set of synonyms into the base template to generate a complete structured prompt word template.

[0012] Optionally, the domain knowledge base uses the UMLS Unified Medical Language System or SNOMED CT as the underlying terminology source and builds a local cache; the domain knowledge base provides the following calling interfaces: get_synonyms: Returns a list of synonyms and abbreviations for the specified concept; get_semantic_constraint: Returns the semantic constraint rules for the specified concept; get_parent_concepts: Returns the parent concept of the specified concept for context inference.

[0013] Optionally, the step of dynamically constructing an output JSON Schema that matches the field structure of the target dynamic form includes: Parse the metadata of the target dynamic form to identify the nested structure of fields, grouping logic, and repeating group identifiers; Iterate through all fields in the form and extract the field paths and field type information; Recursively construct nested JSON object structures according to field paths, and construct JSON array structures for fields containing repeating groups; Based on the field validation rules, data type constraints and value range constraints are added to the corresponding fields in the JSON Schema to form a complete output JSON Schema.

[0014] Optionally, after inputting the structured intermediate data and structured prompt word templates into the large language model, the process also includes output parsing and error handling steps: Receive the raw text output by the large language model and extract JSON format data fragments from it using regular expressions; If the extracted JSON format data fragment does not meet the predefined structure requirements, the model temperature parameter will be automatically reduced and the large language model will be called again. If no valid output is obtained after retrying, the field will be marked as pending manual review based on its required attributes. Successfully extracted structured data fragments are stored according to field paths, and confidence level and source location information are recorded.

[0015] Optionally, the preprocessing of the multimodal raw data to obtain structured intermediate data includes: For text data, a named entity recognition model trained on a medical corpus is used to extract the entity list and entity locations; For image data, an OCR engine is used to extract text content and text region coordinates from the image, and a layout analysis model is combined to identify the table structure and field correspondence. For audio data, an automatic speech recognition engine is used to transcribe the audio into text, retaining timestamp information, and the transcribed text is then aligned with acoustic features. For PDF data, it is parsed into text blocks and image blocks, preserving page layout information and reading order, and extracting table structure and cross-page relationships.

[0016] Optionally, the step of automatically filling the field values ​​from the standard JSON data packet into the corresponding controls of the target dynamic form according to semantic matching, type validation, and logical constraints includes: Perform multi-level matching between the key names in the JSON data packet and the field identifiers of the target dynamic form, including exact matching, alias matching, and semantic similarity matching based on the text embedding model; For successfully matched field values, perform type validation based on the field's control type, including numeric range validation, date format validation, and option enumeration validation. For field values ​​that pass type validation, further cross-field logical constraint validation is performed, including conditional dependency constraints and consistency constraints. Populate the validated field values ​​into the corresponding controls of the target dynamic form and record the population log.

[0017] Optionally, it also includes a semantic mapping step, specifically including: Map the field labels of the target dynamic form to standard terminology codes. If the field labels cannot be directly matched, retrieve the closest standard term from the domain knowledge base using a string similarity algorithm. Entities in the structured intermediate data are mapped to standard terminology codes, and abbreviations and spelling variations are converted using a thesaurus of the domain knowledge base. The standard terminology code of the field is matched with the standard terminology code of the entity. If there is an exact match, the field is filled directly. If there is a synonym match, the confidence level is lowered before filling. In cases where there are multiple candidate entities, the most suitable entity is selected by combining the semantic constraint rules of the fields with contextual information through rule engine or large model reasoning.

[0018] Optionally, it is characterized in that, The rule engine is used to process simple context patterns, including time modifier matching, negative word detection, and unit conversion; The large model inference is used to handle complex context scenarios. By inputting the original text fragments and field descriptions into the large language model, the large language model outputs the matching judgment results and extracted values.

[0019] Secondly, this application provides a dynamic form automatic mapping system for multimodal data, used to implement the dynamic form automatic mapping method for multimodal data described in the first aspect, the system comprising: The data acquisition module is used to acquire multimodal raw data and the metadata of the target dynamic form. The multimodal raw data includes at least one of text data, image data, audio data, or PDF data. The metadata of the target dynamic form includes the identifier, label, control type, and validation rules of each field. The data preprocessing module is used to preprocess the multimodal raw data to obtain structured intermediate data. The preprocessing includes performing OCR recognition on image data to extract text and coordinate information, performing ASR transcription on audio data to extract text and timestamp information, and performing entity recognition on text data. The prompt word generation module is used to dynamically generate a structured prompt word template corresponding to each field of the target dynamic form, based on the control type, tag semantics and validation rules of the field, and combined with synonyms and semantic constraints in the domain knowledge base. The large model extraction module is used to input the structured intermediate data and the structured prompt word template into the large language model to obtain the structured data fragments output by the large language model. The structured data fragments include field values, confidence scores and source location information. The JSON processing module is used to dynamically construct an output JSON Schema that matches the field structure of the target dynamic form, and organize the structured data fragments into a standard JSON data packet according to the output JSON Schema; The data population module is used to automatically populate the field values ​​in the standard JSON data packet into the corresponding controls of the target dynamic form according to semantic matching, type validation and logical constraints, so as to complete the automatic mapping of multimodal data to dynamic forms.

[0020] Compared with the prior art, this application has the following beneficial effects: 1. Significantly improved efficiency: It can automatically enter dynamic forms, which can greatly improve efficiency compared to manual entry.

[0021] 2. Zero-configuration adaptive: No need to modify the template or retrain the model for new forms, it can automatically adapt to the fields, controls, nested and repetitive group structures of any dynamic form.

[0022] 3. Full modality compatibility: Unifies the processing of data such as text, images, audio, and PDF, and achieves fusion extraction through cross-modal semantic alignment.

[0023] 4. Strong semantic understanding: Based on the domain knowledge base, it solves the problems of synonym, abbreviation and context inference, avoiding semantic mismatch.

[0024] 5. Compliance and Traceability: Each piece of data includes a source evidence chain and confidence level, supporting audit traceability and meeting medical data compliance requirements.

[0025] 6. Precise structural alignment: Adaptive output JSON Schema ensures that the model output perfectly matches the form structure, eliminating the possibility of filling in the wrong position. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Furthermore, these drawings and textual descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments.

[0027] Figure 1 A flowchart illustrating a dynamic form automatic mapping method for multimodal data provided in this application embodiment; Figure 2 This application provides an overall architecture diagram of a dynamic form automatic mapping system for multimodal data, as shown in the embodiments of this application. Figure 3 This is a schematic diagram of the structure of a dynamic form automatic mapping system for multimodal data provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the embodiments of this application. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0029] Example 1 This embodiment provides a method for automatic dynamic form mapping for multimodal data. This method can be implemented through a software system, the architecture of which is as follows: Figure 2 As shown in the figure. This embodiment describes in detail the specific execution steps of the method, as follows. Figure 1 and Figure 2 As shown, the method in this embodiment specifically includes the following steps: Step S1: Obtain the raw multimodal data and the metadata of the target dynamic form.

[0030] Specifically, the multimodal raw data includes at least one of text data, image data, audio data, or PDF data, covering all data sources in clinical research, such as medical record text, test report images, consultation recordings, and examination report PDFs.

[0031] The metadata of the target dynamic form includes the identifiers, labels, control types, and validation rules for each field. Optional additional information includes the form's nested structure, grouping logic, repeating group identifiers, required attributes, units, date formats, enumeration options, and other extended information.

[0032] In practice, multimodal raw data can be obtained through methods such as API import, local upload, and database retrieval. Metadata is input using a standardized JSON Schema format, compatible with mainstream form engines such as REDCap and Viedoc.

[0033] This step provides a unified input benchmark for subsequent preprocessing, prompt word generation, and structured extraction, ensuring the basic compatibility between the form and the data.

[0034] Step S2: Preprocess the multimodal raw data to obtain structured intermediate data.

[0035] This step is a core preliminary step in multimodal unified semantic processing. It employs dedicated processing algorithms for different modalities, and the algorithm selection is based on the advantages of accuracy, efficiency, and adaptability in the medical scenario.

[0036] In summary, preprocessing includes performing OCR (Optical Character Recognition) on image data to extract text and coordinate information, performing ASR (Automatic Speech Recognition) on audio data to extract text and timestamp information, and performing entity recognition on text data.

[0037] In some embodiments, step S2 preprocesses the multimodal raw data to obtain structured intermediate data, specifically including: For text data, a named entity recognition model trained on a medical corpus is used to extract the entity list and entity locations. Specifically, the PubMedBERT named entity recognition model trained on a medical corpus can be used to extract the entity list, entity locations, and contextual text. PubMedBERT achieves over 30% higher accuracy in medical entity recognition compared to the general-purpose BERT, and is suitable for professional texts such as medical records and test reports. The final output includes, but is not limited to, entity names, entity types, start and end positions, and confidence scores.

[0038] For image data, an OCR engine is used to extract text content and text region coordinates from the image, and a layout analysis model is combined to identify table structure and field correspondences. Specifically, the PaddleOCR engine can be used to extract text and coordinate information, and the LayoutLMv3 layout analysis model can be used to identify table structure, field correspondences, and reading order. PaddleOCR has a text recognition rate of ≥98% for medical test reports; LayoutLMv3 can integrate text and layout features to solve problems such as field misalignment and table cross-column issues in images. The final output includes, but is not limited to, text content, region coordinates, table row and column relationships, and field association pairs.

[0039] For audio data, an automatic speech recognition engine is used to transcribe the audio into text, preserving timestamp information and performing acoustic feature alignment on the transcribed text. Specifically, the Whisper ASR engine can be used to transcribe the audio into text, preserving timestamp information and performing acoustic feature alignment. Whisper supports medical speech transcription, with timestamp accuracy ≤100ms, accurately locating the occurrence time of key fields in the audio. The final output includes, but is not limited to, transcribed text, timestamps, and corresponding audio segments.

[0040] For PDF data, it is parsed into text blocks and image blocks, preserving page layout information and reading order, and extracting table structure and cross-page relationships. Specifically, the PDF can be parsed into text blocks and image blocks, preserving page layout and reading order, and extracting table structure and cross-page relationships. The final output includes, but is not limited to, block text, page coordinates, table structure, and cross-page field relationship identifiers.

[0041] Finally, the four types of modal data are standardized in format to eliminate modal differences and form a standardized input that can be directly processed by a large language model. The format is, for example: {content, location / timestamp, source type, confidence level, context information}.

[0042] Step S3: For each field of the target dynamic form, based on the control type, tag semantics, and validation rules of the field, and combined with synonyms and semantic constraints in the domain knowledge base, dynamically generate a structured prompt word template corresponding to the field.

[0043] This step is one of the core innovative aspects of this invention. Its function is to transform the field definitions (meta-information) of the target dynamic form into query instructions (prompt words) that can be accurately understood by the large language model.

[0044] Furthermore, the prompts are dynamically generated based on the form structure, rather than using fixed prompts. In traditional methods, whenever the form structure changes (such as adding fields, modifying labels, or adjusting validation rules), the rules need to be manually reconfigured or the model retrained, which is costly and difficult to scale. This step, through a dynamic generation mechanism, enables the system to possess the following capabilities: Real-time response to form changes: When a user uploads a new dynamic form, the system automatically parses its metadata without any manual intervention and generates corresponding prompt word templates for each field in real time.

[0045] Eliminate pre-configuration dependencies: No longer do we need to write fixed prompts or rule scripts for each project or form, achieving a zero-configuration adaptation mode of "form as configuration".

[0046] Lowering the barrier to entry: Non-technical personnel (such as clinical research coordinators) can directly upload any pre-designed CRF form, and the system will automatically complete all subsequent mapping work without needing to understand the underlying technical details.

[0047] Furthermore, by incorporating domain knowledge (such as synonyms, abbreviations, and semantic constraints) into the prompt words, the shortcomings of the traditional "field name matching" method are effectively addressed. This semantically enhanced prompt word generation mechanism results in a significantly higher accuracy rate for extraction results compared to traditional keyword matching or fixed prompt word schemes.

[0048] In some embodiments, the specific steps for dynamically generating structured prompt word templates include: Step S31: Select the corresponding basic template from the predefined prompt word template library according to the control type of the field.

[0049] Specifically, different control types (numeric input boxes, date pickers, radio buttons, checkboxes, text fields, etc.) require different information extraction strategies and output formats. For example, numeric fields need to specify units and ranges, date fields need to specify formats, and radio button fields need to explicitly list options. Using a uniform template will result in many fields failing to be extracted correctly. Therefore, it is necessary to select an appropriate base template as a standardized skeleton for subsequent filling based on the control type. Furthermore, it is crucial to ensure that the generated prompts have a uniform structure that conforms to business specifications, reducing the difficulty of parsing large language models.

[0050] Step S32: Fill the field's label, unit, and value range into the corresponding positions in the basic template.

[0051] Specifically, this step injects the specific business attributes of the fields into the template skeleton, realizing the personalized transformation from a general template to a specific field.

[0052] In implementation, the label text, unit information, and numerical range (if present) of the fields are extracted from the metadata of the target dynamic form. This information is then filled into the corresponding placeholder positions of the base template selected in step S31 using string replacement or a template engine. This ensures that the generated prompts have a clear target focus, allowing the large language model to clearly understand the specific task, such as which field needs to be extracted, the expected unit, and the effective range. This significantly improves the accuracy of extraction and reduces false and invalid extractions.

[0053] Step S33: Call the domain knowledge base interface to calculate the semantic similarity between the field label and the domain knowledge base, and obtain a set of synonyms related to the field label. The set of synonyms includes abbreviations, aliases and multilingual expressions of standard terms.

[0054] Specifically, field labels in clinical studies often have multiple expressions (e.g., "blood glucose" can be written as "GLU", "glucose", or "glucose"). Traditional "field name matching" methods cannot handle this semantic diversity.

[0055] In this step, medical expertise is integrated into the prompt word generation process, enabling the large language model to identify synonym variants and distinguish similar concepts, effectively solving the "semantic gap" problem and significantly improving the recall and accuracy of extraction.

[0056] For semantic similarity calculation, an embedding model such as PubMedBert can be used, and the similarity threshold can be set to 0.8.

[0057] In some embodiments, the domain knowledge base uses the UMLS Unified Medical Language System or SNOMED CT as the underlying terminology source and builds a local cache; the two terminology bases are standard terminology systems in the medical field, covering synonyms, abbreviations, and semantic constraints, and solving the problem of ambiguity in medical terminology.

[0058] More specifically, the domain knowledge base provides the following API calls: get_synonyms: Returns a list of synonyms (including aliases and multilingual expressions) and abbreviations for the specified concept; get_semantic_constraint: Returns the semantic constraint rules for the specified concept (e.g., "fasting blood glucose must include the fasting identifier"). get_parent_concepts: Returns the parent concept of the specified concept for context inference.

[0059] Step S34: Construct a weighted semantic vector based on the synonym set and generate constraint conditions by integrating field validation rules.

[0060] Specifically, in practice, different contexts may correspond to different meanings (e.g., "blood sugar" may be fasting blood sugar or postprandial blood sugar). Based on this step, constraints are generated by merging field validation rules, such as validating whether "blood sugar" is "fasting blood sugar" through context.

[0061] Step S35: Encode the constraints into JSON format and integrate them with the thesaurus into the base template to generate a complete structured prompt word template.

[0062] In this step, all the information obtained in the previous steps (basic template, field attributes, domain knowledge) is integrated to form a complete prompt word that can be directly executed by the large language model.

[0063] In implementation, the synonym set is formatted as prompt text such as "possible synonyms: GLU, glucose, glucose," and inserted into the appropriate position in the template (usually in the semantic constraint description section). The semantic constraint rules are formatted as natural language instructions (such as "ignore if 'fasting' is not explicitly marked"), encoded in JSON format, and incorporated into the constraint condition section of the template. Finally, all placeholders are replaced, and a complete, structured natural language prompt word template is output, namely the structured prompt word template mentioned above.

[0064] Step S4: Input the structured intermediate data and the structured prompt word template into the large language model to obtain the structured data fragments output by the large language model. These structured data fragments include field values, confidence levels, and source location information.

[0065] The structured intermediate data and structured prompt word templates obtained in the aforementioned steps are input into the large language model, and then the output of the large language model is obtained.

[0066] In specific implementations, large language models can be Qwen2-72B-Instruct, GPT-4o, or ChatGLM3-6B, and must support long contexts (≥32k tokens) and JSON Mode / Function Calling.

[0067] During input, structured intermediate data can be concatenated with structured prompt word templates before input. For extremely long texts (texts exceeding the context length limit of the large language model), sliding window chunking can be used. The sliding window chunking process is as follows: divide the content into multiple chunks according to paragraphs or pages, with the length of each chunk not exceeding 80% of the context limit of the large language model; for each chunk, call the large language model to extract field values ​​to obtain local extraction results; for cases where the same field has multiple candidate values ​​in multiple chunks, select the candidate value with the highest confidence as the final result, and record all candidate values ​​and their source chunks.

[0068] The output of the large language model is a structured data fragment containing field values, confidence levels, and source location information. The confidence level and source location information provide a traceable and verifiable data source for subsequent data entry.

[0069] Step S5: Based on the field structure of the target dynamic form, dynamically construct an output JSON Schema that matches the field structure of the target dynamic form, and organize the structured data fragments into a standard JSON data packet according to the output JSON Schema.

[0070] This step addresses the industry pain point of mismatch between the output of a large language model and the form structure. The structured data fragments output by the large language model are reorganized into a standardized data packet that is strictly aligned with the target form structure. These structured data fragments are flat and unordered, while the target dynamic form often has a complex structure—fields may be organized in groups (e.g., a "Patient Basic Information" group), may form nested structures (e.g., "Patient → Family Information"), and may contain duplicate groups (e.g., an array structure of multiple laboratory tests). Through step S5, these scattered data fragments can be reorganized according to the field structure (grouping, nesting, duplicate groups) of the target form into a JSON data packet that is completely aligned with the form structure.

[0071] In some embodiments, an output JSON schema matching the field structure of the target dynamic form is dynamically constructed based on the field structure of the target dynamic form, specifically including: Step S51: Parse the metadata of the target dynamic form and identify the nested structure of the fields, grouping logic, and repeating group identifiers.

[0072] In practice, the form's field structure is recursively parsed to identify the following information: Nested grouping: This means that a large group contains smaller groups, forming a hierarchical structure of "parent group → child group → field", which is used to classify strongly related sub-module data.

[0073] Grouping logic is a structured management rule in dynamic forms that divides multiple fields into "ordinary groups / nested groups / repeating groups" according to business semantics, data ownership, filling scenarios, and structural relationships. It is used to clarify the hierarchical ownership, display relationship, data organization method, and validation scope of fields.

[0074] Repeat group: This indicates that a group of fields can be filled in multiple times. It is used in scenarios such as "multiple examinations, multiple medications, and multiple adverse events" and is the most commonly used grouping logic in medical forms.

[0075] Step S52: Traverse all fields of the form and extract field paths and field type information.

[0076] The field path indicates the path to each field. Field types include: string, numeric, date, boolean, etc.

[0077] Step S53: Recursively construct nested JSON object structures according to field paths, and construct JSON array structures for fields containing repeating groups.

[0078] That is, recursively construct nested JSON objects according to field paths; construct JSON array structures for repeated group fields.

[0079] A JSON Schema that strictly corresponds to the form structure includes: Field path (e.g., patient.name) Data type constraints (e.g., type: "string") Numerical range constraints (e.g., minimum: 0, maximum: 30) Enumerate options (e.g., enum: ["Yes", "No"]) Step S54: Based on the field validation rules, add data type constraints and value range constraints to the corresponding fields in the JSON Schema to form a complete output JSON Schema.

[0080] The validation rules include: numerical range, enumeration options, regular expressions, etc., to ensure that the fields meet the requirements.

[0081] The final generated JSON Schema is perfectly aligned with the target dynamic form structure, thus avoiding problems such as incorrect field placement, disordered nested structures, and missing duplicate groups from the root.

[0082] Step S6: Automatically populate the field values ​​in the standard JSON data packet into the corresponding controls of the target dynamic form according to semantic matching, type validation, and logical constraints, thus completing the automatic mapping of multimodal data to dynamic forms.

[0083] This step is responsible for accurately writing the cleaned, extracted, and reorganized standardized data into the designated controls of the target form. It acts as a converter from "data" to "form filling action," transforming abstract data values ​​into concrete operations on form controls.

[0084] Semantic matching maps field values ​​from the JSON data packet to the correct control positions in the form, preventing misalignment issues such as filling in "doctor's name" instead of "patient's name". Type validation and logical constraints ensure that the filled data conforms to the form's format requirements and business rules, avoiding format errors (such as incorrect date format), range errors (such as a blood glucose level of 500), and logical contradictions (such as "pregnant" but gender listed as "male").

[0085] In some embodiments, step S6 specifically includes the following steps: Step S61: Perform multi-level matching between the key names in the JSON data packet and the field identifiers of the target dynamic form, including exact matching, alias matching, and semantic similarity matching based on the text embedding model.

[0086] This step is responsible for establishing the mapping relationship between the key names in the JSON data packet and the form field identifiers, determining which control each value should be filled into.

[0087] Because the key names in the JSON data packet (such as fasting_glucose) may not exactly match the form field identifiers (such as blood glucose_fasting), multi-level matching is used to resolve matching failures caused by inconsistent field names, synonyms, abbreviations, etc. The multi-level matching is as follows: Level 1: Exact match, meaning the JSON key name exactly matches the form field ID. This is the fastest and most direct way to cover standard naming scenarios. For example, fasting_glucose → fasting_glucose.

[0088] Level 2: Alias ​​matching, which maps names using a predefined alias table. This handles known naming differences without computational overhead. For example, fasting_glucose → blood glucose_fasting (configured in the alias table). Level 3: Semantic similarity matching, which uses a text embedding model to calculate semantic similarity. This can handle unknown naming differences and has generalization ability. For example, the vector similarity between fasting_glucose and fasting blood glucose value is >0.85, so it is considered a match.

[0089] Step S62: For successfully matched field values, perform type validation based on the field's control type, including numeric range validation, date format validation, and option enumeration validation.

[0090] In this step, multiple validations are used to verify whether the value to be filled conforms to the format requirements of the form field, preventing invalid data from being written. Numeric range validation ensures that the value of numeric fields is within a preset range, date format validation ensures that the value of date fields conforms to a specified format, and option enumeration validation ensures that the value of single / multiple selection fields is in the option list.

[0091] Step S63: For field values ​​that pass type validation, further perform cross-field logical constraint validation, which includes conditional dependency constraints and consistency constraints.

[0092] In this step, further cross-field logical constraint validation ensures that the filled form data conforms to the business logic. Conditional dependency constraints are used to validate the dependencies between fields (for example, if "Pregnancy status" is "yes", then "Whether to take folic acid" must be filled in; if not filled in, a conflict is triggered), and consistency constraints validate the semantic consistency between fields (for example, if "gender" is "male", "pregnancy status" cannot be "yes", otherwise a conflict is marked).

[0093] This ensures the business rationality of the populated data, avoids errors in subsequent data analysis due to logical conflicts between fields, guarantees the data quality of clinical research, and meets the data consistency requirements of GCP (Good Clinical Practice).

[0094] Step S64: Fill the validated field values ​​into the corresponding controls of the target dynamic form and record the filling log.

[0095] The final step in this process is to perform data entry, enabling end-to-end unattended data entry. By recording data entry logs, medical data compliance requirements (such as FDA 21 CFR Part 11) can be met, providing a basis for manual review and supporting post-event traceability and problem localization.

[0096] Based on the process of steps S1 to S6, through dynamic prompt word generation driven by table cell information, adaptive JSONSchema construction, multimodal unified semantic fusion, hierarchical semantic matching and dual verification mechanism, it is possible to achieve zero-configuration, high-precision and automatic mapping and filling of multimodal raw data to any dynamic form, which can greatly improve data entry efficiency, reduce error rate and meet the compliance and practicality requirements of medical clinical research.

[0097] In some embodiments, after inputting the structured intermediate data and structured prompt word templates into the large language model (step S4), the method further includes output parsing and error-tolerant processing steps: Step S41: Receive the raw text output by the large language model and extract the JSON format data fragments using regular expressions.

[0098] Specifically, when generating output, large language models often add additional text descriptions outside the JSON structure, such as: "Based on the text you provided, I found the following information: {'value': 5.6, 'unit': 'mmol / L', 'source': 'Page 3, Paragraph 2'} Please note that this value is fasting blood glucose." Content outside the curly braces {...} is not in valid JSON format, and this information may cause subsequent JSON parsing to fail. Therefore, in this step, regular expressions are used to accurately extract JSON-formatted data fragments from the raw text output by the large language model, filtering out irrelevant and redundant information.

[0099] Step S42: If the extracted JSON format data fragment does not meet the predefined structural requirements, the model temperature parameter will be automatically reduced and the large language model will be called again.

[0100] Specifically, even if S41 successfully extracts a JSON fragment, the fragment may still not meet the requirements. For example, the JSON structure may be incomplete (missing necessary fields), the field types may be incorrect (e.g., numeric fields are output as strings), or the output content may completely deviate from the expected structure. This situation usually occurs because the model lacks determinism during generation—the model's temperature parameter is too high, leading to excessive randomness in the output. Therefore, when the extracted JSON fragment does not meet the predefined structure requirements, the output quality can be improved by adjusting the model parameters and re-invoking the model.

[0101] Step S43: If no valid output can be obtained after calling again, decide whether to mark the field as pending manual review based on the required attributes of the field.

[0102] Specifically, despite the retry mechanism, the following situations may still prevent automatic recovery: the original data does not contain the information for that field; the data is too ambiguous for the large language model to accurately determine; or the model's capabilities are limited, making it unable to understand complex semantics. In these cases, forcibly filling in the data may result in incorrect data, while a direct failure will prevent the entire form from being generated. This step provides a third option: mark it for manual review. First, it determines whether the field is a required field. If it is, it is marked as "pending review"; if it is not, it is allowed to be left blank and does not need to be marked.

[0103] Step S44: Store the successfully extracted structured data fragments according to the field paths, and record the confidence level and source location information.

[0104] This step stores the successfully extracted structured data fragments according to field paths and records metadata, providing structured input for subsequent JSON template construction and mapping, and providing a basis for manual review and auditing.

[0105] In some embodiments, a semantic mapping step S7 is further included, executed between steps S4 and S5, to address the problems of synonym, abbreviation, and context inference, specifically including: Step S71: Map the field labels of the target dynamic form to standard term codes. If the field labels cannot be directly matched, retrieve the closest standard term from the domain knowledge base using a string similarity algorithm.

[0106] Specifically, in practice, form field labels may have multiple expressions ("fasting blood glucose", "fasting glucose"), or different organizations' CRFs may use different names for the same concept. Therefore, it is necessary to unify them to a standard terminology system for matching.

[0107] This step converts the business labels of form fields (such as "fasting blood glucose") into standard terminology codes (such as LOINC2345-7). There are two main matching methods: Exact match: Search the domain knowledge base for standard terms that exactly match the field label; Fuzzy matching: If an exact match fails, use a string similarity algorithm (such as Levenshtein distance or Jaro-Winkler) to retrieve the closest standard term.

[0108] Step S72: Map entities in the structured intermediate data to standard term codes. For abbreviations and spelling variations, convert them using the thesaurus of the domain knowledge base.

[0109] The raw data contains various expressions ("GLU", "glucose", "glucose"), which need to be normalized to standard terms in order to match them with form fields.

[0110] Specifically, this can be achieved in the following way: Synonym mapping: Convert entity words into standard terms through the get_synonyms interface of the domain knowledge base.

[0111] Abbreviation expansion: For abbreviations (such as "GLU"), look up the corresponding complete concept.

[0112] Spelling variant handling: For spelling errors or variants, use edit distance for error-tolerant matching.

[0113] Step S73: Match the standard term code of the field with the standard term code of the entity. If there is an exact match, fill directly; if there is a synonym match, lower the confidence level and fill.

[0114] An exact match means that the field code and the entity code are exactly the same. This type of data is filled directly with the original confidence level. A synonym match means that the field code and the entity code are synonyms. This type of data can be filled after reducing the confidence level by 5-10 percentage points.

[0115] Step S74: In the case of multiple candidate entities, the most suitable entity is selected by combining the semantic constraint rules of the field and the context information through the rule engine or large language model reasoning.

[0116] Specifically, when there are multiple candidate entities for the same field (such as text containing both fasting blood glucose and postprandial blood glucose), the most suitable one is selected based on the context information.

[0117] In practice, a hybrid strategy of rule engine + large language model is adopted.

[0118] The rule engine is used to handle simple contextual patterns, including time modifier matching, negative word detection, and unit conversion; the large language model inference is used to handle complex contextual scenarios. By inputting the original text fragments and field descriptions into the large language model, the large language model outputs the matching judgment results and extracted values.

[0119] To verify the effectiveness of the method, experimental validation was conducted. In pilot applications in one IIT study and one RWS study, the average field filling accuracy of this solution reached 98.2%, with a manual review rate of <5%. Compared to the traditional OCR + rule engine solution (accuracy 82%, review rate >40%), the efficiency is improved by more than 20 times. Compared to manual data entry, the efficiency is improved by 10 times (actual test: a complex CRF document was reduced from 30 minutes to less than 3 minutes). Furthermore, it eliminates the need for pre-defined form templates, supports any dynamic form design engine, requires no manual configuration of field mapping rules, and supports simultaneous processing of text, images, audio, and PDFs, as well as mixed input. In addition, it saves at least 1 million RMB in research funding per project (based on an IIT project with 10 centers, a one-year duration, and each center equipped with a CRC responsible for data collection and entry, the annual cost is at least 100,000 RMB, with total CRC personnel costs exceeding 1 million RMB).

[0120] Example 2 This embodiment provides a method described in Embodiment 1 of a dynamic form automatic mapping system for multimodal data. The system adopts a modular architecture, with each module independently encapsulated and interacting collaboratively. Figure 3 As shown, the specific structure and functions are as follows: The data acquisition module 110 is used to acquire multimodal raw data and the metadata of the target dynamic form. The multimodal raw data includes at least one of text data, image data, audio data or PDF data. The metadata of the target dynamic form includes the identifier, label, control type and validation rules of each field. The data preprocessing module 120 is used to preprocess the multimodal raw data to obtain structured intermediate data. The preprocessing includes performing OCR recognition on image data to extract text and coordinate information, performing ASR transcription on audio data to extract text and timestamp information, and performing entity recognition on text data. The prompt word generation module 130 is used to dynamically generate a structured prompt word template corresponding to each field of the target dynamic form, based on the control type, tag semantics and validation rules of the field, and combined with synonyms and semantic constraints in the domain knowledge base. The large model extraction module 140 is used to input the structured intermediate data and the structured prompt word template into the large language model to obtain the structured data fragments output by the large language model. The structured data fragments include field values, confidence scores and source location information. The JSON processing module 150 is used to dynamically construct an output JSON Schema that matches the field structure of the target dynamic form based on the field structure of the target dynamic form, and organize the structured data fragments into a standard JSON data packet according to the output JSON Schema; The data population module 160 is used to automatically populate the field values ​​in the standard JSON data packet into the corresponding controls of the target dynamic form according to semantic matching, type validation and logical constraints, so as to complete the automatic mapping of multimodal data to dynamic form.

[0121] The modular architecture of this system allows each module to be developed, debugged, and updated independently, facilitating system maintenance and upgrades. It also supports distributed deployment, allowing different modules to be deployed on different servers according to actual application scenarios, thereby improving system processing efficiency.

[0122] In summary, the dynamic form automatic mapping method and system for multimodal data proposed in this application can achieve zero-configuration, high-precision, and automated mapping and filling of multimodal raw data to any dynamic form, greatly improving data entry efficiency, reducing error rate, and meeting the compliance and practicality requirements of medical clinical research.

[0123] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0124] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means at least two.

[0125] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0126] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0127] Those skilled in the art will understand that all or part of the steps of the methods implementing the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0128] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.

[0129] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0130] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for automatic dynamic form mapping for multimodal data, characterized in that, Includes the following steps: Obtain the metadata of multimodal raw data and the target dynamic form. The multimodal raw data includes at least one of text data, image data, audio data, or PDF data. The metadata of the target dynamic form includes the identifier, label, control type, and validation rules of each field. The multimodal raw data is preprocessed to obtain structured intermediate data. The preprocessing includes OCR recognition of image data to extract text and coordinate information, ASR transcription of audio data to extract text and timestamp information, and entity recognition of text data. For each field of the target dynamic form, based on the control type, tag semantics and validation rules of that field, and combined with synonyms and semantic constraints in the domain knowledge base, a structured prompt word template corresponding to that field is dynamically generated; The structured intermediate data and the structured prompt word template are input into the large language model to obtain the structured data fragment output by the large language model. The structured data fragment includes field values, confidence scores and source location information. Based on the field structure of the target dynamic form, dynamically construct an output JSON schema that matches the field structure of the target dynamic form, and organize the structured data fragments into a standard JSON data packet according to the output JSON schema; The field values ​​in the standard JSON data packet are automatically populated into the corresponding controls of the target dynamic form according to semantic matching, type validation, and logical constraints, thus completing the automatic mapping of multimodal data to dynamic forms.

2. The method according to claim 1, characterized in that, The specific steps for dynamically generating structured prompt word templates include: Based on the control type of the field, select the corresponding basic template from the predefined prompt word template library; Fill the field's label, unit, and value range into the corresponding positions in the basic template; The domain knowledge base interface is called to calculate the semantic similarity between the field tags and the domain knowledge base, and to obtain a set of synonyms related to the field tags; the set of synonyms includes abbreviations, aliases and multilingual expressions of standard terms; A weighted semantic vector is constructed based on a set of synonyms, and constraints are generated by integrating field validation rules. The constraints are encoded into JSON format and integrated with the set of synonyms into the base template to generate a complete structured prompt word template.

3. The method according to claim 2, characterized in that, The domain knowledge base uses UMLS (Unified Medical Language System) or SNOMED CT as the underlying terminology source and builds a local cache; the domain knowledge base provides the following calling interfaces: get_synonyms: Returns a list of synonyms and abbreviations for the specified concept; get_semantic_constraint: Returns the semantic constraint rules for the specified concept; get_parent_concepts: Returns the parent concept of the specified concept for context inference.

4. The method according to claim 1, characterized in that, The step of dynamically constructing an output JSON schema that matches the field structure of the target dynamic form includes: Parse the metadata of the target dynamic form to identify the nested structure of fields, grouping logic, and repeating group identifiers; Iterate through all fields in the form and extract the field paths and field type information; Recursively construct nested JSON object structures according to field paths, and construct JSON array structures for fields containing repeating groups; Based on the field validation rules, data type constraints and value range constraints are added to the corresponding fields in the JSON Schema to form a complete output JSON Schema.

5. The method according to claim 1, characterized in that, After inputting the structured intermediate data and structured prompt word templates into the large language model, the process also includes output parsing and error handling steps. Receive the raw text output by the large language model and extract JSON format data fragments from it using regular expressions; If the extracted JSON format data fragment does not meet the predefined structure requirements, the model temperature parameter will be automatically reduced and the large language model will be called again. If no valid output is obtained after retrying, the field will be marked as pending manual review based on its required attributes. Successfully extracted structured data fragments are stored according to field paths, and confidence level and source location information are recorded.

6. The method according to claim 1, characterized in that, The preprocessing of the multimodal raw data to obtain structured intermediate data includes: For text data, a named entity recognition model trained on a medical corpus is used to extract the entity list and entity locations; For image data, an OCR engine is used to extract text content and text region coordinates from the image, and a layout analysis model is combined to identify the table structure and field correspondence. For audio data, an automatic speech recognition engine is used to transcribe the audio into text, retaining timestamp information, and the transcribed text is then aligned with acoustic features. For PDF data, it is parsed into text blocks and image blocks, preserving page layout information and reading order, and extracting table structure and cross-page relationships.

7. The method according to claim 1, characterized in that, The step of automatically filling the field values ​​from the standard JSON data packet into the corresponding controls of the target dynamic form according to semantic matching, type validation, and logical constraints includes: Perform multi-level matching between the key names in the JSON data packet and the field identifiers of the target dynamic form, including exact matching, alias matching, and semantic similarity matching based on the text embedding model; For successfully matched field values, perform type validation based on the field's control type, including numeric range validation, date format validation, and option enumeration validation. For field values ​​that pass type validation, further cross-field logical constraint validation is performed, including conditional dependency constraints and consistency constraints. Populate the validated field values ​​into the corresponding controls of the target dynamic form and record the population log.

8. The method according to claim 7, characterized in that, It also includes a semantic mapping step, specifically including: Map the field labels of the target dynamic form to standard terminology codes. If the field labels cannot be directly matched, retrieve the closest standard term from the domain knowledge base using a string similarity algorithm. Entities in the structured intermediate data are mapped to standard terminology codes, and abbreviations and spelling variations are converted using a thesaurus of the domain knowledge base. The standard terminology code of the field is matched with the standard terminology code of the entity. If there is an exact match, the field is filled directly. If there is a synonym match, the confidence level is lowered before filling. In cases where there are multiple candidate entities, the most suitable entity is selected by combining the semantic constraint rules of the fields with contextual information through rule engine or large model reasoning.

9. The method according to claim 8, characterized in that, The rule engine is used to process simple contextual patterns, including time modifier matching, negative word detection, and unit conversion; The large model inference is used to handle complex context scenarios. By inputting the original text fragments and field descriptions into the large language model, the large language model outputs the matching judgment results and extracted values.

10. A dynamic form automatic mapping system for multimodal data, characterized in that, The system for implementing the method according to any one of claims 1 to 9, the system comprising: The data acquisition module is used to acquire multimodal raw data and the metadata of the target dynamic form. The multimodal raw data includes at least one of text data, image data, audio data, or PDF data. The metadata of the target dynamic form includes the identifier, label, control type, and validation rules of each field. The data preprocessing module is used to preprocess the multimodal raw data to obtain structured intermediate data. The preprocessing includes performing OCR recognition on image data to extract text and coordinate information, performing ASR transcription on audio data to extract text and timestamp information, and performing entity recognition on text data. The prompt word generation module is used to dynamically generate a structured prompt word template corresponding to each field of the target dynamic form, based on the control type, tag semantics and validation rules of the field, and combined with synonyms and semantic constraints in the domain knowledge base. The large model extraction module is used to input the structured intermediate data and the structured prompt word template into the large language model to obtain the structured data fragments output by the large language model. The structured data fragments include field values, confidence scores and source location information. The JSON processing module is used to dynamically construct an output JSON Schema that matches the field structure of the target dynamic form, and organize the structured data fragments into a standard JSON data packet according to the output JSON Schema; The data population module is used to automatically populate the field values ​​in the standard JSON data packet into the corresponding controls of the target dynamic form according to semantic matching, type validation and logical constraints, so as to complete the automatic mapping of multimodal data to dynamic forms.

Images