Image segmentation and simulation linkage method and system based on large language model driving

Abstract

The present application relates to a large language model driven image segmentation and simulation linkage method and system, belonging to the computer vision technical field. The present application solves the problems of segmentation and simulation fragmentation, strategy automatic judgment, time sequence stability and simulation configuration in the existing large language model image segmentation. The method comprises the following steps: S1: inputting an image; S2: performing segmentation processing on the input image to extract a segmentation mask; S3: performing post-processing on the segmentation mask; S4: extracting the geometric parameters of the post-processed segmentation mask and inputting the large language model; S5: executing a simulation task through the large language model.

Description

Technical Field

[0001] This invention relates to a method and system for image segmentation and simulation linkage driven by a large language model, belonging to the field of computer vision technology. Background Technology

[0002] In fields such as materials science, biomedicine, and industrial inspection, researchers need to accurately segment target regions (such as particles, pores, and cells) in microscope or industrial images, and use the segmentation results (target area, quantity, morphological parameters) as input to drive subsequent simulation calculations (such as finite element method, molecular dynamics, and electrochemical simulations). Existing technologies suffer from the following main drawbacks: (1) Segmentation and simulation are separated: Image segmentation and simulation calculation are usually done by different tools, and the transfer of intermediate parameters depends on manual operation, which is inefficient and prone to errors; (2) Segmentation type judgment depends on expert experience: The same dataset may require binary classification (foreground / background) or tri-class classification (foreground / outside background / background holes) segmentation, and manual judgment is time-consuming and inconsistent; (3) Poor timing stability: When performing frame-by-frame segmentation on video or sequence images, the result has timing jitter, which leads to unstable simulation input; (4) High threshold for simulation parameter configuration: Users need to be familiar with the parameter system of the simulation software and cannot describe their needs in natural language and automatically complete the simulation configuration. Summary of the Invention

[0003] This invention addresses the problems of segmentation and simulation separation, automatic strategy judgment, temporal stability, and simulation configuration in existing large language model-driven image segmentation, and proposes a method and system for image segmentation and simulation linkage based on large language model.

[0004] The technical solution adopted by this invention to solve the above problems is: the image segmentation and simulation linkage method based on a large language model proposed in this invention includes: Step 1: Based on the semantic features of the input image, use a large language model to determine the topological structure type of the target region in order to determine the corresponding segmentation strategy; Step 2: Perform image segmentation processing on the input image according to the determined segmentation strategy to obtain the segmentation mask; Step 3: Perform post-processing on the segmentation mask to obtain a target mask that meets topological constraints and is time-stable; Step 4: Extract geometric parameters based on the target mask, and input the geometric parameters into the large language model in the form of keywords; Step 5: Based on the keywords and the user's natural language requirements, use a large language model to generate simulation task configurations to drive downstream simulation tools to execute simulations.

[0005] Furthermore, step 1 specifically includes: Based on the multi-peak grayscale distribution of the input image, the nested relationship of connected components, and the topological keywords in the user's natural language description, a large language model is used to determine whether there are hole structures inside the target region. When determining the presence of hole structures, a three-class segmentation strategy is adopted to divide image pixels into three categories: foreground, background, and background holes. When it is determined that there are no holes, a binary classification segmentation strategy is adopted to divide the image pixels into foreground and background categories.

[0006] Furthermore, in step 2, after determining that a three-class segmentation strategy is adopted, the original mask is mapped to a three-class encoding according to the naming prefix pairing relationship between the source image and the mask. The U-Net segmentation network is trained based on the mapped three-class classification code, where the loss function is a weighted combination of cross-entropy loss and Dice loss, and the Dice loss weight of the background hole class is higher than that of the foreground class. The trained U-Net is used to infer the input image and obtain the three-class logits output; Based on the output of the three-class logits, the three-class segmentation mask is obtained through argmax decision and post-processing.

[0007] Furthermore, after determining the three-class segmentation strategy in step 3, the three-class segmentation mask is sequentially subjected to ring hole morphological closure filling, constraint check that the hole must be within the foreground expansion range, small connected region removal, and hole removal that contacts the background, in order to reconstruct the three-class mask. The hole convexity and model confidence are calculated based on the reconstructed three-class mask. When the hole convexity is lower than the preset threshold or the model confidence is lower than the preset threshold, the prompting segmentation model is triggered to refine the hole region and obtain the refined three-class mask.

[0008] Furthermore, in step 2, after determining that a binary classification segmentation strategy is adopted, the mask is binarized into foreground and background classes based on the naming prefix pairing relationship between the source image and the mask, according to a preset threshold. The U-Net segmentation network is trained based on the binarized mask, where the loss function is a combination of binary cross-entropy loss and soft Dice loss. The trained U-Net is used to infer the input image and obtain single-channel logits output; The single-channel logits output is activated by sigmoid and thresholded to obtain a binary classification segmentation mask.

[0009] Furthermore, in step 3, after determining that a binary classification segmentation strategy is adopted, morphological closing operation, hole filling, small connected component filtering, and edge connected component removal are sequentially performed on the binary classification segmentation mask to obtain a binary classification target mask that meets the boundary quality requirements.

[0010] Furthermore, after obtaining the target mask that conforms to the topological constraints, a topological quality control check is performed on the target mask to verify whether the hole is contained within the foreground and whether the hole is connected to the background. Based on the verification results, a quality control index file is generated, and holes that meet the preset abnormal conditions are automatically repaired to obtain the target mask after quality control.

[0011] Based on the pixel-level category distribution of the segmentation masks of each frame within the preset sliding window of the target mask after quality control, a majority vote is performed at each pixel position to obtain a stable foreground mask. Based on the stable foreground mask, the region outside the background is determined by filling inwards from the boundary, and the background hole region is determined based on pixels that are neither foreground nor outside the background, and the three-class stable mask is reconstructed. Based on the intersection-union ratio (IUU) detection results of the current frame and its neighboring frames, multi-frame majority voting replacement and morphological cleaning are performed on abnormal frames to obtain a temporally stable target mask sequence.

[0012] Furthermore, step 4 includes: Extract at least one geometric parameter from the following: target quantity, target area, total area ratio, porosity, equivalent diameter, and convexity, based on the stabilized target mask: Organize the geometric parameters into keyword parameters in key-value pair form and input them into the large language model.

[0013] Furthermore, step 5 specifically includes: Based on the keyword parameters and the user's input of natural language simulation requirements, a simulation task configuration including solver selection, parameter settings, and running scripts is generated using a large language model. Based on the simulation task configuration, the simulation calculation is performed in a containerized environment, and the simulation results are returned to the user.

[0014] Furthermore, this invention also proposes an image segmentation and simulation linkage system driven by a large language model, comprising: The strategy determination module is used to determine the topological structure type of the target region based on the semantic features of the input image and using a large language model, so as to determine the corresponding segmentation strategy. The image segmentation module is used to perform image segmentation processing on the input image according to the segmentation strategy determined by the strategy judgment module, and obtain the segmentation mask. The segmentation mask processing module is used to perform post-processing on the segmentation mask obtained by the image segmentation module to obtain the target mask that conforms to the topological constraints. The parameter extraction module is used to extract geometric parameters from the target mask obtained by the post-processing module and input the geometric parameters into the large language model in the form of keywords. The simulation linkage module is used to generate simulation task configurations based on keywords and user natural language descriptions input from the parameter extraction module, using a large language model to drive downstream simulation tools to execute simulations.

[0015] The beneficial effects of this invention are: (1) High degree of automation: The method proposed in this invention is fully automated from image input to simulation result output, without the need for manual intervention in segmentation strategy selection and simulation parameter configuration, which significantly reduces the threshold for use.

[0016] (2) High segmentation accuracy: The three-classification path in this invention effectively handles the hole structure through topological constraints and SAM refinement; the two-classification path ensures the boundary quality through morphological post-processing.

[0017] (3) Strong timing stability: The present invention adopts a majority voting combined with an abnormal frame replacement mechanism to eliminate inter-frame jitter in sequence image segmentation and ensure the timing consistency of simulation input.

[0018] (4) Flexible simulation linkage: This invention uses natural language keywords as the interface and supports a variety of simulation solvers (PyBaMM, ASE, LAMMPS, Impedance, etc.), so users do not need to master the details of the simulation software.

[0019] (5) Strong scalability: The image segmentation module, segmentation mask processing module, parameter extraction module and simulation linkage module in the system of the present invention can all be replaced independently, which is convenient for customization for different application scenarios. Attached Figure Description

[0020] Figure 1 The flowchart shows a method for image segmentation and simulation linkage driven by a large language model. Figure 2 This is a flowchart of the three-class classification and segmentation path; Figure 3 This is a flowchart of the binary classification and segmentation path; Figure 4 Flowchart for timing stability processing; Figure 5 This is a schematic diagram illustrating the linkage between segmentation parameter extraction and LLM simulation. Figure 6 This is a block diagram of an image segmentation and simulation linkage system driven by a large language model. Detailed Implementation Specific Implementation Method 1 like Figure 1As shown, the steps of the image segmentation and simulation linkage method based on a large language model driven by this embodiment include: S1: Input image; S2: Perform segmentation processing on the input image and extract the segmentation mask; like Figure 2 The diagram illustrates the steps and logical relationships of data pairing, mask mapping, model training, post-inference processing, and SAM refinement. The source image and mask are paired according to the V4-XXXX prefix. KMeans (k=3) clustering is performed on the original grayscale mask, and it is mapped to a three-class encoding (0=bg_out, 1=fg, 2=bg_hole) sorted by brightness. Semantic swapping is automatically detected and corrected. If the original mask is already a three-valued mask, it is used directly; otherwise… Figure 3 The diagram illustrates the steps of data preparation, binary classification UNet training, inference, and morphological post-processing. Otherwise, KMeans (k=3) clustering is used and sorted by brightness, with the darkest mapping assigned to 0, the middle mapping to 1, and the brightest mapping to 2. Semantic interchange is automatically detected and corrected based on the proportion of holes outside the foreground.

[0022] S3: Post-process the segmentation mask; (1) Three-class segmentation path (UNet+SAM3) S301: Model training: The standard U-Net architecture is used to output 3 types of logits. The loss function is a weighted combination of cross-entropy loss and Dice loss (L = CE + Dice_fg + 2*Dice_hole, with the weight of hole class doubled).

[0023] S302: Post-inference processing: The inference results are processed sequentially as follows: morphological closure filling of annular holes, constraint that holes must be within the foreground expansion range, removal of small connected components, removal of holes that are in contact with the background, and finally reconstruction of the three-class mask.

[0024] S303: Optional SAM3 Refinement: When the hole solidity is below the threshold (default 0.70) or the model confidence is below the threshold (default 0.55), SAM (Segment Anything Model) is triggered to refine the hole region and output the refinement mask.

[0025] (2) Binary classification segmentation path (UNet_2_refine): S304: Data preprocessing: Scan the source image and mask directory, pair them according to the naming prefix, and binarize the mask to {0,1} with a threshold of 128.

[0026] S305: Model Training: U-Net outputs single-channel logits, with the loss function being BCEWithLogits + soft Dice.

[0027] S306: Post-inference processing: After obtaining the probability map using sigmoid, thresholding is performed, followed by morphological closing, flood-fill, small connected component filtering, and edge-connected component removal.

[0028] S307: Perform topological constraint checks on the segmentation results: verify whether holes are contained within the foreground and whether holes are connected to the background, etc.; generate fail_frames.csv records for abnormal frames; perform automatic repair on donut-type abnormal holes according to area / ratio range. QC results are output as pred_qc.csv, containing metrics such as hole_solidity and hole_outside_fg_ratio.

[0029] S308: The frame-by-frame segmentation results of the image sequence are time-stable, such as... Figure 4 The diagram illustrates the steps and logic for majority voting, hole repair, abnormal frame replacement, and area anomaly correction. The specific steps are as follows: 1. Foreground Priority Majority Voting: In a sliding window, a majority vote is performed on each pixel position within the default 5 frames to obtain a stable foreground mask. Hole pixels require a high number of consistent votes to prevent flickering, while ring / foreground pixels employ a lenient retention strategy (to prevent topological breaks).

[0030] 2. Reconstruct a three-class mask from a stable foreground: outside the background, the flood-fill is determined from the boundary inwards; inside the background = neither foreground nor outside the background.

[0031] 3. Hole Repair Heuristic: If a hole is missing in the current frame, and there is a hole in the previous frame but the number of holes in the current frame is very small, then the hole shape / union is borrowed from the neighboring frame for repair; if the hole is connected to the boundary, then the union / reconstruction of the neighboring frames is used for repair.

[0032] 4. Replacement of Temporally Abnormal Frames: If the IoU between the current frame and the two frames before and after it is lower than the threshold (default 0.65), but the IoU between the two frames before and after it is higher than the threshold, then the current frame is determined to be an abnormal frame. It is replaced with the majority vote result of the three frames before, current and after, and morphological cleanup is performed again.

[0033] 5. Foreground area anomaly correction: When the foreground area of ​​the current frame is abnormally large relative to the neighboring frames, it is tightened to the intersection and union of the neighboring frames and the largest connected component is retained; in the case of multiple holes, the largest hole is retained.

[0034] S4: Extract the geometric parameters of the post-processed segmentation mask and input them into the large language model; like Figure 5 The diagram illustrates the keyword parameter organization method, LLM interface call, simulation script generation, and containerized execution process. Target geometric parameters are extracted from the stabilized segmentation mask, including but not limited to: the number of targets, the area of ​​each target (pixels / physical unit), the proportion of the total area, porosity, equivalent diameter, convexity, and other morphological indicators.

[0035] S5: Perform simulation tasks using a large language model; The parameters extracted from S4 are organized as keywords (key=value pairs) and input into the Large Language Model (LLM). Based on the user's natural language requirements and the keyword parameters, the LLM automatically generates the simulation task configuration (solver selection, parameter settings, and execution script), executes the simulation through a containerized environment (Docker), and finally returns the simulation results to the user.

[0036] Furthermore, this implementation also proposes an image segmentation and simulation linkage system driven by a large language model, such as... Figure 6 As shown, it includes a strategy judgment module, an image segmentation module, a post-processing module, a parameter extraction module, and a simulation linkage module connected in sequence.

[0037] The strategy determination module uses a large language model to determine the topological structure type of the target region based on the semantic features of the input image, thereby determining the corresponding segmentation strategy. The image segmentation module performs image segmentation processing on the input image according to the segmentation strategy determined by the strategy determination module, obtaining a segmentation mask. The post-processing module performs post-processing on the segmentation mask obtained by the image segmentation module to obtain a target mask that conforms to topological constraints. The parameter extraction module extracts geometric parameters from the target mask obtained by the post-processing module and inputs the geometric parameters into the large language model in the form of keywords. The simulation linkage module generates a simulation task configuration using the large language model based on the keywords input by the parameter extraction module and the user's natural language requirement description, thereby driving downstream simulation tools to perform simulations.

[0038] Example 1 The user inputs a sequence of 50 porous ceramic cross-section microscope images. The system's LLM module analyzes the images, identifies obvious pore structures, and automatically selects a three-class segmentation strategy.

[0039] Step 1: Data preprocessing.

[0040] The source image and mask are paired according to the V4-XXXX prefix. KMeans (k=3) clustering is performed on the original grayscale mask, and the image is mapped to a three-class classification code according to the brightness. The semantic interchange is automatically detected and corrected.

[0041] Step 2: UNet training and inference.

[0042] The three-class UNet was trained with the loss weight set to CE+Dice_fg+2*Dice_hole. The three-class mask was obtained by inferring frame by frame on 50 images.

[0043] Step 3: Post-processing and QC.

[0044] Perform ring hole filling, hole range constraint, and small connected component removal on the inference mask, and perform topology QC checks to generate pred_qc.csv. Trigger SAM3 refinement for frames with hole_solidity < 0.70.

[0045] Step 4: Timing stable.

[0046] Perform majority voting with a window size of 5 on 50 frames of masks, repair missing frames with holes, replace frames with abnormal IoU, and output a stable mask sequence.

[0047] Step 5: Parameter extraction and simulation.

[0048] Extract parameters such as porosity = 0.23, average pore area = 1250 um^2, and number of pores = 18 from the stable mask. Input these parameters into LLM as keywords. The user describes the calculation of the equivalent thermal conductivity of the porous structure. LLM automatically generates the LAMMPS simulation script and executes it in a Docker container, returning the equivalent thermal conductivity calculation results.

[0049] Example 2 The user inputs a SEM image of a lithium battery electrode cross-section (single frame, no holes). LLM classifies it as a binary classification segmentation task.

[0050] Step 1: Binary classification UNet inference.

[0051] Perform sigmoid thresholding on the image to obtain a binary foreground mask for the active material particles.

[0052] Step 2: Morphological post-processing.

[0053] Perform closing operations (kernel=5x5, iter=2), hole filling, and small connected component filtering (min_area=100 px^2).

[0054] Step 3: Parameter extraction.

[0055] The parameters extracted include an active material volume fraction of 0.62, a particle count of 47, and an average particle size of 8.3 μm.

[0056] Step 4: LLM simulation linkage.

[0057] The user inputs the discharge curve of the electrode at a 1C rate. LLM combines the extracted parameters to automatically configure the PyBaMM simulation parameters (active material volume fraction, particle radius, etc.), generate the simulation script and execute it, and output the voltage-time curve.

[0058] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the scope of the present invention, based on the technical essence of the present invention and within the spirit and principles of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A method for image segmentation and simulation linkage driven by a large language model, characterized in that, include: Step 1: Based on the semantic features of the input image, use a large language model to determine the topological structure type of the target region in order to determine the corresponding segmentation strategy; Step 2: Perform image segmentation processing on the input image according to the determined segmentation strategy to obtain the segmentation mask; Step 3: Perform post-processing on the segmentation mask to obtain a target mask that meets topological constraints and is time-stable; Step 4: Extract geometric parameters based on the target mask, and input the geometric parameters into the large language model in the form of keywords; Step 5: Based on the keywords and the user's natural language requirements, use a large language model to generate simulation task configurations to drive downstream simulation tools to execute simulations.

2. The image segmentation and simulation linkage method based on a large language model as described in claim 1, characterized in that, Step 1 specifically includes: Based on the multi-peak grayscale distribution of the input image, the nested relationship of connected components, and the topological keywords in the user's natural language description, a large language model is used to determine whether there are hole structures inside the target region. When determining the presence of hole structures, a three-class segmentation strategy is adopted to divide image pixels into three categories: foreground, background, and background holes. When it is determined that there are no holes, a binary classification segmentation strategy is adopted to divide the image pixels into foreground and background categories.

3. The image segmentation and simulation linkage method based on a large language model as described in claim 2, characterized in that, In step 2, after determining that a three-class segmentation strategy is to be adopted, the original mask is mapped to a three-class encoding according to the naming prefix pairing relationship between the source image and the mask. The U-Net segmentation network is trained based on the mapped three-class classification code, where the loss function is a weighted combination of cross-entropy loss and Dice loss, and the Dice loss weight of the background hole class is higher than that of the foreground class. The trained U-Net is used to infer the input image and obtain a three-class logits output; Based on the output of the three-class logits, the three-class segmentation mask is obtained through argmax decision and post-processing.

4. The image segmentation and simulation linkage method based on a large language model as described in claim 3, characterized in that, After determining the three-class segmentation strategy in step 3, the three-class segmentation mask is sequentially subjected to the following steps: morphological closure filling of annular holes, constraint check that holes must be within the foreground expansion range, removal of small connected regions, and removal of holes that are in contact with the background, in order to reconstruct the three-class mask. The hole convexity and model confidence are calculated based on the reconstructed three-class mask. When the hole convexity is lower than a preset threshold or the model confidence is lower than a preset threshold, the prompting segmentation model is triggered to refine the hole region and obtain the refined three-class mask.

5. The image segmentation and simulation linkage method based on a large language model as described in claim 2, characterized in that, In step 2, after determining that a binary classification segmentation strategy is to be adopted, the mask is binarized into foreground and background classes based on the naming prefix pairing relationship between the source image and the mask, according to a preset threshold. The U-Net segmentation network is trained based on the binarized mask, where the loss function is a combination of binary cross-entropy loss and soft Dice loss. The trained U-Net is used to infer the input image and obtain single-channel logits output; The single-channel logits output is activated by sigmoid and thresholded to obtain a binary classification segmentation mask.

6. The image segmentation and simulation linkage method based on a large language model as described in claim 5, characterized in that, In step 3, after determining that a binary classification segmentation strategy is adopted, morphological closing operation, hole filling, small connected component filtering, and edge connected component removal are sequentially performed on the binary classification segmentation mask to obtain a binary classification target mask that meets the boundary quality requirements.

7. The image segmentation and simulation linkage method based on a large language model as described in claim 4 or 6, characterized in that, After obtaining the target mask that meets the topological constraints, a topological quality control check is performed on the target mask to verify whether the holes are contained within the foreground and whether the holes are connected to the background. Based on the verification results, a quality control index file is generated, and holes that meet the preset abnormal conditions are automatically repaired to obtain the target mask after quality control. Based on the pixel-level category distribution of the segmentation masks of each frame within the preset sliding window of the target mask after quality control, a majority vote is performed at each pixel position to obtain a stable foreground mask. Based on the stable foreground mask, the region outside the background is determined by filling inwards from the boundary, and the background hole region is determined based on pixels that are neither foreground nor outside the background, and the three-class stable mask is reconstructed. Based on the intersection-union ratio (IUU) detection results of the current frame and its neighboring frames, multi-frame majority voting replacement and morphological cleaning are performed on abnormal frames to obtain a temporally stable target mask sequence.

8. The image segmentation and simulation linkage method based on a large language model as described in claim 1, characterized in that, Step 4 includes: Extract at least one geometric parameter from the following: target quantity, target area, total area ratio, porosity, equivalent diameter, and convexity, based on the stabilized target mask: The geometric parameters are organized into keyword parameters in the form of key-value pairs and input into the large language model.

9. The image segmentation and simulation linkage method based on a large language model as described in claim 1, characterized in that, Step 5 specifically includes: Based on the keyword parameters and the user's input of natural language simulation requirements, a simulation task configuration including solver selection, parameter settings, and running scripts is generated using a large language model. Based on the simulation task configuration, the simulation calculation is performed in a containerized environment, and the simulation results are returned to the user.

10. A large language model-driven image segmentation and simulation linkage system, applied to the large language model-driven image segmentation and simulation linkage method according to any one of claims 1-9, characterized in that, include: The strategy determination module is used to determine the topological structure type of the target region based on the semantic features of the input image and using a large language model, so as to determine the corresponding segmentation strategy. The image segmentation module is used to perform image segmentation processing on the input image according to the segmentation strategy determined by the strategy judgment module, and obtain the segmentation mask. The segmentation mask processing module is used to perform post-processing on the segmentation mask obtained by the image segmentation module to obtain the target mask that conforms to the topological constraints. The parameter extraction module is used to extract geometric parameters from the target mask obtained by the post-processing module and input the geometric parameters into the large language model in the form of keywords. The simulation linkage module is used to generate simulation task configurations based on keywords and user natural language descriptions input from the parameter extraction module, using a large language model to drive downstream simulation tools to execute simulations.

Images