Semantic segmentation model training method, system, equipment and medium
By using a semantic segmentation model training method, feature vectors are generated using an image encoder and a table feature extractor. End-to-end training is then performed using an adaptive semantic rule-constrained loss function. This solves the problems of high data dependency and insufficient generalization ability in table segmentation, and achieves efficient and interpretable table cell segmentation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JILIN PROVINCE JIKE SOFT INFORMATION TECH CO LTD
- Filing Date
- 2026-01-21
- Publication Date
- 2026-04-28
AI Technical Summary
Existing technologies are highly dependent on data and lack generalization ability in table segmentation. Furthermore, traditional methods require strict preprocessing, resulting in poor performance in segmenting complex tables.
A semantic segmentation model training method is adopted, which generates visual and numerical feature vectors through an image encoder and a table feature extractor. End-to-end training is performed by combining a feature fusion module and a decoder. The weights are dynamically adjusted using an adaptive semantic rule constraint loss function to achieve accurate segmentation of table cells.
It improves the ability to generalize across different table types, reduces the dependence on data quality and quantity, directly outputs structured results without complex post-processing, and has good interpretability and scalability.
Smart Images
Figure CN121938007A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of model training technology, and in particular to a semantic segmentation model training method, system, device and medium. Background Technology
[0002] Existing technical solutions first locate the table area and then directly use a deep learning model to segment cells within the area; however, they lack explicit guidance for the model and rely solely on data distribution, which places high demands on both data quality and quantity. Existing technical solutions use large models for table extraction, which can avoid dependence on data, but at the same time increases the dependence on the capabilities of the large model itself, and the current large models' ability to perform on complex tables still needs improvement. Existing technical solutions only use line detection to determine cells, and this traditional method has poor generalization and requires strict preprocessing to be effective. Summary of the Invention
[0003] In view of this, this application provides a semantic segmentation model training method, system, device and medium.
[0004] This application discloses a semantic segmentation model training method, which includes: Step 1: Encode the original image using an image encoder to obtain visual feature vectors; the original image includes the table to be segmented; Step 2: Load the corresponding table feature extractor according to the table type ID, and convert the table data into a numerical feature vector; the table type ID is a unique identifier assigned to different table types, used to distinguish different table structures, field meanings and semantic information; Step 3: The visual feature vector and the numerical feature vector are fused using the feature fusion module to obtain the fused feature vector; Step 4: Decode the fused feature vector using a decoder to output a segmentation map; the segmentation map is the cell image obtained after segmenting the table. Step 5: Construct a loss function based on the segmentation map and the labeled map of the original table. Calculate the total loss function value using the adaptive semantic rule constraint loss function module. The adaptive semantic rule constraint loss function module selects the corresponding semantic constraint rule library based on the input table type ID and dynamically adjusts the weights of different semantic constraints. Step 6: Perform gradient backpropagation based on the total loss function value calculated in Step 5, update the neural network parameters, realize end-to-end training, and complete the semantic segmentation model training; the neural network includes an image encoder, a table feature extractor, a feature fusion module, and a decoder connected in sequence; the neural network parameters include the weights of the image encoder, the parameters of the table feature extractor, the fusion weights of the feature fusion module, and the upsampling layer parameters of the decoder.
[0005] Further, step 2 includes: Based on the table type ID, load the corresponding feature extractor from the predefined rule base; The feature extractor is used to perform structural parsing on the table data and extract the cell structure information of the table. Based on the field mapping rules, the text content in the table is mapped to semantic vectors; The semantic vector is encoded by a semantic understanding model to obtain a numerical feature vector.
[0006] Further, step 3 includes: Through an attention mechanism, the attention weights between image features and table features are calculated, and weighted fusion is performed. The image feature vector is concatenated with the table feature vector to obtain the fused feature vector.
[0007] Furthermore, in step 4, the decoder adopts an upsampling network structure, gradually recovering the resolution of the feature map through deconvolutional layers or transposed convolutional layers, and finally outputting a segmentation map of the same size as the input image.
[0008] Further, step 5 includes: The adaptive semantic rule constraint loss function module selects the corresponding semantic constraint set from the preset semantic constraint rule library based on the currently input table type ID, and calculates the weight of each semantic constraint in the semantic constraint set. The weight is adaptively adjusted based on the current training round number. The semantic constraint set includes column number constraint, column width ratio constraint, and table header and body constraint. The column number constraint is used to constrain the number of columns in the segmentation graph, the column width ratio constraint is used to constrain the column width ratio of the segmentation graph, and the table header and body constraint is used to constrain the table header and body of the segmentation graph. Each semantic constraint in the semantic constraint set is weighted and fused according to its adaptive weight. The weighted fused value is then summed with the pixel-level segmentation value to obtain the total loss function value. During the training process of the model, the error between the segmentation map and the labeled map is minimized by minimizing the total loss function value. The network parameters are then updated through backpropagation to achieve accurate cell segmentation of the table.
[0009] Furthermore, in step 5, the expression for the loss function is:
[0010] Where Ctotal is the total loss function value, which is the target to be minimized during network training, and is composed of a weighted average of pixel-level segmentation loss and multiple semantic rule constraint losses; Lseg is the pixel-level segmentation loss; α is the dynamic weight coefficient of the column number constraint loss; Ncol is the number of columns the model is currently predicting; k is the maximum number of columns allowed for this table type; |Ncol k| represents the column number deviation penalty; the greater the difference between the predicted column number output by the model and the column number in the labeled graph, the larger the total loss function value, guiding the model to generate a column number that meets the requirements; β is the dynamic weight coefficient of the column width ratio constraint loss; r is the column width ratio vector predicted by the model; r template The template column width ratio vector represents the preset standard column width ratio; cos(r,r) template ) represents the cosine similarity; 1 cos(r,r template ) represents the penalty for differences in column width ratio; γ represents the dynamic weighting coefficient of the header and body constraint loss; N head N represents the number of header rows predicted by the model. max The maximum allowed number of header rows; max(0,N) head N max () is a penalty for exceeding the limit for the number of header rows.
[0011] Furthermore, the method also includes: After the semantic segmentation model is trained, save the trained model parameters; During the inference phase, the image to be segmented and its corresponding table type ID are input into the trained semantic segmentation model, which directly outputs the segmentation image as the final result.
[0012] This application also discloses a semantic segmentation model training system that implements the above-described method, comprising: The multimodal input and feature fusion module is used to encode the original image to obtain a visual feature vector; and to convert the table data into a numerical feature vector according to the table type ID. The visual feature vector and the numerical feature vector are fused to obtain the fused feature vector; The decoder is used to decode the fused feature vector and output a segmentation map; The adaptive semantic rule constraint loss function module is used to select the corresponding semantic constraint rule library based on the input table type ID, dynamically adjust the weights of different semantic constraints, and calculate the total loss function value. The training module is used to perform gradient backpropagation based on the total loss function value, update the neural network parameters, realize end-to-end training, and complete the training of the semantic segmentation model. The neural network includes an image encoder, a table feature extractor, a feature fusion module, and a decoder connected in sequence. The neural network parameters include the weights of the image encoder, the parameters of the table feature extractor, the fusion weights of the feature fusion module, and the upsampling layer parameters of the decoder.
[0013] This application also discloses an electronic device, including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method described above.
[0014] This application also discloses a computer-readable storage medium comprising a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described above.
[0015] Due to the adoption of the above technical solution, this application has the following advantages: 1. Strong generalization ability across different table types: It adapts to different table styles through semantic rule constraints; 2. No complex post-processing required: End-to-end training, directly outputting structured results; 3. Good interpretability: Explicit semantic rules provide an interpretable constraint mechanism; 4. Excellent scalability: It is easy to add new table types and semantic rules. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.
[0017] Figure 1 This is a flowchart illustrating a semantic segmentation model training method according to an embodiment of this application. Detailed Implementation
[0018] The present application will be further described in conjunction with the accompanying drawings and embodiments. The described embodiments are only some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of the present application.
[0019] See Figure 1 This application provides an embodiment of a semantic segmentation model training method, which includes: Step 1: Encode the original image using an image encoder to obtain visual feature vectors; the original image includes the table to be segmented; Step 2: Load the corresponding table feature extractor according to the table type ID, and convert the table data into a numerical feature vector; the table type ID is a unique identifier assigned to different table types, used to distinguish different table structures, field meanings and semantic information; Step 3: The visual feature vector and the numerical feature vector are fused using the feature fusion module to obtain the fused feature vector; Step 4: Decode the fused feature vector using a decoder to output a segmentation map; the segmentation map is the cell image obtained after segmenting the table. Step 5: Construct a loss function based on the segmentation map and the labeled map of the original table. Calculate the total loss function value using the adaptive semantic rule constraint loss function module. The adaptive semantic rule constraint loss function module selects the corresponding semantic constraint rule library based on the input table type ID and dynamically adjusts the weights of different semantic constraints. Step 6: Perform gradient backpropagation based on the total loss function value calculated in Step 5, update the neural network parameters, realize end-to-end training, and complete the semantic segmentation model training; the neural network includes an image encoder, a table feature extractor, a feature fusion module, and a decoder connected in sequence; the neural network parameters include the weights of the image encoder, the parameters of the table feature extractor, the fusion weights of the feature fusion module, and the upsampling layer parameters of the decoder.
[0020] For example, the visual feature vector and the numerical feature vector are fused through the feature fusion module, corresponding to... Figure 1 Multimodal feature fusion in [the context of the text].
[0021] Optionally, step 2 includes: Based on the table type ID, load the corresponding feature extractor from the predefined rule base; The feature extractor is used to perform structural parsing on the table data and extract the cell structure information of the table. Based on the field mapping rules, the text content in the table is mapped to semantic vectors; The semantic vector is encoded by a semantic understanding model to obtain a numerical feature vector.
[0022] Optionally, step 3 includes: Through an attention mechanism, the attention weights between image features and table features are calculated, and weighted fusion is performed. The image feature vector is concatenated with the table feature vector to obtain the fused feature vector.
[0023] Optionally, in step 4, the decoder adopts an upsampling network structure, gradually recovering the resolution of the feature map through deconvolution layers or transposed convolution layers, and finally outputting a segmentation map of the same size as the input image.
[0024] Optionally, step 5 includes: The adaptive semantic rule constraint loss function module selects the corresponding semantic constraint set from the preset semantic constraint rule library based on the currently input table type ID, and calculates the weight of each semantic constraint in the semantic constraint set. The weight is adaptively adjusted based on the current training round number. The semantic constraint set includes column number constraint, column width ratio constraint, and table header and body constraint. The column number constraint is used to constrain the number of columns in the segmentation graph, the column width ratio constraint is used to constrain the column width ratio of the segmentation graph, and the table header and body constraint is used to constrain the table header and body of the segmentation graph. Each semantic constraint in the semantic constraint set is weighted and fused according to its adaptive weight. The weighted fused value is then summed with the pixel-level segmentation value to obtain the total loss function value. During the training process of the model, the error between the segmentation map and the labeled map is minimized by minimizing the total loss function value. The network parameters are then updated through backpropagation to achieve accurate cell segmentation of the table.
[0025] For example, the segmentation map is the result of the model's calculations. Through the segmentation map, the final segmentation result can be obtained. For instance, a pixel value of 1 represents a header cell, and a pixel value of 2 represents a body cell. By counting the number of pixels with a value of 1, the number of header cells can be determined, thus indicating the number of columns. Similarly, the label of the original image is manually labeled to indicate which areas are header cells and which are body cells.
[0026] The loss function is calculated by comparing the difference between the result of the model's operation (segmentation map) and the label of the original manually annotated image. Then, the network parameters are updated through backpropagation, which is the training process.
[0027] Optionally, in step 5, the expression for the loss function is:
[0028] Where Ctotal is the total loss function value, which is the target to be minimized during network training. It is composed of a weighted average of pixel-level segmentation loss and multiple semantic rule constraint losses; Lseg is the conventional pixel-level segmentation loss, which is the basic loss for table detection and ensures that the model can correctly segment table regions. It usually uses cross-entropy or Dice loss; α is the dynamic weight coefficient of the column number constraint loss, which is dynamically adjusted according to the number of training epochs; Ncol is the number of table columns currently predicted by the model (an integer); k is the maximum number of columns allowed for this table type (a preset constant); |Ncol k| represents the column number deviation penalty; the greater the difference between the predicted column number output by the model and the column number in the labeled graph, the larger the total loss function value, guiding the model to generate a column number that meets the requirements; β is the dynamic weight coefficient of the column width ratio constraint loss, which is also dynamically adjusted with the training process; r is the column width ratio vector predicted by the model, for example, a vector composed of the proportion of each column width to the total width; r template The template column width ratio vector represents the preset standard column width ratio; cos(r,r) template Cosine similarity measures the consistency of the directions of two vectors, with a range of [-1, 1]. A larger value indicates greater similarity. cos(r,r template ) represents the penalty for differences in column width ratio; the greater the difference between the predicted ratio and the template ratio, the larger the total loss function value, ensuring that the column width ratio meets the requirements; γ is the dynamic weighting coefficient of the header and body constraint loss, which is also dynamically adjusted; N head N is the number of header rows predicted by the model (integer). max The maximum allowed number of header rows (a preset constant); max(0,N) head N max The loss function is penalized for exceeding the limit on the number of header rows; the loss only occurs when the number of predicted rows exceeds the limit, otherwise it is zero; subsequently, if new rules are added for other types of tables, the weighting terms of the loss function can be further expanded.
[0029] Optionally, the method further includes: After the semantic segmentation model is trained, save the trained model parameters; During the inference phase, the image to be segmented and its corresponding table type ID are input into the trained semantic segmentation model, which directly outputs the segmentation image as the final result.
[0030] This application also provides an embodiment of a semantic segmentation model training system, which implements the methods described in the above embodiments, and includes: The multimodal input and feature fusion module is used to encode the original image to obtain a visual feature vector; and to convert the table data into a numerical feature vector according to the table type ID. The visual feature vector and the numerical feature vector are fused to obtain the fused feature vector; The decoder is used to decode the fused feature vector and output a segmentation map; The adaptive semantic rule constraint loss function module is used to select the corresponding semantic constraint rule library based on the input table type ID, dynamically adjust the weights of different semantic constraints, and calculate the total loss function value. The training module is used to perform gradient backpropagation based on the total loss function value, update the neural network parameters, realize end-to-end training, and complete the training of the semantic segmentation model. The neural network includes an image encoder, a table feature extractor, a feature fusion module, and a decoder connected in sequence. The neural network parameters include the weights of the image encoder, the parameters of the table feature extractor, the fusion weights of the feature fusion module, and the upsampling layer parameters of the decoder.
[0031] This application also provides an embodiment of an electronic device, including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the methods described in the above embodiments.
[0032] This application also provides an embodiment of a computer-readable storage medium, the computer-readable storage medium including a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the above embodiments.
[0033] This application enables the model to perform end-to-end cell segmentation of different types of table images by explicitly adding various semantic rule constraints, thereby avoiding complex post-processing and improving the model's generalization ability among different types of table images.
[0034] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this application. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this application should be covered within the protection scope of the claims of this application.
Claims
1. A semantic segmentation model training method, characterized in that, include: Step 1: Encode the original image using an image encoder to obtain visual feature vectors; the original image includes the table to be segmented; Step 2: Load the corresponding table feature extractor according to the table type ID, and convert the table data into a numerical feature vector; the table type ID is a unique identifier assigned to different table types, used to distinguish different table structures, field meanings and semantic information; Step 3: The visual feature vector and the numerical feature vector are fused using the feature fusion module to obtain the fused feature vector; Step 4: Decode the fused feature vector using a decoder to output a segmentation map; the segmentation map is the cell image obtained after segmenting the table. Step 5: Construct a loss function based on the segmentation map and the labeled map of the original table. Calculate the total loss function value using the adaptive semantic rule constraint loss function module. The adaptive semantic rule constraint loss function module selects the corresponding semantic constraint rule library based on the input table type ID and dynamically adjusts the weights of different semantic constraints. Step 6: Perform gradient backpropagation based on the total loss function value calculated in Step 5, update the neural network parameters, realize end-to-end training, and complete the semantic segmentation model training; the neural network includes an image encoder, a table feature extractor, a feature fusion module, and a decoder connected in sequence; the neural network parameters include the weights of the image encoder, the parameters of the table feature extractor, the fusion weights of the feature fusion module, and the upsampling layer parameters of the decoder.
2. The method according to claim 1, characterized in that, Step 2 includes: Based on the table type ID, load the corresponding feature extractor from the predefined rule base; The feature extractor is used to perform structural parsing on the table data and extract the cell structure information of the table. Based on the field mapping rules, the text content in the table is mapped to semantic vectors; The semantic vector is encoded by a semantic understanding model to obtain a numerical feature vector.
3. The method according to claim 1, characterized in that, Step 3 includes: Through an attention mechanism, the attention weights between image features and table features are calculated, and weighted fusion is performed. The image feature vector is concatenated with the table feature vector to obtain the fused feature vector.
4. The method according to claim 1, characterized in that, In step 4, the decoder adopts an upsampling network structure, which gradually restores the resolution of the feature map through deconvolution layers or transposed convolution layers, and finally outputs a segmentation map of the same size as the input image.
5. The method according to claim 1, characterized in that, Step 5 includes: The adaptive semantic rule constraint loss function module selects the corresponding semantic constraint set from the preset semantic constraint rule library based on the currently input table type ID, and calculates the weight of each semantic constraint in the semantic constraint set. The weight is adaptively adjusted based on the current training round number. The semantic constraint set includes column number constraint, column width ratio constraint, and table header and body constraint. The column number constraint is used to constrain the number of columns in the segmentation graph, the column width ratio constraint is used to constrain the column width ratio of the segmentation graph, and the table header and body constraint is used to constrain the table header and body of the segmentation graph. Each semantic constraint in the semantic constraint set is weighted and fused according to its adaptive weight. The weighted fused value is then summed with the pixel-level segmentation value to obtain the total loss function value. During the training process of the model, the error between the segmentation map and the annotation map is minimized by minimizing the total loss function value. The network parameters are then updated through backpropagation to achieve accurate cell segmentation of the table.
6. The method according to claim 1, characterized in that, In step 5, the expression for the loss function is: Where Ctotal is the total loss function value, which is the target to be minimized during network training, and is composed of a weighted average of pixel-level segmentation loss and multiple semantic rule constraint losses; Lseg is the pixel-level segmentation loss; α is the dynamic weight coefficient of the column number constraint loss; Ncol is the number of columns the model is currently predicting; k is the maximum number of columns allowed for this table type; |Ncol k| represents the column number deviation penalty; the greater the difference between the predicted column number output by the model and the column number in the labeled graph, the larger the total loss function value, guiding the model to generate a column number that meets the requirements; β is the dynamic weight coefficient of the column width ratio constraint loss; r is the column width ratio vector predicted by the model; r template The template column width ratio vector represents the preset standard column width ratio; cos(r,r) template ) represents the cosine similarity; 1 cos(r,r template ) represents the penalty for differences in column width ratio; γ represents the dynamic weighting coefficient of the header and body constraint loss; N head N represents the number of header rows predicted by the model. max The maximum allowed number of header rows; max(0,N) head N max () is a penalty for exceeding the limit for the number of header rows.
7. The method according to claim 1, characterized in that, The method further includes: After the semantic segmentation model is trained, save the trained model parameters; During the inference phase, the image to be segmented and its corresponding table type ID are input into the trained semantic segmentation model, which directly outputs the segmentation image as the final result.
8. A semantic segmentation model training system, implementing the method described in any one of claims 1 to 7, characterized in that, include: The multimodal input and feature fusion module is used to encode the original image to obtain a visual feature vector; and to convert the table data into a numerical feature vector according to the table type ID. The visual feature vector and the numerical feature vector are fused to obtain the fused feature vector; The decoder is used to decode the fused feature vector and output a segmentation map; The adaptive semantic rule constraint loss function module is used to select the corresponding semantic constraint rule library based on the input table type ID, dynamically adjust the weights of different semantic constraints, and calculate the total loss function value. The training module is used to perform gradient backpropagation based on the total loss function value, update the neural network parameters, realize end-to-end training, and complete the training of the semantic segmentation model. The neural network includes an image encoder, a table feature extractor, a feature fusion module, and a decoder connected in sequence. The neural network parameters include the weights of the image encoder, the parameters of the table feature extractor, the fusion weights of the feature fusion module, and the upsampling layer parameters of the decoder.
9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it implements the method of any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 7.