Image semantic structured decomposition method based on convolutional neural network

Abstract

The invention discloses an image semantic structured decomposition method based on a convolutional neural network. The image semantic structured decomposition method comprises the steps of 1, collecting a target image and executing standardization processing; 2, generating a fused semantic feature graph and a semantic ridge response graph through an improved HRNet network; 3, constructing a smooth semantic gradient field through gradient extraction based on the fused semantic feature map; 4, based on the semantic ridge response graph, performing semantic gradient flow tracking on the smooth semantic gradient field to generate a semantic candidate region set; 5, calculating and generating a semantic attribute vector set corresponding to the semantic candidate region set; step 6, constructing a semantic hierarchical structure chart according to the semantic candidate region set and the semantic attribute vector set; and 7, converting the semantic hierarchical structure graph into a structure graph file in a JSON format. According to the invention, through the improved HRNet network and semantic gradient flow tracking, the precision and hierarchical expression capability of image semantic structured decomposition are improved.

Description

Technical Field

[0001] This invention relates to the field of image intelligent analysis technology, and in particular to an image semantic structure decomposition method based on convolutional neural networks. Background Technology

[0002] With the increasing demand for structured image understanding in industrial image processing and intelligent manufacturing scenarios, image semantic decomposition methods for semantic entity recognition and structural relationship expression in complex images have attracted widespread attention. Existing image recognition and segmentation technologies mostly rely on fully convolutional neural networks or attention-enhanced models for object detection and region segmentation, but they generally suffer from the following problems in practical applications:

[0003] Extracted semantic features often lack precise perception of structural details and boundary ridges in images, resulting in blurred region edges and incomplete label coverage, making it difficult to meet the needs of semantic hierarchical expression and structural relationship modeling. Multi-scale semantic features suffer from semantic misalignment and spatial inconsistency during fusion, leading to discontinuous regional semantics or weakened contextual relationships, affecting the subsequent construction of structured maps. Existing region generation methods mostly use saliency responses or fixed rules to divide candidate regions, lacking gradient flow guidance and structural boundary constraints, which easily causes region adhesion, incomplete segmentation, or excessive fragmentation, making it difficult to ensure the coherence and semantic consistency of structural decomposition.

[0004] Therefore, how to provide a method for image semantic structured decomposition based on convolutional neural networks is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose an image semantic structure decomposition method based on convolutional neural networks. This invention fully integrates an improved HRNet network and a semantic gradient flow tracing mechanism, detailing the extraction of multiple semantic regions from images, generation of semantic attributes, and construction of a semantic hierarchical structure map. This invention extracts multi-scale semantic features and semantic ridge response maps from images using an improved HRNet network, generates structurally continuous semantic candidate regions through a semantic gradient flow tracing mechanism, and infers the spatial and semantic relationships between regions by combining region semantic attributes and boundary geometric features. It outputs a semantic hierarchical structure map oriented towards interpretable visual tasks, possessing advantages such as clear structural representation, accurate boundary modeling, reasonable semantic organization, and strong machine resolvability.

[0006] An image semantic structure decomposition method based on a convolutional neural network according to an embodiment of the present invention includes the following steps: Step 1: Acquire the target image and perform normalization processing to generate a standard image tensor; Step 2: Input the standardized image tensor into the improved HRNet network to perform high-resolution feature extraction, multi-scale semantic modeling and structural ridge detection, and output a fused semantic feature map and a semantic ridge response map; the improved HRNet network includes a high-resolution backbone convolution module, a multi-scale parallel convolution module and a semantic ridge perception module; Step 3: Based on the fused semantic feature map, construct a smooth semantic gradient field of the target image through gradient extraction and bilateral filtering; Step 4: Based on the semantic ridge response map, perform semantic gradient flow tracing on the smooth semantic gradient field to generate a set of semantic candidate regions for the target image; Step 5: Based on the set of semantic candidate regions and the fused semantic feature map, calculate the average feature vector of the region and the boundary geometric features, and output the set of semantic attribute vectors; Step 6: Based on the set of semantic candidate regions and the set of semantic attribute vectors, infer the spatial inclusion relationship, semantic adjacency relationship and structural hierarchy relationship between the semantic candidate regions, and construct the semantic hierarchy structure diagram of the target image; Step 7: Convert the semantic hierarchy structure diagram of the target image into a JSON format structure map file and output it.

[0007] Optionally, step one specifically includes: The standardization process includes size normalization, color space conversion, noise suppression, and contrast enhancement. The size normalization specifically involves adjusting the size of the target image to a set size using a bilinear interpolation algorithm to obtain a size-normalized image. The color space conversion specifically involves converting the color space of the size-normalized image from RGB space to YCbCr space to obtain a luminance channel image and a chrominance channel image. The noise suppression specifically involves: suppressing high-frequency noise signals in the luminance channel image using a mean filter based on a Gaussian kernel to obtain a smooth channel image; The contrast enhancement specifically involves: using a linear histogram stretching method to normalize the gray values ​​of the smooth channel image to the [0,1] interval, thereby obtaining a normalized channel image; The normalized channel image and the chroma channel image are concatenated to obtain the standard image tensor.

[0008] Optionally, the high-resolution backbone convolution module includes a feature initialization unit, a position-guided residual enhancement unit, and a boundary-aware gating adjustment unit. It generates a high-resolution feature map from the standard image tensor through feature initialization, position-guided residual enhancement, and boundary-aware gating adjustment. Specifically: In the feature initialization unit, the standard image tensor is used to generate an initial feature map through a 3×3 convolution unit; the 3×3 convolution unit includes a two-dimensional convolution with a kernel size of 3×3, batch normalization and ReLU activation; the initial feature map is downsampled through a convolution operation with a stride of 2 to obtain an initial backbone feature map; In the location-guided residual enhancement unit, the location encoding vector of each pixel position in the initial backbone feature map is generated by calculating the sine and cosine functions. The location encoding vectors of all pixel locations are combined to form a location encoding map, and the initial backbone feature map and the location encoding map are concatenated along the channel dimension to generate a location-enhanced feature map; The location augmentation feature map is generated into a convolution output feature map through a 3×3 convolution unit, and the convolution output feature map is residually concatenated with the location augmentation feature map. The output residual augmentation feature map is then activated by ReLU. In the boundary-aware gating adjustment unit, the initial backbone feature map is used to generate the horizontal boundary response map and the vertical boundary response map by using the horizontal Sobel operator and the vertical Sobel operator respectively. The gradient magnitude of the horizontal boundary response map and the vertical boundary response map is calculated to obtain the boundary response map. The boundary response map is then used to generate the boundary gating map by passing a 1× convolution and sigmoid activation. Based on the boundary-gated map, the residual enhanced feature map is gated and fused with the initial backbone feature map to obtain a high-resolution feature map.

[0009] Optionally, the multi-scale parallel convolution module includes a scale construction unit, a frequency-guided modulation unit, and a cross-scale fusion unit, which generates a fused semantic feature map by scale construction, frequency modulation, and cross-scale fusion of the high-resolution feature map, specifically: In the scale construction unit, the high-resolution feature map is downsampled three times by a 3×3 convolution unit with a stride of 2 to generate medium-resolution feature maps, low-resolution feature maps and very low-resolution feature maps. In the frequency-guided modulation unit, two-dimensional fast Fourier transforms are performed on the medium-resolution feature map, low-resolution feature map, and ultra-low-resolution feature map, respectively, and the frequency domain amplitude is calculated to obtain the medium-resolution frequency domain amplitude map, low-resolution frequency domain amplitude map, and ultra-low-resolution frequency domain amplitude map. The medium-resolution frequency domain amplitude map, low-resolution frequency domain amplitude map, and ultra-low-resolution frequency domain amplitude map are respectively generated by 1×1 convolution and Sigmoid activation to generate medium-resolution frequency weight map, low-resolution frequency weight map, and ultra-low-resolution frequency weight map. Depth-separable convolutions are performed on the medium-resolution feature map, low-resolution feature map, and ultra-low-resolution feature map respectively to obtain the medium-resolution semantic feature map, low-resolution semantic feature map, and ultra-low-resolution semantic feature map. The medium-resolution semantic feature map, low-resolution semantic feature map, and ultra-low-resolution semantic feature map are modulated channel by channel with the medium-resolution frequency weight map, low-resolution frequency weight map, and ultra-low-resolution frequency weight map, respectively, to generate the medium-resolution frequency enhancement feature map, the medium-resolution frequency enhancement feature map, and the medium-resolution frequency enhancement feature map. In the cross-scale fusion unit, the medium-resolution frequency-enhanced feature map, the medium-resolution frequency-enhanced feature map and the medium-resolution frequency-enhanced feature map are upsampled to the same spatial size as the high-resolution feature map to obtain three scale-aligned feature maps; The three scale-aligned feature maps are concatenated with the high-resolution feature map along the channel dimension, and a fused semantic feature map is generated by a 3×3 convolutional unit.

[0010] Optionally, the semantic ridge perception module will fuse the semantic feature maps and generate a semantic ridge response map by constructing a structure matrix and filtering gradient extrema, specifically as follows: The maximum channel feature value at each pixel location is extracted along the channel dimension of the fused semantic feature map to construct a dimension-reduced semantic response map. Based on the dimensionality-reduced semantic response map, the horizontal gradient value and vertical gradient value of each pixel position in the horizontal and vertical directions are calculated by using a one-dimensional Gaussian derivative kernel, and the structure matrix of each pixel position is constructed based on the horizontal gradient value and vertical gradient value. The principal and secondary eigenvalues ​​of the structure matrix are calculated using the eigenvalue decomposition method, and the square root of the difference between the principal and secondary eigenvalues ​​is used as the semantic gradient magnitude at each pixel location; the semantic gradient magnitudes at all pixel locations are then combined to form a semantic gradient magnitude map. Non-maximum suppression is performed on the semantic gradient magnitude map to preserve local extrema along the main gradient direction, thus obtaining a ridge response candidate map. Normalize each pixel value in the ridge response candidate map to obtain the semantic ridge response map.

[0011] Optionally, step three specifically includes: The number of channels in the fused semantic feature map is compressed to 1 by using a 1×1 convolution to obtain a semantic activation map; Calculate the first-order gradient of each pixel position in the semantic activation map in the horizontal and vertical directions to obtain the horizontal semantic gradient component and the vertical semantic gradient component. The horizontal and vertical semantic gradient components of each pixel are combined to form a semantic gradient direction vector, and the semantic gradient direction vectors of all pixel locations are combined to form a semantic gradient field. The semantic gradient field of the target image is smoothed by using a bilateral filtering method.

[0012] Optionally, step four specifically includes: Set the seed threshold, gradient stopping threshold, boundary blocking threshold, and maximum number of tracking steps; Construct a seed point set by: if the pixel response value of the semantic ridge response map is greater than the seed threshold, then the corresponding pixel position is used as a seed point; Using each seed point as the starting pixel position, semantic gradient flow tracing is performed in a smooth semantic gradient field, and the next pixel position is obtained by iterative update. In each iteration update, if the termination condition is met, the semantic gradient flow tracking will stop. The termination condition is as follows: the gradient magnitude of the current pixel position is less than the gradient stopping threshold; the boundary response value of the current pixel is greater than the boundary blocking threshold; the current tracking step number reaches the maximum tracking step number. The gradient magnitude is the square root of the sum of the squares of the horizontal and vertical semantic gradient components at each pixel location, and the boundary response value is the square root of the sum of the squares of the edge gradient components calculated based on the horizontal and vertical Sobel operators. By semantic gradient flow tracing, a set of connected pixel locations is formed around each seed point, and the set of connected pixel locations is used as a semantic candidate region. The semantic candidate regions corresponding to all seed points are combined into a set of semantic candidate regions for the target image.

[0013] Optionally, step five specifically includes: Obtain the channel vectors of all pixel positions within the semantic candidate region in the fused semantic feature map, and perform average pooling on all channel vectors to obtain the average feature vector of the region; Based on the boundary contour of each semantic candidate region, the boundary geometric features are calculated, including the region boundary length, region area, principal axis orientation angle of the region's minimum bounding rectangle, and aspect ratio. The main axis direction angle is the angle between the main axis and the horizontal axis, and the aspect ratio is the ratio of the main axis length to the secondary axis length of the smallest bounding rectangle of the region. The average feature vector of each semantic candidate region and the boundary geometric features are combined to form a semantic attribute vector, and the semantic attribute vectors of all semantic candidate regions are combined to form a set of semantic attribute vectors.

[0014] Optionally, step six specifically includes: Based on the minimum bounding rectangle of each semantic candidate region, the region boundary coordinate range is extracted, which includes the minimum horizontal coordinate, the maximum horizontal coordinate, the minimum vertical coordinate, and the maximum vertical coordinate. Construct a spatial inclusion relation matrix: If the region boundary coordinate range of semantic candidate region i completely contains the region boundary coordinate range of semantic candidate region j, then it is determined that semantic candidate region i spatially contains semantic candidate region j, and the corresponding element of the spatial inclusion relation matrix is ​​set to 1; otherwise, it is set to 0. Set a similarity threshold and construct a semantic adjacency matrix: if the cosine similarity between the semantic attribute vectors of semantic candidate region i and semantic candidate region j is greater than the similarity threshold, then semantic candidate region i and semantic candidate region j are semantically close adjacent regions, and the corresponding element of the semantic adjacency matrix is ​​set to 1; otherwise, it is set to 0. The average feature vector of each semantic candidate region is obtained and fed into the MLP classifier to generate the semantic category label of each semantic candidate region. Construct a semantic hierarchical structure graph, where the graph nodes are semantic candidate regions, the feature vectors of the graph nodes are semantic attribute vectors, and the semantic labels of the graph nodes are semantic category labels; if the corresponding element of the spatial containment relation matrix is ​​equal to 1, or the corresponding element of the semantic adjacency relation matrix is ​​equal to 1, then establish graph edges between the corresponding graph nodes of the semantic hierarchical structure graph.

[0015] The beneficial effects of this invention are: First, this invention effectively enhances the structural decomposition capability of semantic elements in images by constructing an improved HRNet network. The improved HRNet network, based on the traditional high-resolution convolutional structure, introduces a position-guided residual enhancement unit and a boundary-aware gating adjustment unit, thereby enhancing the network's response to spatial location information and boundary structures, ensuring that high-resolution feature maps accurately reflect the semantic expression of structurally complex regions in the image. In the multi-scale parallel convolution module, a frequency-guided modulation unit performs frequency domain response modeling and channel modulation on medium, low, and very low-resolution feature maps, improving the stability and resolution of cross-scale semantic features. In the semantic ridge perception module, by combining structure matrix construction and non-maximum suppression operations, the main semantic boundary ridges in the image are effectively extracted, and a semantic ridge response map is output, providing a clear structural guidance signal for region structural segmentation.

[0016] Secondly, the semantic gradient field and semantic gradient flow tracing mechanism proposed in this invention further improve the continuity and structural consistency of semantic candidate region generation. By performing channel compression and gradient extraction on the fused semantic feature map, a smooth semantic gradient field is constructed, effectively suppressing local noise interference and enhancing the gradient direction guidance capability. Based on the seed point set set set by the semantic ridge response map, combined with step size control and termination condition constraints, accurate semantic gradient flow tracing in the gradient field is achieved, obtaining a set of candidate regions with clear structural boundaries and semantic coherence, significantly improving the adhesion or breakage problems existing in traditional region growing methods.

[0017] Furthermore, by fusing semantic feature maps to extract the average feature vectors of semantic candidate regions and boundary geometric features, a set of more expressive semantic attribute vectors is constructed. Based on the spatial inclusion relationship and semantic adjacency relationship between the region boundary coordinate range and the semantic attribute vectors, a semantic hierarchical structure graph is constructed, realizing complete semantic expression from the region level to the structure level. Finally, the semantic hierarchical structure graph is converted into a JSON format structure graph file, supporting downstream tasks for hierarchical analysis and semantic reconstruction of image content.

[0018] In summary, this invention significantly improves the accuracy and representational completeness of the structured decomposition of complex semantic elements in images by using an improved HRNet network and a semantic gradient flow tracing mechanism. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0020] Figure 1 This is a schematic diagram of an image semantic structure decomposition method based on convolutional neural networks proposed in this invention; Figure 2 This is a flowchart of the improved HRNet network structure in the image semantic structured decomposition method based on convolutional neural networks proposed in this invention; Figure 3 This is a flowchart of the process for generating a set of semantic candidate regions in an image semantic structured decomposition method based on convolutional neural networks proposed in this invention. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0022] refer to Figures 1-3 A method for semantic structural decomposition of images based on convolutional neural networks includes the following steps: Step 1: Acquire the target image and perform normalization processing to generate a standard image tensor; Step 2: Input the standardized image tensor into the improved HRNet network to perform high-resolution feature extraction, multi-scale semantic modeling and structural ridge detection, and output a fused semantic feature map and a semantic ridge response map; the improved HRNet network includes a high-resolution backbone convolution module, a multi-scale parallel convolution module and a semantic ridge perception module; Step 3: Based on the fused semantic feature map, construct a smooth semantic gradient field of the target image through gradient extraction and bilateral filtering; Step 4: Based on the semantic ridge response map, perform semantic gradient flow tracing on the smooth semantic gradient field to generate a set of semantic candidate regions for the target image; Step 5: Based on the set of semantic candidate regions and the fused semantic feature map, calculate the average feature vector of the region and the boundary geometric features, and output the set of semantic attribute vectors; Step 6: Based on the set of semantic candidate regions and the set of semantic attribute vectors, infer the spatial inclusion relationship, semantic adjacency relationship and structural hierarchy relationship between the semantic candidate regions, and construct the semantic hierarchy structure diagram of the target image; Step 7: Convert the semantic hierarchy structure diagram of the target image into a JSON format structure map file and output it.

[0023] In this embodiment, step one specifically includes: The standardization process includes size normalization, color space conversion, noise suppression, and contrast enhancement. The size normalization specifically involves adjusting the size of the target image to a set size using a bilinear interpolation algorithm to obtain a size-normalized image. The color space conversion specifically involves converting the color space of the size-normalized image from RGB space to YCbCr space to obtain a luminance channel image and a chrominance channel image. The noise suppression specifically involves: suppressing high-frequency noise signals in the luminance channel image using a mean filter based on a Gaussian kernel to obtain a smooth channel image; The contrast enhancement specifically involves: using a linear histogram stretching method to normalize the gray values ​​of the smooth channel image to the [0,1] interval, thereby obtaining a normalized channel image; The normalized channel image and the chroma channel image are concatenated to obtain the standard image tensor.

[0024] In this embodiment, the high-resolution backbone convolution module includes a feature initialization unit, a position-guided residual enhancement unit, and a boundary-aware gating adjustment unit. It generates a high-resolution feature map from the standard image tensor through feature initialization, position-guided residual enhancement, and boundary-aware gating adjustment. Specifically: In the feature initialization unit, the standard image tensor is used to generate an initial feature map through a 3×3 convolution unit; the 3×3 convolution unit includes a two-dimensional convolution with a kernel size of 3×3, batch normalization and ReLU activation; the initial feature map is downsampled through a convolution operation with a stride of 2 to obtain an initial backbone feature map; In the location-guided residual enhancement unit, the location encoding vector of each pixel position in the initial backbone feature map is generated by calculating the sine and cosine functions: ; ; Where i represents the position encoding dimension index, with a value ranging from 0 to... d represents the total dimension of the set position encoding vector. This represents the encoded value corresponding to the 2i-th dimension of the pixel position (x, y) in the position encoding vector. This represents the encoded value corresponding to the (2i+1)th dimension of the pixel position (x, y) in the position encoding vector; The location encoding vectors of all pixel locations are combined to form a location encoding map, and the initial backbone feature map and the location encoding map are concatenated along the channel dimension to generate a location-enhanced feature map; The location augmentation feature map is generated into a convolution output feature map through a 3×3 convolution unit, and the convolution output feature map is residually concatenated with the location augmentation feature map. The output residual augmentation feature map is then activated by ReLU. In the boundary-aware gating adjustment unit, the initial backbone feature map is used to generate the horizontal boundary response map and the vertical boundary response map by using the horizontal Sobel operator and the vertical Sobel operator respectively. The gradient magnitude of the horizontal boundary response map and the vertical boundary response map is calculated to obtain the boundary response map. The boundary response map is then used to generate the boundary gating map by passing a 1× convolution and sigmoid activation. Based on the boundary-gated map, the residual enhanced feature map is gated and fused with the initial backbone feature map to obtain a high-resolution feature map.

[0025] In this embodiment, the multi-scale parallel convolution module includes a scale construction unit, a frequency-guided modulation unit, and a cross-scale fusion unit. It generates a fused semantic feature map by performing scale construction, frequency modulation, and cross-scale fusion on the high-resolution feature map. Specifically: In the scale construction unit, the high-resolution feature map is downsampled three times by a 3×3 convolution unit with a stride of 2 to generate medium-resolution feature maps, low-resolution feature maps and very low-resolution feature maps. In the frequency-guided modulation unit, two-dimensional fast Fourier transforms are performed on the medium-resolution feature map, low-resolution feature map, and ultra-low-resolution feature map, respectively, and the frequency domain amplitude is calculated to obtain the medium-resolution frequency domain amplitude map, low-resolution frequency domain amplitude map, and ultra-low-resolution frequency domain amplitude map. The medium-resolution frequency domain amplitude map, low-resolution frequency domain amplitude map, and ultra-low-resolution frequency domain amplitude map are respectively generated by 1×1 convolution and Sigmoid activation to generate medium-resolution frequency weight map, low-resolution frequency weight map, and ultra-low-resolution frequency weight map. Depth-separable convolutions are performed on the medium-resolution feature map, low-resolution feature map, and ultra-low-resolution feature map respectively to obtain the medium-resolution semantic feature map, low-resolution semantic feature map, and ultra-low-resolution semantic feature map. The medium-resolution semantic feature map, low-resolution semantic feature map, and ultra-low-resolution semantic feature map are modulated channel by channel with the medium-resolution frequency weight map, low-resolution frequency weight map, and ultra-low-resolution frequency weight map, respectively, to generate the medium-resolution frequency enhancement feature map, the medium-resolution frequency enhancement feature map, and the medium-resolution frequency enhancement feature map. In the cross-scale fusion unit, the medium-resolution frequency-enhanced feature map, the medium-resolution frequency-enhanced feature map and the medium-resolution frequency-enhanced feature map are upsampled to the same spatial size as the high-resolution feature map to obtain three scale-aligned feature maps; The three scale-aligned feature maps are concatenated with the high-resolution feature map along the channel dimension, and a fused semantic feature map is generated by a 3×3 convolutional unit.

[0026] In this embodiment, the semantic ridge perception module generates a semantic ridge response map by fusing semantic feature maps and constructing a structure matrix and filtering gradient extrema, specifically as follows: The maximum channel feature value at each pixel location is extracted along the channel dimension of the fused semantic feature map to construct a dimension-reduced semantic response map. Based on the dimensionality-reduced semantic response map, the horizontal and vertical gradient values ​​of each pixel position are calculated using a one-dimensional Gaussian derivative kernel. A structure matrix for each pixel position is then constructed based on the horizontal and vertical gradient values. The structure matrix is ​​a 2×2 structure matrix, where the top-left element is the square of the horizontal gradient value, the top-right element is the product of the horizontal and vertical gradient values, the bottom-left element is the product of the horizontal and vertical gradient values, and the bottom-right element is the square of the vertical gradient value. The principal and secondary eigenvalues ​​of the structure matrix are calculated using the eigenvalue decomposition method, and the square root of the difference between the principal and secondary eigenvalues ​​is used as the semantic gradient magnitude at each pixel location; the semantic gradient magnitudes at all pixel locations are then combined to form a semantic gradient magnitude map. Non-maximum suppression is performed on the semantic gradient magnitude map to retain local extrema along the main gradient direction, thus obtaining a ridge response candidate map. Specifically, for each pixel position in the semantic gradient magnitude map, the gradient magnitudes of two adjacent pixel positions are obtained along the main gradient direction of the pixel position. If the gradient magnitude of the current pixel position is less than the gradient magnitude of either adjacent pixel position, the gradient magnitude of the current pixel position is set to 0; otherwise, the original gradient magnitude is retained. For example, the main gradient direction of pixel P=(1,1) is determined to be 45°. Two adjacent pixels along this direction are obtained: the previous adjacent pixel is (0,2), and the next adjacent pixel is (2,0). The magnitude of the current pixel P is G(1,1)=0.8, and the magnitudes of the two adjacent pixels are G(0,2)=0.4 and G(2,0)=0.5. Since G(1,1)>G(0,2) and G(1,1)>G(2,0), the magnitude of the current pixel is retained. Normalize each pixel value in the ridge response candidate map to obtain the semantic ridge response map.

[0027] In this embodiment, step three specifically includes: The number of channels in the fused semantic feature map is compressed to 1 by using a 1×1 convolution to obtain a semantic activation map; Calculate the first-order gradient of each pixel position in the semantic activation map in the horizontal and vertical directions to obtain the horizontal semantic gradient component and the vertical semantic gradient component. The horizontal and vertical semantic gradient components of each pixel are combined to form a semantic gradient direction vector, and the semantic gradient direction vectors of all pixel locations are combined to form a semantic gradient field. The semantic gradient field of the target image is smoothed by using a bilateral filtering method.

[0028] In this embodiment, step four specifically includes: Set the seed threshold, gradient stopping threshold, boundary blocking threshold, and maximum number of tracking steps; Construct a seed point set by: if the pixel response value of the semantic ridge response map is greater than the seed threshold, then the corresponding pixel position is used as a seed point; Using each seed point as the starting pixel position, semantic gradient flow tracing is performed in a smooth semantic gradient field to iteratively update and obtain the next pixel position. : ; ; in, These are step size control parameters. This represents the x-coordinate of the current pixel position. The vertical coordinate of the current pixel position. This represents the smooth horizontal semantic gradient component at the current pixel position. This represents the smooth vertical semantic gradient component at the current pixel position. This represents the x-coordinate of the next pixel position. This represents the y-coordinate of the next pixel position. Indicates the number of tracking steps; In each iteration update, if the termination condition is met, the semantic gradient flow tracking will stop. The termination condition is as follows: the gradient magnitude of the current pixel position is less than the gradient stopping threshold; the boundary response value of the current pixel is greater than the boundary blocking threshold; the current tracking step number reaches the maximum tracking step number. The gradient magnitude is the square root of the sum of the squares of the horizontal and vertical semantic gradient components at each pixel location, and the boundary response value is the square root of the sum of the squares of the edge gradient components calculated based on the horizontal and vertical Sobel operators. By semantic gradient flow tracing, a set of connected pixel locations is formed around each seed point, and the set of connected pixel locations is used as a semantic candidate region. The semantic candidate regions corresponding to all seed points are combined into a set of semantic candidate regions for the target image.

[0029] In this embodiment, step five specifically includes: Obtain the channel vectors of all pixel positions within the semantic candidate region in the fused semantic feature map, and perform average pooling on all channel vectors to obtain the average feature vector of the region; Based on the boundary contour of each semantic candidate region, the boundary geometric features are calculated, including the region boundary length, region area, principal axis orientation angle of the region's minimum bounding rectangle, and aspect ratio. The main axis direction angle is the angle between the main axis and the horizontal axis, and the aspect ratio is the ratio of the main axis length to the secondary axis length of the smallest bounding rectangle of the region. The average feature vector of each semantic candidate region and the boundary geometric features are combined to form a semantic attribute vector, and the semantic attribute vectors of all semantic candidate regions are combined to form a set of semantic attribute vectors.

[0030] In this embodiment, step six specifically includes: Based on the minimum bounding rectangle of each semantic candidate region, the region boundary coordinate range is extracted, which includes the minimum horizontal coordinate, the maximum horizontal coordinate, the minimum vertical coordinate, and the maximum vertical coordinate. Construct a spatial inclusion relation matrix: If the region boundary coordinate range of semantic candidate region i completely contains the region boundary coordinate range of semantic candidate region j, then it is determined that semantic candidate region i spatially contains semantic candidate region j, and the corresponding element of the spatial inclusion relation matrix is ​​set to 1; otherwise, it is set to 0. The complete inclusion is specifically defined as follows: if the minimum horizontal coordinate of semantic candidate region i is less than or equal to the minimum horizontal coordinate of semantic candidate region j, and the maximum horizontal coordinate of semantic candidate region i is greater than or equal to the maximum horizontal coordinate of semantic candidate region j, and the minimum vertical coordinate of semantic candidate region i is less than or equal to the minimum vertical coordinate of semantic candidate region j, and the maximum vertical coordinate of semantic candidate region i is greater than or equal to the maximum vertical coordinate of semantic candidate region j, then it is determined that semantic candidate region i spatially includes semantic candidate region j. Set a similarity threshold and construct a semantic adjacency matrix: if the cosine similarity between the semantic attribute vectors of semantic candidate region i and semantic candidate region j is greater than the similarity threshold, then semantic candidate region i and semantic candidate region j are semantically close adjacent regions, and the corresponding element of the semantic adjacency matrix is ​​set to 1; otherwise, it is set to 0. The average feature vector of each semantic candidate region is obtained and fed into the MLP classifier to generate the semantic category label of each semantic candidate region. Construct a semantic hierarchical structure graph, where the graph nodes are semantic candidate regions, the feature vectors of the graph nodes are semantic attribute vectors, and the semantic labels of the graph nodes are semantic category labels; if the corresponding element of the spatial containment relation matrix is ​​equal to 1, or the corresponding element of the semantic adjacency relation matrix is ​​equal to 1, then establish graph edges between the corresponding graph nodes of the semantic hierarchical structure graph.

[0031] Example 1 To verify the feasibility of this invention in practice, the method was applied to the automatic semantic structure parsing task of industrial product nameplate images in an industrial quality inspection platform. In this application scenario, industrial nameplates typically contain multiple texts, identifiers, borders, and functional areas. Traditional OCR methods or rule-based image segmentation methods struggle to accurately segment multi-line nested structures, distinguish semantically adjacent regions, and lack the ability to express structural hierarchy, resulting in parsing results that fail to meet the needs of downstream image structure modeling or digital twin label reconstruction.

[0032] In actual deployment, the acquired high-resolution nameplate images undergo image size normalization, noise suppression, and brightness enhancement to form a standard image tensor, which is then input into an improved HRNet network to extract high-resolution feature maps and perform multi-scale semantic modeling to generate semantic ridge response maps. A semantic gradient field is constructed based on the fused semantic feature maps, and semantic candidate regions are generated by tracing the semantic gradient flow guided by the ridges. Subsequently, the average feature vector and boundary geometric properties of each semantic candidate region are calculated, and a semantic hierarchical structure map is constructed based on the spatial inclusion relationship and semantic similarity between the semantic candidate regions, outputting a JSON format map file.

[0033] To evaluate the performance of this invention in real-world scenarios, two commonly used comparative methods in enterprises were selected as references: Comparison Method A: Traditional OCR + connected component segmentation method; Comparison Method B: Semantic segmentation method based on U-Net structure. Comparative experiments were constructed based on the three invention methods, using a nameplate dataset from five manufacturing companies, totaling 1627 images with an average resolution of 1800×1200, covering typical styles such as motor nameplates, inverter nameplates, and pump / valve nameplates. On the same test set, six comparative metrics of the different invention methods in the structural recognition task were compared. The experimental results are shown in Table 1.

[0034] Table 1. Performance Comparison of the Invention Method and Comparative Methods in Nameplate Image Structure Analysis

[0035] As shown in Table 1, the method of this invention exhibits significant advantages over the comparative methods in image semantic structure decomposition tasks. In terms of region segmentation accuracy, the method of this invention reaches 96.4%, an improvement of nearly 13 percentage points compared to 83.5% of comparative method A, and an improvement of over 7 percentage points compared to 89.2% of comparative method B. This indicates that the method of this invention has higher accuracy in semantic region recognition and is more suitable for industrial image scenarios with complex boundaries and nested regions. In terms of structural hierarchy restoration rate, the method of this invention reaches 92.3%, significantly better than the comparative methods. This shows that the method can not only segment regions but also accurately infer the hierarchical inclusion and adjacency relationships between regions, meeting the requirements of semantic structure modeling. Regarding the average structure IoU index, the method of this invention reaches 0.893, significantly better than 0.621 of comparative method A and 0.732 of comparative method B. This further indicates a higher degree of overlap between the segmentation results and the actual structure annotations, with significant improvements in boundary accuracy and region coverage.

[0036] Furthermore, regarding boundary fitting error, the method of this invention has an error of 4.1 pixels, which is significantly lower than the 14.7 pixels of representative method A and the 8.2 pixels of comparative method B, reflecting the stronger control capability of the method of this invention in boundary awareness, ridge extraction, and semantic gradient tracking. In terms of JSON structure matching rate, this invention achieves 94.8%, significantly higher than the comparative method, demonstrating extremely high semantic structure graph reconstruction accuracy, which can provide a high-quality structural foundation for subsequent machine parsing, graph retrieval, and visual knowledge fusion. Although the comparative method A consumes less processing time than the method of this invention, the method of this invention maintains both high accuracy and good inference efficiency.

[0037] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for semantic structural decomposition of images based on convolutional neural networks, characterized in that, Includes the following steps: Step 1: Acquire the target image and perform normalization processing to generate a standard image tensor; Step 2: Input the standardized image tensor into the improved HRNet network to perform high-resolution feature extraction, multi-scale semantic modeling and structural ridge detection, and output a fused semantic feature map and a semantic ridge response map; the improved HRNet network includes a high-resolution backbone convolution module, a multi-scale parallel convolution module and a semantic ridge perception module; Step 3: Based on the fused semantic feature map, construct a smooth semantic gradient field of the target image through gradient extraction and bilateral filtering; Step 4: Based on the semantic ridge response map, perform semantic gradient flow tracing on the smooth semantic gradient field to generate a set of semantic candidate regions for the target image; Step 5: Based on the set of semantic candidate regions and the fused semantic feature map, calculate the average feature vector of the region and the boundary geometric features, and output the set of semantic attribute vectors; Step 6: Based on the set of semantic candidate regions and the set of semantic attribute vectors, infer the spatial inclusion relationship, semantic adjacency relationship and structural hierarchy relationship between the semantic candidate regions, and construct the semantic hierarchy structure diagram of the target image; Step 7: Convert the semantic hierarchy structure diagram of the target image into a JSON format structure map file and output it.

2. The image semantic structure decomposition method based on convolutional neural networks according to claim 1, characterized in that, Step one specifically includes: The standardization process includes size normalization, color space conversion, noise suppression, and contrast enhancement. The size normalization specifically involves adjusting the size of the target image to a set size using a bilinear interpolation algorithm to obtain a size-normalized image. The color space conversion specifically involves converting the color space of the size-normalized image from RGB space to YCbCr space to obtain a luminance channel image and a chrominance channel image. The noise suppression specifically involves: suppressing high-frequency noise signals in the luminance channel image using a mean filter based on a Gaussian kernel to obtain a smooth channel image; The contrast enhancement specifically involves: using a linear histogram stretching method to normalize the gray values ​​of the smooth channel image to the [0,1] interval, thereby obtaining a normalized channel image; The normalized channel image and the chroma channel image are concatenated to obtain the standard image tensor.

3. The image semantic structure decomposition method based on convolutional neural networks according to claim 1, characterized in that, The high-resolution backbone convolutional module includes a feature initialization unit, a position-guided residual enhancement unit, and a boundary-aware gating adjustment unit. It generates a high-resolution feature map from the standard image tensor through feature initialization, position-guided residual enhancement, and boundary-aware gating adjustment. Specifically: In the feature initialization unit, the standard image tensor is used to generate an initial feature map through a 3×3 convolution unit; the 3×3 convolution unit includes a two-dimensional convolution with a kernel size of 3×3, batch normalization and ReLU activation; the initial feature map is downsampled through a convolution operation with a stride of 2 to obtain an initial backbone feature map; In the location-guided residual enhancement unit, the location encoding vector of each pixel position in the initial backbone feature map is generated by calculating the sine and cosine functions. The location encoding vectors of all pixel locations are combined to form a location encoding map, and the initial backbone feature map and the location encoding map are concatenated along the channel dimension to generate a location-enhanced feature map; The location augmentation feature map is generated into a convolution output feature map through a 3×3 convolution unit, and the convolution output feature map is residually concatenated with the location augmentation feature map. The output residual augmentation feature map is then activated by ReLU. In the boundary-aware gating adjustment unit, the initial backbone feature map is used to generate the horizontal boundary response map and the vertical boundary response map by using the horizontal Sobel operator and the vertical Sobel operator respectively. The gradient magnitude of the horizontal boundary response map and the vertical boundary response map is calculated to obtain the boundary response map. The boundary response map is then used to generate the boundary gating map by passing a 1× convolution and sigmoid activation. Based on the boundary-gated map, the residual enhanced feature map is gated and fused with the initial backbone feature map to obtain a high-resolution feature map.

4. The image semantic structure decomposition method based on convolutional neural networks according to claim 1, characterized in that, The multi-scale parallel convolution module includes a scale construction unit, a frequency-guided modulation unit, and a cross-scale fusion unit. It generates a fused semantic feature map by scaling, frequency modulation, and cross-scale fusion of the high-resolution feature map. Specifically: In the scale construction unit, the high-resolution feature map is downsampled three times by a 3×3 convolution unit with a stride of 2 to generate medium-resolution feature maps, low-resolution feature maps and very low-resolution feature maps. In the frequency-guided modulation unit, two-dimensional fast Fourier transforms are performed on the medium-resolution feature map, low-resolution feature map, and ultra-low-resolution feature map, respectively, and the frequency domain amplitude is calculated to obtain the medium-resolution frequency domain amplitude map, low-resolution frequency domain amplitude map, and ultra-low-resolution frequency domain amplitude map. The medium-resolution frequency domain amplitude map, low-resolution frequency domain amplitude map, and ultra-low-resolution frequency domain amplitude map are respectively generated by 1×1 convolution and Sigmoid activation to generate medium-resolution frequency weight map, low-resolution frequency weight map, and ultra-low-resolution frequency weight map. Depth-separable convolutions are performed on the medium-resolution feature map, low-resolution feature map, and ultra-low-resolution feature map respectively to obtain the medium-resolution semantic feature map, low-resolution semantic feature map, and ultra-low-resolution semantic feature map. The medium-resolution semantic feature map, low-resolution semantic feature map, and ultra-low-resolution semantic feature map are modulated channel by channel with the medium-resolution frequency weight map, low-resolution frequency weight map, and ultra-low-resolution frequency weight map, respectively, to generate the medium-resolution frequency enhancement feature map, the medium-resolution frequency enhancement feature map, and the medium-resolution frequency enhancement feature map. In the cross-scale fusion unit, the medium-resolution frequency-enhanced feature map, the medium-resolution frequency-enhanced feature map and the medium-resolution frequency-enhanced feature map are upsampled to the same spatial size as the high-resolution feature map to obtain three scale-aligned feature maps; The three scale-aligned feature maps are concatenated with the high-resolution feature map along the channel dimension, and a fused semantic feature map is generated by a 3×3 convolutional unit.

5. The image semantic structure decomposition method based on convolutional neural networks according to claim 1, characterized in that, The semantic ridge perception module will generate a semantic ridge response map by fusing semantic feature maps and constructing a structure matrix and filtering gradient extrema, specifically: The maximum channel feature value at each pixel location is extracted along the channel dimension of the fused semantic feature map to construct a dimension-reduced semantic response map. Based on the dimensionality-reduced semantic response map, the horizontal gradient value and vertical gradient value of each pixel position in the horizontal and vertical directions are calculated by using a one-dimensional Gaussian derivative kernel, and the structure matrix of each pixel position is constructed based on the horizontal gradient value and vertical gradient value. The principal and secondary eigenvalues ​​of the structure matrix are calculated using the eigenvalue decomposition method, and the square root of the difference between the principal and secondary eigenvalues ​​is used as the semantic gradient magnitude at each pixel location; the semantic gradient magnitudes at all pixel locations are then combined to form a semantic gradient magnitude map. Non-maximum suppression is performed on the semantic gradient magnitude map to preserve local extrema along the main gradient direction, thus obtaining a ridge response candidate map. Normalize each pixel value in the ridge response candidate map to obtain the semantic ridge response map.

6. The image semantic structure decomposition method based on convolutional neural networks according to claim 1, characterized in that, Step three specifically includes: The number of channels in the fused semantic feature map is compressed to 1 by using a 1×1 convolution to obtain a semantic activation map; Calculate the first-order gradient of each pixel position in the semantic activation map in the horizontal and vertical directions to obtain the horizontal semantic gradient component and the vertical semantic gradient component. The horizontal and vertical semantic gradient components of each pixel are combined to form a semantic gradient direction vector, and the semantic gradient direction vectors of all pixel locations are combined to form a semantic gradient field. The semantic gradient field of the target image is smoothed by using a bilateral filtering method.

7. The image semantic structure decomposition method based on convolutional neural networks according to claim 1, characterized in that, Step four specifically includes: Set the seed threshold, gradient stopping threshold, boundary blocking threshold, and maximum number of tracking steps; Construct a seed point set by: if the pixel response value of the semantic ridge response map is greater than the seed threshold, then the corresponding pixel position is used as a seed point; Using each seed point as the starting pixel position, semantic gradient flow tracing is performed in a smooth semantic gradient field, and the next pixel position is obtained by iterative update. In each iteration update, if the termination condition is met, the semantic gradient flow tracking will stop. The termination condition is as follows: the gradient magnitude of the current pixel position is less than the gradient stopping threshold; the boundary response value of the current pixel is greater than the boundary blocking threshold; the current tracking step number reaches the maximum tracking step number. The gradient magnitude is the square root of the sum of the squares of the horizontal and vertical semantic gradient components at each pixel location, and the boundary response value is the square root of the sum of the squares of the edge gradient components calculated based on the horizontal and vertical Sobel operators. By semantic gradient flow tracing, a set of connected pixel locations is formed around each seed point, and the set of connected pixel locations is used as a semantic candidate region. The semantic candidate regions corresponding to all seed points are combined into a set of semantic candidate regions for the target image.

8. The image semantic structure decomposition method based on convolutional neural networks according to claim 1, characterized in that, Step five specifically includes: Obtain the channel vectors of all pixel positions within the semantic candidate region in the fused semantic feature map, and perform average pooling on all channel vectors to obtain the average feature vector of the region; Based on the boundary contour of each semantic candidate region, the boundary geometric features are calculated, including the region boundary length, region area, principal axis orientation angle of the region's minimum bounding rectangle, and aspect ratio. The main axis direction angle is the angle between the main axis and the horizontal axis, and the aspect ratio is the ratio of the main axis length to the secondary axis length of the smallest bounding rectangle of the region. The average feature vector of each semantic candidate region and the boundary geometric features are combined to form a semantic attribute vector, and the semantic attribute vectors of all semantic candidate regions are combined to form a set of semantic attribute vectors.

9. The image semantic structure decomposition method based on convolutional neural networks according to claim 1, characterized in that, Step six specifically includes: Based on the minimum bounding rectangle of each semantic candidate region, the region boundary coordinate range is extracted, which includes the minimum horizontal coordinate, the maximum horizontal coordinate, the minimum vertical coordinate, and the maximum vertical coordinate. Construct a spatial inclusion relation matrix: If the region boundary coordinate range of semantic candidate region i completely contains the region boundary coordinate range of semantic candidate region j, then it is determined that semantic candidate region i spatially contains semantic candidate region j, and the corresponding element of the spatial inclusion relation matrix is ​​set to 1; otherwise, it is set to 0. Set a similarity threshold and construct a semantic adjacency matrix: if the cosine similarity between the semantic attribute vectors of semantic candidate region i and semantic candidate region j is greater than the similarity threshold, then semantic candidate region i and semantic candidate region j are semantically close adjacent regions, and the corresponding element of the semantic adjacency matrix is ​​set to 1; otherwise, it is set to 0. The average feature vector of each semantic candidate region is obtained and fed into the MLP classifier to generate the semantic category label of each semantic candidate region; Construct a semantic hierarchical structure graph, where the graph nodes are semantic candidate regions, the feature vectors of the graph nodes are semantic attribute vectors, and the semantic labels of the graph nodes are semantic category labels; if the corresponding element of the spatial containment relation matrix is ​​equal to 1, or the corresponding element of the semantic adjacency relation matrix is ​​equal to 1, then establish graph edges between the corresponding graph nodes of the semantic hierarchical structure graph.

Images