Image feature extraction-based topic token generation system and method

Abstract

The application relates to the technical field of data processing, and discloses a subject Token generation system and method based on image feature extraction. The system integrates and divides semantic regions by hierarchically extracting and fusing multi-level visual features, obtaining a fused feature map, and providing basic feature data for subsequent semantic matching and feature quantization. According to domain subject knowledge, a subject semantic structure is built, and a quantization codebook is initialized. A mapping relationship between codebook entries and subject semantics is established, so that the codebook is self-adaptive to different image semantic scenes. In the attention interaction link, the mapping relationship is introduced to impose semantic constraints, and a sparse subject Token set is generated, thereby improving the authenticity and effectiveness of subject representation content. The saliency score of the subject Token is calculated by fusing the visual saliency and attention weight of the semantic region. After global deduplication and sorting, a subject Token sequence is formed, which comprehensively and objectively reflects the core subject information of the image.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a topic token generation system and method based on image feature extraction. Background Technology

[0002] With the continuous iteration of computer vision and image semantic parsing technologies, content theme parsing and keyword extraction based on image feature extraction have become the mainstream application direction. At present, the common practice is to first perform convolutional sampling on the image, extract visual features at different levels, and then complete feature fusion. The image region is cut according to fixed division rules, and then the corresponding theme representation unit is generated by combining a preset vocabulary and clustering coding method.

[0003] At the semantic matching and encoding quantization level, existing solutions mostly use static codebooks to complete feature vector quantization matching, use attention mechanisms to associate image regions with semantic terms, and finally filter and output image topic content based on the matching results. This type of mode can basically realize the transformation from visual image to text topic, and meet the usage needs of simple image content recognition and shallow topic annotation in general scenarios.

[0004] However, most current methods only mechanically segment regions based on pixel texture and contour boundaries. The resulting regions suffer from semantic confusion and low boundary fit, making subsequent feature representations unable to accurately correspond to independent thematic content. At the same time, the quantization codebook uses a static initialization mode, which cannot adapt to the semantic expression differences of different image scenes. The fitting deviation generated during the feature quantization process cannot be fed back to optimize the codebook structure. After long-term use, the semantic coverage and matching accuracy of the codebook gradually decline. Moreover, ignoring the visual performance features of the image itself, it is impossible to take into account both semantic association and visual characteristics, resulting in a blurred hierarchy of the output thematic sequence.

[0005] To address the shortcomings of the existing technologies, the technical problem this application aims to solve is how to extract semantically unified image features to achieve accurate generation of topic tokens. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this application is to provide a topic token generation system and method based on image feature extraction, so as to adapt to topic parsing work for images in multiple target domains.

[0007] To achieve the above objectives, this application adopts the following technical solution:

[0008] In the first aspect, this application provides a topic token generation method based on image feature extraction, which extracts multi-level visual features of the input image, integrates them to obtain a fused feature map, and divides the fused feature map into regional features of multiple semantic regions;

[0009] Acquire topic knowledge in the target domain, construct a topic semantic structure, initialize a quantized codebook based on the topic semantic structure, and establish a mapping relationship between codebook entries in the quantized codebook and topic semantics;

[0010] Multiple sets of preset topic query vectors are interacted with codebook entries through attention. Semantic constraints are imposed on the interaction process based on the mapping relationship. A sparse topic token set is generated for each semantic region, and the quantization residual of each region feature is calculated.

[0011] The codebook entries are adjusted based on the quantization residuals, and the codebook entries are structurally adjusted based on the distribution characteristics of the quantization residuals.

[0012] The visual saliency of each semantic region is calculated based on the fused feature map. The attention weight of each topic token is combined to calculate the saliency score of each topic token. The topic tokens of all semantic regions are then globally deduplicated and sorted to form a topic token sequence.

[0013] Furthermore, the regional features for dividing the semantic region include:

[0014] The input image is analyzed to obtain multi-level visual features. Fusion weights are assigned according to the spatial correlation between the visual features, and the fusion feature map is generated by integrating them.

[0015] The fused feature map is split into shallow contour components, mid-layer texture components, and deep semantic components. Semantic anchors are determined based on the deep semantic components. The coverage of the corresponding mid-layer texture components is matched based on the semantic anchors to preliminarily determine the semantic interval range.

[0016] Within the semantic interval, the gradient difference between components at adjacent coordinate points is calculated, and the segmentation boundary is checked in conjunction with the morphological trend of the shallow contour components to determine the region contour.

[0017] By combining the distribution of semantic anchor points with the region outline, semantic regions are divided, and the regional features of the semantic regions are obtained.

[0018] Furthermore, establishing the mapping relationship includes:

[0019] Extract entities and relationships from the subject knowledge of the target domain, and construct a subject semantic structure based on the relationships between entities, with entities as nodes.

[0020] The nodes within the topic semantic structure are converted into semantic vectors of a unified dimension. Based on the spatial distribution characteristics of the semantic vectors, codebook entries are generated by clustering, thus completing the initialization operation of the quantized codebook.

[0021] Calculate the correlation between codebook entries and semantic vectors, complete the binding according to the correlation, and establish the mapping relationship between codebook entries and topic semantics.

[0022] Furthermore, a topic semantic structure is constructed through a weighted topology networking method, using the semantic distance between any nodes as the connection weight, and integrating relationships including subordinate, parallel, and derived relationships;

[0023] The spatial distribution characteristics of semantic vectors are analyzed by density peak clustering algorithm, and the cluster center vectors obtained by the solution are set as codebook entries in sequence.

[0024] Set an association threshold and compare the association degree with the association threshold to distinguish and bind the semantic association attributes of strong and weak semantic nodes.

[0025] Furthermore, the calculation of the quantization residual for each region feature includes:

[0026] Multiple sets of preset topic query vectors are respectively subjected to attention interaction operations with codebook entries to obtain the attention weights between topic query vectors and codebook entries;

[0027] Based on the mapping relationship, semantic constraints are corrected on the attention weights, and item combinations are selected from the corrected codebook items according to the filtering conditions to generate a sparse topic token set corresponding to the semantic region.

[0028] Using the corrected attention weights as weighting coefficients, the codebook entries corresponding to the topic tokens are merged to reconstruct the regional features. The quantized residuals of each regional feature are obtained by comparing them with the original regional features.

[0029] Furthermore, attention weights between topic query vectors and codebook entries are calculated using cosine similarity; semantic constraint corrections of attention weights are completed based on the semantic association attributes bound to the codebook entries and the mapping relationship.

[0030] The semantic vectors of the codebook entries corresponding to the topic token are fused using weighted coefficients to obtain the reconstructed regional features. The deviation between the feature differences and the semantic constraints is combined and superimposed to calculate the quantized residuals of the corresponding regional features.

[0031] Furthermore, the structural adjustment of the codebook entries includes:

[0032] Summarize the quantization residuals corresponding to all semantic regions, statistically analyze the residual distribution intervals and discrete clustering patterns, combine the preset judgment criteria to determine the degree of residual deviation, and determine whether there is an adjustment requirement for the current codebook entry;

[0033] The codebook entries to be adjusted are selected based on the degree of residual deviation, and the deviation causes are classified in combination with the semantic association attributes bound to the codebook entries.

[0034] Based on the category of deviation cause, match the corresponding adjustment method to perform structural adjustment, and replace the original codebook entries with the structurally adjusted codebook entries.

[0035] Furthermore, the formation of the theme token sequence includes:

[0036] Based on the pixel distribution state of the fused feature map and the boundary range of the semantic region, the texture density and gray-level change amplitude of each semantic region are statistically analyzed, and the visual saliency of the corresponding semantic region is obtained by coupling operation.

[0037] The salience score of each topic token is calculated by weighting the attention weight of each topic token with the visual salience of its semantic region.

[0038] Redundancy detection and global deduplication are performed on topic tokens in all semantic regions, and the deduplicated topic tokens are uniformly sorted according to a set order to form a topic token sequence.

[0039] Furthermore, all topic tokens within all semantic regions are traversed, and the semantic similarity and spatial affiliation of topic tokens are compared to identify and eliminate redundant topic tokens with duplicate representations. The remaining topic tokens are then sorted in descending order based on their salience scores.

[0040] Secondly, this application provides a topic token generation system based on image feature extraction, including a partitioning module, a codebook module, a quantization module, an adjustment module, and a generation module;

[0041] The segmentation module is used to extract multi-level visual features from the input image, fuse them to obtain a fused feature map, and divide the fused feature map into regional features of multiple semantic regions;

[0042] The codebook module is used to acquire topic knowledge of the target domain, construct a topic semantic structure, initialize a quantized codebook based on the topic semantic structure, and establish a mapping relationship between codebook entries and topic semantics in the quantized codebook.

[0043] The quantization module is used to perform attention interaction between multiple sets of preset topic query vectors and codebook entries, apply semantic constraints to the interaction process based on the mapping relationship, generate a sparse topic token set for each semantic region, and calculate the quantization residual of each region feature.

[0044] The adjustment module is used to determine the adjustment requirements of codebook entries based on the quantization residuals, and to perform structural adjustments on codebook entries based on the distribution characteristics of the quantization residuals.

[0045] The generation module is used to calculate the visual saliency of each semantic region based on the fused feature map, integrate the attention weight of each topic token, calculate the saliency score of each topic token, and perform global deduplication and sorting of topic tokens in all semantic regions to form a topic token sequence.

[0046] Compared with the prior art, the beneficial effects achieved by this application are as follows:

[0047] Multi-level visual features are extracted and fused in layers to obtain a fused feature map and semantic regions are divided, providing basic feature data for subsequent semantic matching and feature quantization. Based on domain topic knowledge, a topic semantic structure is built and the quantization codebook is initialized. The mapping relationship between codebook entries and topic semantics is established, allowing the codebook to adapt to different image semantic scenarios and continuously optimize the feature quantization accuracy and domain semantic coverage.

[0048] By introducing a mapping relationship to impose semantic constraints in the attention interaction stage, a sparse topic token set is generated, which effectively reduces semantic misalignment and content redundancy, and improves the authenticity and effectiveness of topic representation. The visual saliency of semantic regions and attention weights are combined to calculate the saliency score of topic tokens. After global deduplication and sorting, a topic token sequence is formed, which takes into account both image visual features and semantic relationships, and comprehensively and objectively reflects the core topic information of the image. Attached Figure Description

[0049] Figure 1 Here is a flowchart of a topic token generation method based on image feature extraction;

[0050] Figure 2 A flowchart illustrating the logic for forming a sequence of themed tokens;

[0051] Figure 3 This is a schematic diagram of a topic token generation system based on image feature extraction. Detailed Implementation

[0052] The technical solution of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of this application, rather than limitations thereof.

[0053] Example 1

[0054] like Figure 1 As shown, this embodiment provides a topic token generation method based on image feature extraction, including: S1, extracting multi-level visual features of the input image, integrating them to obtain a fused feature map, and dividing the fused feature map into regional features of multiple semantic regions.

[0055] Specifically, the regional features for dividing semantic regions include:

[0056] The input image is analyzed to obtain multi-level visual features. Fusion weights are assigned according to the spatial correlation between the visual features, and the fusion feature map is generated by integrating them.

[0057] The fused feature map is split into shallow contour components, mid-layer texture components, and deep semantic components. Semantic anchors are determined based on the deep semantic components. The coverage of the corresponding mid-layer texture components is matched based on the semantic anchors to preliminarily determine the semantic interval range.

[0058] Within the semantic interval, the gradient difference between components at adjacent coordinate points is calculated, and the segmentation boundary is checked in conjunction with the morphological trend of the shallow contour components to determine the region contour.

[0059] By combining the distribution of semantic anchor points with the region outline, semantic regions are divided, and the regional features of the semantic regions are obtained.

[0060] Single-level features can only capture local dimensional information and cannot simultaneously take into account the three core image information types: contour shape, texture details, and semantic connotation. Direct segmentation of regions is prone to boundary offset and semantic classification errors. By extracting differentiated features layer by layer and then completing weighted fusion through quantitative correlation, a unified feature carrier with complete information is constructed, providing compliant basic data for subsequent feature component splitting. Feature parsing is based on a multi-layer convolutional network structure to mine information at different abstract levels. Multi-level visual features are fixedly divided into three types of matrix data: shallow contour features, mid-level texture features, and deep semantic features. The degree of spatial correlation is quantified by the similarity of visual features at the same coordinate position. The fusion weight is determined by the information contribution ratio based on the similarity interval. The fused feature map is a feature matrix of the same specification obtained after weighted operation of multiple types of features.

[0061] In actual execution, the image to be processed with a resolution of 512×512 is input into the convolutional neural network, and convolution and downsampling operations are performed sequentially. Those skilled in the art can adjust the parameters of the number of convolutional layers, the size of the convolutional kernel, and the downsampling ratio in the convolutional neural network accordingly during the actual process, and output contour feature matrix, texture feature matrix, and semantic feature matrix with a size of 512×512 respectively. All coordinate points in the matrix are traversed row by row and column by column, and the matching value of the three types of visual feature data at the same coordinate position is calculated using Euclidean similarity. The calculation result is normalized to the interval of 0~1 and then used as the degree of spatial correlation.

[0062] Based on the empirical distribution features of multi-level visual features, two judgment thresholds of 0.85 and 0.4 are set. Those skilled in the art can adjust them accordingly in practice. Specifically, 0.85 corresponds to the upper bound of the high confidence interval of spatial correlation, indicating that the three types of features at this coordinate position are highly consistent, and the corresponding level contributes to the fusion result. A fusion weight of 0.6 is assigned to strengthen the features of this layer. 0.4 corresponds to the lower bound of the low confidence interval of spatial correlation, indicating that the three types of features at this coordinate position are significantly different, and the corresponding level only provides auxiliary reference information. A fusion weight of 0.1 is assigned to suppress noise interference. In the medium correlation region between the two, a fusion weight of 0.3 is assigned to maintain balanced participation. The normalization constraint of the sum of the three weights makes the fusion values ​​of each coordinate point comparable. According to the single-pixel weighted calculation, each coordinate position is substituted into the corresponding fusion weight to complete the summation operation, and the fusion feature value of each pixel point is calculated in turn. After all points are calculated, they are combined to form a fusion feature matrix, i.e., a fusion feature map.

[0063] This addresses the shortcomings of single-layer feature information, aggregates multi-dimensional visual information to form a globally unified feature representation, and fuses the original image information corresponding to the feature map, ensuring the consistency of data traceability in subsequent component decomposition.

[0064] The fused feature map stores multiple types of attribute data, making it impossible to retrieve single-dimensional information to locate the core region. It splits into independent components to achieve information classification and retrieval. Semantic data is used to determine the region's attribution benchmark, and texture data is used to define the actual space occupied by the entity, compressing the scope of subsequent boundary calculations and reducing the interference of invalid data on the judgment results. Component splitting classifies and separates the fused feature map according to the data representation attributes. The shallow contour component stores the values ​​related to the shape boundary, the mid-layer texture component stores the values ​​of surface detail distribution, and the deep semantic component stores the confidence values ​​of topic attribution. The semantic anchor point is defined as the spatial coordinate point corresponding to the maximum built-in confidence value of the deep semantic component.

[0065] Specifically, the feature map is traversed and fused according to attribute classification. Based on the numerical representation meaning, three groups of non-overlapping feature components with consistent specifications are decomposed. All coordinate points of the deep semantic components are traversed, and the semantic confidence value corresponding to each point is read one by one. The semantic confidence value comparison and retrieval are performed sequentially to select the global maximum confidence value. The two-dimensional coordinates corresponding to the confidence value are determined as the semantic anchor point of the current semantic region. With the two-dimensional coordinates of the semantic anchor point as the center, the values ​​of the middle-layer texture components in the four neighboring areas above, below, left, and right are retrieved sequentially. The texture value deviation between adjacent coordinate points is calculated. When the texture value deviation between coordinate points is less than a preset threshold, it is determined to be a continuous homogeneous texture region. A preset threshold is set based on 5% of the range of texture component values. In this embodiment, the preset threshold is set to 0.1. The retrieval is continuously expanded outward until the deviation exceeds the preset threshold. The closed space enclosed by the retrieval termination position is defined as the initial semantic interval range. Those skilled in the art can adjust the value of the preset threshold accordingly in the actual process.

[0066] This enables the separation and control of multi-attribute features, accurately locates the semantic core coordinates, determines the effective computational space range, and defines the execution range of subsequent gradient calculation and boundary correction based on the completed semantic interval, thus defining the basic spatial boundary for the shaping of the region outline.

[0067] Relying solely on the rules of feature mutation to define boundaries can easily cause the formed lines to deviate from the actual shape and structure of the object. Combining the inherent extension trajectory of the shallow contour to correct the boundary coordinates improves the fit between the boundary and the object shape, ensuring that the region division has objective rationality. Adjacent coordinate points refer to sampling pixels that are spatially adjacent. The gradient difference represents the change range of features of adjacent points. Mutation points serve as candidate positions for segmentation boundaries. The shape trend is constrained by the continuous coordinate change trend of the shallow contour components to guide the boundary direction.

[0068] Specifically, sampling points are uniformly selected at 8-pixel intervals within the semantic interval, and paired to form adjacent sampling combinations. The corresponding values ​​of the three sets of feature components at the sampling points are extracted. The gradient difference of each set of sampling points is calculated using a difference algorithm, and all candidate boundary points with gradient differences greater than the difference threshold are recorded. All candidate boundary points are connected sequentially according to the spatial arrangement of the sampling points to generate a polygonal initial segmentation edge. The coordinate change slope of the shallow contour components is extracted segment by segment, and the slope deviation of the initial segmentation edge is compared. When the slope deviation of two segments is greater than the deviation threshold, the inflection point coordinates of the segmentation edge are fine-tuned. This process is repeated until the slope change trend of the segmentation edge matches the natural contour trend, ultimately forming a closed and neat regional contour. In this embodiment, the deviation threshold is set to 0.15, and the sampling points are uniformly selected at 8-pixel intervals within the semantic interval. The difference threshold is set to 0.2. Those skilled in the art can adjust the difference threshold and deviation threshold accordingly in actual practice.

[0069] This effectively reduces the probability of boundary segmentation distortion. The corrected and finalized region outline, combined with the previously locked semantic anchor points, forms a dual basis for judgment, supporting the subsequent segmentation of independent semantic regions.

[0070] Spatial contours alone can only define the physical range and cannot guarantee the uniformity of semantic attributes within the region. Combining semantic anchor points with collaborative segmentation maintains the theme consistency of sub-regions, collects and integrates feature data within the region, generates standardized feature units, and adapts to the computational requirements of subsequent semantic matching and codebook quantization. A semantic region is an image sub-region with closed boundaries and uniform internal semantic attributes. The regional feature is a representation matrix formed after regularizing and operating all effective feature data within the sub-region.

[0071] Specifically, the closed region contour is used as the cutting boundary. Spatial segmentation is performed on the fused feature map to obtain multiple initial sub-regions in batches. The semantic category labels of the built-in semantic anchors of each sub-region are retrieved. The semantic confidence of all points in the sub-region is traversed, and the proportion of semantic categories in the region is calculated. Based on the consistency requirement within the semantic region (i.e., the main semantic category of the region should dominate), a consistency threshold of 70% is preset. Regions with a category proportion of less than 70% are judged as semantically scattered blocks and are removed. Valid sub-regions with unified semantic attributes are retained. All features within the valid sub-regions are extracted. Irrelevant pixel data outside the contour boundary is removed. The row and column arrangement specifications of all data are unified. The mean value of the features in the region is calculated and used as a benchmark to correct the discrete values. After normalization, the region features corresponding to each semantic region are generated.

[0072] This completes the region decomposition of the entire image, forming computationally computable feature units with standardized boundaries and unified semantics. Multiple sets of region feature matrices directly serve as input data for subsequent topic knowledge matching, semantic structure construction, and codebook mapping relationship construction, supporting subsequent full-process semantic interaction and feature quantization operations.

[0073] S2. Obtain topic knowledge of the target domain, construct a topic semantic structure, initialize the quantization codebook based on the topic semantic structure, and establish the mapping relationship between codebook entries and topic semantics in the quantization codebook.

[0074] Specifically, establishing the mapping relationship includes:

[0075] Extract entities and relationships from the subject knowledge of the target domain, and construct a subject semantic structure based on the relationships between entities, with entities as nodes.

[0076] The nodes within the topic semantic structure are converted into semantic vectors of a unified dimension. Based on the spatial distribution characteristics of the semantic vectors, codebook entries are generated by clustering, thus completing the initialization operation of the quantized codebook.

[0077] Calculate the correlation between codebook entries and semantic vectors, complete the binding according to the correlation, and establish the mapping relationship between codebook entries and topic semantics.

[0078] Among them, the topic semantic structure is constructed by weighted topology networking, with the semantic distance between any nodes as the connection weight, and the association relationship including subordinate, parallel and derived relationships is integrated;

[0079] The spatial distribution characteristics of semantic vectors are analyzed by density peak clustering algorithm, and the cluster center vectors obtained by the solution are set as codebook entries in sequence.

[0080] Set an association threshold and compare the association degree with the association threshold to distinguish and bind the semantic association attributes of strong and weak semantic nodes.

[0081] The original domain subject knowledge exists in the form of scattered texts, without a standardized logical framework. The entity concepts are independent and scattered, and cannot be directly converted into a calculable data form, nor can the logical connection rules within the knowledge be intuitively reflected. First, the core entities and their relationships are extracted, and then a topological network structure is built to integrate the fragmented knowledge into an ordered system, clarify the knowledge hierarchy and interaction rules, and provide a standardized processing carrier for subsequent semantic vector encoding.

[0082] The subject knowledge entity refers to the conceptual subject extracted from domain-specific materials and labeled texts; the relationship is divided into three basic logical types: subordinate, parallel, and derivative. Subordinate represents a hierarchical relationship, parallel represents a coexistence relationship at the same level, and derivative represents a content expansion and derivation relationship; semantic distance is used to quantify the degree of difference in the semantic content of two entities; weighted topology networking is a network construction mode in which entities are used as nodes and semantic distance is used as the weight of the node connection.

[0083] Specifically, the original subject knowledge obtained from the domain is cleaned by removing meaningless and redundant statements and ambiguous expressions. Independent concepts within the text are identified based on entity extraction algorithms, and these are aggregated into a discrete knowledge entity set. Simultaneously, the inherent logic between entities is analyzed, and the pairwise relationships between all entities are categorized into three types of association relationships. Those skilled in the art can adjust the parameters of the entity extraction algorithm accordingly during the actual process. For any two different entity nodes within the entity set, the semantic difference between them is calculated using cosine distance based on the semantic vectors corresponding to the entities, i.e., semantic distance = 1 - cos(θ) = 1 - (v... i ·v j ) / (‖v i ‖×‖v j ‖), v i With v j The semantic vectors corresponding to the two entity nodes are used as the connection weights of the topological edges between the two nodes.

[0084] The node connectivity is determined based on the type of association. Hierarchical connectivity links are built for subordinate nodes, horizontal connectivity links are built for parallel nodes, and extended connectivity links are built for derived nodes. All nodes are interconnected and networked by combining the corresponding connection weights, and finally a weighted topological semantic structure with hierarchical structure and weight attributes is generated.

[0085] For example, a single processing step extracts 18 valid knowledge entities, and the semantic distance between entities is calculated in the range of 2 to 27. The entire network is then constructed according to the association type and weight, forming a semantic topology architecture.

[0086] This integrates scattered knowledge into a structured system with logical constraints, unifies the knowledge organization form, fully preserves the various inherent relationship characteristics between entities, and directly converts the completed topological semantic nodes into semantic vectors in batches, ensuring that the converted vector data can inherit the logical relationship characteristics of the original knowledge.

[0087] Knowledge entities in topological networks are textual concepts and cannot participate in subsequent numerical operations such as attention interaction and regional feature quantification. They must be converted into vector data to meet the requirements of algorithm operations. All nodes are uniformly converted into semantic vectors of the same dimension to achieve standardized digital representation of knowledge. Density peak clustering is used to select representative vectors of spatial distribution as codebook entries to condense the core semantic features of the domain, avoid the semantic bias problem caused by random initialization, and make the initial codebook fit the theme attributes of the target domain.

[0088] The unified-dimensional semantic vector refers to the numerical representation vector with completely consistent dimensional specifications after encoding all entity nodes; the density peak clustering algorithm determines the cluster center vector based on two indicators: local vector density and relative distance between vectors, which is an unsupervised feature clustering method; the codebook entry is the basic vector unit that makes up the quantized codebook; specifically, all entity nodes inside the weighted topological topic semantic structure are traversed, and encoding operations are performed on each node. The vector dimension is uniformly set to 128 dimensions, and all nodes output semantic vectors with consistent specifications, which are then summarized to form the semantic vector set of the nodes.

[0089] The density peak clustering algorithm is initiated to traverse all semantic vector samples, and the local density parameter ρ of each individual semantic vector is calculated one by one. i =Σχ(d ij -d c ), where d ij Let d be the Euclidean distance between vectors i and j. c For the cutoff distance, χ(d) ij -d c ) is the truncation function, in (d ij -d c If ) < 0, take 1; otherwise, take 0. Cutoff distance dc After sorting all vectors by pairwise distances, the upper bound of the top 2% of distance values ​​is selected, i.e., d c Within a neighborhood of radius d, the distance to vector i is less than d c The number of adjacent vectors, ρ i A larger value indicates a higher local density around the vector; and the relative distance parameter δ between any two semantic vectors... i =min{d ij |ρ j >ρ i The high-density vector takes the global maximum distance δ and filters out the vectors that have cluster representativeness in the spatial distribution. The filtered vectors are the cluster center vectors. According to the order of cluster generation, all cluster center vectors are stored into the quantization codebook in sequence, and combined to form a complete entry cluster, thus completing the quantization codebook initialization work.

[0090] The calculation rule for the relative distance parameter is as follows: For vector i, from all vectors with a local density higher than it (i.e., satisfying ρ... j >ρ i Among vectors j, find the one closest to i; for the vector with the highest local density globally (where ρ does not exist), find the one with the highest density globally. j >ρ i (a vector), whose δ i Take the global maximum distance among all pairwise distances of vectors, i.e., δ i =max{d ij};δ i It is the shortest distance from vector i to a higher density region, δ i The larger the value, the farther the vector is from denser regions, and the more likely it is to be an independent cluster center.

[0091] The criterion for selecting the cluster center vector is: simultaneously satisfying ρ i With δ i Vectors with larger average values ​​are identified as cluster center vectors, meaning these vectors have high local density (ρ). i Large) and far from other higher density areas (δ) i The large value indicates that it has cluster representativeness in spatial distribution; in specific screening, a decision map is drawn with ρ as the horizontal axis and δ as the vertical axis, and the discrete point located in the upper right corner of the decision map is selected as the cluster center.

[0092] Assuming there are 6 semantic vector samples, the calculated cutoff distance d c=5, the ρ and δ of each vector are as follows: Vector V1 has a local density ρ=5 and a relative distance δ=8.2; Vector V2 has a local density ρ=4 and a relative distance δ=3.1; Vector V3 has a local density ρ=3 and a relative distance δ=4.5; Vector V4 has a local density ρ=2 and a relative distance δ=2.8; Vector V5 has a local density ρ=2 and a relative distance δ=7.6; Vector V6 has a local density ρ=1 and a relative distance δ=6.3. Vector V1 has the highest local density and the global maximum relative distance. The vector with the highest local density closest to Vector V2 is V1, and d(V2,V1)=3.1. The nearest higher local density vector to vector V3 is V1, d(V3,V1)=4.5; the nearest higher local density vector to vector V4 is V3, d(V4,V3)=2.8; the nearest higher local density vector to vector V5 is V3, d(V5,V3)=7.6; and the nearest higher local density vector to vector V6 is V4, d(V6,V4)=6.3. Among them, V1 and V5 are located in the upper right corner of the decision graph and are selected as cluster center vectors. V1 represents the center of the first semantic cluster, and V5 represents the center of the second independent cluster because it is far away from all higher density regions.

[0093] This enables the standardized transformation of textual knowledge into numerical vectors. By using a clustering mechanism to filter and retain core semantic features of the domain, the overall attributes of the initial codebook match the target domain's topic category. It also performs correlation calculations with the semantic vectors of topological nodes, providing basic computational data for subsequent semantic mapping and binding processes.

[0094] The initialized codebook entries only have independent vector features and do not have a fixed correspondence with semantic nodes in the domain, so they cannot be subject to effective semantic constraints in the subsequent attention interaction process. The degree of fit between the two is calculated by quantification, and the strength of the association is divided to clarify the semantic scope of a single codebook entry, build a searchable and callable correspondence system, and meet the usage conditions of subsequent semantic rule constraints.

[0095] The correlation degree is used to quantify the degree of content matching between the codebook entry vector and the node semantic vector; the correlation threshold is the critical judgment value for classifying the strength of the correlation, and those skilled in the art can adjust the correlation threshold accordingly in the actual process; a strongly correlated semantic node refers to a core corresponding node with a high degree of matching with the codebook entry, and a weakly correlated semantic node refers to an auxiliary correlated node with a relatively low degree of matching; the mapping relationship refers to the queryable and callable binding correspondence system between codebook entries and domain topic semantics.

[0096] Specifically, each initialized codebook entry vector is retrieved and its correlation is calculated with the semantic vectors of all nodes within the topology. The correlation is calculated using cosine similarity, i.e., correlation = (e·v) / (‖e‖×‖v‖), where e is the codebook entry vector and v is the node semantic vector. The correlation of each pairing is recorded. Two levels of correlation thresholds are preset, defining the correlation level by numerical range. If the correlation is greater than the higher threshold, it is determined to be a strongly correlated semantic node; if the correlation is in the middle range of the two thresholds, it is determined to be a weakly correlated semantic node. All correlated nodes corresponding to a single codebook entry are sorted from high to low according to their correlation values. The entry is prioritized to establish a core binding relationship with strongly correlated semantic nodes, and the corresponding weakly correlated semantic nodes are simultaneously attached as extended correlation objects.

[0097] After traversing all codebook entries and completing hierarchical binding, a clear hierarchy and multi-node linkage correspondence system is formed, constructing a stable and effective semantic mapping relationship between codebook entries and topics. Based on the semantic density distribution characteristics of knowledge entities in the target domain, a preset association threshold is set. Specifically, the numerical distribution of the association degree between all codebook entries and node semantic vectors is statistically analyzed. The natural breakpoint of the distribution curve is used as the threshold division basis. The high association threshold of 0.72 corresponds to the vicinity of the upper quartile of the association degree distribution, ensuring that strongly associated nodes cover about 25% of high-fit matching pairs. The low association threshold of 0.35 corresponds to the vicinity of the lower quartile of the association degree distribution, excluding obviously irrelevant matching pairs. Based on the numerical comparison results, two types of association attributes are divided to complete the multi-semantic node matching and binding.

[0098] This allows for the definition of the semantic boundaries of codebook entries using quantitative standards, the division of primary and secondary association levels, and the construction of a semantic mapping system with hierarchical attributes. This system serves as the core criterion for semantic constraint correction in the attention interaction process, enabling the verification and correction of attention weights and ensuring that the interaction process conforms to the domain's semantic logic rules.

[0099] S3. Perform attention interaction between multiple sets of preset topic query vectors and codebook entries, apply semantic constraints to the interaction process based on the mapping relationship, generate a sparse topic token set for each semantic region, and calculate the quantization residual of each region feature.

[0100] Specifically, calculating the quantization residuals for each region feature includes:

[0101] Multiple sets of preset topic query vectors are respectively subjected to attention interaction operations with codebook entries to obtain the attention weights between topic query vectors and codebook entries;

[0102] Based on the mapping relationship, semantic constraints are corrected on the attention weights, and item combinations are selected from the corrected codebook items according to the filtering conditions to generate a sparse topic token set corresponding to the semantic region.

[0103] Using the corrected attention weights as weighting coefficients, the codebook entries corresponding to the topic tokens are merged to reconstruct the regional features. The quantized residuals of each regional feature are obtained by comparing them with the original regional features.

[0104] Specifically, the attention weight between the topic query vector and the codebook entry is calculated using cosine similarity; the semantic constraint correction of the attention weight is completed based on the semantic association attributes bound to the codebook entry and the mapping relationship.

[0105] The semantic vectors of the codebook entries corresponding to the topic token are fused using weighted coefficients to obtain the reconstructed regional features. The deviation between the feature differences and the semantic constraints is combined and superimposed to calculate the quantized residuals of the corresponding regional features.

[0106] Relying solely on topic concepts and semantic entries is insufficient to intuitively determine the degree of matching between them; they must be transformed into numerical correlation indicators to support subsequent logical verification and filtering operations. Through vector-level similarity interaction operations, the degree of fit between the query topic and the semantic content carried by the codebook is quantified, generating attention weights to provide the original computational basis for subsequent semantic constraint correction. Here, the preset topic query vector refers to a fixed-dimensional numerical vector predefined according to the topic type of the target domain, the codebook entries are semantic vector units initialized by the previous semantic clustering, the attention weights are used to quantify the degree of matching between the semantic content of the query vector and the codebook entries, and the cosine similarity is used to measure the content overlap between the two sets of vectors.

[0107] Let the topic query vector be x, and the single barcode entry vector be y, with a uniform vector dimension of d. The corresponding attention weight calculation formula is: w=(x·y) / ‖x‖‖y‖, where x·y represents the vector dot product operation, and ‖x‖ and ‖y‖ are the magnitudes of the two sets of vectors, respectively. Specifically, the topic query vector and the barcode entry vector are uniformly set to 128 dimensions. All sets of topic query vectors to be processed and the set of initialized barcode entries are retrieved, and the two sets of vectors are paired up. For each pair of paired vectors, numerical calculation is performed based on cosine similarity, and a single pairing operation generates an attention weight. After traversing all pairings and completing the calculation, the attention weights are summarized and formed.

[0108] This allows for the establishment of a correspondence between query topics and codebook semantics under a unified numerical scale, objectively reflecting the matching differences between different entries, and serving as the processing object for the next stage of semantic constraint correction operations.

[0109] Weights obtained solely through vector similarity only reflect numerical matching results and are not constrained by the inherent semantic logic of the domain. This can easily lead to invalid matches across semantic levels and those that violate entity association rules. Such abnormal weights can interfere with the accuracy of subsequent topic extraction. By combining the previously constructed mapping system with the strong and weak association attributes to verify the weights, abnormal data that does not conform to objective semantic rules can be eliminated, ensuring that the weight results align with the domain knowledge logic. Semantic association attributes include two types of binding attributes: strongly associated semantic nodes and weakly associated semantic nodes. The mapping relationship refers to the corresponding binding system between codebook entries and domain semantic nodes. Semantic constraints are corrected by referring to semantic logic rules, and the process of filtering compliant weights and eliminating abnormal and invalid weights is carried out.

[0110] Define the criteria for semantic logic compliance. Let S be the set of semantic nodes bound to a codebook entry, and p be the semantic node to which the query vector belongs. If node p has no subordinate, parallel, or derived relationship with any node in set S, the current weight is determined to be an abnormal weight and is directly removed. If a node has a legitimate relationship, the corresponding attention weight is retained. Specifically, retrieve the initial attention weight and the corresponding query vector and codebook entry for each weight. Based on the binding information stored in the codebook entry, locate the semantic node to which it belongs and the association level. Compare the three basic association relationships (subordinate, parallel, and derived) within the topological semantic structure to determine whether the current query topic and the semantics to which the codebook belongs have legitimate connectivity logic. Weight data that exceeds the scope of semantic association or violates the hierarchical affiliation rules are directly removed, and valid attention weights that meet the requirements of semantic binding relationship and association attribute are retained, thus completing the correction process for the entire set of attention weights.

[0111] This filters out illogical matching results, ensures that the weight distribution aligns with the real semantic architecture of the domain, and the remaining effective attention weights after correction will serve as the core criterion for filtering sparse topic token entries.

[0112] The revised valid codebook entries are too large in number and contain a lot of redundant content with overlapping meanings. The set of entries cannot concisely express the core theme of a single semantic region. To address this, a unified selection criterion for the number of entries is set to remove low-contribution entries and form sparse theme expression units that meet the representation requirements of regional theme features. The selection criteria are pre-set weight judgment standards, and the sparse theme token set is a combination of codebook entries that independently represent the core theme of a single semantic region after removing redundant entries.

[0113] Specifically, all codebook entries carrying effective attention weights are retrieved. Based on the empirical ratio of sparsity in topic token representation, a weight screening threshold of 0.4 is preset. The effective attention weight corresponding to each entry is compared with the threshold value. Codebook entries with attention weights greater than the threshold value are retained, while low-association entries with weight values ​​that do not meet the standard are discarded. The remaining entries after screening are uniformly grouped and classified according to the semantic region division range, and sparse topic token sets corresponding to each semantic region are generated.

[0114] This effectively reduces the data volume of semantic representation, removes redundant and interfering information, and forms a representative combination of topic items. The codebook semantic vectors in the sparse topic token set are used for subsequent regional feature reconstruction operations.

[0115] The original region features are derived from the image visual dimension extraction results, while the reconstructed features are the representation results generated by fitting the domain semantic codebook. The deviation between the two types of data can intuitively reflect the feature quantization fitting error. The semantic level matching deviation is superimposed to comprehensively evaluate the quantization defects from both visual representation and semantic adaptation dimensions. The resulting residual data can objectively reflect the fitting quality of the current codebook system. Among them, the weighting coefficient refers to the effective attention weight after correction. The reconstructed region features are the fitted feature data obtained after weighted fusion of multiple sets of codebook semantic vectors. The feature difference represents the numerical deviation between the original visual region features and the reconstructed fitted features. The degree of semantic constraint deviation represents the deviation between the actual item matching relationship and the standard semantic binding rule. The quantization residual is a comprehensive error index obtained after integrating the two-dimensional deviations.

[0116] Let y be the codebook vector corresponding to the sparse tokens within the region. i The corresponding effective attention weight is w i Reconstructing the regional feature formula F r =∑w i y i The original region features are denoted as F, and the visual feature difference is E. v =‖FF r The semantic constraint deviation is E. s The deviation between the actual binding relationship and the standard mapping relationship is represented by the numerical value, and the combined quantitative residuals are superimposed to obtain E=αE. v +βE s α and β are fixed balance coefficients. Let α = 0.7 and β = 0.3, that is, the weight of visual feature difference is higher than the weight of semantic constraint deviation, to ensure that codebook adjustment responds first to visual quantization deviation.

[0117] Specifically, a sparse topic token set corresponding to a single semantic region is selected, and the semantic vectors of all codebook entries within the set are extracted. Using the effective attention weights corresponding to the entries as weighting coefficients, a linear weighted fusion operation is performed to obtain the reconstructed region features corresponding to that region. The reconstructed region features are then compared with the original region features obtained in the previous segmentation, and the feature difference between the two sets of features is calculated. Simultaneously, the difference between the current entry matching combination and the standard semantic mapping relationship is checked, and the degree of semantic constraint deviation is calculated. According to the above superposition operation rules, the feature difference and the degree of semantic deviation are integrated and calculated, and finally the quantized residual corresponding to the single semantic region feature is obtained. All residual calculations are completed by traversing each region.

[0118] This comprehensively reflects the visual and semantic matching errors that exist in the feature quantization process, forming an objective and complete error evaluation index. The quantization residuals calculated in each region will serve as the basis for subsequent judgment of codebook adjustment needs and codebook entry structure adjustments.

[0119] S4. Determine the adjustment requirements of codebook entries based on the quantization residuals, and perform structural adjustments on the codebook entries based on the distribution characteristics of the quantization residuals. Use the structurally adjusted codebook entries for subsequent quantization.

[0120] Specifically, structural adjustments to codebook entries include:

[0121] Summarize the quantization residuals corresponding to all semantic regions, statistically analyze the residual distribution intervals and discrete clustering patterns, combine the preset judgment criteria to determine the degree of residual deviation, and determine whether there is an adjustment requirement for the current codebook entry;

[0122] The codebook entries to be adjusted are selected based on the degree of residual deviation, and the deviation causes are classified in combination with the semantic association attributes bound to the codebook entries.

[0123] Based on the category of deviation cause, match the corresponding adjustment method to perform structural adjustment, and replace the original codebook entries with the structurally adjusted codebook entries for subsequent codebook quantization.

[0124] Quantization residuals in a single region can only reflect local quantization errors and cannot objectively assess the overall adaptability of the entire codebook. Relying on local abnormal data can easily lead to incorrect adjustment decisions. By integrating and statistically analyzing all residual data, the overall deviation characteristics are summarized from two dimensions: numerical coverage and data distribution pattern. Then, the deviation level is measured with reference to a unified judgment standard to objectively identify the overall operating status of the codebook and reasonably determine whether a structural adjustment process needs to be initiated. Among them, quantization residuals are the comprehensive error index of each region output by the previous operation, residual distribution intervals refer to the numerical range actually covered by all residual values, discrete clustering pattern is used to describe the concentrated or scattered arrangement characteristics of residual data, preset judgment benchmark is a pre-set codebook adaptation qualification reference standard, and residual deviation degree is used to characterize the gap level between the actual overall state of residuals and the qualification benchmark.

[0125] Specifically, the quantized residuals calculated from all semantic regions are collected to form a residual dataset. All values ​​within the dataset are traversed, and the maximum and minimum values ​​are extracted to define the residual distribution range. Simultaneously, the clustered and scattered points of the statistical data are analyzed to clarify the discrete clustering pattern of the residuals. The statistically obtained distribution characteristics are compared and analyzed with a preset judgment benchmark, and the degree of residual deviation is determined based on the deviation span. When the residual deviation exceeds the preset judgment benchmark, it is determined that the current codebook entry requires adjustment. If the overall deviation does not exceed the preset judgment benchmark, the codebook is determined to be running normally, and no structural adjustment is required. For example, if the preset judgment benchmark range for codebook adaptation is preset to 0~0.5, when the overall distribution of all residuals exceeds 0.5, it is determined that the codebook requires adjustment, using the position of the overall range of all residuals as the criterion for determining the adjustment requirement.

[0126] This avoids the judgment bias caused by single-point data, uses global data characteristics as the core basis for evaluation, improves the objectivity and rationality of the judgment results of adjustment requirements, determines whether the subsequent item screening process should be started, and can reduce meaningless subsequent calculation operations.

[0127] After determining that the codebook needs adjustment at the global level, we only know the overall defect status and cannot pinpoint the specific codebook entries that caused the error. At the same time, the root cause of the residual deviation is directly related to the strong and weak semantic association attributes and topological association relationships bound to the entry itself. By combining the degree of residual deviation, we can narrow down the range of abnormal entries, and then trace the cause of deviation based on the inherent semantic binding information of the entry to accurately locate the problem object. At the same time, we can distinguish different defect causes and classify them as a basis for subsequent targeted adjustments. The codebook entries to be adjusted are the vector representation units that cause quantitative residual anomalies. The semantic association attributes include two types of inherent attributes: strong association semantic node binding and weak association semantic node binding. The deviation cause category is the codebook defect type obtained by combining data features and semantic logic.

[0128] Specifically, based on the degree of residual deviation, an association mapping relationship is established between residual data and corresponding codebook entries participating in quantization operations. Codebook entries with high residual deviation are selected as target entries to be adjusted. The semantic binding information of the entries to be adjusted is retrieved in advance to confirm the strong and weak association node attributes of the entries. At the same time, combined with the subordinate, parallel and derived topological association relationships established above, the internal causes of deviation are analyzed. According to the analysis results, the causes are classified into categories, and different defect types such as semantic entry overlap and redundancy, domain semantic coverage gaps and excessive deviation of entry vector representation are distinguished. According to the differences in the binding association level of entries, the causes are further subdivided into core semantic matching failure and subordinate semantic association misalignment.

[0129] This allows for precise identification of abnormal carriers, clear differentiation of different causes of defects, and avoidance of using a uniform approach to process all codebook entries. The categorized abnormal entries and their corresponding cause categories serve as the direct basis for subsequent matching and adjustment measures, ensuring that structural adjustment actions are targeted.

[0130] The codebook defects caused by different factors are fundamentally different, and a single fixed adjustment method cannot simultaneously solve various problems such as item redundancy, semantic missingness, and vector bias. According to the defect cause matching optimization strategy after classification, the number of codebook entries, vector values, and semantic binding range are modified respectively to repair various defects at the structural level. After the adjustment is completed, the original codebook system is replaced, and the optimized representation units can be used for subsequent feature quantization related operations. Among them, structural adjustment refers to the structural optimization and modification of the number of codebook entries, vector value parameters, and semantic binding association range. Item replacement is the replacement of the original operation codebook with the entire optimized new codebook. Subsequent codebook quantization refers to the subsequent series of quantization processing processes such as image semantic region feature encoding and topic token generation.

[0131] Specifically, for the cause of semantic representation overlap and redundancy, an entry merging and simplification process is performed. Multiple codebook entries with highly overlapping semantic content (i.e., cosine similarity between entries greater than 0.9) are integrated into a single entry. The new entry after merging is the mean vector of the vectors of the entries participating in the merging, reducing invalid and redundant units within the codebook. For the cause of domain semantic coverage gaps, corresponding codebook entries are added by comparing with the complete semantic system of the target domain to fill in the missing semantic representation range of the original codebook. For the cause of excessive deviation in entry vector representation, the error direction of the quantization residual feedback is used, i.e., the quantization... The gradient direction corresponding to the visual feature difference in the residual is used to iteratively correct the codebook entry vector along the negative gradient direction with a preset learning rate (e.g., 0.01) until the residual change in two consecutive iterations is less than 0.001, thus optimizing the accuracy of the semantic expression of the entries. After all entry adjustment operations are completed, the mapping and binding relationship between the adjusted codebook entries and the topic semantics is verified to confirm that there are no errors or breaks in the subordinate, parallel, and derived association logic. After the verification result is qualified, the optimized codebook is used to replace the original codebook entry system in the system and is connected to the computing link to participate in subsequent processing.

[0132] This will fix various structural defects, comprehensively optimize the semantic representation capabilities and overall structural rationality of the codebook, and the updated codebook will serve as the basic computational carrier for subsequent attention interaction, regional feature quantification and topic token generation steps, affecting the data computation accuracy and semantic logic compliance of subsequent processes.

[0133] S5. Calculate the visual saliency of each semantic region based on the fused feature map, combine the attention weights of each topic token, calculate the saliency score of each topic token, and perform global deduplication and sorting of topic tokens in all semantic regions to form a topic token sequence.

[0134] Specifically, such as Figure 2 As shown, the sequence of theme tokens includes:

[0135] Based on the pixel distribution state of the fused feature map and the boundary range of the semantic region, the texture density and gray-level change amplitude of each semantic region are statistically analyzed, and the visual saliency of the corresponding semantic region is obtained by coupling operation.

[0136] The salience score of each topic token is calculated by weighting the attention weight of each topic token with the visual salience of its semantic region.

[0137] Redundancy detection and global deduplication are performed on topic tokens in all semantic regions, and the deduplicated topic tokens are uniformly sorted according to a set order to form a topic token sequence.

[0138] This process involves traversing all topic tokens within all semantic regions, comparing the semantic similarity and spatial affiliation of topic tokens, identifying and eliminating redundant topic tokens with duplicate representations, and sorting the remaining topic tokens in descending order based on their salience scores.

[0139] A single visual evaluation metric can only reflect local image features and cannot comprehensively measure the overall visual prominence of a region. Judging regional salience solely based on texture arrangement or grayscale changes is prone to bias. This approach incorporates two types of feature metrics for statistical calculations and then integrates the results through coupling. It objectively defines the visual hierarchy of a region from two dimensions: detail texture representation and contour grayscale contrast, providing reliable basic parameters for subsequent theme token importance assessment. The fused feature map is a unified feature carrier formed after integrating multiple layers of visual features. The semantic region boundary range is used to delineate the pixel function range of independent image sub-regions. Texture density refers to the density distribution of texture elements within a region. Grayscale abrupt change amplitude represents the overall fluctuation difference in grayscale values ​​between adjacent pixels within the region. Visual salience is a quantitative indicator of regional visual performance obtained after coupling the two metrics.

[0140] Specifically, the corresponding pixel range is determined according to the boundaries of the defined semantic region, and the calculation range is constrained within the sub-region. All pixel points are traversed within the defined interval, the frequency of texture element arrangement and distribution spacing are counted, and the statistical results are summarized to obtain the texture density of the current region. Simultaneously, the gray values ​​of adjacent pixels are compared point by point, the overall amplitude level of gray value jump in the entire region is counted, and the gray value jump amplitude is obtained.

[0141] Let N be the total number of pixels in a single semantic region, and N be the number of effective texture pixels. t Texture density calculation formula: I t =N t / N, select the gray-level difference sequence of adjacent pixels within the semantic region, and take the extreme value difference of the sequence as the gray-level change amplitude I. g Texture density reflects the richness of structural details within a region and is a sustained factor in attracting visual attention. It contributes more stably and persistently to thematic token identification, thus receiving a higher weight. Gray-scale abrupt change amplitude reflects the local contrast of the contour edge; although it has instantaneous salience, its support for thematic semantic attribution is weak, therefore it receives a lower weight. The sum of the two coefficients is normalized to 1. Therefore, in this embodiment, the coupling weighting coefficients γ1=0.6 and γ2=0.4 are set, meaning that texture density contributes more to visual salience than gray-scale abrupt change amplitude. The visual salience coupling calculation is S. v =γ1I t +γ2I gIn actual execution, the range of pixels to be operated on is locked according to the region boundary. Following the above coupled operation logic, the two statistical indicators are converted and processed, and finally the visual saliency corresponding to a single semantic region is output.

[0142] This objectively distinguishes the visual differences in different semantic regions, and visual salience serves as the core input parameter for the subsequent weighted calculation of topic token salience scores.

[0143] Attention weight alone can only reflect the degree of matching and association between the topic content and the semantic region, and cannot judge the value of the token expression in combination with the actual visual presentation characteristics of the image. The two types of parameters are independent and cannot comprehensively reflect the importance level of the topic. By combining the semantic matching weight with the visual salience characteristics of the region and performing a weighted calculation, the semantic relevance and visual prominence are taken into account to form a multi-dimensional comprehensive evaluation index to classify the importance level of different topic tokens. Among them, the attention weight is the effective correlation coefficient obtained after the previous attention interaction and semantic constraint correction, the semantic region refers to the sub-region to which the image belongs in a single topic token, and the salience score is the quantitative result of the integration of semantic relevance and visual features, which represents the importance of the topic expression.

[0144] Specifically, each item in the sparse topic token set is retrieved individually, and the effective attention weight corresponding to each item is extracted. The visual saliency result of matching the semantic region to which the item belongs is then used. Let w be the semantic matching weight corresponding to a single token. s That is, the effective attention weight after semantic constraint correction and sparse filtering, and the visual salience of matching the semantic region to which the item belongs is S. v Based on the hierarchical weighting principle of semantic association as the primary factor and visual features as a secondary factor, a balanced weighting coefficient η is set, and the semantic matching weight w is set. s Visual significance S is directly used as the baseline component of the significance score (coefficient is 1). v After scaling by η, it is superimposed as an auxiliary correction factor. In this embodiment, η=0.5, that is, the correction magnitude of visual saliency is 50% of the semantic matching weight. Under the premise of ensuring that semantic matching dominates the ranking, visual features are allowed to exert a moderate degree of influence on the ranking results.

[0145] In specific application domains, the value of η is adjusted according to the characteristics of the actual scene. If the semantic structure of the image is clear and the visual performance differences are significant, η is appropriately reduced to reduce the interference of visual features on the semantic-dominant ranking. If the image has distinct visual layers but sparse semantic annotations, η is appropriately increased to enhance the corrective influence of visual features. The significance score is Score=w s +η×S v The algorithm retrieves all topic tokens within the region one by one, matches the corresponding weights and regional salience, and substitutes them into the formula to complete the calculation, thus obtaining the salience scores of all tokens in batches.

[0146] For example, a sparse topic token set for a semantic region contains 3 entries, and the semantic matching weight w of entry A is... sA =0.75 (cosine similarity is retained after semantic constraint correction), w for item B sB =0.62, w of entry C sC =0.45; the visual salience S of this region v =0.6, substituting into the calculation gives the Score A =0.75 + 0.5 × 0.6 = 1.05, Score B =0.62 + 0.5 × 0.6 = 0.92, Score C =0.45+0.5×0.6=0.75. After global deduplication and descending sorting, the theme token sequence is A→B→C.

[0147] This improves the thematic evaluation dimensions, accurately distinguishes the actual expressive value of different thematic tokens, and the salience score is the direct basis for judging the item ranking after the subsequent global deduplication.

[0148] Judging duplicate entries solely based on semantic content similarity cannot identify redundant content such as synonymous expressions across regions, and simply checking the spatial attribution range cannot eliminate the problem of multiple entries with overlapping meanings in the same region. The single identification dimension has omission defects. We adopt a dual dimension of semantic content and spatial location to simultaneously verify, so as to comprehensively cover various repetitive expression scenarios, eliminate redundant entries that do not have independent meaning functions, and simplify the overall theme expression volume. Semantic similarity is used to measure the overlap ratio of the core theme content carried by two theme tokens, and spatial attribution range refers to the image semantic region location boundary corresponding to the entry. Redundant theme tokens are invalid entries with repeated meaning content and cannot add effective theme information.

[0149] Specifically, a full-domain traversal search is initiated, and the topic tokens generated from all semantic regions of the entire image are uniformly collected and paired for dual-condition comparison. The semantic overlap is calculated by comparing the semantic representation information corresponding to the entries, and the spatial region boundaries of each entry are checked. When two entries have highly overlapping semantics and their spatial affiliations are related by the same source or have overlapping ranges, they are judged as redundant and duplicate entries. A single valid entry is retained according to the established retention rules, and the remaining duplicate representation entries are removed. A semantic similarity reference threshold is set as a reference condition for duplicate judgment. In this embodiment, it is set to 0.8, which is a preset based on the distinguishability requirement of semantic representation. Those skilled in the art can adjust the reference threshold accordingly in the actual process.

[0150] This process comprehensively identifies all types of duplicate entries, retains thematic entries with independent semantic value, and the set of valid thematic tokens obtained after deduplication directly enters the final ordered sorting process.

[0151] Unsorted topic tokens merely list topic content in a scattered manner, failing to reflect the hierarchy and important arrangement of topics within the image, which is detrimental to subsequent unified analysis and information retrieval. Using comprehensive saliency score as the sorting standard, the items are arranged in descending order of value, and the content hierarchy is organized according to the importance of the topics, forming a well-structured and logically clear topic expression sequence. The descending order sorting refers to the orderly arrangement of items from high to low based on numerical value. The topic token sequence is an ordered combination of items that fully summarize all the core topics of the image after sorting.

[0152] Specifically, all valid topic tokens after deduplication are extracted, and each item is bound to its corresponding salience score data. Using the score value as the sole sorting criterion, the order of items is adjusted sequentially according to the descending order rule. After all items are arranged, they are integrated and organized to form the final topic token sequence.

[0153] This clearly demonstrates the hierarchical relationship of image themes, ensures a logically consistent arrangement of themes, and allows the theme token sequence to directly connect to subsequent upper-level application processes such as image semantic interpretation and content retrieval and recognition.

[0154] Example 2

[0155] like Figure 3 As shown, this embodiment provides a topic token generation system based on image feature extraction, including a segmentation module, a codebook module, a quantization module, an adjustment module, and a generation module.

[0156] The segmentation module is used to extract multi-level visual features from the input image, fuse them to obtain a fused feature map, and divide the fused feature map into regional features of multiple semantic regions;

[0157] The codebook module is used to acquire topic knowledge of the target domain, construct a topic semantic structure, initialize a quantized codebook based on the topic semantic structure, and establish a mapping relationship between codebook entries and topic semantics in the quantized codebook.

[0158] The quantization module is used to perform attention interaction between multiple sets of preset topic query vectors and codebook entries, apply semantic constraints to the interaction process based on the mapping relationship, generate a sparse topic token set for each semantic region, and calculate the quantization residual of each region feature.

[0159] The adjustment module is used to determine the adjustment requirements of codebook entries based on the quantization residuals, and to perform structural adjustments on the codebook entries based on the distribution characteristics of the quantization residuals, so that the structurally adjusted codebook entries can be used for subsequent quantization.

[0160] The generation module is used to calculate the visual saliency of each semantic region based on the fused feature map, integrate the attention weight of each topic token, calculate the saliency score of each topic token, and perform global deduplication and sorting of topic tokens in all semantic regions to form a topic token sequence.

[0161] The above description is merely a preferred embodiment of this application. The scope of protection of this application is not limited to the above embodiments. All technical solutions falling within the scope of this application's concept are within the scope of protection of this application. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of this application should also be considered within the scope of protection of this application.

Claims

1. A topic token generation method based on image feature extraction, characterized in that, include: Multi-level visual features of the input image are extracted, integrated to obtain a fusion feature map, and the fusion feature map is divided into regional features of multiple semantic regions; Acquire topic knowledge in the target domain, construct a topic semantic structure, initialize a quantized codebook based on the topic semantic structure, and establish a mapping relationship between codebook entries in the quantized codebook and topic semantics; Multiple sets of preset topic query vectors are interacted with codebook entries through attention. Semantic constraints are imposed on the interaction process based on the mapping relationship. A sparse topic token set is generated for each semantic region, and the quantization residual of each region feature is calculated. The codebook entries are adjusted based on the quantization residuals, and the codebook entries are structurally adjusted based on the distribution characteristics of the quantization residuals. The visual saliency of each semantic region is calculated based on the fused feature map. The attention weight of each topic token is combined to calculate the saliency score of each topic token. The topic tokens of all semantic regions are then globally deduplicated and sorted to form a topic token sequence.

2. The topic token generation method based on image feature extraction according to claim 1, characterized in that, The regional features for dividing the semantic region include: The input image is analyzed to obtain multi-level visual features. Fusion weights are assigned according to the spatial correlation between the visual features, and the fusion feature map is generated by integrating them. The fused feature map is split into shallow contour components, mid-layer texture components, and deep semantic components. Semantic anchors are determined based on the deep semantic components. The coverage of the corresponding mid-layer texture components is matched based on the semantic anchors to preliminarily determine the semantic interval range. Within the semantic interval, the gradient difference between components at adjacent coordinate points is calculated, and the segmentation boundary is checked in conjunction with the morphological trend of the shallow contour components to determine the region contour. By combining the distribution of semantic anchor points with the region outline, semantic regions are divided, and the regional features of the semantic regions are obtained.

3. The topic token generation method based on image feature extraction according to claim 2, characterized in that, Establishing the mapping relationship includes: Extract entities and relationships from the subject knowledge of the target domain, and construct a subject semantic structure based on the relationships between entities, with entities as nodes. The nodes within the topic semantic structure are converted into semantic vectors of a unified dimension. Based on the spatial distribution characteristics of the semantic vectors, codebook entries are generated by clustering, thus completing the initialization operation of the quantized codebook. Calculate the correlation between codebook entries and semantic vectors, complete the binding according to the correlation, and establish the mapping relationship between codebook entries and topic semantics.

4. The topic token generation method based on image feature extraction according to claim 3, characterized in that, Thematic semantic structures are constructed using a weighted topology networking approach, with semantic distance between any nodes used as connection weights, and incorporating relationships including subordinate, parallel, and derived relationships. The spatial distribution characteristics of semantic vectors are analyzed by density peak clustering algorithm, and the cluster center vectors obtained by the solution are set as codebook entries in sequence. Set an association threshold and compare the association degree with the association threshold to distinguish and bind the semantic association attributes of strong and weak semantic nodes.

5. The topic token generation method based on image feature extraction according to claim 4, characterized in that, The calculation of the quantization residual for each region feature includes: Multiple sets of preset topic query vectors are respectively subjected to attention interaction operations with codebook entries to obtain the attention weights between topic query vectors and codebook entries; Based on the mapping relationship, semantic constraints are corrected on the attention weights, and item combinations are selected from the corrected codebook items according to the filtering conditions to generate a sparse topic token set corresponding to the semantic region. Using the corrected attention weight as the weighting coefficient, the codebook entries corresponding to the topic token are merged to reconstruct the regional features. The quantized residual of each regional feature is obtained by comparing it with the original regional features.

6. The topic token generation method based on image feature extraction according to claim 5, characterized in that, The attention weight between the topic query vector and the codebook entry is calculated using cosine similarity; the semantic constraint correction of the attention weight is completed based on the semantic association attributes bound to the codebook entry and the mapping relationship. The semantic vectors of the codebook entries corresponding to the topic token are fused using weighted coefficients to obtain the reconstructed regional features. The deviation between the feature differences and the semantic constraints is combined and superimposed to calculate the quantized residuals of the corresponding regional features.

7. The topic token generation method based on image feature extraction according to claim 6, characterized in that, The structural adjustment of codebook entries includes: Summarize the quantization residuals corresponding to all semantic regions, statistically analyze the residual distribution intervals and discrete clustering patterns, combine the preset judgment criteria to determine the degree of residual deviation, and determine whether there is an adjustment requirement for the current codebook entry; The codebook entries to be adjusted are selected based on the degree of residual deviation, and the deviation causes are classified in combination with the semantic association attributes bound to the codebook entries. Based on the category of deviation cause, match the corresponding adjustment method to perform structural adjustment, and replace the original codebook entries with the structurally adjusted codebook entries.

8. The topic token generation method based on image feature extraction according to claim 7, characterized in that, The sequence of topic tokens includes: Based on the pixel distribution state of the fused feature map and the boundary range of the semantic region, the texture density and gray-level change amplitude of each semantic region are statistically analyzed, and the visual saliency of the corresponding semantic region is obtained by coupling operation. The salience score of each topic token is calculated by weighting the attention weight of each topic token with the visual salience of its semantic region. Redundancy detection and global deduplication are performed on topic tokens in all semantic regions, and the deduplicated topic tokens are uniformly sorted according to a set order to form a topic token sequence.

9. The topic token generation method based on image feature extraction according to claim 8, characterized in that, Traverse all topic tokens within all semantic regions, compare the semantic similarity and spatial affiliation of topic tokens, identify and eliminate redundant topic tokens with duplicate representations, and sort the remaining topic tokens in descending order according to the magnitude of the salience score.

10. A topic token generation system based on image feature extraction, used to implement the topic token generation method based on image feature extraction as described in any one of claims 1-9, characterized in that, include: The module consists of a partitioning module, a codebook module, a quantization module, an adjustment module, and a generation module. The segmentation module is used to extract multi-level visual features from the input image, fuse them to obtain a fused feature map, and divide the fused feature map into regional features of multiple semantic regions; The codebook module is used to acquire topic knowledge of the target domain, construct a topic semantic structure, initialize a quantized codebook based on the topic semantic structure, and establish a mapping relationship between codebook entries and topic semantics in the quantized codebook. The quantization module is used to perform attention interaction between multiple sets of preset topic query vectors and codebook entries, apply semantic constraints to the interaction process based on the mapping relationship, generate a sparse topic token set for each semantic region, and calculate the quantization residual of each region feature. The adjustment module is used to determine the adjustment requirements of codebook entries based on the quantization residuals, and to perform structural adjustments on codebook entries based on the distribution characteristics of the quantization residuals. The generation module is used to calculate the visual saliency of each semantic region based on the fused feature map, integrate the attention weight of each topic token, calculate the saliency score of each topic token, and perform global deduplication and sorting of topic tokens in all semantic regions to form a topic token sequence.

Images