Label setting method and system for unstructured data

Abstract

The invention relates to the technical field of data labels, and particularly discloses a label setting method and system for unstructured data, and the method comprises the steps: carrying out the multi-dimensional preprocessing of the unstructured data, and obtaining standardized data; performing feature extraction on the standardized data by adopting a hybrid model fused with a feature extraction operator to obtain deep feature information of the data; generating a dynamic label set based on the deep feature information; matching the dynamic label set with the standardized data through a semantic association degree calculation method to obtain a preliminary label; and monitoring the dynamic change of the data and new business requirements by means of a time sequence feature tracking module, and updating and expanding the preliminary label through a label evolutionary algorithm to obtain a final label. According to the method, a multi-dimensional preprocessing technology is adopted, the quality and pertinence of preprocessed data are improved, and a good foundation is laid for subsequent feature extraction and label setting.

Description

Technical Field

[0001] This invention relates to the field of data tagging technology, and specifically to a method and system for tagging unstructured data. Background Technology

[0002] In data management and processing, tagging systems are widely used to classify and label various types of data to achieve efficient data retrieval, analysis, and utilization. Effective tagging is particularly crucial for unstructured data, such as text, images, audio, and video. Existing technologies offer various tagging schemes for unstructured data. Some schemes employ a predefined, fixed tagging system, determining a set of tags manually or semi-automatically in a specific domain or initial application scenario. For example, in early news text classification systems, tags such as "politics," "economics," "sports," and "entertainment" were preset based on common news categories to label news articles. This approach can achieve basic classification management functions when the data scale is small and the application scenario is relatively simple and stable.

[0003] With the rapid development of information technology, data volume is exploding, and application scenarios are becoming increasingly complex and diverse. For example, some large e-commerce platforms need to tag massive amounts of unstructured data such as user review texts, product description texts, and product images to support various business operations such as precision marketing, product recommendations, and user profiling analysis. Traditional fixed tagging systems have revealed significant limitations. On the one hand, their flexibility is severely insufficient, making it difficult to adapt to emerging business needs and data characteristics. For instance, when e-commerce platforms expand into new business areas, such as cross-border e-commerce, involving special factors such as the culture and regulations of different countries, the original tagging system cannot quickly cover new tagging requirements such as "cross-border logistics methods," "import tariff policies," and "consumer preferences in different countries." On the other hand, for dynamic changes in data, such as the frequent appearance of emerging internet slang in user review texts, fixed tagging systems lack the ability to automatically update and expand, leading to a gradual decrease in the matching degree between tags and data content, failing to accurately reflect the actual characteristics of the data, and thus affecting the accuracy of data analysis and the scientific nature of business decisions.

[0004] Some existing technologies employ simple rule-based tagging methods, such as adding tags to text data through keyword matching rules. In some social media text analysis systems, rules are set to automatically add the tag "Sports-Football Event" when keywords like "football" or "match" appear in the text. This method can quickly achieve partial tagging when the rules are clearly defined and the data characteristics are relatively obvious. However, its scalability is extremely poor. When facing new business scenarios, such as analyzing health and wellness text on social media, the rich and complex knowledge in the health and wellness field, with new concepts and terms constantly emerging, makes it difficult to cover all relevant content with limited keyword matching rules. For example, for some newly emerging health and wellness methods, such as "meditation yoga therapy," if relevant keywords are not pre-defined in the rules, it is impossible to correctly add tags to texts involving such content. Furthermore, this simple rule-based method has a superficial understanding of the data and cannot handle semantically complex or ambiguous data, easily leading to mislabeling and omissions, significantly reducing the accuracy and effectiveness of tagging. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for setting labels for unstructured data, so as to solve the above-mentioned technical problems.

[0006] The objective of this invention can be achieved through the following technical solutions: A method for labeling unstructured data includes the following steps: S1: Perform multi-dimensional preprocessing on unstructured data to obtain standardized data; S2: Use a hybrid model with fusion feature extraction operators to extract features from the standardized data and obtain deep feature information of the data; S3: Based on the deep feature information, combined with the preset dynamic expansion tag library and tag evolution model, a dynamic tag set is generated, wherein the tag evolution model includes tag gene chains and tag mutation operators; S4: The dynamic tag set is matched with the standardized data using a semantic relevance calculation method, and the data is optimized based on the matching results and adaptive optimization rules to obtain preliminary tags; S5: By using the time-series feature tracking module to monitor the dynamic changes of data and new business requirements, the initial labels are updated and expanded through the label evolution algorithm to obtain the final labels. The label evolution algorithm includes label crossover, label mutation and label selection operations.

[0007] As a further aspect of the present invention: the multi-dimensional preprocessing includes differential cleaning based on data source, adaptive format conversion, and intelligent deduplication. The differential cleaning based on data source adopts different cleaning rules for unstructured data from different sources. The adaptive format conversion automatically selects the optimal conversion format according to the data type. The intelligent deduplication determines duplicates by calculating the multi-feature hash value of the data.

[0008] As a further aspect of the present invention, the feature extraction operators include a context weighting operator for text, a region importance operator for images, an emotional fluctuation operator for audio, and a dynamic change operator for video.

[0009] As a further aspect of the present invention: in step S2, the contextual weight operator of the text assigns different weights based on the frequency and importance of words in different contexts; the regional importance operator of the image determines the regional weight based on the contribution of regions in the image to the expression of the theme; the emotional fluctuation operator of the audio extracts emotional features and assigns weights by analyzing changes in tone, speed and volume of the audio; and the dynamic change operator of the video determines the dynamic feature weights based on the amplitude and frequency of motion of the images in the video.

[0010] As a further aspect of the present invention: in step S3, the dynamically expanded tag library has a self-growing attribute, which is manifested as follows: When new feature information is detected and existing tags cannot cover it, new tags are automatically created and associations with related tags are established; the tag gene chain records the historical evolution information and feature attributes of the tags, and the tag mutation operator performs mutation adjustments on the tags according to the new feature information to adapt to the new data features.

[0011] As a further aspect of the present invention: In step S5, the temporal feature tracking module monitors the feature changes of the data in real time through a sliding window. When the feature change exceeds a preset threshold, the label evolution algorithm is triggered. The label crossover operation fuses the features of two related labels to generate a new label. The label mutation operation randomly adjusts the features of the labels to generate new features. The label selection operation retains labels with high fitness and eliminates labels with low fitness.

[0012] As a further aspect of the present invention: in S4, the semantic relevance calculation method integrates semantic distance, feature similarity, and contextual fit.

[0013] A labeling system for unstructured data, comprising: The multi-dimensional preprocessing module is used to perform differential cleaning, adaptive format conversion, and intelligent deduplication on unstructured data based on the data source, so as to obtain standardized data; The hybrid feature extraction module uses a hybrid model that integrates original feature extraction operators to extract features from the standardized data and obtain deep feature information of the data. The original feature extraction operators include the context weight operator for text, the regional importance operator for images, the emotional fluctuation operator for audio, and the dynamic change operator for video. The dynamic tag generation module generates a dynamic tag set based on the deep feature information, combined with a preset dynamic extended tag library and an original tag evolution model. The dynamic extended tag library has self-growing properties, and the tag evolution model includes tag gene chains and tag mutation operators. The semantic matching optimization module matches the dynamic tag set with the standardized data using a unique semantic relevance calculation method, and optimizes the data based on the matching results and adaptive optimization rules to obtain preliminary tags. The semantic relevance calculation method integrates semantic distance, feature similarity, and contextual fit. The temporal evolution module monitors the dynamic changes in data and new business requirements with the help of the temporal feature tracking module. It updates and expands the initial labels through the label evolution algorithm to obtain the final labels. The label evolution algorithm includes label crossover, label mutation and label selection operations.

[0014] The beneficial effects of this invention are as follows: The use of multi-dimensional preprocessing technology improves the quality and relevance of preprocessed data, laying a solid foundation for subsequent feature extraction and label setting; the hybrid model integrating feature extraction operators can more accurately extract deep features from unstructured data, providing a more reliable basis for label generation; the dynamic expansion of the label library and label evolution model enables the label system to have self-growth and evolution capabilities, quickly adapting to new data features and business needs, solving the problem of insufficient flexibility in fixed label systems; the semantic relevance calculation method improves the accuracy of label-data matching, reducing mislabeling and omissions; and the temporal feature tracking and label evolution algorithm realizes dynamic updating and expansion of labels, enhancing the scalability and adaptability of the label system, and better meeting ever-changing business needs. Attached Figure Description

[0015] The invention will now be further described with reference to the accompanying drawings.

[0016] Figure 1 This is a flowchart illustrating a label setting method for unstructured data according to the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1: Please see Figure 1 As shown, this embodiment takes the unstructured data processing of a cross-border e-commerce platform as an example to explain in detail the method for setting tags for unstructured data. The specific steps are as follows: S1: Perform multi-dimensional preprocessing on unstructured data from e-commerce platforms: The platform's unstructured data includes user review texts, product images, customer service audio recordings, and product promotional videos.

[0019] The source identification unit identifies the data type through a feature matching algorithm. The accuracy rate is calculated as follows: (number of correctly identified data / total number of data) × 100% = (9850 / 10000) × 100% = 98.5%.

[0020] The intelligent deduplication unit uses a multi-feature hash value calculation model, the formula of which is: ; in: f i (v) is the i-th feature value of data v; w i It is the feature weight, and satisfies ; Hash() is the MurmurHash3 function; For user review texts, three features are selected: keyword frequency, length, and publication time. Weights are calculated using the entropy weighting method. Information entropy ,in, x ij It is the value of the i-th feature in the j-th sample; when the difference in hash values ​​between two user review texts is less than 10 -6 If the data is found to be duplicated, it is considered a duplicate.

[0021] After differential cleaning, the noise removal rate of text data = (amount of noise data removed / amount of noise data before preprocessing) × 100%; the noise removal rate of image data = (amount of noise data removed / amount of noise data before preprocessing) × 100%. S2: The hybrid feature extraction module extracts features from standardized data. The text feature extraction unit applies the context weight operator, with the following formula: ; Where α+β+γ=1; The context score is calculated as follows: ;c i is the context words within a window size k, and Sim is the cosine similarity of the word vectors.

[0022] Context-aware word vectors based on BERT: ; The attention score is as follows: ; Where Q and K are the query and key vectors, respectively, and d is the vector dimension.

[0023] Similarly, we can also extract regional importance operators for images, emotional fluctuation operators for audio, and dynamic change operators for videos.

[0024] S3: Based on the deep feature information, combined with the preset dynamic expansion tag library and tag evolution model, a dynamic tag set is generated, wherein the tag evolution model includes tag gene chains and tag mutation operators; The dynamically expanded tag library has a self-growing attribute, which is manifested as follows: When new feature information is detected and existing tags cannot cover it, new tags are automatically created and associations with related tags are established; the tag gene chain records the historical evolution information and feature attributes of the tags, and the tag mutation operator performs mutation adjustments on the tags according to the new feature information to adapt to the new data features.

[0025] S4: The dynamic tag set is matched with the standardized data using a semantic relevance calculation method, and the data is optimized based on the matching results and adaptive optimization rules to obtain preliminary tags; The semantic relevance calculation method integrates semantic distance, feature similarity, and contextual fit.

[0026] S5: By using the time-series feature tracking module to monitor the dynamic changes of data and new business requirements, the initial labels are updated and expanded through the label evolution algorithm to obtain the final labels. The label evolution algorithm includes label crossover, label mutation and label selection operations.

[0027] The time-series feature tracking module monitors the feature changes of the data in real time through a sliding window. When the feature change exceeds a preset threshold, the label evolution algorithm is triggered. The label crossover operation merges the features of two related labels to generate a new label. The label mutation operation randomly adjusts the features of the labels to generate new features. The label selection operation retains labels with high fitness and eliminates labels with low fitness.

[0028] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. A method for setting labels for unstructured data, characterized in that, Includes the following steps: S1: Perform multi-dimensional preprocessing on unstructured data to obtain standardized data; S2: Use a hybrid model with fusion feature extraction operators to extract features from the standardized data and obtain deep feature information of the data; S3: Based on the deep feature information, combined with the preset dynamic expansion tag library and tag evolution model, a dynamic tag set is generated, wherein the tag evolution model includes tag gene chains and tag mutation operators; S4: The dynamic tag set is matched with the standardized data using a semantic relevance calculation method, and the data is optimized based on the matching results and adaptive optimization rules to obtain preliminary tags; S5: By using the time-series feature tracking module to monitor the dynamic changes of data and new business requirements, the initial labels are updated and expanded through the label evolution algorithm to obtain the final labels. The label evolution algorithm includes label crossover, label mutation and label selection operations.

2. The labeling method for unstructured data according to claim 1, characterized in that, The multi-dimensional preprocessing includes differential cleaning based on data source, adaptive format conversion, and intelligent deduplication. The differential cleaning based on data source adopts different cleaning rules for unstructured data from different sources. The adaptive format conversion automatically selects the optimal conversion format according to the data type. The intelligent deduplication determines duplicates by calculating the multi-feature hash value of the data.

3. The labeling method for unstructured data according to claim 1, characterized in that, The feature extraction operators include context weighting operators for text, region importance operators for images, sentiment fluctuation operators for audio, and dynamic change operators for video.

4. The labeling method for unstructured data according to claim 3, characterized in that, In step S2, the context weighting operator of the text assigns different weights based on the frequency and importance of words in different contexts; the region importance operator of the image determines the region weight based on the contribution of regions in the image to the expression of the theme; the emotional fluctuation operator of the audio extracts emotional features and assigns weights by analyzing changes in tone, speech rate and volume of the audio; and the dynamic change operator of the video determines the dynamic feature weights based on the amplitude and frequency of motion of the images in the video.

5. A method for setting tags for unstructured data according to claim 1, characterized in that, In step S3, the dynamically expanded tag library has a self-growing attribute, which is manifested as follows: When new feature information is detected and existing tags cannot cover it, new tags are automatically created and associations with related tags are established; the tag gene chain records the historical evolution information and feature attributes of the tags, and the tag mutation operator performs mutation adjustments on the tags according to the new feature information to adapt to the new data features.

6. The labeling method for unstructured data according to claim 1, characterized in that, In step S5, the temporal feature tracking module monitors the feature changes of the data in real time through a sliding window. When the feature change exceeds a preset threshold, the label evolution algorithm is triggered. The label crossover operation merges the features of two related labels to generate a new label. The label mutation operation randomly adjusts the features of the labels to generate new features. The label selection operation retains labels with high fitness and eliminates labels with low fitness.

7. The labeling method for unstructured data according to claim 1, characterized in that, In S4, the semantic relevance calculation method integrates semantic distance, feature similarity, and contextual fit.

8. A labeling system for unstructured data, characterized in that, include: The multi-dimensional preprocessing module is used to perform differential cleaning, adaptive format conversion, and intelligent deduplication on unstructured data based on the data source, so as to obtain standardized data; The hybrid feature extraction module uses a hybrid model that integrates original feature extraction operators to extract features from the standardized data and obtain deep feature information of the data. The original feature extraction operators include the context weight operator for text, the regional importance operator for images, the emotional fluctuation operator for audio, and the dynamic change operator for video. The dynamic tag generation module generates a dynamic tag set based on the deep feature information, combined with a preset dynamic extended tag library and an original tag evolution model. The dynamic extended tag library has self-growing properties, and the tag evolution model includes tag gene chains and tag mutation operators. The semantic matching optimization module matches the dynamic tag set with the standardized data using a unique semantic relevance calculation method, and optimizes the data based on the matching results and adaptive optimization rules to obtain preliminary tags. The semantic relevance calculation method integrates semantic distance, feature similarity, and contextual fit. The temporal evolution module monitors the dynamic changes in data and new business requirements with the help of the temporal feature tracking module. It updates and expands the initial labels through the label evolution algorithm to obtain the final labels. The label evolution algorithm includes label crossover, label mutation and label selection operations.

Images