Market supervision multi-field data label generation method and system

Abstract

The invention relates to a market supervision multi-field data label generation method and system, and the method comprises the steps: obtaining market supervision multi-field historical text data, and carrying out the data preprocessing, and obtaining market supervision text data; training the large language model in a supervision fine tuning mode, and extracting keywords of the market supervision text data through the trained large language model; extracting keywords of the market supervision text data through a natural language processing technology; and recombining the keywords obtained in various modes, carrying out natural language processing and synonym recommendation processing, and integrating into a final tag text set. According to the method, data are processed by adopting a multi-language model, and a natural language processing technology and a text processing technology based on a large model are included. The same part of data is processed through different models, and more diversified and comprehensive keywords and abstracts can be extracted, so that the accuracy and coverage rate of tag generation are improved.

Description

Technical Field

[0001] The present invention relates to the field of data information processing technology, and in particular to a method and system for generating multi-field data labels for market supervision. Background Art

[0002] With the continuous development of big data and artificial intelligence technology, market regulators need to process and analyze massive amounts of data every day to achieve effective market supervision. In order to manage and utilize this data more efficiently, labeling has become an important data processing method. By labeling the data, it is easy to classify, retrieve and analyze the data, thereby improving the efficiency and accuracy of market supervision.

[0003] At present, in the field of market supervision, the creation of data labels mainly relies on manual methods. Supervisors manually add labels to data based on their own experience and understanding. Although this method can meet the needs of labeling to a certain extent, it has obvious limitations. Manually creating labels requires a lot of time and energy, and is greatly affected by personal subjective factors, making it difficult to ensure the consistency and accuracy of labels. Summary of the invention

[0004] In view of this, it is necessary to provide a method for generating multi-field data labels for market supervision to solve the above-mentioned defects of the prior art.

[0005] In order to solve the above problems, in a first aspect, an embodiment of the present invention provides a method for generating multi-field data labels for market supervision, comprising:

[0006] Obtain historical text data in multiple fields of market supervision and perform data preprocessing to obtain market supervision text data;

[0007] Create a training dataset based on market supervision text data, train the large language model using supervised fine-tuning, and extract keywords from the market supervision text data through the trained large language model;

[0008] Extract keywords from market regulatory text data through natural language processing technology;

[0009] The keywords obtained in various ways are recombined, natural language processing and synonym recommendation are performed, and integrated into the final label text set.

[0010] Preferably, the historical text data in multiple fields of market supervision include consumer complaint text data, and supervision record data generated in daily supervision activities such as food, medicine, cosmetics, and special equipment.

[0011] Preferably, the data preprocessing includes:

[0012] Data cleaning: remove noise, missing values, outliers or duplicate values ​​in the data, and ensure data compliance and privacy protection;

[0013] Data standardization: Use text conversation data format to label the cleaned data.

[0014] Preferably, the method of training the large language model by supervised fine-tuning and extracting keywords from the market supervision text data by using the trained large language model specifically includes:

[0015] Create a training dataset based on text conversation data;

[0016] Design supervised fine-tuning (SFT) tasks based on the task scenarios in the market supervision vertical field and train the large language model ERNIELite;

[0017] Verify the performance of large language models through multiple evaluation metrics;

[0018] Directly extract keywords from market supervision text data through a trained large language model; or, after restatement of the market supervision text data through a trained large language model, extract keywords from the restatement text; wherein the restatement is to simplify the text, adjust grammar, and eliminate ambiguity.

[0019] Preferably, after obtaining the final label text set, the method further includes:

[0020] Compare the newly generated labels with the original labels, identify duplicate labels, and perform deduplication processing;

[0021] After confirming that the newly generated label does not overlap with the original label, perform the label storage operation.

[0022] Preferably, after executing the label storage operation, the method further includes:

[0023] Provides tag search, tag version control, user access control, identity authentication, and log monitoring functions for tag management.

[0024] Preferably, after executing the label storage operation, the method further includes:

[0025] Data association: Users can reference and apply customized data tags in business processes.

[0026] Data sharing: Provides internal data label sharing and external link generation and sharing functions to facilitate data sharing and communication; and provides label data sharing monitoring and label data task monitoring functions;

[0027] Data analysis: Conduct label profile construction analysis, label monitoring alarm analysis, label call frequency analysis, label synonym analysis, and label data association analysis on label data.

[0028] In a second aspect, an embodiment of the present invention provides a market supervision multi-field data label generation system, the system is used to execute the market supervision multi-field data label generation method described in the first aspect embodiment, the system comprising:

[0029] The data preparation module is used to obtain historical text data in multiple fields of market supervision and perform data preprocessing to obtain market supervision text data;

[0030] The model training module is used to create a training data set based on market supervision text data, train the large language model using supervised fine-tuning, and extract keywords from the market supervision text data through the trained large language model;

[0031] A natural language processing module, which is used to extract keywords from market regulatory text data through natural language processing technology;

[0032] The tag generation module is used to recombine the keywords obtained in various ways, perform natural language processing and synonym recommendation processing, and integrate them into the final tag text set.

[0033] In a third aspect, the present invention further provides an electronic device, including a memory and a processor, wherein:

[0034] The memory is used to store programs;

[0035] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the method for generating multi-field data labels for market supervision as described in the embodiment of the first aspect of the present invention.

[0036] In a fourth aspect, the present invention further provides a computer-readable storage medium for storing computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the method for generating multi-field data labels for market supervision as described in the embodiment of the first aspect of the present invention.

[0037] The beneficial effects of adopting the above embodiment are:

[0038] The present invention combines natural language processing technology (NLP) with a large model trained on data in specific fields of market supervision scenarios, and designs a method for generating multi-field data labels for market supervision. This method can automatically generate data labels based on proprietary data in vertical fields to form a label set for vertical fields of market supervision. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1A flow chart of the method for generating multi-field data labels for market supervision provided by the present invention;

[0040] Figure 2 A schematic diagram of the original text data of the market supervision topic provided by the present invention;

[0041] Figure 3 A schematic diagram of the label generation process provided by the present invention;

[0042] Figure 4 A complete flow chart of the method for generating multi-domain data labels for market supervision provided by the present invention;

[0043] Figure 5 A complete structural diagram of the market supervision multi-field data label generation system provided by the present invention;

[0044] Figure 6 This is a structural block diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0045] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0046] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0047] At present, in the field of market supervision, the creation of data labels mainly relies on manual methods. Supervisors manually add labels to data based on their own experience and understanding. Although this method can meet the needs of labeling to a certain extent, it has obvious limitations. Manually creating labels requires a lot of time and energy, and is greatly affected by personal subjective factors, making it difficult to ensure the consistency and accuracy of labels.

[0048] In view of this, the present invention provides a method for generating multi-domain data labels for market supervision. By combining the application of natural language processing technology (NLP) with a large model trained on data in specific fields of market supervision scenarios, a method for generating a label set in a vertical field of market supervision is designed to achieve automatic generation and efficient management of data labels. The following will be explained and introduced through multiple embodiments.

[0049] Figure 1 The flowchart of the method for generating multi-domain data labels for market supervision provided by the present invention is as follows. Figure 1 As shown, the method for generating multi-field data labels for market supervision includes at least the following steps S1 to S4:

[0050] Step S1, obtaining historical text data in multiple fields of market supervision and performing data preprocessing to obtain market supervision text data.

[0051] Specifically, first, historical text data in multiple fields of market supervision are obtained as original text data. Figure 2 This is a schematic diagram of the original text data of the market supervision topic provided by the present invention. Figure 2 , Figure 2 The external special data in the data is the content of the vertical field of market supervision, that is, the original text data. The original text data includes but is not limited to consumer complaint text data, supervision record data generated in daily supervision activities such as food, medicine, cosmetics, special equipment, double random inspection record data, and other market supervision text record data. The demonstration data used in the embodiment of the present invention is consumer complaint data.

[0052] Then, the original text data in the vertical field of market supervision is preprocessed, which includes data cleaning and data standardization.

[0053] Among them, data cleaning is the first step in data preprocessing, which is mainly to remove noise, missing values, outliers or duplicate values ​​in the data, and to deal with inconsistent data. Data cleaning specifically includes: Removing noise: for example, deleting or replacing invalid or unreasonable data in the data set. Dealing with missing values: Depending on the specific circumstances of the data, you can choose to delete records with missing values, or fill in missing values ​​with methods such as mean, median, mode, etc. Dealing with outliers: You can use the IQR (interquartile range) method or other statistical methods to detect and deal with outliers. Deleting duplicate values: Ensure that each record in the data set is unique. For market regulatory data, extra attention may need to be paid to data compliance and privacy protection to ensure that the processed data does not violate relevant laws and regulations.

[0054] Data standardization includes labeling the cleaned data in a text dialogue data format. In this embodiment, the Prompt+Respons data format, i.e., the text dialogue data format, is adopted, and the questions and answers correspond one to one. The cleaned data is labeled. The standardized data is used for training the large language model.

[0055] Step S2: Create a training data set based on the market supervision text data, train the large language model using supervised fine-tuning, and extract keywords from the market supervision text data through the trained large language model.

[0056] In a preferred embodiment of the present invention, step S2 specifically includes the following steps S21 to S24;

[0057] S21, creating a training data set based on the text conversation data;

[0058] This embodiment adopts the text dialogue (Prompt+Response) mode. According to the multiple rounds of text dialogue data provided, a single question and answer correspond one-to-one as the training input for subsequent SFT, PromptTuning and deltatuning. Among them, PromptTuning is a fine-tuning technology based on "Prompt". The output behavior of the model is adjusted by modifying the "Prompt" in the input text (that is, the context or instruction that guides the model output). This method does not require large-scale adjustment of model parameters, but guides the model to generate the required output by optimizing the "Prompt". By adjusting the "Prompt", the model can be flexibly guided to complete different tasks. Compared with comprehensive fine-tuning, Prompt Tuning only needs to optimize a small number of "Prompt" parameters, which greatly reduces computational complexity and resource consumption. Delta-Tuning is a fine-grained parameter optimization technology that aims to maximize model performance by fine-tuning the parameters of the pre-trained model to adapt to specific tasks.

[0059] S22, designs supervised fine-tuning SFT tasks based on task scenarios in the vertical field of market supervision, and trains the large language model ERNIE Lite.

[0060] Specifically, SFT (Supervised Fine-Tuning) is a supervised learning method that fine-tunes a pre-trained large language model by providing labeled data (Prompt and Response pairs).

[0061] In this embodiment, the large language model uses ERNIE Lite, and the version selected is ERNIE-Lite-8K-0308. A single piece of data supports 8192 tokens. ERNIE-Lite-8K-0308 is the latest version of ERNIE Lite, which optimizes both the effect and performance, and extends support for the LoRA training method. LoRA only updates the parameters of the low-rank part during training, requires fewer computing resources, and the training process is faster, which can reduce the risk of overfitting. LoRA is a fine-tuning technology based on "low-rank decomposition", which approximates the parameters that need to be adjusted by adding a low-rank matrix to the pre-trained model, thereby achieving fine-tuning without significantly changing the overall structure of the model. This method can significantly reduce the number of parameters that need to be optimized and improve the efficiency of fine-tuning. LoRA reduces the number of parameters that need to be optimized through low-rank decomposition, reduces computational complexity and resource consumption. Due to the reduction in the number of parameters, LoRA can accelerate the training process of the model.

[0062] Use Prompt as follows:

[0063] Task 1: Generate text summary

[0064] Please read and analyze the following text carefully. If the text is more than 50 words long, please use no more than 30% of the text (round down if the result is not an integer) to extract and write a summary that can highly summarize the main idea and core information of the text based on your understanding and analysis. If the text is less than or equal to 50 words, please directly describe its core information in a short and concise sentence.

[0065] {content}

[0066] Task 2: Extract keywords

[0067] Next, based on the above text content, select as many keywords (or phrases) as possible that can represent the text theme, core ideas or key information. These keywords should be able to fully and accurately reflect the text content and help readers or search engines quickly understand its essence. Please try to ensure that each keyword is independent, specific and meaningful.

[0068] The training parameter configuration of the large language model ERNIE Lite is shown in Table 1 below

[0069] Table 1

[0070]

[0071]

[0072]

[0073] S23, verify the performance of the large language model through multiple evaluation indicators;

[0074] Specifically, the model performance of the large language model output summary and keywords can be evaluated in the following ways:

[0075] BLEU-4 score: 66.82% is a relatively high score, indicating that the summary generated by the model matches the reference summary at the n-gram (here 4-gram) level. BLEU is a commonly used evaluation indicator for machine translation and text summarization. The higher the score, the better the quality of the generated text.

[0076] ROUGE score:

[0077] ROUGE-1 (81.48%): measures the F1 score based on word overlap between the model summary and the reference summary. A higher score indicates that the model can capture the key words in the reference summary well.

[0078] ROUGE-2 (70.89%): Based on the overlap of word pairs, the model's ability to capture phrases or word groups was further evaluated.

[0079] ROUGE-L (78.79%): The F1 score based on the longest common subsequence (LCS), which takes into account sentence-level similarity, is also high, indicating that the model performs well in generating coherent summaries.

[0080] A decrease in LOSS from 1.2 to 0.2 indicates that the model successfully reduced the prediction error during training, i.e., the model is more accurate in generating the desired output when given an input.

[0081] Perplexity reduction: The reduction from 3.24 to 1.3 is a clear sign of improved model performance. Perplexity is an indicator of the uncertainty of the model about the test data. A lower Perplexity value means that the model is more confident in predicting the test data, that is, the model can generalize better to new data.

[0082] Embedding distance: A value of 0.04 may indicate that the text generated by the model is close to the reference text in the embedding space, which is generally a good sign because it may mean that the model is able to capture similar semantic information. However, the specific interpretation of this metric may depend on the nature of the embedding space and how it is calculated.

[0083] Edit distance: A value of 16.00 may indicate that a certain number of edits (such as insertions, deletions, or substitutions) are required between the generated summary and the reference summary to achieve complete consistency. While this value itself does not directly indicate the quality of the summary (because the summary usually does not need to be exactly the same as the reference summary), it can be used as a quantitative indicator to evaluate the difference between the generated summary and the reference summary.

[0084] In summary, the large language model performs well in processing text data to output summaries and keywords, with high BLEU and ROUGE scores, and significantly reduced LOSS and Perplexity values. At the same time, the Embedding distance and edit distance also provide useful information about the model performance.

[0085] S24, directly extracting keywords from market supervision text data through a trained large language model; or, extracting keywords from the reformulated text after reformulating the market supervision text data through a trained large language model; wherein the reformulation is to simplify the text, adjust grammar, and eliminate ambiguity.

[0086] Specifically, after the large language model is trained, it needs to be deployed to the production environment, provide services through the API interface, and continuously monitor its performance. The trained large language model is used to extract text summaries and keywords from market regulatory text data.

[0087] Step S3, extracting keywords from market supervision text data through natural language processing technology.

[0088] Specifically, the API interface is called to extract keywords, natural language processing is performed on the text data, and keywords and text summaries are extracted.

[0089] The following is a specific example of using natural language processing technology to call the API interface to extract keywords:

[0090] Interface Description: Extract keywords from given text data.

[0091] Request example: HTTP method: POST

[0092] Request URL: https: / / xxx.api.com / nlp / 1.0 / v0.1 / txt_keywords

[0093] URL parameters:

[0094] parameter value access_token Custom access_token

[0095] Header:

[0096] parameter value Content-Type application / json

[0097] Body request example:

[0098] {

[0099] "text":[

[0100] "Because I needed to buy some local specialties, I casually entered this fruit store. When I was about to buy some bananas, I found that there were many snacks on the shelf to the right of the cash register. Then I saw that there was a discount of 100 yuan for 3 packs of beef jerky, so I bought 15 packs and a pack of handmade taro strips, and paid 500 yuan through WeChat. So today, I found that the beef on the outer packaging was abnormal. Then I checked the label and found out that the food label was: hand-torn beef jerky, the product implementation standard is GB / T23586, and the shelf life is 6 months. I called the manufacturer but couldn't get through, so I searched online and found out that GB / T23586 does not apply to beef jerky, so it was the first violation. Then I checked its license number with the State Administration for Market Regulation and found out that the production scope is braised meat products. I request the Market Supervision Administration to strictly check the merchant's purchase vouchers and safeguard the legitimate rights and interests of consumers."

[0101] ],

[0102] "num":5

[0103] }

[0104] Request format: POST method call requires the use of a JSON structure to describe the specific content of a request.

[0105] The entire text content of the Body supports UTF-8 encoding.

[0106] If the text needs to be encoded in UTF-8 format, please add charset=UTF-8 (case sensitive) in the url parameter.

[0107] Request Parameters

[0108]

[0109] Return format: JSON format. The return encoding format is consistent with the input encoding format.

[0110] Return parameter:

[0111]

[0112] Return example:

[0113]

[0114] Step S4, recombining the keywords obtained in various ways, performing natural language processing and synonym recommendation processing, and integrating them into a final label text set.

[0115] Specifically, Figure 3 The schematic diagram of the label generation process provided by the present invention is shown in FIG. Figure 3 The embodiment of the present invention cleans the historical data of each regulatory field separately, extracts keywords from the processed text through multiple language models, and collects the keywords as tags for personnel to use.

[0116] As an optional implementation, in actual operation, the text history data of different market regulatory fields are collected into the original text data data_0, and each text set is preprocessed to obtain data_00 (different regulatory fields each form a separate text data set, and each is processed to obtain the final label set). The preprocessed data is copied into 2 copies, including copy_data_01 and copy_data_02, which are processed by the large language model and natural language processing technology (NLP) respectively.

[0117] Reference Figure 3, copy_data_02 is subjected to natural language processing to extract keywords and text summaries respectively.

[0118] The copy_data_01 text is output using the large language model technology, and the keywords and text summary are extracted by direct output (directly extracting keywords) and re-expressing the text and then outputting it (first re-express and compile it using the language model, and extract keywords from the expression and compilation results).

[0119] Finally, the keyword sets obtained in the three ways are reassembled into the data_0_new keyword combination, and then natural language processing and synonym recommendation processing are performed to integrate them into the final label text set.

[0120] The present invention uses multiple language models to process data, including natural language processing (NLP) technology and text processing technology based on large models. By processing the same data with different models, more diverse and comprehensive keywords and summaries can be extracted, thereby improving the accuracy and coverage of tag generation.

[0121] The present invention combines natural language processing technology (NLP) with a large model trained on data in specific fields of market supervision scenarios, and designs a method for generating multi-field data labels for market supervision. This method can automatically generate data labels based on proprietary data in vertical fields to form a label set for vertical fields of market supervision.

[0122] Figure 4 This is a complete flow chart of the method for generating multi-field data labels for market supervision provided by the present invention. After obtaining the final label text set, the method also includes label comparison and deduplication and label storage. Among them:

[0123] Label comparison and deduplication: Compare the newly generated labels with the original labels, identify duplicate labels, and perform deduplication.

[0124] Specifically, first, extract the information of the original label from the system, including the unique identifier of the label (such as the label hash value) and its associated item information. Then, use a suitable comparison algorithm (such as hash value comparison) to compare the newly generated label with the original label. This step aims to identify the original labels that are repeated or similar to the new label information. Next, for the identified duplicate labels, perform deduplication processing. This may include deleting the newly generated duplicate labels and updating the information of the original labels (if the new information is more accurate or complete).

[0125] Label storage: After confirming that the newly generated labels are not duplicated with the original labels, perform the label storage operation. Ensure that the data information in the label management system is consistent and keep other related systems synchronized.

[0126] The present invention generates a preliminary tag set by aggregating and reprocessing the keywords and abstracts obtained by processing multiple models, and then ensures the uniqueness and accuracy of the tags and avoids the generation of duplicate tags through comparison algorithms and deduplication processing.

[0127] In a preferred embodiment of the present invention, referring to Figure 4 After executing the label storage operation, the method for generating multi-field data labels for market supervision provided by the present invention further includes:

[0128] Provides tag search, tag version control, user access control, identity authentication, and log monitoring functions for tag management.

[0129] The tag management functions include:

[0130] 1.1. Data label search

[0131] Users can search for data tags in their data tag storage space by keywords. The system supports both fuzzy search and precise search.

[0132] 1.1.1. Fuzzy search

[0133] In order to deal with the user's possible incomplete memory or input errors, the system needs to integrate fuzzy search technology and start the fuzzy matching algorithm, which is based on advanced text matching technologies such as Levenshtein distance and Jaccard similarity. Sort by matching degree and display the data label columns associated with the input keywords. When the user enters partial keywords or there are spelling errors, the fuzzy search algorithm can intelligently identify and recommend the most likely data labels.

[0134] 1.1.2. Precise Search

[0135] For users who know the data tag name or specific attributes, the precise search function is more applicable. Use search engine technologies such as Elasticsearch to build an index library for data tags. Provide API interfaces and front-end interfaces based on keywords, tag attributes, and other search methods. Simply enter the complete data tag name or specific attributes, and the system will directly locate the corresponding data tag record through efficient database indexing technology, quickly retrieve and display detailed results.

[0136] 1.2. Tag version control

[0137] 1.1.3. Tag Data Version Control

[0138] Each data change will generate a new data version for easy tracking and rollback. Comparison between versions is supported to understand data changes.

[0139] 1.1.4. Data version control

[0140] A version control mechanism is implemented in the data storage layer, and a new version record is generated each time the data is changed. Version viewing and comparison functions are provided, allowing users to view historical versions and compare the differences between different versions.

[0141] The present invention provides two search modes: fuzzy search and precise search, which facilitate users to quickly locate the required tags. At the same time, it also implements version control of tag data, and each data change will generate a new version record, which is convenient for tracking and rolling back.

[0142] 1.3. User access control requirements

[0143] Implement a strict access control mechanism to ensure that each user can only access their own data tag storage space and cannot access other users' data. This is achieved through technical means such as identity authentication, authorization checking and data encryption.

[0144] 1.4. Access Control and Authentication

[0145] 1.1.5. Authentication

[0146] The system uses encryption algorithms such as bcrypt to hash passwords and compare them with the hash values ​​stored in the database to verify identities.

[0147] 1.1.6. Access token generation

[0148] After the identity verification is passed, the system generates a JWT (JSON Web Token) as an access token. The token contains the user's unique identification, role permissions and other information, and is digitally signed to prevent tampering.

[0149] 1.1.7. Authorization Check

[0150] Each time a user requests access to the data tag storage space, they must carry a JWT token. The system verifies the legitimacy and validity of the token, including checking the signature, expiration time, etc. Based on the role permission information in the token, it determines whether the user has the right to access the requested resource.

[0151] 1.1.8. Fine-grained access control

[0152] In the data tag storage space, attribute-based access control (ABAC) is further implemented to control the read and write permissions of specific data tags based on the user's attributes (such as department, position, etc.).

[0153] 1.5. Logging monitoring requirements

[0154] Record all users' access requests and operation logs for subsequent audit and tracing. Monitor abnormal access behaviors in real time, such as frequent login failures, illegal access attempts, etc., and trigger corresponding security response mechanisms. Through the above detailed technical features and verification mechanisms, the present invention ensures that an independent, secure and strictly authorized data tag storage space can be created for each user during the system initialization phase.

[0155] The present invention ensures that each user can only access his or her own data tag storage space through technical means such as identity authentication, access token generation, and authorization check, thereby ensuring the security and privacy of the data. At the same time, it also records the access requests and operation logs of all users for subsequent audit and tracing.

[0156] In a preferred embodiment of the present invention, referring to Figure 4 After the label storage operation is performed, the method for generating multi-field data labels for market supervision provided by the present invention also includes data association, data sharing and data analysis.

[0157] Data association: Users can reference and apply customized data tags in business processes.

[0158] Specifically, users can reference and apply custom data labels in their business processes, such as adding additional descriptive information to regulatory objects and regulatory content. Through APIs or built-in system functions, users can easily query, filter, and export data records with custom data labels. Users can also perform advanced applications such as data analysis and report generation based on custom data labels.

[0159] Data sharing: Provides internal data label sharing and external link generation and sharing functions to facilitate data sharing and communication; and provides label data sharing monitoring and label data task monitoring functions.

[0160] In this embodiment, the data sharing function specifically includes:

[0161] 1) Internal data label sharing

[0162] In the data tag management interface, users can select the data tags to be shared by checking or searching. In the sharing configuration page, users can enter the user ID or user name to share with, or select a preset sharing group. The selected tag is used to set specific sharing permissions, such as read-only, edit, etc. The validity period of the sharing can be set to ensure the security and timeliness of the data.

[0163] 2) Generate and share external links

[0164] The user shares the data tag with external users, and the specification provides the generation and management of shared links. You can select the data tag to be shared and generate a unique shared link for it. When generating a link, the user can set the access rights and validity period of the link. The generated link can be shared with external users via email, and the shared link can be revoked or updated at any time to ensure data security. In order to ensure the real-time and consistency of shared data, the real-time synchronization and update function of shared data must be supported. If multiple users edit the same data at the same time, the preset conflict resolution strategy must be used to handle these conflicts to ensure data consistency.

[0165] 3) Label data sharing monitoring

[0166] Data sharing monitoring mainly includes data topics, sharing frequency, number of data users, amount of abnormal data, total shared data volume, amount of recently pushed data, etc.

[0167] 4) Label data task monitoring

[0168] The data sharing task monitoring content mainly includes task name, task duration, task end time, total amount of task data, task data scale, current status of the task, etc.

[0169] Data analysis: Conduct label profile construction analysis, label monitoring alarm analysis, label call frequency analysis, label synonym analysis, and label data association analysis on label data.

[0170] Data analysis functions include:

[0171] 1) Label image analysis

[0172] By analyzing the label data of market entities, we can build a portrait of market entities, understand the correlation and relevance between labels revealed by market entities, and help discover potential connections and patterns in the data. Based on historical label data, we can predict future trends and changes and provide data support for decision-making. We use data mining technology to conduct in-depth analysis of label data to discover patterns and associations hidden in it. This can provide valuable insights and decision-making support for enterprises. Label data analysis is widely used in the fields of regulatory object analysis and regulatory focus analysis. Through label data analysis, we can understand the user's regulatory focus, discover regulatory issues, and reveal research hotspots and trends in the regulatory field.

[0173] 2) Tag monitoring alarm analysis

[0174] A tag monitoring and analysis function is set up. When the usage frequency, number of associated resources, etc. of a specific tag reaches the preset threshold, the system automatically triggers an alarm for analysis. This is very useful for monitoring hot spots of complaints, timely discovering abnormal situations, or conducting market trend analysis.

[0175] 3) Tag call frequency analysis

[0176] Count the data resources associated with the tags and the frequency of tag calls and usage to understand which tags have large amounts of associated data and are frequently used, thereby revealing the themes and characteristics of the tag data.

[0177] 4) Tag synonym analysis

[0178] Allow users to define synonyms for tags so that searches can more accurately match relevant resources. This helps resolve search issues caused by inconsistent tag naming.

[0179] 5) Tag data association analysis

[0180] The root data resource calculates the association between tags. For example, a unified resource may be associated with multiple tags. The co-occurrence frequency and co-occurrence probability of tags are calculated to reveal the association rules and potential association networks between tags.

[0181] In a preferred embodiment of the present invention, this embodiment provides a market supervision multi-field data label generation system, and the market supervision multi-field data label generation system at least includes:

[0182] The data preparation module is used to obtain historical text data in multiple fields of market supervision and perform data preprocessing to obtain market supervision text data;

[0183] The model training module is used to create a training data set based on market supervision text data, train the large language model using supervised fine-tuning, and extract keywords from the market supervision text data through the trained large language model;

[0184] A natural language processing module, which is used to extract keywords from market regulatory text data through natural language processing technology;

[0185] The tag generation module is used to recombine the keywords obtained in various ways, perform natural language processing and synonym recommendation processing, and integrate them into the final tag text set.

[0186] Figure 5 A complete structural diagram of the multi-domain data label generation system for market supervision, see Figure 5 The market supervision multi-field data label generation system is used to execute the market supervision multi-field data label generation method provided in the above embodiments. The market supervision multi-field data label generation method has been described in detail in the above embodiments, and this embodiment will not be repeated here.

[0187] Figure 6 The structural block diagram of the electronic device provided by the present invention is as follows: Figure 6As shown, the present invention also provides an electronic device, and the electronic device 600 can be a computing device such as a mobile terminal, a desktop computer, a notebook, a PDA, and a server. The electronic device 600 includes a processor 601 and a memory 602, wherein the memory 602 stores a market supervision multi-field data label generation program 603.

[0188] In some embodiments, the memory 602 may be an internal storage unit of a computer device, such as a hard disk or memory of a computer device. In other embodiments, the memory 602 may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Furthermore, the memory 602 may also include both an internal storage unit of a computer device and an external storage device. The memory 602 is used to store application software and various types of data installed on the computer device, such as program codes for installing the computer device. The memory 602 can also be used to temporarily store data that has been output or is to be output. In one embodiment, when the market supervision multi-field data label generation program 603 is executed by the processor 601, the following steps are implemented:

[0189] Obtain historical text data in multiple fields of market supervision and perform data preprocessing to obtain market supervision text data;

[0190] Create a training dataset based on market supervision text data, train the large language model using supervised fine-tuning, and extract keywords from the market supervision text data through the trained large language model;

[0191] Extract keywords from market regulatory text data through natural language processing technology;

[0192] The keywords obtained in various ways are recombined, natural language processing and synonym recommendation are performed, and integrated into the final label text set.

[0193] In some embodiments, the processor 601 can be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code stored in the memory 602 or process data, such as executing a market supervision multi-field data label generation program.

[0194] This embodiment also provides a computer-readable storage medium, on which a market supervision multi-field data label generation program is stored. When the market supervision multi-field data label generation program is executed by a processor, the following steps are implemented:

[0195] Obtain historical text data in multiple fields of market supervision and perform data preprocessing to obtain market supervision text data;

[0196] Create a training dataset based on market supervision text data, train the large language model using supervised fine-tuning, and extract keywords from the market supervision text data through the trained large language model;

[0197] Extract keywords from market regulatory text data through natural language processing technology;

[0198] The keywords obtained in various ways are recombined, natural language processing and synonym recommendation are performed, and integrated into the final label text set.

[0199] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating multi-field data labels for market supervision, characterized in that: include: Obtain historical text data in multiple fields of market supervision and perform data preprocessing to obtain market supervision text data; Create a training dataset based on market supervision text data, train the large language model using supervised fine-tuning, and extract keywords from the market supervision text data through the trained large language model; Extract keywords from market regulatory text data through natural language processing technology; The keywords obtained in various ways are recombined, natural language processing and synonym recommendation are performed, and integrated into the final label text set.

2. The method for generating multi-field data labels for market supervision according to claim 1 is characterized in that: The historical text data in multiple fields of market supervision include consumer complaint text data, and supervision record data generated in daily supervision activities such as food, medicine, cosmetics, and special equipment.

3. The method for generating multi-field data labels for market supervision according to claim 1 is characterized in that: The data preprocessing includes: Data cleaning: remove noise, missing values, outliers or duplicate values ​​in the data, and ensure data compliance and privacy protection; Data standardization: Use text conversation data format to label the cleaned data.

4. The method for generating multi-domain data labels for market supervision according to claim 3 is characterized in that: The supervised fine-tuning method is used to train the large language model, and the keywords of the market supervision text data are extracted through the trained large language model, specifically including: Create a training dataset based on text conversation data; Design supervised fine-tuning (SFT) tasks based on the task scenarios in the market supervision vertical field and train the large language model ERNIELite; Verify the performance of large language models through multiple evaluation metrics; Directly extract keywords from market supervision text data through a trained large language model; or, after restatement of the market supervision text data through a trained large language model, extract keywords from the restatement text; wherein the restatement is to simplify the text, adjust grammar, and eliminate ambiguity.

5. The method for generating multi-domain data labels for market supervision according to claim 1 is characterized in that: After obtaining the final label text set, the method further includes: Compare the newly generated labels with the original labels, identify duplicate labels, and perform deduplication processing; After confirming that the newly generated label does not overlap with the original label, perform the label storage operation.

6. The method for generating multi-domain data labels for market supervision according to claim 5 is characterized in that: After executing the label storage operation, the method further includes: Provides tag search, tag version control, user access control, identity authentication, and log monitoring functions for tag management.

7. The method for generating multi-domain data labels for market supervision according to claim 5 is characterized in that: After executing the label storage operation, the method further includes: Data association: Users can reference and apply customized data tags in business processes. Data sharing: Provides internal data label sharing and external link generation and sharing functions to facilitate data sharing and communication; and provides label data sharing monitoring and label data task monitoring functions; Data analysis: Conduct label profile construction analysis, label monitoring alarm analysis, label call frequency analysis, label synonym analysis, and label data association analysis on label data.

8. A market supervision multi-field data label generation system applied to the market supervision multi-field data label generation method according to any one of claims 1 to 7, characterized in that: include: The data preparation module is used to obtain historical text data in multiple fields of market supervision and perform data preprocessing to obtain market supervision text data; The model training module is used to create a training data set based on market supervision text data, train the large language model using supervised fine-tuning, and extract keywords from the market supervision text data through the trained large language model; A natural language processing module, which is used to extract keywords from market regulatory text data through natural language processing technology; The tag generation module is used to recombine the keywords obtained in various ways, perform natural language processing and synonym recommendation processing, and integrate them into the final tag text set.

9. An electronic device, It is characterized in that comprising a memory and a processor, wherein: The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the market supervision multi-field data label generation method described in any one of claims 1 to 7 above.

10. A computer-readable storage medium, characterized in that: Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the method for generating multi-field data labels for market supervision as described in any one of claims 1 to 7 above.

Images