High-performance automobile industry word right perception re-method

Abstract

The invention belongs to the technical field of high-performance perceptual automobiles, and discloses a high-performance perceptual automobile industry word right re-method. The method comprises the following specific steps: S1, planning a noise check strategy; s2, sorting high-frequency vocabulary entity categories; s3, single character entity category special examination is carried out; s4, screening and processing missed label data; s5, determining a data enhancement thought; s6, carrying out word segmentation sequence rearrangement operation; and S7, confirming a data set expansion result. The scheme of the invention has multiple advantages of providing a unique data set expansion and adjustment method, processing noise, enhancing data and creating a high-quality data set according to actual requirements, increasing data volume, ensuring accurate classification and improving F1 score by 4% in combination with scenes, and improving the classification accuracy by 4% on the basis of an automobile industry application language. And the entities are accurately classified and labeled according to an accurate and personalized principle, so that the development of related tasks can be effectively assisted.

Description

Technical Field

[0001] The present invention belongs to the technical field of high-performance perception automobiles, and specifically relates to a high-performance perception automobile industry word weight re-authorization method. Background Art

[0002] The popularity of the Internet has made search an essential tool. Accurately obtaining information from massive data is a challenge for search engines. Named entity recognition can help them understand queries, filter out irrelevant information, and improve query understanding and other performance. There are many common named entity recognition technologies. Although deep learning and pre-trained models are excellent, they require a lot of computing resources and labeled data, which makes data set construction a difficult point in the project.

[0003] The NER model based on deep learning requires a large amount of annotated data support. NER datasets in multiple languages, biomedicine and other fields are commonly used, but there are fewer suitable for Chinese. Commonly used large-scale datasets include CLUENER, ResumeNER and open datasets in the e-commerce field, etc., covering multiple entity categories. Manually annotated datasets are manually labeled data of various types for machine learning, etc., and are often used for supervised learning. The establishment of NER datasets includes processes such as data collection, cleaning, manual annotation and quality assessment. The annotator marks the named entities in the text in units of characters or words according to the defined entity categories, thereby constructing the dataset.

[0004] Most of the existing data sets are in English, and there are gaps in Chinese data sets and corpora. There are few NER data sets suitable for Chinese and they lack universal applicability, resulting in poor application effects. Manually annotated data sets have disadvantages such as high cost and annotation noise. Due to the huge amount of data, large-scale annotation is expensive and time-consuming. Moreover, since the NER task involves subjective judgment, different annotators are prone to produce inconsistent annotation results, misjudgments and missed judgments, resulting in noise affecting model performance. Summary of the invention

[0005] The purpose of the present invention is to provide a high-performance automotive industry-aware word re-weighting method to solve the problems raised in the above-mentioned background technology.

[0006] In order to achieve the above-mentioned purpose, the present invention provides the following technical solutions: The specific steps of a high-performance perception automobile industry word weight re-authorization method are as follows:

[0007] S1: Noise check strategy planning: clarify the overall direction and key focus of noise check, including frequency analysis of high-frequency lexical entity category distribution, single-word entity category inspection, and planning for missing data processing;

[0008] S2: High-frequency vocabulary entity category sorting: frequency analysis of high-frequency vocabulary categories, identification of anomalies and correction of wrong categories, to ensure that the text vocabulary entities are unified and correct;

[0009] S3: Special review of single-word entity categories: Focus on single-word entity categories, analyze the differences between automobile and daily words, and conduct a secondary check on "Tang" and "Song" to prevent misjudgment of categories;

[0010] S4: Screening and processing of missing data: Create a word segmentation table based on the annotations, conduct a detailed search in the data set, find missing query, re-annotate or delete data as needed to ensure the accuracy of the data set;

[0011] S5: The idea of ​​data enhancement is to rearrange the word order of the text and double the size without changing the semantics, so as to add diversity to the data set and enhance the data richness.

[0012] S6: Word segmentation order rearrangement operation: For the above dataset, the word segmentations in the text are rigorously rearranged in order under the constraints of semantically unchanged rules, starting the dataset expansion process;

[0013] S7: Confirmation of data set expansion results: After rearrangement, the data set is checked in detail to verify the scale and diversity, successfully completing the entire data processing process.

[0014] Preferably, the specific steps of noise inspection strategy planning in S1 are as follows:

[0015] Step 1: Determine the direction of high-frequency vocabulary inspection: Make it clear that we need to focus on the high-frequency words in the query, analyze the distribution frequency of the corresponding named entity categories, sort out the vocabulary performance in different frequency ranges, and prepare for the subsequent identification of abnormal situations;

[0016] Step 2: Focus on the entity category verification of single words: Focus on the entity category of single words, carefully distinguish the differences between special words and common words in daily life in specific scenarios in the automotive field, and focus on checking single words with special meanings for car models, such as "Tang" and "Song", to ensure accurate classification of single word entities;

[0017] Step 3: Plan a solution for missing data: Build a word segmentation table based on the labeling results, and use the word segmentation table to systematically search for missing queries in the data set text. At the same time, plan in advance whether to re-label or delete related data for missing data to ensure the completeness and accuracy of the data set labeling.

[0018] Preferably, the special review of single-word entity categories in S2 refers to: for high-frequency words in the Query, accurately calculating the distribution frequency of their corresponding named entity categories, carefully identifying minority class anomalies, determining abnormal Targets caused by human errors, and preparing for corrections to ensure that the entity categories of the same word in different texts are unified.

[0019] Preferably, the special review of single-word entity categories in S3 refers to: focusing on the entity categories of single words, deeply analyzing the differences between special words in the automotive field and commonly used words in life, and conducting a secondary strict inspection of typical classified words such as "Tang" and "Song" to prevent misjudgment of entity categories.

[0020] Preferably, the missing data screening process in S4 refers to: constructing a word segmentation table based on the labeling results, carefully checking the words in the word segmentation table one by one in the data set text, trying to find out the missing queries, and then taking the operation of re-labeling or deleting related data to ensure the accuracy of the data set.

[0021] Preferably, the data enhancement idea determination in S5 refers to: determining to expand the data set by reordering the text segmentations in the data set, and expanding the data set to twice the original size without changing the original semantics to improve the diversity of the data set.

[0022] Preferably, the specific steps of the word segmentation order rearrangement operation in S6 are as follows:

[0023] Step 1: Establish the principle of data set expansion: clearly take the existing data set as the object, uphold a rigorous and responsible attitude, take semantic constancy as the core rule, and lay the foundation for the subsequent reconstruction of word segmentation order;

[0024] Step 2: Perform word segmentation order reconstruction: Based on the established principles, carefully design a plan to reconstruct the order of text segmentation, use clever permutation and combination techniques, and carefully adjust the word segmentation order to ensure that the semantics is not damaged;

[0025] Step 3: Start the data set expansion action: After completing the word segmentation order reconstruction, promote the data set expansion process in an orderly manner according to the plan to enrich the data resources, thereby improving the overall quality and availability of the data and achieving the ultimate goal of expanding the data set.

[0026] Preferably, the confirmation of the data set expansion result in S7 refers to a comprehensive check of the expanded data set after the word segmentation order is rearranged to ensure that it meets the expected double scale and the data diversity is effectively improved, thereby completing the entire data processing flow.

[0027] The beneficial effects of the present invention are as follows:

[0028] 1. Through this scheme, the present invention proposes a unique method for expanding and adjusting data sets. For existing manually annotated data sets, according to the specific needs of actual scenarios, the present invention rigorously carries out noise processing work and cleverly implements data enhancement measures, and is committed to creating a data set with higher quality and better results.

[0029] 2. Through this solution, the present invention not only achieves an increase in data volume, but also effectively guarantees the accuracy of data entity classification, and closely combines it with real application scenarios. With these advantages, the F1 score is ultimately improved by 4% compared to the original data set, with significant results.

[0030] 3. The present invention has unique advantages through this solution. It takes the actual application language of the automotive industry as its basis. When carrying out entity classification and labeling work, it adheres to the principles of precision and personalization and can accurately classify and label various entities. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flow chart of a high-performance perception automobile industry word weight re-authorization method of the present invention;

[0032] Figure 2 Constructing a flow chart for the dataset of the present invention;

[0033] Figure 3 This is a data enhancement flow chart of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] like Figures 1 to 3 As shown, the specific steps of the embodiment of the present invention providing a high-performance perception automobile industry word weight re-authorization method are as follows:

[0036] S1: Noise check strategy planning: clarify the overall direction and key focus of noise check, including frequency analysis of high-frequency lexical entity category distribution, single-word entity category inspection, and planning for missing data processing;

[0037] S2: High-frequency vocabulary entity category sorting: frequency analysis of high-frequency vocabulary categories, identification of anomalies and correction of wrong categories, to ensure that the text vocabulary entities are unified and correct;

[0038] S3: Special review of single-word entity categories: Focus on single-word entity categories, analyze the differences between automobile and daily words, and conduct a secondary check on "Tang" and "Song" to prevent misjudgment of categories;

[0039] S4: Screening and processing of missing data: Create a word segmentation table based on the annotations, conduct a detailed search in the data set, find missing query, re-annotate or delete data as needed to ensure the accuracy of the data set;

[0040] S5: The idea of ​​data enhancement is to rearrange the word order of the text and double the size without changing the semantics, so as to add diversity to the data set and enhance the data richness.

[0041] S6: Word segmentation order rearrangement operation: For the above dataset, the word segmentations in the text are rigorously rearranged in order under the constraints of semantically unchanged rules, starting the dataset expansion process;

[0042] S7: Confirmation of data set expansion results: After rearrangement, the data set is checked in detail to verify the scale and diversity, successfully completing the entire data processing process.

[0043] NER: Named Entity Recognition Query: text segmentation, named entity object Target: named entity category F1Score: the harmonic mean of precision and recall.

[0044] The specific steps of noise inspection strategy planning in S1 are as follows:

[0045] Step 1: Determine the direction of high-frequency vocabulary inspection: Make it clear that we need to focus on the high-frequency words in the query, analyze the distribution frequency of the corresponding named entity categories, sort out the vocabulary performance in different frequency ranges, and prepare for the subsequent identification of abnormal situations;

[0046] Step 2: Focus on the entity category verification of single words: Focus on the entity category of a single word, carefully distinguish the differences between special words and common words in daily life in specific scenarios such as the automotive field, and focus on checking single words with special meanings such as car model references, such as "Tang" and "Song", to ensure accurate classification of single word entities;

[0047] Step 3: Plan a solution for missing data: Build a word segmentation table based on the labeling results, and use the word segmentation table to systematically search for missing queries in the data set text. At the same time, plan in advance whether to re-label or delete related data for missing data to ensure the completeness and accuracy of the data set labeling.

[0048] First, lock in the high-frequency words in the query, analyze the frequency distribution of named entity categories, lay the foundation for anomaly recognition, focus on single-word entities, distinguish the differences between specific and common words, check for special words, build tables based on annotations to check for missed queries, plan strategies for re-annotating or deleting data, and ensure that the dataset is accurately annotated.

[0049] The special review of single-word entity categories in S2 refers to: for high-frequency words in the query, accurately calculate the distribution frequency of their corresponding named entity categories, carefully identify minority class anomalies, determine the abnormal Target caused by human errors, and prepare for corrections to ensure the unity of the entity categories of the same word in different texts.

[0050] Special review of single-word entity categories, that is, for high-frequency Query words, accurately calculate the distribution frequency of their named entity categories, identify anomalies, correct errors, and ensure the uniformity of vocabulary entity categories.

[0051] Among them, the special review of single-word entity categories in S3 refers to: focusing on the entity categories of single words, deeply analyzing the differences between special words in the automotive field and commonly used words in life, and conducting secondary strict inspections on typical classified words such as "Tang" and "Song" to prevent misjudgment of entity categories.

[0052] The special review of single-word entity categories focuses on the entity categories of single words, deeply explores the differences between the automotive field and commonly used words in life, and conducts a second strict inspection of typical "Tang" and "Song" characters to avoid misjudgment of entity categories.

[0053] The missing data screening process in S4 refers to: constructing a word segmentation table based on the labeling results, carefully checking the words in the word segmentation table one by one in the data set text, trying to find out the missing queries, and then taking the operation of re-labeling or deleting the relevant data to ensure the accuracy of the data set.

[0054] After building a word segmentation table based on the annotation results, the words in the table are carefully checked in the data set text to find out the missing queries, and the corresponding operations are performed to ensure the accuracy of the data set.

[0055] The data enhancement idea in S5 refers to: determining to expand the data set by reordering the text word segments in the data set, and expanding the data set to twice the original size without changing the original semantics, so as to improve the diversity of the data set.

[0056] This means that the dataset is expanded by rearranging the order of the text segments in the dataset, doubling its original size while ensuring that the semantics remain unchanged, thereby improving diversity.

[0057] The specific steps of the word segmentation order rearrangement operation in S6 are as follows:

[0058] Step 1: Establish the principle of data set expansion: clearly take the existing data set as the object, uphold a rigorous and responsible attitude, take semantic constancy as the core rule, and lay the foundation for the subsequent reconstruction of word segmentation order;

[0059] Step 2: Perform word segmentation order reconstruction: Based on the established principles, carefully design a plan to reconstruct the order of text segmentation, use clever permutation and combination techniques, and carefully adjust the word segmentation order to ensure that the semantics is not damaged;

[0060] Step 3: Start the data set expansion action: After completing the word segmentation order reconstruction, promote the data set expansion process in an orderly manner according to the plan to enrich the data resources, thereby improving the overall quality and availability of the data and achieving the ultimate goal of expanding the data set.

[0061] First, establish the expansion principle, take the established set as the object, and adhere to the semantic invariance rule; second, perform the reconstruction operation, carefully adjust the order to preserve the semantics; third, start the expansion action, enrich the resources and improve the data quality and availability.

[0062] The confirmation of the data set expansion result in S7 refers to a comprehensive check of the expanded data set after the word segmentation order is rearranged to ensure that it meets the expected double scale and the data diversity is effectively improved, thus completing the entire data processing flow.

[0063] The results of the dataset expansion are confirmed, that is, the expanded set is fully checked after rearrangement to verify that it has doubled in size and improved in diversity, thus completing the entire data processing process.

[0064] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0065] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A high-performance automotive industry-aware word weight re-approach, characterized by: The specific steps of this high-performance perception automobile industry word weight re-authorization method are as follows: S1: Noise check strategy planning: clarify the overall direction and key focus of noise check, including frequency analysis of high-frequency lexical entity category distribution, single-word entity category inspection, and planning for missing data processing; S2: High-frequency vocabulary entity category sorting: frequency analysis of high-frequency vocabulary categories, identification of anomalies and correction of wrong categories, to ensure that the text vocabulary entities are unified and correct; S3: Special review of single-word entity categories: Focus on single-word entity categories, analyze the differences between automobile and daily words, and conduct a secondary check on "Tang" and "Song" to prevent misclassification of categories; S4: Screening and processing of missing data: Create a word segmentation table based on the annotations, conduct a detailed search in the data set, find missing query, re-annotate or delete data as needed to ensure the accuracy of the data set; S5: The idea of ​​data enhancement is to rearrange the word order of the text and double the size without changing the semantics, so as to add diversity to the data set and enhance the data richness. S6: Word segmentation order rearrangement operation: For the above dataset, the word segmentations in the text are rigorously rearranged in order under the constraints of semantically unchanged rules, starting the dataset expansion process; S7: Confirmation of data set expansion results: After rearrangement, the data set is checked in detail to verify the scale and diversity, successfully completing the entire data processing process.

2. According to claim 1, a high-performance perception automobile industry word weight re-authorization method is characterized by: The specific steps of noise inspection strategy planning in S1 are as follows: Step 1: Determine the direction of high-frequency vocabulary inspection: Make it clear that we need to focus on the high-frequency words in the query, analyze the distribution frequency of the corresponding named entity categories, sort out the vocabulary performance in different frequency ranges, and prepare for the subsequent identification of abnormal situations; Step 2: Focus on the entity category verification of single words: Focus on the entity category of single words, carefully distinguish the differences between special words and common words in daily life in specific scenarios in the automotive field, and focus on checking single words with special meanings for car models, such as "Tang" and "Song", to ensure accurate classification of single word entities; Step 3: Plan a solution for missing data: Build a word segmentation table based on the labeling results, and use the word segmentation table to systematically search for missing queries in the data set text. At the same time, plan in advance whether to re-label or delete related data for missing data to ensure the completeness and accuracy of the data set labeling.

3. The high-performance automotive industry-aware word weight re-authorization method according to claim 1 is characterized by: The special review of single-word entity categories in S2 refers to: for high-frequency words in the query, accurately calculate the distribution frequency of their corresponding named entity categories, carefully identify minority class anomalies, determine the abnormal Target caused by human errors, and prepare for corrections to ensure the consistency of entity categories of the same word in different texts.

4. The high-performance automotive industry-aware word weight re-authorization method according to claim 1 is characterized by: The special review of single-word entity categories in S3 refers to: focusing on the entity categories of single words, deeply analyzing the differences between special words in the automotive field and commonly used words in life, and conducting a secondary strict inspection of typical classified words such as "Tang" and "Song" to prevent misjudgment of entity categories.

5. The high-performance automotive industry-aware word re-weighting method according to claim 1 is characterized by: The missing data screening process in S4 refers to: constructing a word segmentation table based on the labeling results, carefully checking the words in the word segmentation table one by one in the data set text, trying to find out the missing queries, and then taking the operation of re-labeling or deleting related data to ensure the accuracy of the data set.

6. The high-performance automotive industry-aware word re-weighting method according to claim 1, characterized in that: The data enhancement idea determination in S5 refers to: determining to expand the data set by re-arranging the order of text segmentations in the data set, and expanding the data set to twice the original size without changing the original semantics, so as to improve the diversity of the data set.

7. The high-performance automotive industry-aware word weight re-authorization method according to claim 1, characterized in that: The specific steps of the word segmentation order rearrangement operation in S6 are as follows: Step 1: Establish the principle of data set expansion: clearly take the existing data set as the object, uphold a rigorous and responsible attitude, take semantic constancy as the core rule, and lay the foundation for the subsequent reconstruction of word segmentation order; Step 2: Perform word segmentation order reconstruction: Based on the established principles, carefully design a plan to reconstruct the order of text segmentation, use clever permutation and combination techniques, and carefully adjust the word segmentation order to ensure that the semantics is not damaged; Step 3: Start the data set expansion action: After completing the word segmentation order reconstruction, promote the data set expansion process in an orderly manner according to the plan to enrich the data resources, thereby improving the overall quality and availability of the data and achieving the ultimate goal of expanding the data set.

8. The high-performance automotive industry-aware word re-weighting method according to claim 1 is characterized by: The confirmation of the data set expansion result in S7 refers to a comprehensive check of the expanded data set after the word segmentation order is rearranged to ensure that it meets the expected double scale and the data diversity is effectively improved, thus completing the entire data processing flow.

Images