Intelligent industry classification and coding method and device in social investigation

Abstract

An intelligent industry classification and coding method and device in social investigation belong to the technical field of natural language processing, machine learning and data classification. The accuracy and efficiency of industry coding are significantly improved by combining hierarchical classification and semantic matching technology. Technical scheme: based on the BERT model, the industry text to be coded is judged to generate the top five candidate classes; for each class, the key word matching, shallow semantic matching and deep semantic model are used to calculate the semantic similarity of the small class by combining the international standard and the self-built corpus; finally, the highest recommended small class coding is output by integrating the semantic weight and business analysis. The device includes a class judgment module, a small class data processing module, a multi-modal semantic matching module and a coding decision module. Advantageous effects: the invention solves the problems of complex industry text classification ambiguity and high artificial dependence by using hierarchical classification strategy and multi-modal semantic fusion, and significantly improves the automation level of large-scale data processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, machine learning and data classification, and is especially suitable for the automatic classification and coding of unstructured industry text in social surveys. In particular, the present application relates to an intelligent industry classification and coding method and device in social surveys. BACKGROUND

[0002] In the field of social surveys and data analysis, industry coding is a key link for converting unstructured text into structured information, and its accuracy and efficiency directly affect the reliability and timeliness of subsequent research. Current industry coding mainly relies on manual operation or automated methods based on machine learning, but both have significant technical bottlenecks. Manual coding requires professional personnel to match text according to industry classification standards, and in the face of emerging industries or composite descriptions, coders need to repeatedly consult materials, which is inefficient and subjective, resulting in significant differences in classification results for the same text by different coders. Although automated methods reduce human intervention, traditional end-to-end models directly predict small class codes, which increases model complexity due to the need to handle the high-dimensional space of thousands of small classes, and semantic overlap between small classes easily causes confusion, making it difficult to achieve high-precision classification.

[0003] Existing automated solutions mostly use a single deep learning model (such as BERT, LSTM) for end-to-end prediction, which has the limitation of over-reliance on global semantic representation, ignoring the importance of explicit features and business rules. For example, for sub-scenarios such as "battery manufacturing" and "battery recycling", relying solely on semantic similarity can easily lead to misjudgment. In addition, some technologies attempt to narrow the search range through hierarchical classification, but their design has obvious defects: the first-level classification often relies on a single result, and once the first-level classification is wrong, the subsequent small class matching will completely deviate from the correct path. For example, "intelligent driving system development" may be incorrectly classified into "manufacturing industry", making it impossible to match the correct code under "technology service industry".

[0004] Analyzing existing hierarchical methods, they mostly use rule templates or single semantic models in the small class matching stage, lacking deep integration of multi-dimensional features. Rule templates can capture explicit terms, but their generalization ability is insufficient in the face of the diversity and dynamic changes of industry terms; while single semantic models can analyze context semantics, they have difficulty in distinguishing subtle differences between business scenarios. Such methods fail to combine multi-modal features such as keyword matching, shallow semantics and deep semantics, resulting in limited classification accuracy for complex text and inability to adapt to the characteristics of different industries.

[0005] In view of the above problems, there is an urgent need for an intelligent coding scheme that balances efficiency and accuracy. SUMMARY

[0006] The main purpose of the present application is to overcome the defects of the prior art, face the social investigation situation, and provide an intelligent industry coding method and device to at least solve the above technical problems existing in the current industry coding. The present application effectively breaks through the bottleneck of the prior art through the innovative design of hierarchical classification and multi-modal semantic fusion: a multi-candidate set is generated in the middle class prediction stage to avoid global deviation caused by single error; keyword matching, shallow semantic analysis and deep semantic analysis are integrated in the small class matching, and dynamic weight adjustment is used to adapt to different industry demands. In addition, the business rule library is introduced to calibrate the semantic matching result twice, which significantly improves the classification robustness of fuzzy text. Compared with the prior art, the present application realizes accurate analysis of complex industry description while maintaining high processing capacity, and provides a more reliable solution for the automatic coding of large-scale social survey data.

[0007] To achieve the above object, the present application provides the following technical scheme:

[0008] The first aspect of the present application is to provide an intelligent industry coding method in social investigation. It includes:

[0009] Based on the pre-trained BERT model, the industry middle class of the input industry text is judged, and the top five middle class candidate sets are generated;

[0010] For each middle class candidate, an international standard corpus and a self-built industry word library are combined to construct a small class semantic matching data set;

[0011] Through the fusion calculation of keyword matching, shallow semantic model and deep semantic model, the semantic similarity weight of industry text and small class data is generated;

[0012] Based on the semantic weight and business rule analysis, the industry small class coding with the highest recommendation degree is output.

[0013] In the above method, the step of generating industry middle class candidate set based on pre-trained BERT model specifically includes:

[0014] According to the international industry classification standard and the self-built industry problem library, a training corpus set of middle class dimension is constructed;

[0015] The BERT model is used to train the corpus set to generate an industry middle class classifier;

[0016] The industry text to be coded is input into the classifier, and the probability distribution of each middle class is output, and the top five middle classes with the highest probability are selected as the candidate set.

[0017] In the above method, the step of constructing a small class semantic matching data set specifically includes:

[0018] For each mid-class candidate, extract the description text and keywords of the corresponding sub-class from the international standard file and the self-built industry vocabulary;

[0019] Preprocess the sub-class text, including word segmentation, stop word removal, and standardized expression;

[0020] Generate a structured sub-class semantic matching dataset containing sub-class codes, text descriptions, and keyword labels.

[0021] In the above method, the implementation of the multi-modal semantic matching algorithm includes the following sub-steps:

[0022] Keyword matching: extract keywords from industry text and sub-class corpus based on TF-IDF algorithm, and calculate intersection weight;

[0023] Shallow semantic matching: convert industry text and sub-class description into TF-IDF vectors, and calculate similarity score through cosine similarity;

[0024] Deep semantic matching: use pre-trained Sentence-BERT model to generate sentence vector, and calculate cosine similarity score at semantic level;

[0025] Fuse the above three matching scores according to the preset weight to generate comprehensive semantic similarity weight.

[0026] In the above method, the step of determining the target code according to semantic weight and business rule analysis includes:

[0027] According to the business scenario requirements, dynamically adjust the weight distribution proportion of keywords, shallow semantics and deep semantics;

[0028] Sort the comprehensive similarity scores of each sub-class candidate in descending order, and select the top three candidate codes;

[0029] Determine the final target industry sub-class code from the candidate codes in combination with artificial review rules or business priority.

[0030] In the above method, the weight distribution proportion is: keyword matching weight 30%, shallow semantic matching weight 20%, and deep semantic matching weight 50%.

[0031] The second aspect of the present application is to provide an intelligent industry classification and coding device, characterized in that it comprises:

[0032] Mid-class classification module: load pre-trained BERT model to perform mid-class probability prediction and candidate selection of industry text;

[0033] Sub-class corpus management module: store international standard and self-built sub-class corpus data, support dynamic update and retrieval;

[0034] Multi-modal matching engine: integrate keyword matching, shallow semantic matching and deep semantic matching algorithm, output comprehensive similarity weight;

[0035] Coding decision module: based on weight fusion result and business rule base, generate and output target industry coding.

[0036] The multi-modal matching engine further comprises:

[0037] Keyword extraction unit, based on TF-IDF and regular expression to extract core terms;

[0038] Shallow semantic calculation unit, calling Scikit-learn library to complete TF-IDF vectorization and similarity calculation;

[0039] Deep semantic modeling unit, loading Sentence-BERT model to generate sentence vector and calculate cosine distance.

[0040] The third aspect of the application is to provide a computer readable storage medium, the storage medium stores a computer program, the computer program is implemented when being executed the intelligent industry classification and coding method in social investigation of the application.

[0041] The fourth aspect of the application is to provide an electronic device, comprising:

[0042] Processor;

[0043] Memory;

[0044] The memory stores instructions executable by the processor, which, when executed, implement the intelligent industry classification and coding method in social investigation of the application.

[0045] Compared with the prior art, the intelligent industry classification and coding method and device in social investigation of the application have the following technical characteristics and beneficial effects:

[0046] (1) Significantly improve the classification accuracy and robustness

[0047] Solve the semantic ambiguity problem: through the hierarchical classification strategy, avoid the global coding deviation caused by single middle class prediction error.

[0048] Multi-modal semantic fusion: combine keyword matching, shallow semantic analysis and deep semantic model, solve the misjudgment problem of traditional single model to complex industry text.

[0049] Business rule secondary calibration: artificial review rule and dynamic weight adjustment mechanism further correct the semantic matching result, enhance the fault tolerance of fuzzy text.

[0050] (2) Greatly improve the coding efficiency

[0051] Reduce dependence on manual: automated processes replace traditional manual piece-by-piece matching, especially for emerging industries and complex descriptions, reducing subjective differences and time costs.

[0052] Layered complexity reduction: two-stage processing of medium class first and small class second, avoiding the performance bottleneck of end-to-end model directly processing high-dimensional small class space.

[0053] (3) Enhance industry adaptability

[0054] Dynamic corpus support: continuously incorporate emerging terms into self-built industry word library, solving the problem of insufficient generalization of traditional rule templates.

[0055] Flexible weight configuration: dynamically adjust multi-modal weights according to business scenarios to adapt to different industry characteristics.

[0056] Manual review interface: results with confidence less than 80% are automatically pushed to manual review to balance automation and reliability.

[0057] (4) Standardization and scalability

[0058] International standard compatibility: corpus construction is based on international industry classification standards to ensure coding standardization.

[0059] Modular device design: medium class classification, multi-modal matching engine and other modules support independent upgrade, facilitating technology iteration. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the present application will be described in detail below in combination with the drawings and detailed embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained from these drawings without creative labor. Among them:

[0061] Fig. 1 A flowchart of an intelligent industry classification and coding method in social investigation provided by the embodiment of the present application;

[0062] Fig. 2 A multi-modal semantic matching architecture diagram provided by the embodiment of the present application;

[0063] Fig. 3 A dynamic weight adjustment and rule library coupling diagram provided by the embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application. The present application is described below in combination with the drawings Figs. 1-3 The intelligent industry classification and coding method and device in social investigation are further described.

[0065] Example 1: Method example

[0066] The present embodiment provides an intelligent industry coding method in social investigation, as shown in the following table, the embodiment includes the following steps: Fig. 1

[0067] Step 1, data collection and arrangement: a large amount of industry text data is obtained from social investigation, which is generated by enterprises or individuals when filling in industry information, and has the characteristics of diversification and unstructured. The collected original data is preliminarily screened to remove obviously incorrect, repeated or irrelevant data records. For example, records containing random codes, pure advertising content irrelevant to the industry are removed.

[0068] Step 2, middle class data preparation: according to the national standard file, the industry middle class related data is comprehensively cleaned. The accuracy and completeness of the data are carefully checked, the incorrect information is corrected, and the missing content is supplemented. At the same time, combined with the industry vocabulary maintained for a long time, the emerging words, professional terms and aliases in the industry are supplemented to the data. For example, for the “information technology” industry, emerging terms such as “big data” and “artificial intelligence” and aliases such as “AI” and “big data technology” are included in the vocabulary. After the arrangement, a rich and accurate middle class dimension corpus data is constructed, for example:

[0069] Table 1

[0070]

[0071] Step 3, model training: using the prepared middle class dimension corpus data, the BERT model is trained. In the training process, the training parameters such as learning rate and training number of rounds are reasonably set to optimize the performance of the model. Through multiple iterations of training, the model is fully trained to learn the characteristics and patterns of different industry middle class texts, so that it can accurately identify and distinguish various industry middle classes. Finally, a mature trained model that can be used for industry middle class judgment is obtained.

[0072] ​Step 4, middle class judgment: Take the example of "a company specializing in the research and development and production of new solar panels", input this text into the trained BERT model. The model analyzes and calculates the input text, and outputs the probability distribution of each middle class. From the output results, select the top five middle classes with the highest probability as the candidate middle class set. In this example, the top five middle classes that may be obtained include "manufacturing industry", "technology service industry", "electricity, heat, gas and water production and supply industry", "wholesale and retail industry", "scientific research and technology service industry", and the like, as shown in the following example:

[0073] Table 2

[0074]

[0075] Step 5, small class data processing: For each middle class candidate, such as "manufacturing industry" and "technology service industry", extract the detailed information of the corresponding small classes from the national standard files and the industry data maintained by itself. For "manufacturing industry", extract the description text and keywords of small classes such as "battery manufacturing", "new energy equipment manufacturing", "electrical machinery and equipment manufacturing", etc. For "technology service industry", extract the relevant data of small classes such as "technology research and development services", "technology promotion services", "information technology consulting services", etc.

[0076] Preprocess these small class texts, including word segmentation, splitting sentences into individual words; remove stop words such as "of", "has been", "in", etc. without actual meaning; standardize the terminology expression, standardize different expressions but the same meaning of the terminology. For example, "solar panels" and "photovoltaic panels" are unified as "solar panels". After processing, structured small class semantic matching data sets are generated, each containing small class code, detailed text description and keyword label, as shown in the following example:

[0077] Table 3

[0078]

[0079] Step 6, semantic judgment:

[0080] Step 6.1, keyword matching:

[0081] Based on the TF-IDF algorithm, extract the keywords of the enterprise description text and the small class corpus respectively. For "a company specializing in the research and development and production of new solar panels", extract the keywords "solar panels", "research and development", "production", "new type", etc. Then calculate the intersection weight of these keywords and the keywords of each small class corpus. If the "new energy equipment manufacturing" small class corpus contains keywords such as "solar panels" and "production", the intersection weight will increase accordingly.

[0082] Step 6.2, Shallow Semantic Matching:

[0083] Convert the enterprise description text and the subcategory description into TF-IDF vectors, and calculate the similarity score between the two through cosine similarity. For example, the cosine similarity between the "new energy equipment manufacturing" subcategory description and the enterprise text in the TF-IDF vector space is high, indicating that they are similar in shallow semantics, i.e., they have certain relevance from the perspective of the distribution and frequency of words.

[0084] Step 6.3, Deep Semantic Matching:

[0085] Use the pre-trained Sentence-BERT model to generate sentence vectors for the enterprise description text and the subcategory description, and then calculate the cosine similarity score at the semantic level. This model can understand the deep semantic relationships of the text, such as recognizing the semantic similarity between "solar cell panel" and "photovoltaic cell panel". The score obtained in this way can more accurately reflect the semantic association between texts.

[0086] Step 6.4, Integrated Weight Calculation:

[0087] According to the preset weights, the three matching scores are fused. Assuming that the keyword matching weight is 30%, the shallow semantic matching weight is 20%, and the deep semantic matching weight is 50%, the integrated semantic similarity weight of each subcategory and the enterprise description text is calculated through weighted calculation. For example, the keyword matching score of the "new energy equipment manufacturing" subcategory is 80 points, the shallow semantic matching score is 70 points, and the deep semantic matching score is 90 points, so the integrated semantic similarity weight is 80 x 30% + 70 x 20% + 90 x 50% = 83 points.

[0088] The final result is as follows:

[0089] Table 4

[0090]

[0091] Step 7, Subcategory Judgment:

[0092] The comprehensive weight obtained by combining the semantic judgment and the business rule analysis is used to sort the small class candidates. Considering that the enterprise explicitly mentions "research and development and production", the business is more focused on the manufacturing of actual products. The "new energy equipment manufacturing" small class not only has a higher score in semantic matching, but also is consistent with the business focus of the enterprise. Therefore, the "new energy equipment manufacturing" small class data is ranked in the front, and the corresponding industry small class code is determined as the closest small class data, and the corresponding industry small class code is output. In this example, the "new energy equipment manufacturing" corresponds to the industry small class code "3865" (assuming that the code represents this small class in the coding system), and the classification and coding of the industry text of the enterprise are completed.

[0093] Embodiment 2: Device embodiment

[0094] This embodiment provides an intelligent industry coding device in social investigation, as shown in Fig. 3 The device comprises the following modules:

[0095] A text preprocessing module is used to clean the input text by using regular expression matching and Jieba word segmentation tool, remove redundant characters, and generate standardized text;

[0096] A middle class classification module is loaded with an NVIDIA T4 GPU and loaded with a pre-trained BERT model to output Top5 middle class candidates;

[0097] A multi-modal matching engine module uses TF-IDF algorithm to extract keywords, uses deep semantic model Sentence-BERT to perform keyword, shallow semantic and deep semantic matching, and generates a comprehensive score;

[0098] A dynamic decision module adjusts the weight according to the business rules, outputs the final code, and supports API dynamic update;

[0099] An artificial review interface module pushes the results with a confidence of less than 80% to the artificial review queue.

[0100] Embodiment 3: Computer program product embodiment

[0101] This embodiment is a computer program product that can execute the above-mentioned method embodiment, which contains program instructions that can execute the steps in the above-mentioned method embodiment when running. The computer program product can use one or more program languages to write program codes, such as C, C++, Java, Python, etc. The program codes can be executed entirely or partially, and can be executed on a local computer device, a remote computing device or a server.

[0102] Embodiment 4: Computer readable storage medium embodiment

[0103] The embodiment is a computer readable storage medium, which stores program instructions capable of executing the steps in the above method embodiments when running. The computer readable storage medium can adopt one readable medium or a combination of multiple readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable signal medium includes but is not limited to infrared rays, magnetism, semiconductor, etc. The readable storage medium includes but is not limited to a hard disk, a portable disk, a card type memory (such as an SD card, etc.), a read-only memory (ROM), a random access memory (RAM), etc.

[0104] The above embodiment is only a preferred embodiment of the intelligent industry coding method and device in the social investigation of the present application, and does not limit the form and scope of the present application. Any disassembly, recombination, equivalent structure and equivalent process made by using the content of the specification and drawings of the present application are also included in the patent protection scope of the present application.

[0105] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can make equivalent replacement or change according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be included in the protection scope of the present application.

Claims

1. A method for classifying and coding intelligent industries in social surveys, characterized in that, In a social survey context, the steps are as follows: S1. Based on the pre-trained BERT model, class determination is performed on the input industry text to generate a set of the top five class candidates. S2. For each candidate class, construct a semantic matching dataset for the minor class by combining international standard corpora and industry thesaurus; S3. Generate semantic similarity weights between industry text and sub-category data through keyword matching, fusion calculation of shallow semantic model and deep semantic model; In step S3, the implementation of the multimodal semantic matching algorithm includes the following sub-steps: Keyword matching: Extract keywords from industry texts and subcategory corpora based on the TF-IDF algorithm, and calculate the intersection weight; Shallow semantic matching: Convert industry text and subcategory descriptions into TF-IDF vectors, and calculate similarity scores using cosine similarity; Deep semantic matching: Generate sentence vectors using a pre-trained Sentence-BERT model and calculate the cosine similarity score at the semantic level; The three matching scores are combined according to preset weights to generate a comprehensive semantic similarity weight; S4. Based on semantic weight and business rule analysis, sort and output the industry sub-category codes with the highest recommendation degree; Step S4, which involves determining the target code based on semantic weights and business rules, includes: S41. Dynamically adjust the weight distribution ratio of keywords, shallow semantics, and deep semantics according to business scenario requirements; S42. Sort the comprehensive similarity scores of each sub-category candidate in descending order and select the top three candidate codes; S43. Combine manual review rules or business priorities to determine the final target industry sub-category code from the candidate codes.

2. The intelligent industry classification and coding method in social surveys according to claim 1, characterized in that, The specific steps of step S1 are as follows: S11. Based on international industry classification standards and industry question databases, construct a training corpus set for the middle category dimension; S12. The BERT model is used to train the corpus to generate an industry-specific classifier; S13. Input the industry text to be encoded into the classifier, output the probability distribution of each class, and select the top five classes with the highest probabilities as the candidate set.

3. The intelligent industry classification and coding method in social surveys according to claim 1, characterized in that, Step S2, specifically the step of constructing the small-class semantic matching dataset, includes: S21. For each candidate subclass, extract the description text and keywords of the corresponding subclass from international standard documents and industry thesaurus; S22. Preprocess the sub-class texts, including word segmentation, stop word removal, and standardized expression; S23. Generate a structured subclass semantic matching dataset, including subclass codes, text descriptions, and keyword labels.

4. The intelligent industry classification and coding method in social surveys according to claim 1, characterized in that, The weighting distribution is as follows: keyword matching weight 30%, shallow semantic matching weight 20%, and deep semantic matching weight 50%.

5. A smart industry classification and coding device for social surveys, used to implement the method described in any one of claims 1-4, characterized in that, include: Mid-class classification module: Loads pre-trained BERT models and performs mid-class probability prediction and candidate selection for industry texts; Subcategory Corpus Management Module: Stores international standard and self-built subcategory corpus data, and supports dynamic updates and retrieval; Multimodal matching engine module: integrates keyword matching, shallow semantic matching and deep semantic matching algorithms, and outputs comprehensive similarity weights; Coding Decision Module: Based on the weighted fusion results and the business rule base, it generates and outputs the target industry code.

6. The intelligent industry classification and coding device for social surveys according to claim 5, characterized in that, The multimodal matching engine module includes: The keyword extraction unit extracts core terms based on TF-IDF and regular expressions. The shallow semantic computation unit calls the Scikit-learn library to complete TF-IDF vectorization and similarity calculation; The deep semantic modeling unit loads the Sentence-BERT model to generate sentence vectors and calculates the cosine distance.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed, implements the intelligent industry classification and coding method in the social survey as described in any one of claims 1-4.

8. An electronic device, characterized in that, include: processor; Memory; The memory stores instructions that can be executed by a processor, which, when executed, implement the intelligent industry classification and coding method in the social survey as described in any one of claims 1-4.

Images