Method, device and equipment for constructing dynamic expansion word bank and medium

Abstract

The invention relates to the technical field of passenger service, and discloses a construction method and device of a dynamic extension word bank, equipment and a medium. The method comprises the following steps: acquiring multi-source heterogeneous original knowledge data related to civil aviation passenger service, and converting the multi-source heterogeneous original knowledge data into a standard format text; processing the unstructured data in the standard format text based on the adaptive word segmentation model and the stop word list to obtain a plurality of candidate words; performing multi-dimensional corpus feature quantitative analysis on each candidate word to determine candidate keywords; and performing similarity calculation based on the entity word segmentation and the candidate keywords, determining effective candidate keywords, and adding the effective candidate keywords into the dynamic expansion word bank. By means of the method and device, the technical problems that in the prior art, a word bank used by a civil aviation service system is usually based on a static vocabulary or depends on manual compiling and updating, the static word bank cannot be updated in real time, the characteristics of the civil aviation field cannot be flexibly handled, and the maintenance cost is high due to manual updating and maintenance are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of passenger service, and in particular to a method and device for constructing a dynamically expanded vocabulary, and a medium. BACKGROUND

[0002] Currently, the departure system includes flight processing, passenger service, baggage management, boarding processing and other business scenarios, involving a large number of professional terms and dynamically changing business scenarios. In order to ensure that various civil aviation service systems can accurately and efficiently process these complex information, a precise and dynamically updated vocabulary needs to be constructed so that the system can understand and process the queries and requests of passengers.

[0003] In the prior art, the vocabulary used by many civil aviation service systems is often based on a static vocabulary table or relies on manual preparation and updating. This traditional method has the following problems: 1. Unable to update in real time: As the business continues to change and develop, new terms and vocabulary continue to emerge, and static vocabulary cannot reflect these changes in a timely manner; 2. Poor domain characteristics: Traditional vocabulary construction methods are usually based on universally preset rules and cannot flexibly adjust and handle the characteristics of the civil aviation field; 3. High maintenance cost: Many system vocabularies still rely on manual intervention for updating and maintenance, increasing the maintenance cost and easily leading to outdated vocabulary content. SUMMARY

[0004] The present application provides a method and device for constructing a dynamically expanded vocabulary, which can solve the technical problems of the existing civil aviation service system vocabulary that cannot be updated in real time, poor domain characteristics, and high maintenance cost.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions: A method for constructing a dynamically expanded vocabulary, the method comprising: Obtaining multi-source heterogeneous original knowledge data related to civil aviation passenger service, and converting the multi-source heterogeneous original knowledge data into standard format text; Processing the unstructured data in the standard format text based on an adaptive segmentation model and a stop word list to obtain a plurality of candidate words; Performing multi-dimensional corpus feature quantization analysis on each candidate word to determine a candidate key word; Performing similarity calculation based on entity segmentation and the candidate key word to determine an effective candidate key word, and adding the effective candidate key word to the dynamically expanded vocabulary.

[0006] In an embodiment, the multi-source heterogeneous original knowledge data includes: structured data and unstructured data; wherein the structured data includes: IATA coding standards and operation behavior record tables of the departure system, and the unstructured data includes: operation manuals, report materials, business description documents, customer service question and answer records, process guidelines and user feedback texts.

[0007] In an embodiment, the unstructured data in the standard format text is processed based on the adaptive word segmentation model and the stop word table to obtain a plurality of candidate words, including: The unstructured data in the standard format text is dynamically segmented using the adaptive word segmentation model to obtain segmented unstructured data; The segmented unstructured data is filtered based on the stop word table to obtain a plurality of candidate words.

[0008] In an embodiment, the multi-dimensional corpus feature quantitative analysis is performed on each candidate word to determine a candidate keyword, including: Word frequency statistics are performed on each candidate word to obtain a normalized word frequency statistical score corresponding to each candidate word; Word position statistics are performed on each candidate word to obtain a word position statistical score corresponding to each candidate word; Distribution statistics of each candidate word in different sentences are performed to obtain a word distribution statistical score corresponding to each candidate word; Context relevance of each candidate word is calculated to obtain a context relevance score corresponding to each candidate word; Based on the normalized word frequency statistical score, the word position statistical score, the word distribution statistical score and the context relevance score, an importance score of each candidate word is obtained through a preset formula; Based on the importance score of each candidate word, a candidate keyword is determined; The preset formula is as follows:

[0009] In the formula, represents the importance score of the candidate word ω, represents the context relevance score, represents the word position statistical score, represents the normalized word frequency statistical score, represents the word distribution statistical score.

[0010] In an embodiment, the word frequency statistics are performed on each candidate word to obtain a normalized word frequency statistical score corresponding to each candidate word, including: The normalized term frequency statistical score corresponding to the candidate word ω is obtained based on the number of occurrences of the candidate word ω in the standard format text and the average term frequency of all terms in the standard format text ;

[0011] In the formula, MeanTF represents the average term frequency of all terms in the standard format text, σ represents the standard deviation of the average term frequency, represents the number of occurrences of the candidate word ω in the standard format text; By analogy, the normalized term frequency statistical score corresponding to each candidate word is obtained.

[0012] In an embodiment, the term position statistics of each candidate word is performed respectively to obtain the term position statistical score corresponding to each candidate word, including: The total number of sentences in the standard format text is obtained according to the ordered queue obtained by cutting the characters according to punctuation marks; For any candidate word ω, the index set of all sentences containing the candidate word ω is obtained based on each sentence in the standard format text; The term position statistical score corresponding to the candidate word ω is obtained based on the index set of all sentences containing the candidate word ω and the term position statistical formula ; In the formula, the index set of all sentences containing the candidate word ω is represented by

[0013] In the formula, represents the index set of all sentences containing the candidate word ω, represents the median function; By analogy, the term position statistical score corresponding to each candidate word is obtained.

[0014] In an embodiment, the distribution of each candidate word in different sentences is counted respectively to obtain the term distribution statistical score corresponding to each candidate word, including: The total number of sentences in the standard format text is obtained according to the ordered queue obtained by cutting the characters according to punctuation marks through a preset function; For any candidate word ω, it is judged whether each sentence contains the candidate word ω to obtain the number of sentences containing the candidate word ω in the standard format text; The term distribution statistical score corresponding to the candidate word ω is obtained based on the number of sentences containing the candidate word ω in the standard format text and the total number of sentences in the standard format text and the term distribution statistical formula ; The formula for word distribution statistics is as follows:

[0015] In the formula, SF(ω) represents the number of sentences containing the candidate word ω in the standard format text, and #Sentences represents the total number of sentences in the standard format text; By analogy, the word distribution statistical score corresponding to each candidate word is obtained.

[0016] In one embodiment, the step of calculating the contextual relevance of each candidate word to obtain a contextual relevance score for each candidate word includes: For any candidate word ω, the contextual relevance score corresponding to candidate word ω is calculated based on the distribution of co-occurring words of candidate word ω within the standard format text. ;

[0017] In the formula, S represents the total number of sentences in the standard format text, and oct(ω, C, j) represents the number of times the candidate word ω co-occurs with the context word set C in the j-th sentence; By analogy, the contextual relevance score for each candidate word is obtained.

[0018] In one embodiment, determining candidate keywords based on the importance score of each candidate word includes: The importance score of each candidate word is weighted and adjusted to obtain a comprehensive score for each candidate word; The overall score of each candidate word is compared with a threshold, and candidate words with an overall score greater than the threshold are identified as candidate keywords.

[0019] Based on the same inventive concept, embodiments of this application also provide an apparatus for constructing a dynamically expanded lexicon, the apparatus comprising: The data acquisition module is configured to acquire multi-source heterogeneous raw knowledge data related to civil aviation passenger services and convert the multi-source heterogeneous raw knowledge data into standard format text. The candidate word extraction module is configured to process unstructured data in the standard format text based on an adaptive word segmentation model and a stop word list to obtain several candidate words; The candidate word analysis module is configured to perform multi-dimensional corpus feature quantification analysis on each candidate word to determine candidate keywords; The thesaurus construction module is configured to perform similarity calculations based on entity segmentation and the candidate keywords, determine effective candidate keywords, and add the effective candidate keywords to the dynamically expanding thesaurus.

[0020] Based on the same inventive concept, this application also provides an electronic device, including: a memory and a processor; the processor is used to read and execute a computer program stored in the memory to implement the steps of the aforementioned method for constructing a dynamically expanded lexicon.

[0021] Based on the same inventive concept, this application also provides a computer storage medium storing computer-executable instructions, which, when executed, implement the steps of the aforementioned method for constructing a dynamically expanded lexicon.

[0022] The technical effects and advantages of this application are as follows: 1) Introduce word vector matching to enhance semantic expansion capabilities: By introducing an adaptive word segmentation model to express candidate words semantically, and then calculating their similarity with existing keywords in passenger service business scenarios, a dynamic clustering strategy is adopted to group candidate words with high semantic similarity into the same semantic cluster, thereby achieving automatic expansion and updating of the lexicon. This method effectively makes up for the shortcomings of traditional rule matching methods, such as the inability to identify synonyms and the difficulty in adapting to changes in business context, and significantly improves the lexicon's adaptability to variations in industry terminology, new business vocabulary, and different expression habits. Experimental results show that, through the word vector semantic classification mechanism, the new word recognition capability is improved by approximately 30% compared to traditional statistical methods, significantly enhancing the system's actual coverage and intelligence in passenger service dialogue systems, semantic retrieval, and automatic attribution analysis. 2) Automating the build process improves efficiency and stability: This paper presents a highly automated, dynamically expanding thesaurus construction process by employing a fusion of structured and unstructured data inputs, an adaptive word segmentation model, and a combination of feature scoring and semantic clustering. This avoids the high costs associated with traditional methods that rely on manual annotation and selection. The automation mechanism enables the system to quickly respond to and update the corresponding thesaurus content in the face of dynamically changing service scenarios (such as sudden flight delays or policy adjustments), achieving on-demand expansion and incremental learning. Furthermore, this application exhibits excellent reusability and integrability, making it suitable for unified thesaurus management across multiple civil aviation subsystems (such as departure systems, customer service systems, and intelligent question-and-answer platforms), ensuring semantic consistency. Verification has shown that this application can achieve hourly-level semantic thesaurus construction and updates in large-scale log data environments, improving processing efficiency by over 50% compared to manual construction methods, demonstrating significant advantages in system stability and operational efficiency.

[0023] This application solves the technical problems of existing civil aviation service systems, which often use static vocabulary lists or rely on manual compilation and updates. Static vocabulary lists cannot be updated in real time, cannot flexibly adapt to the characteristics of the civil aviation field, and rely on manual updates and maintenance, resulting in high maintenance costs.

[0024] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating the method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the overall process of an embodiment of this application; Figure 3 for Figure 1 A detailed flowchart of step S30; Figure 4 This is a schematic diagram of the functional modules of an embodiment of the apparatus for constructing a dynamically expanded thesaurus of this application; Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] To address the shortcomings of existing technologies, refer to Figure 1 This application discloses a method for constructing a dynamically expanded thesaurus, including: Step S10: Obtain multi-source heterogeneous raw knowledge data related to civil aviation passenger services, and convert the multi-source heterogeneous raw knowledge data into standard format text; In this embodiment, the multi-source heterogeneous original knowledge data includes: structured data and unstructured data; wherein, the structured data includes: IATA coding standards and departure system operation behavior record tables, and the unstructured data includes: operation manuals, report materials, business description documents, customer service Q&A records, process guides and user feedback texts.

[0029] Reference Figure 2 All data is uniformly converted into analyzable standard format text through a preprocessing process and stored in the corpus to be processed, ensuring that the data foundation for subsequent processing is complete and authentic.

[0030] Step S20: Based on the adaptive word segmentation model and the stop word list, the unstructured data in the standard format text is processed to obtain several candidate words; In some specific embodiments, step S20 includes: An adaptive word segmentation model is used to dynamically segment unstructured data in the standard format text to obtain segmented unstructured data. Based on the stop word list, the unstructured data after word segmentation is filtered to obtain several candidate words.

[0031] In this embodiment, we continue to refer to... Figure 2 The standard-format text in the constructed corpus is segmented into words. An adaptive word segmentation model trained on a civil aviation corpus is used to dynamically segment unstructured data in the standard-format text, resulting in segmented unstructured data. Combined with a custom stop word list, several candidate words are dynamically identified and extracted from the segmented unstructured data text to ensure coverage of a large number of new words, specialized terms, abbreviations, and variant expressions within the industry.

[0032] Build or use a stop word list suitable for Chinese to filter out common but meaningless words. The stop word list includes categories such as sensitive words, Chinese auxiliary words, punctuation marks, and special symbols.

[0033] The list of stop words is shown in Table 1.

[0034] Table 1

[0035] In Chinese, individual characters often have meaning, but in many scenarios, it may be necessary to adjust the minimum word length (e.g., set to at least 2 characters) to improve keyword accuracy. This mode breaks down sentences most precisely, making it suitable for most scenarios such as text analysis.

[0036] For example, unstructured data in standard format text: "How to claim compensation for flight delays? How long is considered a flight delay? Is there compensation for weather-related reasons and flight cancellations?"

[0037] Unstructured data in standard formatted text after word segmentation: Sentence 1: ["flight", "delay", "demand", "compensation"]; Sentence 2: ["How long", "calculate", "flight", "delay"]; Sentence 3: ["weather", "reason", "flight", "cancellation", "compensation"].

[0038] Several candidate words were obtained: flight, delay, request, compensation, how long, calculate, weather, reason, and cancellation.

[0039] Step S30: Perform multi-dimensional corpus feature quantitative analysis on each candidate word to determine candidate keywords; In this embodiment, we continue to refer to... Figure 2 For the obtained candidate words, a multi-feature fusion algorithm model is used to perform quantitative analysis on each candidate word, including: word frequency statistics (the frequency of occurrence of candidate words in the full corpus), word position statistics (the distribution characteristics of words at the beginning, middle, and end of sentences), word distribution statistics (the distribution of candidate words in different sentences), and contextual relevance (the semantic coupling degree with surrounding words).

[0040] Based on the above analysis, after normalizing the scores of each feature, a weighted calculation method is used to calculate the importance score of each candidate word. A custom scoring mechanism is used to identify candidate keywords so that they can proceed to the next step of processing first.

[0041] In some specific embodiments, reference is made to Figure 3 Step S30 includes: Step S301: Perform word frequency statistics on each candidate word to obtain the normalized word frequency statistics score corresponding to each candidate word; In this embodiment, since high-frequency words usually better reflect the theme of a document, but high frequency alone in long documents may also produce bias, it is necessary to normalize the word frequency. This can not only eliminate the word frequency bias in long documents, emphasize the significantly high-frequency theme words in the document, suppress extremely high-frequency but meaningless words, and generate more reasonable word importance weights within the document, but is also more refined and robust than simple TF (Term Frequency).

[0042] For any candidate word ω, the normalized term frequency score is obtained based on the number of times candidate word ω appears in the standard format text and the average term frequency of all words in the standard format text. ;

[0043] In the formula, MeanTF represents the average term frequency of all words in the standard format text, and σ represents the standard deviation of the average term frequency. This indicates the number of times the candidate word ω appears in the standard format text; By analogy, the normalized word frequency statistical score corresponding to each candidate word is calculated.

[0044] Step S302: Perform word position statistics for each candidate word to obtain the word position statistics score for each candidate word; In this embodiment, the beginning of the document usually summarizes the topic better, so words that appear earlier are more likely to be keywords. Therefore, it is necessary to perform word position statistics for each candidate word.

[0045] Specifically, characters are truncated into an ordered queue based on punctuation marks to obtain the total number of sentences in the standard format text. For any candidate word ω, based on each sentence in the standard format text, all sentences containing candidate word ω are numbered to obtain an index set of all sentences containing candidate word ω. Based on the index set of all sentences containing candidate word ω, the word position statistical score corresponding to candidate word ω is obtained through a word position statistical formula. ; The formula for word position statistics is as follows:

[0046] In the formula, This represents the set of indices of all sentences containing the candidate word ω. This represents the median function; By following this method, the word position statistical score corresponding to each candidate word can be obtained.

[0047] Step S303: Statistically analyze the distribution of each candidate word in different sentences to obtain the word distribution statistical score corresponding to each candidate word; In this embodiment, if a word is distributed in more sentences, it is more likely to be the core concept of the document. Therefore, it is necessary to statistically analyze the distribution of each candidate word in different sentences.

[0048] Specifically, based on the characters, a preset function (e.g., the split() function) is used to extract the standard format text into an ordered queue according to the punctuation marks, thus obtaining the total number of sentences in the standard format text; For any candidate word ω, determine whether each sentence contains candidate word ω, and obtain the number of sentences in the standard format text that contain candidate word ω; Based on the number of sentences containing candidate word ω in the standard format text and the total number of sentences in the standard format text, the word distribution statistical score corresponding to candidate word ω is calculated using the word distribution statistical formula. ; The formula for word distribution statistics is as follows:

[0049] In the formula, SF(ω) represents the number of sentences in the standard format text that contain the candidate word ω, and #Sentences represents the total number of sentences in the standard format text; By following this method, the word distribution statistical score corresponding to each candidate word can be obtained.

[0050] Step S304: Calculate the contextual relevance for each candidate word to obtain the contextual relevance score for each candidate word; In this embodiment, if a word co-occurs with many of its left and right neighboring words, it may indicate that its independent information is weak; conversely, the fewer co-occurring words, the more special the word is in the context. Therefore, it is necessary to perform contextual correlation calculation for each candidate word.

[0051] Specifically, for any candidate word ω, the contextual relevance score corresponding to candidate word ω is calculated based on the distribution of co-occurring words of candidate word ω within the standard format text. ;

[0052] In the formula, S represents the total number of sentences in the standard format text, and oct(ω, C, j) represents the number of times the candidate word ω co-occurs with the context word set C in the j-th sentence; By doing so, the contextual relevance score for each candidate word can be obtained.

[0053] It should be noted that words that appear together with the target word within the preset range of the window are considered co-occurring words.

[0054] Step S305: Based on the normalized word frequency statistical score, the word position statistical score, the word distribution statistical score, and the context relevance score, the importance score of each candidate word is obtained through a preset formula; In this embodiment, the preset formula is as follows:

[0055] In the formula, The importance score represents the candidate word ω. Indicates the contextual relevance score. Indicates the positional score of words. This represents the normalized word frequency statistical score. Indicates the statistical score of word distribution.

[0056] In this pre-defined formula, a lower score indicates higher importance, and the combination of various features in the numerator or denominator plays a balancing role: it emphasizes the relative position and relevance of candidate words in the context, while also normalizing word frequency and the number of sentences in which they appear.

[0057] Step S306: Determine candidate keywords based on the importance score of each candidate word.

[0058] In some specific embodiments, step S306 includes: The importance score of each candidate word is weighted and adjusted to obtain a comprehensive score for each candidate word; The overall score of each candidate word is compared with a threshold, and candidate words with an overall score greater than the threshold are identified as candidate keywords.

[0059] In this embodiment, the importance score of each candidate word needs to be fine-tuned and corrected in different dimensions based on the characteristics of the business system, including different dimensions such as the scale of the airport from which the data comes, the type of airport that the data comes into operation, the type of data source, and the type of data policy.

[0060] The weights corresponding to different dimensions are shown in Table 2.

[0061] Table 2

[0062] For example, suppose the multi-source heterogeneous original knowledge data comes from an airport with a passenger volume of 30 million, the airport is in the trial production stage, the data source is user input, the multi-source heterogeneous original knowledge data belongs to the non-policy category, and the importance score of the candidate word ω calculated based on this multi-source heterogeneous original knowledge data is 0.2.

[0063] The score for the airport scale of the knowledge data source is 0.8; the score for the airport commissioning type of the data source is 0.4; the score for the data source type is 0.5; and the score for the data policy type is 1.

[0064] Therefore, the overall score = (0.8×0.3+0.4×0.3+0.5×0.3+1×0.1)×0.2=0.61×0.2=0.122.

[0065] Finally, the output lists all candidate keywords whose scores exceed a configurable threshold.

[0066] For example, since the unstructured data in the standard formatted text after word segmentation is: Sentence 1: ["flight", "delay", "demand", "compensation"]; Sentence 2: ["How long", "calculate", "flight", "delay"]; Sentence 3: ["weather", "reason", "flight", "cancellation", "compensation"].

[0067] Therefore, for step S30, word frequency statistics, word position statistics, word distribution statistics, and contextual relevance calculations are performed on each extracted candidate word, as shown in the following example: The total number of words in the standard formatted text after word segmentation is 4 + 4 + 5 = 13; the total number of sentences is 3.

[0068] Word frequency statistics and normalization: The average word frequency of all words is 1.8 times, and the standard deviation of the average word frequency is approximately 0.748. The candidate word "flight" appears 3 times, and the normalized word frequency score corresponding to the candidate word "flight" is... The candidate word "delay" appeared twice, and its normalized word frequency score was [value missing]. The candidate word "compensation" appeared twice, and its normalized word frequency score was [value missing]. The candidate word "weather" appears once, and its normalized word frequency score is [value missing]. The candidate word "cancel" appears once, and its normalized word frequency score is [value missing]. ; Word position statistics: The candidate word "flight" appears in sentences 1, 2, and 3, and its word position statistics score is 3; the candidate word "delay" appears in sentences 1 and 2, and its word position statistics score is 2; the candidate word "compensation" appears in sentences 1 and 3, and its word position statistics score is 2; the candidate word "weather" appears only in sentence 3, and its word position statistics score is 1; the candidate word "cancel" appears only in sentence 3, and its word position statistics score is 1.

[0069] Word distribution statistics: The candidate word "flight" appears in sentences 1, 2, and 3. The word distribution statistics score corresponding to the candidate word "flight" is . The candidate word "delay" appears in sentences 1 and 2, and the word distribution statistical score corresponding to the candidate word "delay" is [value missing]. The candidate word "requirement" only appears in sentence 1, and the word distribution statistical score corresponding to the candidate word "requirement" is . The candidate word "compensation" appears in sentences 1 and 3, and the word distribution statistical score corresponding to the candidate word "compensation" is [value missing]. Since both the candidate words "weather" and "cancel" appear only in one sentence, their word distribution statistical scores are both [missing information]. ; Contextual relevance calculation: In sentence 1 (["flight", "delay", "demand", "compensation"]), the number of co-occurring words is 3; in sentence 2 (["how long", "calculate", "flight", "delay"]), the number of co-occurring words is 3; in sentence 3 (["weather", "reason", "flight", "cancellation", "compensation"]), the number of co-occurring words is 4.

[0070] For candidate words appearing in multiple sentences, the average co-occurrence count of each candidate word can be taken. Therefore, the average co-occurrence count of the candidate word "flight" (appearing in sentences 1(3), 2(3), and 3(4)) is 1. The average co-occurrence count of the candidate word "delay" (appearing in sentences 1 and 2) is . The average co-occurrence count of the candidate word "compensation" (appearing in sentences 1 and 3) is 1. The average number of co-occurrences of the candidate words "weather" and "cancel" (both appearing only in sentence 3) is 4.

[0071] Weighted Value: Assuming the multi-source heterogeneous original knowledge data comes from an airport with 20-40 million passengers (score: 0.8), the airport's trial operation phase is less than one month (score: 0.6), the content is user-entered (score: 0.5), and it belongs to non-policy-related multi-source heterogeneous original knowledge data (score: 0.5). Weighted Value = 0.8×0.3 + 0.4×0.3 + 0.5×0.3 + 0.5×0.1 = 0.56.

[0072] Based on the above data, we can obtain Table 3: Table 3

[0073] Based on this multi-source heterogeneous original knowledge data, the importance score calculation formula for candidate word ω is obtained, and then weighted, the comprehensive score of candidate word "flight" is calculated as follows: ; Overall score for candidate word "delay" = ; Overall score for the candidate word "compensation" = ; Overall score for the candidate word "weather" = ; Overall score for the candidate word "cancel" = .

[0074] Assuming the threshold is set to 10, the overall score of each candidate word is compared with the threshold of 10. Candidate words with an overall score less than the threshold are identified as candidate keywords. The output candidate keywords are [flight], [delay], and [compensation].

[0075] Step S40: Calculate the similarity between entity segmentation and the candidate keywords to determine the effective candidate keywords, and add the effective candidate keywords to the dynamically expanded thesaurus.

[0076] In this embodiment, we continue to refer to... Figure 2For candidate keywords selected through feature scoring, similarity calculation is performed. The Word2Vec word vector model is used to perform semantic parsing, generate high-dimensional vector representations, and perform semantic similarity calculation with word vectors in the multi-business scenario keyword library established in this system to determine effective candidate keywords. The effective candidate keywords are then added to the dynamically expanded word library.

[0077] The specific steps include: Methods such as cosine similarity are used to measure the vector similarity between valid candidate keywords and existing keywords. If the similarity is higher than the similarity threshold in the business scenario, the semantics of the valid candidate keyword are determined to be valid in the business scenario, and the valid candidate keyword is included in the extended vocabulary of the corresponding business scenario to achieve clustered aggregation and dynamic updating of lexical semantics.

[0078] Semantic similarity can be calculated using the cosine similarity formula based on the vector space model:

[0079] Here, similarity is the semantic similarity between the word vector corresponding to the entity segmentation and the effective candidate keywords in a keyword library for a certain service scenario. It is the word vector representation corresponding to entity segmentation. It is the word vector representation of valid candidate keywords in a keyword library for a certain service scenario. The similarity between the two is measured by calculating the cosine similarity, which ranges from [-1, 1]. The closer the value is to 1, the higher the similarity. Generally speaking, a similarity of ≥0.8 can be considered as the two words being highly similar and considered as a match between the two word vectors.

[0080] The degree of matching between the two word vectors is shown in Table 4.

[0081] Table 4

[0082] For example: The candidate keyword "flight" has a similarity of 0.85 with the keyword "flight delay", thus matching the keyword database.

[0083] The candidate keyword "delay" has a similarity of 0.8 with the keyword "flight delay", and both keywords are included in the keyword database.

[0084] The candidate keyword "compensation" has a similarity of 0.85 with the keyword group "flight delay" which is associated with "payment". Both keywords are also included in the keyword database.

[0085] Keywords with a similarity greater than the similarity threshold are considered valid candidate keywords for the business scenario. Successfully matched valid candidate keywords will be included in the extended thesaurus for the business scenario and used for subsequent semantic retrieval, graph construction, and automatic question answering system training.

[0086] In this embodiment, multi-source heterogeneous raw knowledge data related to civil aviation passenger services is acquired and converted into standard format text. The unstructured data in the standard format text is processed based on an adaptive word segmentation model and a stop word list to obtain several candidate words. Multi-dimensional corpus feature quantification analysis is performed on each candidate word to determine candidate keywords. Similarity calculation is performed based on entity word segmentation and the candidate keywords to determine effective candidate keywords, and the effective candidate keywords are added to a dynamically expanded lexicon.

[0087] 1) Introduce word vector matching to enhance semantic expansion capabilities: By introducing an adaptive word segmentation model to express candidate words semantically, and then calculating their similarity with existing keywords in passenger service business scenarios, a dynamic clustering strategy is adopted to group candidate words with high semantic similarity into the same semantic cluster, thereby achieving automatic expansion and updating of the lexicon. This method effectively makes up for the shortcomings of traditional rule matching methods, such as the inability to identify synonyms and the difficulty in adapting to changes in business context, and significantly improves the lexicon's adaptability to variations in industry terminology, new business vocabulary, and different expression habits. Experimental results show that, through the word vector semantic classification mechanism, the new word recognition capability is improved by approximately 30% compared to traditional statistical methods, significantly enhancing the system's actual coverage and intelligence in passenger service dialogue systems, semantic retrieval, and automatic attribution analysis. 2) Automating the build process improves efficiency and stability: This paper presents a highly automated, dynamically expanding thesaurus construction process by employing a fusion of structured and unstructured data inputs, an adaptive word segmentation model, and a combination of feature scoring and semantic clustering. This avoids the high costs associated with traditional methods that rely on manual annotation and selection. The automation mechanism enables the system to quickly respond to and update the corresponding thesaurus content in the face of dynamically changing service scenarios (such as sudden flight delays or policy adjustments), achieving on-demand expansion and incremental learning. Furthermore, this application exhibits excellent reusability and integrability, making it suitable for unified thesaurus management across multiple civil aviation subsystems (such as departure systems, customer service systems, and intelligent question-and-answer platforms), ensuring semantic consistency. Verification has shown that this application can achieve hourly-level semantic thesaurus construction and updates in large-scale log data environments, improving processing efficiency by over 50% compared to manual construction methods, demonstrating significant advantages in system stability and operational efficiency.

[0088] This embodiment solves the technical problems in the prior art where the thesaurus used by civil aviation service systems is often based on a static vocabulary list or relies on manual compilation and updating. Static thesaurus cannot be updated in real time, cannot flexibly adapt to the characteristics of the civil aviation field, and the maintenance cost is high due to reliance on manual updates and maintenance.

[0089] Based on the same inventive concept, embodiments of this application also provide an apparatus for constructing a dynamically expanded thesaurus.

[0090] In one embodiment, reference is made to Figure 4 , Figure 4 This is a functional module diagram of an embodiment of the apparatus for constructing a dynamically expanded thesaurus according to this application. Figure 4 As shown, the apparatus for building a dynamically expanding lexicon includes: The data acquisition module 10 is configured to acquire multi-source heterogeneous raw knowledge data related to civil aviation passenger services and convert the multi-source heterogeneous raw knowledge data into standard format text. The candidate word extraction module 20 is configured to process unstructured data in the standard format text based on an adaptive word segmentation model and a stop word list to obtain a number of candidate words; The candidate word analysis module 30 is configured to perform multi-dimensional corpus feature quantitative analysis on each candidate word to determine candidate keywords; The thesaurus construction module 40 is configured to perform similarity calculations based on entity segmentation and the candidate keywords, determine effective candidate keywords, and add the effective candidate keywords to the dynamically expanded thesaurus.

[0091] Optionally, in one embodiment, the multi-source heterogeneous original knowledge data includes: structured data and unstructured data; wherein, the structured data includes: IATA coding standards and operation behavior record tables of the departure system, and the unstructured data includes: operation manuals, report materials, business description documents, customer service Q&A records, process guides and user feedback texts.

[0092] Optionally, in one embodiment, the candidate word extraction module 20 is configured to: An adaptive word segmentation model is used to dynamically segment unstructured data in the standard format text to obtain segmented unstructured data. Based on the stop word list, the unstructured data after word segmentation is filtered to obtain several candidate words.

[0093] Optionally, in one embodiment, the candidate word analysis module 30 is configured to: Each candidate word is counted by frequency analysis to obtain the normalized frequency score for each candidate word. Each candidate word is analyzed for its position, and a position score is obtained for each candidate word. The distribution of each candidate word in different sentences was statistically analyzed to obtain the word distribution statistical score for each candidate word. Contextual relevance is calculated for each candidate word to obtain a contextual relevance score for each candidate word; Based on the normalized word frequency statistical score, the word position statistical score, the word distribution statistical score, and the contextual relevance score, the importance score of each candidate word is obtained through a preset formula; Candidate keywords are determined based on the importance score of each candidate word; The preset formula is as follows:

[0094] In the formula, The importance score represents the candidate word ω. Indicates the contextual relevance score. Indicates the positional score of words. This represents the normalized word frequency statistical score. Indicates the statistical score of word distribution.

[0095] Optionally, in one embodiment, the candidate word analysis module 30 is configured to: For any candidate word ω, based on the number of times candidate word ω appears in the standard format text and the average word frequency of all words in the standard format text, the normalized word frequency statistical score corresponding to candidate word ω is obtained. ;

[0096] In the formula, MeanTF represents the average word frequency of all words in the standard format text, and σ represents the standard deviation of the average word frequency. Indicates the number of times the candidate word ω appears in the standard format text; By analogy, the normalized word frequency statistical score corresponding to each candidate word is calculated.

[0097] Optionally, in one embodiment, the candidate word analysis module 30 is configured to: The total number of sentences in the standard format text is obtained by extracting characters into an ordered queue based on punctuation marks. For any candidate word ω, based on each sentence in the standard format text, all sentences containing candidate word ω are numbered to obtain an index set of all sentences containing candidate word ω; Based on the index set of all sentences containing candidate word ω, the word position statistical score corresponding to candidate word ω is obtained through the word position statistical formula. ; The formula for word position statistics is as follows:

[0098] In the formula, This represents the set of indices of all sentences containing the candidate word ω. This represents the median function; By analogy, the word position statistical score corresponding to each candidate word is obtained.

[0099] Optionally, in one embodiment, the candidate word analysis module 30 is configured to: Based on the characters, the standard format text is truncated into an ordered queue according to punctuation marks using a preset function, thus obtaining the total number of sentences in the standard format text; For any candidate word ω, determine whether each sentence contains candidate word ω, and obtain the number of sentences containing candidate word ω in the standard format text; Based on the number of sentences containing candidate word ω in the standard format text and the total number of sentences in the standard format text, the word distribution statistical score corresponding to candidate word ω is calculated using the word distribution statistical formula. ; The formula for word distribution statistics is as follows:

[0100] In the formula, SF(ω) represents the number of sentences containing the candidate word ω in the standard format text, and #Sentences represents the total number of sentences in the standard format text; By analogy, the word distribution statistical score corresponding to each candidate word is obtained.

[0101] Optionally, in one embodiment, the candidate word analysis module 30 is configured to: For any candidate word ω, the contextual relevance score corresponding to candidate word ω is calculated based on the distribution of co-occurring words of candidate word ω within the standard format text. ;

[0102] In the formula, S represents the total number of sentences in the standard format text, and oct(ω, C, j) represents the number of times the candidate word ω co-occurs with the context word set C in the j-th sentence; By analogy, the contextual relevance score for each candidate word is obtained.

[0103] Optionally, in one embodiment, the candidate word analysis module 30 is configured to: The importance score of each candidate word is weighted and adjusted to obtain a comprehensive score for each candidate word; The overall score of each candidate word is compared with a threshold, and candidate words with an overall score greater than the threshold are identified as candidate keywords.

[0104] The functions of each module in the above-mentioned dynamic expansion lexicon construction device correspond to the steps in the above-mentioned dynamic expansion lexicon construction method embodiment, and their functions and implementation processes will not be described in detail here.

[0105] Based on the same inventive concept, embodiments of this application also provide an electronic device, the structure of which is as follows: Figure 5 As shown, it includes: a memory and a processor, wherein the processor is used to read and execute computer programs stored in the memory to implement the aforementioned method for constructing a dynamically expanded lexicon.

[0106] Based on the same inventive concept, this application also provides a computer storage medium storing computer-executable instructions, which, when executed, implement the aforementioned method for constructing a dynamically expanded lexicon.

[0107] Finally, it should be noted that while some processes described in the embodiments of this application include multiple operations or steps that appear in a specific order, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0108] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for constructing a dynamically expandable lexicon, characterized in that, The method includes: Acquire multi-source heterogeneous raw knowledge data related to civil aviation passenger services, and convert the multi-source heterogeneous raw knowledge data into standard format text; Based on the adaptive word segmentation model and the stop word list, the unstructured data in the standard format text is processed to obtain several candidate words; Perform multi-dimensional corpus feature quantitative analysis on each candidate word to determine candidate keywords; Similarity calculations are performed based on entity segmentation and the candidate keywords to determine valid candidate keywords, which are then added to a dynamically expanded thesaurus.

2. The method for constructing a dynamically expanded lexicon according to claim 1, characterized in that, The multi-source heterogeneous original knowledge data includes: structured data and unstructured data; among which, structured data includes: IATA coding standards and departure system operation behavior record tables, and unstructured data includes: operation manuals, report materials, business instruction documents, customer service Q&A records, process guides and user feedback texts.

3. The method for constructing a dynamically expanded lexicon according to claim 1 or 2, characterized in that, The process of processing unstructured data in the standard format text based on an adaptive word segmentation model and a stop word list yields several candidate words, including: An adaptive word segmentation model is used to dynamically segment unstructured data in the standard format text to obtain segmented unstructured data. Based on the stop word list, the unstructured data after word segmentation is filtered to obtain several candidate words.

4. The method for constructing a dynamically expanded lexicon according to claim 1, characterized in that, The process of performing multi-dimensional corpus feature quantification analysis on each candidate word to determine candidate keywords includes: Each candidate word is counted by frequency analysis to obtain the normalized frequency score for each candidate word. Each candidate word is analyzed for its position, and a position score is obtained for each candidate word. The distribution of each candidate word in different sentences was statistically analyzed to obtain the word distribution statistical score for each candidate word. Contextual relevance is calculated for each candidate word to obtain a contextual relevance score for each candidate word; Based on the normalized word frequency statistical score, the word position statistical score, the word distribution statistical score, and the contextual relevance score, the importance score of each candidate word is obtained through a preset formula; Candidate keywords are determined based on the importance score of each candidate word; The preset formula is as follows: In the formula, The importance score represents the candidate word ω. This indicates the score for contextual relevance. Indicates the positional score of words. This represents the normalized word frequency statistical score. Indicates the statistical score of word distribution.

5. The method for constructing a dynamically expanded lexicon according to claim 4, characterized in that, The step of performing word frequency statistics on each candidate word to obtain a normalized word frequency score for each candidate word includes: For any candidate word ω, based on the number of times candidate word ω appears in the standard format text and the average word frequency of all words in the standard format text, the normalized word frequency statistical score corresponding to candidate word ω is obtained. ; In the formula, MeanTF represents the average word frequency of all words in the standard format text, and σ represents the standard deviation of the average word frequency. Indicates the number of times the candidate word ω appears in the standard format text; By analogy, the normalized word frequency statistical score corresponding to each candidate word is calculated.

6. The method for constructing a dynamically expanded lexicon according to claim 4, characterized in that, The step of performing word position statistics for each candidate word to obtain a word position statistical score for each candidate word includes: The total number of sentences in the standard format text is obtained by extracting characters into an ordered queue based on punctuation marks. For any candidate word ω, based on each sentence in the standard format text, all sentences containing candidate word ω are numbered to obtain an index set of all sentences containing candidate word ω; Based on the index set of all sentences containing candidate word ω, the word position statistical score corresponding to candidate word ω is obtained through the word position statistical formula. ; The formula for word position statistics is as follows: In the formula, This represents the set of indices of all sentences containing the candidate word ω. This represents the median function; By analogy, the word position statistical score corresponding to each candidate word is obtained.

7. The method for constructing a dynamically expanded lexicon according to claim 4, characterized in that, The step involves statistically analyzing the distribution of each candidate word in different sentences to obtain a word distribution statistical score for each candidate word, including: Based on the characters, the standard format text is truncated into an ordered queue according to punctuation marks using a preset function, thus obtaining the total number of sentences in the standard format text; For any candidate word ω, determine whether each sentence contains candidate word ω, and obtain the number of sentences containing candidate word ω in the standard format text; Based on the number of sentences containing candidate word ω in the standard format text and the total number of sentences in the standard format text, the word distribution statistical score corresponding to candidate word ω is calculated using the word distribution statistical formula. ; The formula for word distribution statistics is as follows: In the formula, SF(ω) represents the number of sentences containing the candidate word ω in the standard format text, and #Sentences represents the total number of sentences in the standard format text; By analogy, the word distribution statistical score corresponding to each candidate word is obtained.

8. The method for constructing a dynamically expanded lexicon according to claim 4, characterized in that, The step of calculating the contextual relevance of each candidate word to obtain a contextual relevance score for each candidate word includes: For any candidate word ω, the contextual relevance score corresponding to candidate word ω is calculated based on the distribution of co-occurring words of candidate word ω within the standard format text. ; In the formula, S represents the total number of sentences in the standard format text, and oct(ω, C, j) represents the number of times the candidate word ω co-occurs with the context word set C in the j-th sentence; By analogy, the contextual relevance score for each candidate word is obtained.

9. The method for constructing a dynamically expanded lexicon according to claim 4, characterized in that, The process of determining candidate keywords based on the importance score of each candidate word includes: The importance score of each candidate word is weighted and adjusted to obtain a comprehensive score for each candidate word; The overall score of each candidate word is compared with a threshold, and candidate words with an overall score greater than the threshold are identified as candidate keywords.

10. A device for constructing a dynamically expanding lexicon, characterized in that, The device includes: The data acquisition module is configured to acquire multi-source heterogeneous raw knowledge data related to civil aviation passenger services and convert the multi-source heterogeneous raw knowledge data into standard format text. The candidate word extraction module is configured to process unstructured data in the standard format text based on an adaptive word segmentation model and a stop word list to obtain several candidate words; The candidate word analysis module is configured to perform multi-dimensional corpus feature quantification analysis on each candidate word to determine candidate keywords; The thesaurus construction module is configured to perform similarity calculations based on entity segmentation and the candidate keywords, determine effective candidate keywords, and add the effective candidate keywords to the dynamically expanded thesaurus.

11. An electronic device, characterized in that, include: Memory, processor; The processor is configured to read and execute the computer program stored in the memory to implement the steps of the method for constructing a dynamically expanded lexicon according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed, implement the steps of the method for constructing a dynamically expanded lexicon as described in any one of claims 1-9.

Images