Chinese knowledge text word segmentation method and question answering method

Abstract

The invention discloses a Chinese knowledge text word segmentation method and question and answer method, and the method comprises the steps: obtaining a Chinese knowledge corpus material corresponding to a business type, and segmenting the Chinese knowledge corpus material to obtain initial text fragments; for each initial text fragment, performing vocabulary extraction on the initial text fragment according to a pre-generated template vocabulary database to obtain template vocabularies in the initial text fragment; extracting non-template vocabularies according to the template vocabularies in the initial text fragment, and analyzing the non-template vocabularies to determine a professional vocabulary corpus; for each specialized vocabulary corpus, constructing a context relation chain based on the specialized vocabulary corpus, and merging the specialized vocabulary corpus according to nodes in the context relation chain to generate segmented vocabularies; the non-template vocabularies in the initial text fragment are merged according to the word segmentation vocabularies, and the word segmentation result of the initial text fragment is determined based on the merging result and the template vocabularies, so that the problem of inaccurate word segmentation of the Chinese text corpus is solved, and the word segmentation accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a Chinese knowledge text segmentation method and a question-answering method. Background Technology

[0002] With the advent of Large Language Models (LLMs), intelligent operational assistants, built upon LLMs and featuring intelligent interactive question-and-answer functionality, have begun to emerge and empower various vertical industries. Different types of businesses have corresponding proprietary terms. For example, in the field of connected vehicles, there are specialized terms related to connected vehicles, such as car manufacturers, vehicle cards, and vehicle control packages. These proprietary terms cannot be obtained or trained on Chinese internet channels. Therefore, using traditional word segmentation methods will result in truncation of these proprietary terms, making accurate segmentation impossible. Ultimately, this leads to a lack of relevant information in the knowledge base for these terms. Furthermore, if a question contains proprietary terms, the relevance of the text-enhanced retrieval in the knowledge base will be low, resulting in inaccurate prompts fed into the LLM, affecting the quality of the question answers, and even causing hallucinations.

[0003] Existing Large Language Models (LLMs) offer segmentation methods suitable for English expressions, but they are not applicable to the unique sentence segmentation methods of Chinese. This results in inaccurate word segmentation of Chinese knowledge base texts. Either inconsistent or unrelated corpora are segmented into a single text block, or the text containing a fixed word or phrase is segmented, leading to texts expressing the same meaning being divided into different text blocks. This makes the subsequent knowledge base text vectors mixed, directly affecting the quality of the LLM's responses. Therefore, how to accurately segment Chinese text corpora has become a problem that needs to be solved. Summary of the Invention

[0004] This application provides a Chinese knowledge text segmentation method and a question-answering method to solve the problem of inaccurate Chinese text corpus segmentation and improve the accuracy of Chinese text corpus segmentation.

[0005] According to one aspect of this application, a Chinese knowledge text segmentation method is provided, comprising:

[0006] Obtain Chinese knowledge corpus materials corresponding to the business type, segment the Chinese knowledge corpus materials to obtain initial text fragments;

[0007] For each initial text segment, vocabulary is extracted from the initial text segment based on a pre-generated template vocabulary database to obtain the template vocabulary in the initial text segment;

[0008] Non-template words are extracted from the template words in the initial text fragment, and the non-template words are analyzed to determine the professional vocabulary corpus;

[0009] For each of the aforementioned professional vocabulary corpora, a contextual relationship chain is constructed based on the professional vocabulary corpora, and the professional vocabulary corpora are merged according to the nodes in the contextual relationship chain to generate word segmentation vocabulary;

[0010] The non-template words in the initial text segment are merged according to the segmentation words, and the segmentation result of the initial text segment is determined based on the merging result and the template words.

[0011] According to another aspect of this application, a question-and-answer method is provided, comprising:

[0012] Obtain unanswered questions related to connected vehicle services;

[0013] Based on the question to be answered, a prompt text is constructed, and the question to be answered and the prompt text are input into the trained vehicle network big model for question answering. The vehicle network big model is trained by word segmentation using the Chinese knowledge text word segmentation method described in any embodiment of this application.

[0014] The question-and-answer results are determined based on the output of the aforementioned vehicle-to-everything (V2X) model.

[0015] According to another aspect of this application, a Chinese knowledge text segmentation device is provided, comprising:

[0016] The material segmentation module is used to acquire Chinese knowledge corpus materials corresponding to business types, segment the Chinese knowledge corpus materials, and obtain initial text fragments;

[0017] The template vocabulary extraction module is used to extract vocabulary from each initial text fragment based on a pre-generated template vocabulary database to obtain template vocabulary from the initial text fragment.

[0018] The professional terminology identification module is used to extract non-template words based on template words in the initial text fragment, and analyze the non-template words to identify the professional terminology corpus;

[0019] The relation chain construction module is used to construct a context relation chain based on each of the professional vocabulary corpora, and merge the professional vocabulary corpora according to the nodes in the context relation chain to generate word segmentation vocabulary;

[0020] The word segmentation module is used to merge non-template words in the initial text segment according to each of the word segmentation words, and determine the word segmentation result of the initial text segment based on the merging result and the template words.

[0021] According to another aspect of this application, a question-and-answer device is provided, comprising:

[0022] The question retrieval module is used to retrieve unanswered questions related to connected vehicle services.

[0023] The question-answering module is used to construct prompt text based on the question to be answered, and input the question to be answered and the prompt text into the trained vehicle network big model for question answering. The vehicle network big model is trained by word segmentation using the Chinese knowledge text word segmentation method described in any embodiment of this application.

[0024] The question-and-answer result determination module is used to determine the question-and-answer result based on the output of the vehicle network big data model.

[0025] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0026] At least one processor, and a memory communicatively connected to said at least one processor;

[0027] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the Chinese knowledge text segmentation method or question-answering method described in any embodiment of this application.

[0028] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the Chinese knowledge text segmentation method or question-answering method described in any embodiment of this application.

[0029] According to another aspect of this application, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements the Chinese knowledge text segmentation method or question-answering method described in any embodiment of this application.

[0030] The technical solution of this application embodiment involves acquiring Chinese knowledge corpus materials corresponding to business types, segmenting the Chinese knowledge corpus materials to obtain initial text fragments; for each initial text fragment, extracting vocabulary from the initial text fragment according to a pre-generated template vocabulary database to obtain template vocabulary in the initial text fragment; extracting non-template vocabulary from the template vocabulary in the initial text fragment, and analyzing the non-template vocabulary to determine professional vocabulary corpus; for each professional vocabulary corpus, constructing a contextual relationship chain based on the professional vocabulary corpus, merging the professional vocabulary corpus according to the nodes in the contextual relationship chain to generate word segmentation vocabulary; and performing word segmentation on the initial text fragment according to each word segmentation vocabulary. Non-template words in the initial text fragment are merged. Based on the merging result and the template words, the word segmentation result of the initial text fragment is determined, solving the problem of inaccurate word segmentation of Chinese text corpora. The initial text fragment is obtained by segmenting Chinese knowledge corpora. Words are extracted from the initial text fragment using a template word database to quickly obtain template words. Then, non-template words are extracted from the template words. The non-template words are analyzed to determine the professional vocabulary corpus. A contextual relationship chain is constructed based on the professional vocabulary corpus. This contextual relationship chain describes the contextual relationship graph surrounding the professional vocabulary corpus, thereby better identifying fixed collocations or phrases containing professional vocabulary, making the word segmentation result more accurate. The professional vocabulary corpus is merged based on the nodes in the contextual relationship chain, merging fixed collocations to obtain word segmentation vocabulary in the professional field. Finally, non-template words are merged based on the word segmentation vocabulary. Based on the merging result and the template words, an accurate word segmentation result is generated. Word segmentation of the initial text fragment is performed from the perspective of professional vocabulary, improving word segmentation accuracy.

[0031] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart of a Chinese knowledge text segmentation method provided in Embodiment 1 of this application;

[0034] Figure 2 This is a flowchart of a Chinese knowledge text segmentation method provided in Embodiment 2 of this application;

[0035] Figure 3 This is an example diagram of a context relationship chain provided according to Embodiment 2 of this application;

[0036] Figure 4 This is an example diagram illustrating the weights of nodes in a contextual relationship chain according to Embodiment 2 of this application;

[0037] Figure 5 This is a flowchart of a question-and-answer method provided according to Embodiment 3 of this application;

[0038] Figure 6 This is a schematic diagram of the structure of a Chinese knowledge text segmentation device according to Embodiment 4 of this application;

[0039] Figure 7 This is a schematic diagram of a question-and-answer device according to Embodiment 5 of this application;

[0040] Figure 8 This is a schematic diagram of the structure of an electronic device according to Embodiment Six of this application. Detailed Implementation

[0041] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0042] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0043] Example 1

[0044] Figure 1This is a flowchart illustrating a Chinese knowledge text segmentation method provided in Embodiment 1 of this application. This embodiment is applicable to the segmentation of Chinese text. The method can be executed by a Chinese knowledge text segmentation device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0045] S101. Obtain Chinese knowledge corpus materials corresponding to the business type, segment the Chinese knowledge corpus materials, and obtain initial text fragments.

[0046] In this embodiment, the business type can be vehicle networking, power data, etc. The Chinese knowledge corpus can be relevant Chinese materials obtained through different methods. Taking vehicle networking as an example, the Chinese knowledge corpus can be interface documents, instruction documents, and user manuals of the vehicle networking business operation support system; it could also be relevant reporting materials for vehicle networking services; it could be introductory documents for vehicle networking business-related products, etc. Any corpus documents related to vehicle networking services can be collected and summarized into relevant Chinese knowledge corpus materials. The initial text fragment can be understood as the text fragment obtained from the initial segmentation, which can be a single sentence.

[0047] The process begins with identifying the business type and collecting relevant materials and documents to form a Chinese knowledge corpus. This corpus can be in various file formats, such as text files (TXT, WORD, WPS, etc.), spreadsheets (EXCEL), presentation files (PowerPoint), and portable file formats (PDF). Various techniques can be used to extract the text information from these files, generating the Chinese knowledge corpus. The corpus is then analyzed to determine segmentation points, which are then used to segment the corpus into initial text fragments. Alternatively, the corpus can be segmented according to predefined rules to obtain initial text fragments, and so on.

[0048] For example, in Chinese grammar, the basic linguistic unit that can fully express a relatively independent contextual meaning is the sentence. In Chinese expression, a complete sentence usually ends with specific punctuation marks, the most common being the period ".", followed by the question mark "?", the exclamation mark "!", and the semicolon ";" separating two complete sentences. The appearance of these punctuation marks usually indicates the end of a sentence; therefore, these punctuation marks can be used as separators to divide sentences in a complete Chinese corpus, initially segmenting the Chinese knowledge corpus into independent sentences, with each sentence serving as an initial text fragment.

[0049] S102. For each initial text segment, extract vocabulary from the initial text segment based on the pre-generated template vocabulary database to obtain the template vocabulary in the initial text segment.

[0050] In this embodiment, the template vocabulary database can be understood as a pre-built database used to store different template vocabulary. The template vocabulary in this embodiment is usually commonly used Chinese words, including common Chinese phrases, idioms, and common expressions.

[0051] A template vocabulary database is pre-constructed based on commonly used words. For example, frequently used words are obtained from authoritative official Chinese dictionaries such as the *Xinhua Dictionary*, *Modern Chinese Dictionary*, and *Xinhua Idiom Dictionary*, taking into account both simplified and traditional Chinese characters. A large number of template words can also be obtained through methods such as web crawling. These template words are stored to form the template vocabulary database. For each initial text fragment, vocabulary extraction is performed. The initial text fragment is matched with the template words in the template vocabulary database to extract the template words from the initial text fragment.

[0052] S103. Extract non-template words from the template words in the initial text fragment, and analyze the non-template words to determine the professional vocabulary corpus.

[0053] In this embodiment, non-template vocabulary can be understood as vocabulary other than template vocabulary. Specialized vocabulary corpus can be understood as information describing specialized vocabulary in a specific business domain.

[0054] Identify template words in the initial text fragment and extract words other than template words as non-template words; analyze non-template words to determine whether they are professional terms in the business type. If so, non-template words can be used as professional terminology corpus, or the professional terminology corpus corresponding to non-template words can be determined.

[0055] S104. For each professional vocabulary corpus, construct a contextual relationship chain based on the professional vocabulary corpus, and merge the professional vocabulary corpus according to the nodes in the contextual relationship chain to generate word segmentation vocabulary.

[0056] In this embodiment, the contextual relationship chain can be understood as a data chain formed by the context of the specialized vocabulary corpus, used to depict a contextual relationship graph surrounding the specialized vocabulary corpus. Word segmentation vocabulary can be understood as words obtained by merging specialized vocabulary corpora.

[0057] For each professional vocabulary corpus, taking the professional vocabulary corpus as the core, determine the context vocabulary of the professional vocabulary corpus, and construct the context relationship chain of the professional vocabulary corpus. The vocabulary in the context relationship chain can be called nodes. Determine the vocabulary that is fixedly used with the professional vocabulary corpus according to the nodes in the context relationship chain, and merge the professional vocabulary corpus with the vocabulary that is fixedly used with it to generate segmented vocabulary.

[0058] S105. Merge the non-template vocabulary in the initial text segment according to each segmented vocabulary, and determine the segmented result of the initial text segment based on the merged result and the template vocabulary.

[0059] Merge the non-template vocabulary in the initial text segment based on the segmented vocabulary generated from each professional vocabulary corpus. For example, if the segmented vocabulary is "Internet of Vehicles", and the non-template vocabulary in the initial text segment are "vehicle" and "Internet connection" respectively, then based on the segmented vocabulary, the two words "vehicle" and "Internet connection" are merged into one word, resulting in "Internet of Vehicles". Re-segment the initial text segment according to the template vocabulary and the merged non-template vocabulary to obtain the segmented result.

[0060] Exemplarily, the initial text segment is "The vehicle control device will cooperate with XX Province in 2024, assist in completing the construction of the provincial-level secondary platform of the Internet of Vehicles, and provide a one-key diagnosis function". After the text segment is retrieved and matched in the template vocabulary database, the text segmentation is as follows: "The vehicle control device in 2024 with", "XX Province", "carry out", "cooperation,", "assist", "complete", "vehicle", "Internet", "provincial-level secondary", "platform", "construction,", "and", "provide", "one-key", "diagnosis", "function". Among them, the non-template vocabulary are "vehicle", "provincial-level secondary", "and", and "one-key", and the rest of the vocabulary are template vocabulary. By constructing the context relationship chain to merge the non-template vocabulary, "vehicle" and "Internet connection" are merged together to obtain the segmented vocabulary "Internet of Vehicles", and "one-key" and "diagnosis" are merged together to obtain the segmented vocabulary "one-key diagnosis". Re-segment the initial text segment based on the segmented vocabulary and the template vocabulary to generate the corresponding segmented result: "The vehicle control device in 2024 with", "XX Province", "carry out", "cooperation,", "assist", "complete", "Internet of Vehicles", "provincial-level secondary", "platform", "construction,", "and", "provide", "one-key diagnosis", "function", which is basically consistent with the logic of people's sentence segmentation.

[0061] This application provides a method for segmenting Chinese knowledge text, solving the problem of inaccurate segmentation of Chinese text corpora. It segments the Chinese knowledge corpus to obtain initial text fragments, extracts vocabulary from the initial text fragments using a template vocabulary database to quickly obtain template vocabulary, and then extracts non-template vocabulary based on the template vocabulary. The non-template vocabulary is analyzed to determine the professional vocabulary corpus. A contextual relationship chain is constructed based on the professional vocabulary corpus, describing the contextual relationship graph surrounding the professional vocabulary corpus, thereby better identifying fixed collocations or phrases containing professional vocabulary, resulting in more accurate segmentation results. The professional vocabulary corpus is merged based on the nodes in the contextual relationship chain, merging fixed collocations to obtain segmented vocabulary in the professional field. Finally, non-template vocabulary is merged based on the segmented vocabulary, and accurate segmentation results are generated based on the merged results and template vocabulary. This method segments the initial text fragments from the perspective of professional vocabulary, improving segmentation accuracy.

[0062] Example 2

[0063] Figure 2 This is a flowchart of a Chinese knowledge text segmentation method provided in Embodiment 2 of this application. This embodiment is a refinement based on the above embodiments. Figure 2 As shown, the method includes:

[0064] S201. Obtain Chinese knowledge corpus materials corresponding to the business type, segment the Chinese knowledge corpus materials, and obtain initial text fragments.

[0065] S202. For each initial text segment, extract vocabulary from the initial text segment based on the pre-generated template vocabulary database to obtain the template vocabulary in the initial text segment.

[0066] Optionally, the initial text fragment is subjected to word extraction based on a pre-generated template vocabulary database to obtain template vocabulary in the initial text fragment, including: determining the word length corresponding to each template word in the template vocabulary database, and extracting words from the initial text fragment based on the word length in descending order to obtain template vocabulary in the initial text fragment.

[0067] The template vocabulary database stores template vocabularies of different lengths. Commonly, a template vocabulary consists of two characters. There are also template vocabularies composed of three characters, four characters, or even more characters. Moreover, there may be a phenomenon of nested template vocabularies. For example, the length of "气象" is 2, and the length of "气象万千" is 4. Different lengths of template vocabularies may be nested. The word "气象" and the idiom "气象万千" are both template vocabularies. The meanings of these two template vocabularies are completely different, but both contain the two characters "气象". Suppose a sentence text contains the idiom "气象万千". Since the word "气象" also belongs to the template vocabulary, when retrieving, the "气象" word is matched, and the subsequent characters "万千" become non-template vocabularies. Such word segmentation is obviously incorrect.

[0068] In the word segmentation process of the embodiments of the present application, the initial text segment is subjected to vocabulary extraction in the order of the decreasing lengths of the corresponding vocabulary in the template vocabulary database, so as to solve the problem of incorrect word segmentation caused by nested template vocabularies. For the different vocabulary lengths corresponding to the template vocabulary database, the vocabulary lengths are sorted in descending order. First, the vocabulary to be matched in the initial text segment is determined according to the larger vocabulary length, and the vocabulary to be matched is matched with the template vocabulary in the template vocabulary database. If the match is unsuccessful, the next smaller vocabulary length is selected, and the vocabulary to be matched is re-determined and matched. For example, the initial text segment is segmented based on the vocabulary length, and the segmented vocabulary is matched with the template vocabulary in the template vocabulary database to determine the successfully matched template vocabulary.

[0069] Optionally, based on the order of decreasing vocabulary length, vocabulary extraction is performed on the initial text segment to obtain the template vocabulary in the initial text segment, including:

[0070] Sort the vocabulary lengths in descending order, select the first vocabulary length as the current vocabulary length to be matched, and use the first word in the initial text segment as the starting point;

[0071] Determine the vocabulary to be matched in the initial text segment according to the current vocabulary length to be matched and the starting point, and match the vocabulary to be matched with the template vocabulary in the template vocabulary database;

[0072] If the match is successful, one template vocabulary in the initial text segment is extracted, and the next word after the currently extracted template vocabulary in the initial text segment is used as the new starting point, and the first vocabulary length is used as the new current vocabulary length to be matched. Return to execute the step of determining the vocabulary to be matched in the initial text segment according to the current vocabulary length to be matched and the starting point, and matching the vocabulary to be matched with the template vocabulary in the template vocabulary database until all the vocabulary in the initial text segment is successfully matched;

[0073] If the matching fails and there is a next word length corresponding to the current word length to be matched, then use the next word length corresponding to the current word length to be matched as the new current word length to be matched, and return to execute the step of determining the word to be matched in the initial text segment according to the current word length to be matched and the starting point, and matching the word to be matched with the template words in the template word database until all words in the initial text segment are successfully matched;

[0074] If the matching fails and there is no next word length corresponding to the current word length to be matched, use the next word after the starting point as the new starting point, and the first word length as the new current word length to be matched, and return to execute the step of determining the word to be matched in the initial text segment according to the current word length to be matched and the starting point, and matching the word to be matched with the template words in the template word database until all words in the initial text segment are successfully matched.

[0075] Exemplarily, assume that the word lengths corresponding to the template word database are 2 and 3. Taking the initial text segment "The vehicle control device cooperated with XX Province in 2024, assisted in completing the construction of the provincial-level secondary platform for the vehicle network, and provided a one-key diagnosis function" as an example to illustrate the template word extraction process: First, use 3 as the current word length to be matched, take "vehicle" as the starting point, and the word to be matched obtained is "vehicle control device". Match the "vehicle control device" with the template words in the template word database. If the match fails, then use 2 as the updated current word length to be matched, and the new word to be matched is "vehicle control". Match the "vehicle control" with the template words in the template word database. The match still fails, but there is no next word length. Take "control" as the new starting point and 3 as the new current word length to be matched, and continue to determine whether "control device 2024" matches the template word until it is determined that "XX Province" matches the template word. Take the next word "cooperated" after "Province" as the new starting point and 3 as the new current word length to be matched, and continue to determine whether "cooperated with" matches the template word until all words in the initial text segment are judged.

[0076] In the embodiments of this application, by introducing a template vocabulary database, accurate word segmentation is performed on common Chinese words, idioms, phrases, etc. that are prevalent in Chinese text corpora, and retrieval matching is carried out in the retrieval matching order from the highest to the lowest length of the characters to be retrieved, effectively solving the problem of incorrect word segmentation of nested template vocabulary, enabling the word segmentation of Chinese text corpora to be Chinese words with independent and accurate meanings, which can greatly improve the accuracy of word segmentation of Chinese text corpora, and thus reduce the probability of hallucination generation in Chinese large language templates. In view of the characteristics of insufficient and non-fixed professional vocabulary in some emerging fields, and the phenomenon that there is character overlap between many professional vocabulary in the field and common vocabulary, the technical feature of template vocabulary extraction is added. Its purpose is to identify in advance the common Chinese vocabulary with a relatively high frequency of occurrence in the text corpus through the template vocabulary database, reducing the data volume and difficulty of professional vocabulary identification and extraction, thereby improving the accuracy of professional vocabulary identification, enabling the professional vocabulary in the user's question to be effectively identified and accurately understood, and thus reducing the probability of hallucination in the large model.

[0077] S203. Extract non-template vocabulary according to the template vocabulary in the initial text segment.

[0078] S204. Determine the word frequency of each non-template vocabulary. If the word frequency of the non-template vocabulary is greater than the threshold, determine the non-template vocabulary as high-frequency non-template vocabulary.

[0079] For non-template vocabulary, these non-template vocabulary can be divided into three categories. The first category is auxiliary words, such as "的", "地", "得", "了", "啦", etc. These auxiliary words often appear in single-character form, with a very high frequency of occurrence, and these non-template vocabulary do not represent specific semantic meanings; the second category is conjunctions, such as "和", "与", "及", "并", "但", "为", etc. This type of non-template vocabulary also often appears in single-character form, with a very high frequency of occurrence, and these non-template vocabulary only express relationships such as parallelism, progression, and transition in sentences, and do not represent specific semantic meanings; the third category is professional vocabulary corresponding to the business type. For example, in the vehicle networking business, there are professional vocabulary such as "车企", "数智", "车卡", "政企", "话单", "车控宝", etc. These non-template vocabulary often appear in multi-character forms such as two characters and three characters, but the character length is often not very long, generally not exceeding four characters, and the frequency of occurrence is lower than the first two categories of non-template vocabulary. These non-template vocabulary often express the specific meanings of the profession or industry related to the business type. However, due to their professionalism, they have not been included in the revised Chinese dictionary as template vocabulary. This type of professional non-template vocabulary is the professional vocabulary to be identified.

[0080] For each non-template word, count the total number of times it appears in the Chinese knowledge corpus materials, the number of documents in which it appears in different documents of the Chinese knowledge corpus materials, etc., and determine its corresponding word frequency. A threshold is set in advance, and the word frequency is compared with this threshold. If the word frequency of the non-template word is greater than the threshold, the non-template word is determined to be a high-frequency non-template word. High-frequency non-template words refer to non-template words with a relatively high frequency of occurrence. Exemplarily, the way to determine the word frequency of non-template words can be: counting the total number of times each non-template word appears in all the trained Chinese knowledge corpus materials, or the number of documents of the Chinese knowledge corpus materials in which each non-template word appears, or using the total number of times each non-template word appears in all the Chinese knowledge corpus materials divided by the number of documents of the Chinese knowledge corpus materials in which it appears as the frequency.

[0081] Among the high-frequency non-template words extracted in this step, in addition to the professional vocabulary corpus, there are many other non-template words, such as common non-template words that do not represent specific meanings, such as "的", "地", "得", "和", "且", etc., and there are also some professional words in other fields, as well as other non-template words not included in the dictionary. Then, it is necessary to screen out the professional vocabulary corpus from the set of high-frequency non-template words. In the embodiment of the present application, the professional vocabulary corpus is extracted by the method of "text vectorization + clustering".

[0082] S205. Perform text vectorization on each high-frequency non-template word to obtain text vector data.

[0083] In this embodiment, the text vector data can be understood as a kind of feature vector for representing text data. Each high-frequency non-template word is processed by a text vectorization algorithm for text vectorization, and the high-frequency non-template word is mapped into a high-dimensional semantic space to obtain text vector data. Exemplarily, the text vectorization algorithm can be Word2Vec, GloVe, FastText, BERT, etc.

[0084] S206. Cluster each text vector data, and determine the professional vocabulary corpus according to the clustering result.

[0085] Cluster the text vector data according to the clustering algorithm, determine the data with similar semantics according to the clustering result, and use the high-frequency non-template words corresponding to this part of the data as the professional vocabulary corpus. Exemplarily, the clustering algorithm can be Kmeans, KNN, DBSCAN, Hierarchical Clustering, GMM, etc.

[0086] Taking the Internet of Vehicles (IoV) as an example, in the extracted set of high-frequency non-template terms, IoV-related professional terms mainly revolve around concepts such as "vehicle," "SIM card," and "data traffic." Semantically, these terms are quite close to the meanings of these concepts, thus exhibiting a certain degree of clustering in the semantic vector space. Other high-frequency non-template terms, not being within the IoV domain, are more scattered in the semantic vector space. This distinguishing feature allows for effective identification and filtering of IoV-related professional terms. In large language models, the semantic vector space is represented by text vectorization. Text vectorization in large language models is a key technology for transforming text data into numerical vectors. Its core is to convert discrete text symbols (such as words and sentences) into continuous numerical vectors, enabling computers to process and analyze language data. After text vectorization, the text is mapped to a high-dimensional vector space. Within this high-dimensional vector space, texts expressing the same or similar meanings, as well as texts with strong correlations, tend to have relatively close spatial distances. Therefore, the coordinates and distances of spatial vectors can be used to characterize the correlation and similarity between different texts. Therefore, the extracted high-frequency non-template words are mapped into a high-dimensional semantic space using a specific text vectorization algorithm. The professional terms of the Internet of Vehicles will show obvious clustering, so they can be identified and filtered through clustering algorithms. Since the non-template words to be filtered are all professional terms in the field of the Internet of Vehicles, the coordinates of the "core" position of this cluster are known.

[0087] S207. For each professional vocabulary corpus, the professional vocabulary corpus is used as the central vocabulary of the context relationship chain. Chinese knowledge corpus materials are searched, and the preceding and following words are determined according to the predetermined length of the relationship chain and the position of the central vocabulary.

[0088] In this embodiment, the length of the relation chain can be preset to describe the number of words connected to the context of the central word. The preceding word is the word before the central word, and the following word is the word after the central word. The number of preceding words can be 0, 1, or more, and the number of following words can be 0, 1, or more.

[0089] The length of the relational chain is pre-determined. For each professional vocabulary corpus, the professional vocabulary corpus is used as the central vocabulary in the context relational chain. Chinese knowledge corpus materials are searched to determine the contextually related words of the central vocabulary. Based on the pre-determined length of the relational chain, the hierarchy of the words preceding and following the central vocabulary is determined. One unit length of the relational chain represents one level. When the length of the relational chain is N, the number of levels is N. Based on the obtained words at different levels and the position of the central vocabulary, the preceding and following words of the central vocabulary are determined.

[0090] S208. Use the central word as the center of the contextual relationship chain, and use the preceding and following words corresponding to the central word as nodes to construct the contextual relationship chain.

[0091] The central word is used as the center of the contextual relationship chain, and the preceding and following words corresponding to the central word are used as nodes. Each node is connected to its adjacent nodes or central words to construct the contextual relationship chain.

[0092] For example, with a relationship chain length of 1, and "one-click" as the central word, the construction of the contextual relationship chain is explained. When "one-click" is identified as a non-template word, and due to its high frequency, it is identified as a professional vocabulary corpus. The corresponding contextual relationship chain is constructed using "one-click" as the core of this professional vocabulary corpus. The positions of all occurrences of the professional vocabulary "one-click" in the Chinese knowledge corpus are searched, and the template or non-template words appearing immediately before and after each position are extracted. If the relationship chain length is 2, it is necessary to further determine the template or non-template words preceding words such as "provide" and "carry out," and the template or non-template words following words such as "diagnose," "paging," and "button." Figure 3 An example diagram of a contextual relationship chain is provided, taking a relationship chain length of 1 as an example. The diagram also exemplarily shows the word frequency of the nodes. The word frequency of each node includes the number of documents and the total number of times, which are distinguished by " / ".

[0093] Taking the Internet of Vehicles (IoV) as an example, this application's embodiment extracts template words and non-template words. These template words and non-template words do not constitute the final word segmentation words for corpus segmentation. This is because these extracted template words and non-template words are based on template word data matching retrieval and word frequency statistics identification. Each template word or non-template word is independent of each other and there is no correlation between them. This leads to the word segmentation words used for corpus segmentation using these template words or non-template words, which often results in the overall IoV Chinese corpus being segmented too finely. This is not conducive to the large language model understanding the meaning of the IoV business corpus, and reduces the relevance of the search text for relevant questions and knowledge base answers during the construction of the IoV business knowledge vector base. In turn, it reduces the performance of the large language model in question retrieval and related answer retrieval during the IoV business-related retrieval enhancement process. For example, the phrase "one-click diagnosis" in the statement "one-click diagnosis function" is a fixed phrase expressing an independent semantic meaning. It should be segmented as a whole. However, according to the steps described above, because the word "diagnosis" is included in the template vocabulary database, it is identified as a template word during the template vocabulary retrieval and matching process. The characters "one-click," however, are not included in the template vocabulary database and have a relatively high frequency of occurrence, so they are identified as non-template words. But if the corpus is segmented using the "one-click diagnosis" method during word segmentation, the correct segmentation method for this fixed phrase will be lost, resulting in unclear semantics in the segmented words. Similarly, the term "vehicle networking," an important proprietary term in the vehicle networking business, is also segmented into two words, "vehicle networking," because the word "network" is also identified as a template word. This segmentation method is clearly incorrect, and it prevents the large language model from correctly recognizing the inherent professional term "vehicle networking."

[0094] The contextual relationship chain centered on professional vocabulary corpus is a statistical analysis of the template or non-template words and their frequencies surrounding a professional vocabulary corpus in the original Chinese knowledge corpus. Each template or non-template word in the contextual relationship chain centered on this professional vocabulary corpus is called a node in this contextual relationship chain.

[0095] S209. Merge the professional vocabulary corpus according to the nodes in the context relationship chain to generate word segmentation vocabulary.

[0096] Optionally, the specialized vocabulary corpus is merged based on nodes in the contextual relationship chain to generate segmented vocabulary, including at least one of A1 and A2:

[0097] A1. Determine the weights of nodes in the contextual relationship chain, determine the segmentation positions based on the weights of each node, and merge the professional vocabulary corpus based on the segmentation positions to generate segmented vocabulary.

[0098] Calculate the weight of each node in the contextual relationship chain, for example, by assigning weights to nodes based on word frequencies. Compare the weights of each node and determine the splitting position based on the weights, for example, by determining the splitting position based on the node with the highest weight. Figure 3 For example, the node with the highest weight is "diagnosis". "One-click diagnosis" can be used as the segmentation position. Based on the segmentation position, the professional vocabulary corpus is merged, and the two words "one-click" and "diagnosis" are merged into "one-click diagnosis" to obtain the segmented vocabulary.

[0099] A2. Semantically encode the nodes in the context relationship chain, construct a dynamic semantic graph based on the encoding results, query the dynamic semantic graph to determine the segmentation position, merge the professional vocabulary corpus based on the segmentation position, and generate segmented vocabulary.

[0100] In this embodiment, a dynamic semantic graph can be understood as a type of knowledge graph; nodes in a dynamic semantic graph can be the text corpus corresponding to those nodes, and edges can be the relationships between nodes. Semantic encoding algorithms and models are used to semantically encode the nodes in the contextual relationship chain, obtaining corresponding semantic vectors. Based on the encoded semantic vectors, semantically similar nodes are identified, thereby constructing the dynamic semantic graph. The dynamic semantic graph is queried, and segmentation positions are determined based on the nodes and the relationships between them. Based on the segmentation positions, the specialized vocabulary corpus is merged to generate segmented vocabulary.

[0101] Optionally, determine the weights of the nodes in the contextual relationship chain, including B1-B3:

[0102] B1. Calculate the word frequency of each node in the context relationship chain. The word frequency of each node includes the number of documents and the total number of times.

[0103] In this embodiment, the document count can be understood as the number of documents in which the word represented by the node appears. The Chinese knowledge corpus can be composed of multiple documents, and the document count can be incremented by 1 for each word that appears in a document. The total count can be understood as the sum of the number of times the node appears in the Chinese knowledge corpus, and the total count can be incremented by 1 each time the word appears.

[0104] For each node in the contextual relationship chain, determine its position in the Chinese knowledge corpus, and calculate its corresponding word frequency based on its position. Word frequency includes: document frequency and total frequency.

[0105] B2. Determine the type of each node, which can be either a preceding node or a following node.

[0106] In this embodiment, a preceding node can be understood as a node formed by words preceding the central word; a following node can be understood as a node formed by words following the central word. The type of a node is determined based on its positional relationship with the central word; if the word represented by a node is a word preceding the central word, then this node is a preceding node; if the word represented by a node is a word following the central word, then this node is a following node.

[0107] B3. For each node, determine the document frequency weight and total frequency weight based on the node's word frequency and corresponding type. Then, perform a weighted sum of the document frequency weight and total frequency weight to obtain the node's weight.

[0108] For each node, the weight is calculated as follows: Determine the node's type and other nodes belonging to the same type. Determine the document count and total count for each node. Calculate the sum of the document counts and the sum of the total counts for all nodes belonging to the same type. Divide the node's document count by the sum of the document counts for all nodes belonging to the same type to obtain the document count weight. Divide the node's total count by the sum of the total counts for all nodes belonging to the same type to obtain the total count weight. Based on the set weighting coefficients, sum the document count weight and the total count weight to obtain the node's weight.

[0109] In this application embodiment, each node in the contextual relationship chain is assigned a weight. This weight indicates the degree of connection between the corresponding node's vocabulary and the central professional vocabulary corpus. A higher weight indicates a higher degree of connection, making it more likely to be merged into a fixed collocation phrase for word segmentation. As a contextual relationship chain, the length (also called depth) of this chain is also an important indicator. Generally, the fixed collocation phrase most closely related to a professional vocabulary corpus and most likely to be merged is often the closest contextual relationship in the corpus material's contextual position, i.e., the chain length is set to 1. When the chain length is greater than 1, the branches of this contextual relationship chain also increase exponentially, making the structure of this contextual relationship chain very complex. The following uses... Figure 3 This example illustrates how to assign weights to each node in this contextual relationship chain.

[0110] like Figure 3As shown, the contextual relationship chain centered on the professional vocabulary corpus "One-Click" contains multiple nodes. The word frequency statistics for each node are represented by two numbers separated by a " / ". The first number represents the document frequency of the corresponding template or non-template vocabulary in the Chinese knowledge corpus of vehicle networking, while the second number represents the total number of occurrences of the corresponding template or non-template vocabulary. For example, the word frequency of the suffix node "Diagnosis" in this contextual relationship chain is "58 / 853", meaning that the word "diagnosis" appearing immediately after "One-Click" in the professional vocabulary corpus appears 853 times in 58 documents of the Chinese knowledge corpus of vehicle networking. Since the Chinese knowledge corpus related to vehicle networking is relatively static and is not frequently updated, a database can be created in advance to specifically count and store the weights of each document. Once the Chinese corpus is updated, these weights can be recalculated and directly queried during the word frequency statistics process. From the contextual relationship chain constructed around the professional terminology corpus "one-click," it can be seen that the total frequency of words appearing in preceding nodes (also known as prefix nodes) is less than that of words appearing in following nodes (also known as suffix nodes). This indicates that the professional term "one-click" appears a certain number of times at the beginning of a complete segmented sentence, without any preceding prefix node. Furthermore, it can be observed that the word frequencies of the preceding nodes are relatively similar, with no significant differences between them. This suggests that the connection between the professional terminology corpus "one-click" and the aforementioned preceding nodes is relatively loose, and the probability of forming fixed phrases with these preceding nodes is low. Semantically speaking, these preceding node words and the word "one-click" cannot form fixed phrases. Looking at the following nodes in the contextual relationship chain, it can be found that the following node word "diagnosis" appears far more frequently than the other two suffix node words, both in terms of the number of corpus documents it appears in and its total frequency. This indicates that the suffix node word "diagnosis" has a very high primacy in the contextual relationship chain, which provides a basis for assigning weights to each node in the contextual relationship chain.

[0111] The basic principle for assigning weights to each node in a contextual relationship chain is that the higher the word frequency of a node, the higher its weight. There are various implementation methods; below is one specific method for assigning weights to nodes in a contextual relationship chain. First, a specific weighting ratio W is assigned to the two factors of word frequency: the number of documents appearing and the total number of occurrences. D and W C W can be set D =α,W C Since =1-α, the specific weight of each node in the context relation chain can be calculated using the following formula:

[0112] ;

[0113] Where W represents the weight of the node, This indicates the number of documents corresponding to a node. This represents the total number of document occurrences for all nodes of the same type. This represents the total number of times the node corresponds to. This represents the total number of occurrences of all nodes of the same type.

[0114] Based on the above formula for calculating node weights, the weight of each node in this contextual relationship chain can be obtained; for example, Figure 4 An example graph of the weights of nodes in a contextual relationship chain is provided.

[0115] This application's embodiments introduce a contextual relationship chain centered on professional vocabulary corpus, and set the weight of each node in the contextual relationship chain according to word frequency statistics. This allows for the further mining of fixed collocations or phrases from the identified template and non-template vocabulary, avoiding the problem of overly fragmented corpus material after segmentation according to template and non-template vocabulary. This further improves the accuracy of text segmentation and enhances the performance of large language model question answer retrieval.

[0116] Optionally, semantic encoding is performed on the nodes in the context relation chain, and a dynamic semantic graph, including C1-C3, is constructed based on the encoding results:

[0117] C1. Generate text corpus based on nodes in each context relationship chain, and perform semantic encoding on the text corpus based on a pre-trained language model to obtain a semantic encoding vector.

[0118] In this embodiment, the text corpus can be understood as information describing the overall semantics of the contextual relationship chain. The semantic encoding vector can be understood as a feature vector obtained through semantic encoding. The pre-trained language model can be understood as a pre-trained model; for example, the pre-trained language model can be BERT or RoBERTa, etc.

[0119] For each contextual relationship chain, the vocabulary corresponding to all nodes in the chain is determined, and the vocabulary is concatenated according to the contextual relationship to generate a text corpus. The text corpus is then input into a pre-trained language model for semantic encoding, resulting in a semantic encoding vector. Through semantic encoding, a deeper understanding of the underlying meaning of the text corpus, which is composed of contextual relationship chains centered on professional vocabulary, is achieved, leading to more accurate segmentation of professional vocabulary.

[0120] C2. For every two semantic encoding vectors, calculate the similarity between the two semantic encoding vectors, and cluster the text corpus according to the similarity to generate a corpus text set.

[0121] In this embodiment, the corpus text set can be understood as a dataset that stores similar text corpora. Semantic encoding vectors are combined pairwise, and the similarity between each pair of semantic encoding vectors is calculated using a similarity calculation formula. Based on the similarity, the text corpora are clustered, and text corpora with highly consistent similarity are grouped together to generate the corpus text set.

[0122] C3. Construct a dynamic semantic graph based on each corpus text set.

[0123] A dynamic semantic graph is constructed based on a collection of texts of the same type. A corresponding node entity is generated for each text corpus. The specific text corpus and its corresponding semantic encoding vector can be stored in the attributes of its node entity. The similarity between different text corpora can be stored in the corresponding relation entities, thereby constructing a dynamic semantic graph for an open vehicle networking business.

[0124] Text corpora extracted from the same cluster, their corresponding semantic encoding vectors, and the spatial similarity between text corpora constitute a knowledge graph cluster. Within this cluster, text corpora correspond to node entities in the knowledge graph, semantic encoding vectors represent the attributes of the node entities, and similarity corresponds to the attribute values ​​in the relation entities. This application's embodiments introduce a knowledge graph data structure, semantically encoding text corpora in relational chains based on a pre-trained language model and calculating encoding vectors. Similarity is calculated for the semantic encoding vectors of text corpora from every two chains within a set of context relational chains corresponding to a central word. This allows for a deeper understanding and mining of the semantic information of business-specific vocabulary, improving the accuracy of text segmentation. The dynamic semantic graph for generating business types of vehicle-to-everything (V2X) professional segmentation vocabulary, constructed through a pre-trained language model, helps accurately identify word combinations with specific semantics, thereby achieving more precise text segmentation and further enhancing the knowledge representation and understanding capabilities of the retrieval enhancement model.

[0125] In the actual implementation process, the BERT / RoBERTa pre-trained language model is combined to conduct in-depth semantic understanding of the text. After extracting the non-template corpus and counting the word frequencies, these models are used to perform semantic encoding on the non-template corpus and its context to obtain richer semantic information. By analyzing the similarity and relevance between the semantic encoding vectors, it helps to judge the combination relationship between words, rather than relying solely on word frequencies and weights. For example, when processing a sentence like "In 2024, the Vehicle Control Treasure cooperated with Jilin Province to assist in completing the construction of the provincial-level secondary vehicle networking platform and provided a one-key diagnosis function", the pre-trained language model can better understand the semantics contained in the overall concept of "the construction of the provincial-level secondary vehicle networking platform", avoiding wrongly splitting it into "vehicle networking, provincial-level secondary, platform, construction", thus improving the accuracy of text tokenization. After generating the semantic encoding vectors of the text corpus on each chain in this context relationship chain set centered on vehicle networking professional vocabulary, the similarity between the semantic encoding vectors of the text corpus corresponding to each two chains in this relationship chain set can be calculated through the similarity calculation module自带 in the BERT / RoBERTa model. The pairwise similarities between the semantic encoding vectors of the text corpus corresponding to each chain in a context relationship chain set centered on vehicle networking professional vocabulary are calculated, forming a similarity matrix. Clustering is performed through the similarity matrix corresponding to the context relationship chain set to obtain the core clustering and its corresponding corpus text.

[0126] The semantic encoding vector of each text corpus can be regarded as a point in a high-dimensional space, and the similarity between two semantic encoding vectors can be regarded as the distance between two points in the high-dimensional space mapped into a one-dimensional space. The greater the similarity, the smaller the distance between these two points in the high-dimensional space. Therefore, these semantic encoding vectors can be clustered according to the measurement value of their similarity. Commonly used clustering algorithms include K-Means, K-Nearest Neighbor (KNN), DBSCAN, Clara, etc. In this embodiment, the specific clustering algorithm adopted is not limited. After clustering, the text corpus corresponding to each chain in a context relationship chain set centered on vehicle networking professional vocabulary is clustered into one or more clusters. The cluster with the highest intra-class similarity is found as the core clustering cluster, and all the text corpus within the core clustering cluster are gathered for in-cluster similarity clustering to generate the corpus text set corresponding to the core clustering cluster.

[0127] By introducing knowledge graphs as a data structure, a dynamic semantic graph is constructed based on a large-scale vehicle-to-everything (V2X) corpus, representing words and their semantic relationships in graph form. During text segmentation, the text to be processed is matched and compared with the semantic graph. Utilizing the association paths and semantic similarities between words in the graph, a more comprehensive semantic basis is provided for determining the text segmentation location. For example, when processing emerging V2X technology terms or complex business scenario descriptions, the dynamic semantic graph can quickly associate relevant words and concepts, solving the problems of static, fragmented, and inaccurate segmentation in word frequency-based methods that construct contextual relationship chains. This helps to accurately identify word combinations with specific semantics, thereby achieving more precise text segmentation and further improving the knowledge representation and understanding capabilities of the retrieval enhancement model.

[0128] Optionally, the method may also include at least one of D1 and D2:

[0129] D1. Determine the spatial vector coordinates of the central words in the contextual relationship chain; cluster the spatial vector coordinates and generate semantic groups of professional terms based on the clustering results.

[0130] In this embodiment, spatial vector coordinates can be understood as the coordinates of the central word in the vector space. The distance of the central word in the vector space can be analyzed using these spatial vector coordinates. Specialized vocabulary semantic groups can be understood as combinations of words with the same or similar meanings. Since words can have multiple meanings, clustering can be used to identify words with the same or similar meanings and form specialized vocabulary semantic groups. These specialized vocabulary semantic groups can be used to construct prompt text.

[0131] The spatial vector coordinates of the central word in the contextual relationship chain are calculated according to a given coordinate calculation formula. The spatial vector coordinates of the central word can be determined based on information such as the weight of the preceding node (also called the node of the forward relationship), the coordinates of the word corresponding to the preceding node in the word segmentation vector space, the weight of the following node (also called the node of the backward relationship), and the coordinates of the word corresponding to the following node in the word segmentation vector space. The relevant parameters are substituted into the calculation formula to obtain the spatial vector coordinates of the central word. A clustering algorithm is then used to cluster the spatial vector coordinates, grouping central words with the same or similar meanings into a cluster, generating clustering results. Based on the central words in each cluster, a semantic group of specialized vocabulary is formed. Examples of clustering algorithms include KNN, K-means, DBSCAN, and Clara.

[0132] In some emerging fields, specialized terms have not yet formed unified and conventional names, leading to the widespread phenomenon of "multiple words with the same meaning" within these fields. Specifically, in practical questioning, different users may use different words to describe the same physical entity or concept. This application aims to solve the problem of identifying and filtering sets of words with "multiple words with the same meaning." As shown above, the words in a combination constituting "multiple words with the same meaning" refer to the same physical entity or concept. Therefore, in the collected Chinese corpus materials related to connected vehicle services, there is a certain degree of substitutability in the text order among the word combinations constituting "multiple words with the same meaning." For example, consider two sentences selected from the Chinese text corpus collected from the large-scale model of connected vehicle services: Sentence 1: "In 2024, the number of newly issued connected vehicle cards in Guangdong Province exceeded 2.3 million, a year-on-year increase of 8.7%, outperforming the market for three consecutive years"; Sentence 2: "In the fourth quarter of 2023, the number of newly issued vehicle cards nationwide reached 36 million, with double-digit growth for six consecutive quarters, confirming the rapid transformation of the domestic automotive industry towards connectivity and intelligence." In these two sentences, "connected vehicle card" and "vehicle card" are two words with the same meaning but different meanings, referring to the same physical entity. These two words can be substituted for each other without affecting the logical flow of the sentences. Therefore, it can be inferred that the contextual relationship chains corresponding to the words in the set of words with the same meaning are quite similar, or that the words in the contextual relationship chain centered on that word express similar semantic meanings. Therefore, this application designs a word segmentation vector space based on contextual relationship chains and calculates the coordinates of the central word in this word segmentation vector space.

[0133] It is important to note that the word segmentation vector space based on contextual relationship chains referred to in this embodiment does not refer to the text vector space commonly found in large language models, but rather to a vector space constructed using the generated word segments as the vector basis and the branch weights in the contextual relationship chains as weights. Within this word segmentation vector space, the vector space coordinates of each central word are calculated as follows:

[0134] ;

[0135] Where D represents the coordinates of the central word in the calculated context relation chain within the word segmentation vector space based on the context relation chain. The weight of the forward relation chain represents the importance of the words in the forward relation chain to the logical relationship of the central word, and can be determined in advance; The weight of the backward relation chain represents the importance of the words in the backward relation chain to the logical relationship of the central word, and can be determined in advance; This represents the number of nodes in the forward relation chain. This represents the number of nodes in the forward relation chain. This represents the weight of the i-th forward relation node in the context relation chain. The coordinates of the word corresponding to the i-th forward relation node in the context relation chain in the word segmentation vector space can be predetermined; This represents the weight of the j-th backward relation node in the context relation chain. The coordinates of the word corresponding to the j-th backward relation node in the context relation chain in the word segmentation vector space can be predetermined.

[0136] After calculating the word segmentation vector space coordinates of the context relation chain, since the word combinations constituting "multiple words with one meaning" have a certain degree of substitutability in their respective context relation chains, this manifests as the word combinations constituting "multiple words with one meaning" being relatively similar in the context relation chain centered on that word. This means that these word combinations constituting "multiple words with one meaning" are relatively close to each other in the analysis vector space of the context relation chain, exhibiting a clustering phenomenon in the vector space. Clustering algorithms can be used to identify and filter out these word combinations constituting "multiple words with one meaning." In this application scenario, since the word combinations constituting "multiple words with one meaning" all point to the same physical entity or concept, the position of this physical entity or concept in the word segmentation vector space of the context relation chain is the "core" position of these word combinations. Given this core, i.e., the clustering core, the k-means clustering algorithm is quite suitable.

[0137] The kmeans clustering algorithm, also known as k-means clustering, is an iterative clustering analysis algorithm. Its steps are as follows: if data needs to be divided into K groups, K objects are randomly selected as initial cluster centers. Then, the distance between each object and each seed cluster center is calculated, and each object is assigned to the nearest cluster center. The cluster centers and the objects assigned to them represent a cluster. Each time a sample is assigned, the cluster centers are recalculated based on the existing objects in the cluster. This process is repeated until a termination condition is met. This application can use the kmeans clustering algorithm to extract specialized word combinations that constitute "multiple words with one meaning," and these word combinations are called specialized word semantic groups.

[0138] D2. Generate semantic groups of professional terms based on the corpus text set corresponding to the context relationship chain.

[0139] The corpus text set corresponding to the context relation chain can be determined according to steps C1-C2 above. After semantic encoding, words in the context relation chain that point to the same physical entity or concept are mapped to the semantically encoded vector space. In this vector space, the spatial positions of multi-word homonymous word sets representing the same meaning are clustered. Therefore, multi-word homonymous semantic groups can be extracted using clustering algorithms. In this embodiment, the corpus text set obtained by clustering can be used as a semantic group of specialized vocabulary.

[0140] This application provides a Chinese knowledge text word segmentation method. By introducing a template vocabulary database and matching and retrieving Chinese knowledge corpus materials based on the template vocabulary database, template words in the corpus materials are identified and extracted. For the remaining corpus characters, word frequency is statistically analyzed, and words with higher frequencies are extracted as non-template words. Vehicle-to-everything (V2X) professional terms are then selected from the non-template words. A contextual relationship chain is constructed with professional terms as the core, and weights are assigned to each node in the contextual relationship chain. Based on the weights, core professional terms and node words are merged to form new professional terms and fixed collocation phrases, which is more in line with the situation of manual sentence segmentation. This makes the word segmentation of the V2X business knowledge base more accurate, thereby improving the relevance of the retrieval results. Since most of the Chinese corpus materials related to connected vehicle services consist of commonly used Chinese words or phrases, these words can be identified and retrieved by building a template word database. In this way, the remaining few non-template words can be identified by counting word frequencies and establishing contextual relationship chains, which can be used as professional words and phrases for word segmentation. This way, the word segmentation operation can focus on these few non-template words, which greatly reduces the workload and complexity of word segmentation of Chinese corpus materials and improves the efficiency of word segmentation.

[0141] Example 3

[0142] Figure 5 This is a flowchart of a question-and-answer method provided in Embodiment 3 of this application. This embodiment is applicable to answering questions raised by users. The method can be executed by a question-and-answer device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 5 As shown, the method includes:

[0143] S301. Obtain unanswered questions related to vehicle networking services.

[0144] In this embodiment, the vehicle-to-everything (V2X) service can be understood as a service type; the question to be answered can be understood as a question that needs to be answered. The question to be answered can be input by the user through voice, text, or other means. The question to be answered can be related to any service. This embodiment of the application takes the V2X service as an example to conduct question-and-answer sessions for questions related to the V2X service. The user inputs the question to be answered through voice or keyboard input, and this execution entity obtains the question to be answered input by the user.

[0145] S302. Construct prompt text based on the question to be answered, and input the question to be answered and the prompt text into the trained vehicle network big data model for question answering. The vehicle network big data model is trained by word segmentation using the Chinese knowledge text segmentation method described in any embodiment of this application.

[0146] In this embodiment, the prompt text is used to provide guidance on the data processing process of the large language model. The vehicle-to-everything (V2X) large model can be understood as a large language model, which in turn can be understood as a model with a large number of parameters. The V2X large model's word segmentation capability is pre-trained using the Chinese knowledge text segmentation method provided in any embodiment of this application, resulting in a trained V2X large model. During the training process, Chinese knowledge corpus materials with the business type of V2X business are acquired, and then corresponding word segmentation training is performed.

[0147] The process involves analyzing the questions to be answered, extracting key data based on semantic and lexical information, and constructing prompt text based on this key data. The questions and prompt text are then input into a pre-trained vehicle-to-everything (V2X) model for question answering. The V2X model uses the knowledge learned during training to segment the questions and provide corresponding answers.

[0148] S303. Determine the question-and-answer results based on the output of the vehicle-to-everything (V2X) big data model.

[0149] Obtain the output of the large-scale vehicle network model, and use the output of the large-scale vehicle network model as the question and answer result, or extract important content from the output of the large-scale vehicle network model as the question and answer result.

[0150] This application provides a question-answering method that solves the problem of inaccurate question-answering results in large-scale models caused by inaccurate word segmentation of Chinese text corpora. By using the Chinese knowledge text word segmentation method provided in any embodiment of this application to segment and train a large-scale vehicle network model, the large-scale vehicle network model can be segmented from the perspective of professional vocabulary, achieving accurate word segmentation, improving the retrieval performance and answer quality of the large-scale vehicle network model; and providing accurate answers to questions, thereby improving the user experience.

[0151] Optionally, construct prompt text based on the questions to be answered, including E1-E3:

[0152] E1. Match the question to be answered with the words in the pre-determined semantic groups of professional terms, and take the successfully matched semantic groups of professional terms as the target semantic groups of professional terms.

[0153] In this embodiment, the target professional term semantic group can be understood as a professional term semantic group selected from the professional term semantic groups. The professional term semantic group can be generated and saved during the training process.

[0154] Obtain the semantic groups of specialized terms generated during word segmentation training. Each semantic group includes one or more terms. Match the question to be answered with the terms in each semantic group to determine the matching degree. In this embodiment, the question to be answered can be matched with each semantic group separately to determine the matching degree. The semantic group with the highest matching degree is considered a successfully matched semantic group, or the semantic group with the highest matching degree that exceeds a set threshold is considered a successfully matched semantic group. Alternatively, the question to be answered can be matched with the terms in each semantic group sequentially to determine the matching degree. If the matching degree is higher than the set threshold, this semantic group is considered a successfully matched semantic group, and the matching degree between the question to be answered and the remaining semantic groups is not calculated. The successfully matched semantic groups are used as the target semantic group.

[0155] E2. If the target professional terminology semantic group has a corresponding semantic prompt word, construct the prompt text based on the semantic prompt word.

[0156] In this embodiment, semantic prompts can be understood as prompts used to generate prompt text. The target professional vocabulary semantic group can be pre-generated with corresponding semantic prompts, or it can be generated after multiple question-and-answer sessions. It is determined whether the target professional vocabulary semantic group has corresponding semantic prompts. If it does, the prompt text can be directly constructed based on the semantic prompts. For example, the semantic prompts can be filled into the corresponding positions in the prompt template to generate the prompt text.

[0157] E3. If there is no corresponding semantic prompt word in the target professional vocabulary semantic group, select a word from the target professional vocabulary semantic group to construct the prompt text.

[0158] If no corresponding semantic prompt word exists in the target vocabulary semantic group, a word is selected from the target vocabulary semantic group. This selection can be random or sequential. The word chosen from the target vocabulary semantic group is usually different each time. In this embodiment, the words are selected sequentially in a round-robin fashion, starting from the last word and continuing until a semantic prompt word corresponding to the target vocabulary semantic group is generated. For example, if there are 10 words in the target vocabulary semantic group, at least 10 question-and-answer sessions are needed to select all words in the first round. If no corresponding semantic prompt word is generated, the 11th question-and-answer session will select another word from the 10 words to avoid duplication. Prompt text is generated based on the selected word; for example, the selected word is filled into the corresponding position in the prompt word template to generate prompt text.

[0159] Optionally, after selecting a word from the target professional vocabulary semantic group to construct the prompt text, F1-F2 are also included:

[0160] F1. Evaluate the quality of the responses to the prompt text.

[0161] After the vehicle-to-everything (V2X) big data model answers questions based on the prompt text and obtains the answer results, it analyzes the answer results and evaluates the quality of the answer. The evaluation method can be that the user scores the answer results based on satisfaction, or it can be automatically scored by algorithms, models, etc. For example, the scoring indicators can be F1 score, BLEU score, Rouge score, Embedding similarity score, etc.

[0162] F2. Analyze the response quality assessment results of the prompt texts corresponding to different words in the target professional vocabulary semantic group, and determine the semantic prompt words corresponding to the target professional vocabulary semantic group.

[0163] The system determines the response quality assessment result of the prompt text corresponding to each word in the target professional vocabulary semantic group, and analyzes the response quality assessment results for all words. If the semantic prompt word generation conditions are met, the word with the best response quality is selected as the semantic prompt word corresponding to the target professional vocabulary semantic group. If the semantic prompt word generation conditions are not met, the system continues to wait for new prompt text and perform response quality assessment. The semantic prompt word generation conditions may be that the best response quality exceeds a set threshold, or the number of prompt texts corresponding to each word exceeds a set quantity threshold, or the total number of prompt texts corresponding to each word exceeds a set total quantity threshold, etc.

[0164] This application evaluates the response quality of words in a semantic group of specialized vocabulary and selects appropriate semantic prompt words to generate prompt text. Because words in a semantic group of specialized vocabulary often have multiple meanings, and some of these "multiple meanings" overlap with traditional template vocabulary and specialized terms, inappropriate prompt words may be introduced. Based solely on the user's question, the vehicle-to-everything (V2X) model may misunderstand the user's intent and meaning, leading to a "model illusion" and user dissatisfaction with the model's response. Therefore, the key to constructing appropriate prompt words is to identify and filter the set of "multiple meanings" words in the V2X word segmentation set and select excellent prompt words from within this set. After extracting the semantic groups of specialized vocabulary, the application scores the response quality of each semantic group and generates a corresponding semantic prompt word for each group.

[0165] This application assesses the response quality of words in the semantic group of multi-word homonymous business terms, identifies semantic prompt words with better response quality, and thus helps the vehicle network big data model generate better answers. This solves the problem of the "multi-word homonymous" phenomenon in vehicle network business question answering, which causes the vehicle network big data model's intention comprehension bias, reduces the probability of big data model illusion, and improves the quality of question answers and user satisfaction.

[0166] Example 4

[0167] Figure 6 This is a schematic diagram of the structure of a Chinese knowledge text segmentation device provided in Embodiment 4 of this application. Figure 6 As shown, the device includes: a material segmentation module 41, a template vocabulary extraction module 42, a professional vocabulary identification module 43, a relationship chain construction module 44, and a word segmentation module 45.

[0168] The material segmentation module 41 is used to acquire Chinese knowledge corpus materials corresponding to the business type, segment the Chinese knowledge corpus materials, and obtain initial text fragments;

[0169] The template vocabulary extraction module 42 is used to extract vocabulary from each initial text segment based on a pre-generated template vocabulary database to obtain template vocabulary in the initial text segment.

[0170] The professional terminology identification module 43 is used to extract non-template words based on template words in the initial text fragment and analyze the non-template words to identify professional terminology corpus.

[0171] The relation chain construction module 44 is used to construct a context relation chain based on the professional vocabulary corpus for each professional vocabulary corpus, and merge the professional vocabulary corpus according to the nodes in the context relation chain to generate word segmentation vocabulary;

[0172] The word segmentation module 45 is used to merge the non-template words in the initial text segment according to each of the word segmentation words, and determine the word segmentation result of the initial text segment based on the merging result and the template words.

[0173] This application provides a method for segmenting Chinese knowledge text, solving the problem of inaccurate segmentation of Chinese text corpora. It segments the Chinese knowledge corpus to obtain initial text fragments, extracts vocabulary from the initial text fragments using a template vocabulary database to quickly obtain template vocabulary, and then extracts non-template vocabulary based on the template vocabulary. The non-template vocabulary is analyzed to determine the professional vocabulary corpus. A contextual relationship chain is constructed based on the professional vocabulary corpus, describing the contextual relationship graph surrounding the professional vocabulary corpus, thereby better identifying fixed collocations or phrases containing professional vocabulary, resulting in more accurate segmentation results. The professional vocabulary corpus is merged based on the nodes in the contextual relationship chain, merging fixed collocations to obtain segmented vocabulary in the professional field. Finally, non-template vocabulary is merged based on the segmented vocabulary, and accurate segmentation results are generated based on the merged results and template vocabulary. This method segments the initial text fragments from the perspective of professional vocabulary, improving segmentation accuracy.

[0174] Optionally, the template vocabulary extraction module 42 includes:

[0175] The vocabulary length determination unit is used to determine the vocabulary length corresponding to each template vocabulary in the template vocabulary database.

[0176] The template vocabulary extraction unit is used to extract vocabulary from the initial text segment based on the vocabulary length in descending order, so as to obtain the template vocabulary in the initial text segment.

[0177] Optionally, the template vocabulary extraction unit is specifically used for: sorting the lengths of each vocabulary in descending order, selecting the first vocabulary length as the current word to be matched length, and taking the first word in the initial text segment as the starting point; determining the word to be matched in the initial text segment based on the current word to be matched length and the starting point, and matching the word to be matched with the template vocabulary in the template vocabulary database; if the match is successful, extracting a template vocabulary from the initial text segment, taking the next word of the currently extracted template vocabulary in the initial text segment as the new starting point, taking the first vocabulary length as the new current word to be matched length, and returning to execute the steps of determining the word to be matched in the initial text segment based on the current word to be matched length and the starting point, and matching the word to be matched with the template vocabulary in the template vocabulary database until all words in the initial text segment have been matched. If a match fails and a corresponding next word length exists for the current word length to be matched, then the next word length is taken as the new current word length to be matched, and the process returns to the step of determining the word to be matched in the initial text segment based on the current word length to be matched and the starting point, and matching the word to be matched with the template words in the template word database until all words in the initial text segment have been matched; if a match fails and a corresponding next word length does not exist for the current word length to be matched, then the next word of the starting point is taken as the new starting point, and the first word length is taken as the new current word length to be matched, and the process returns to the step of determining the word to be matched in the initial text segment based on the current word length to be matched and the starting point, and matching the word to be matched with the template words in the template word database until all words in the initial text segment have been matched.

[0178] Optional, the specialized terminology identification module 43 includes:

[0179] A high-frequency word determination unit is used to determine the word frequency of each of the non-template words. If the word frequency of the non-template words is greater than a threshold, the non-template words are determined to be high-frequency non-template words.

[0180] The text vector data determination unit is used to perform text vectorization on each of the high-frequency non-template words to obtain text vector data.

[0181] The specialized vocabulary identification unit is used to cluster the text vector data and determine the specialized vocabulary corpus based on the clustering results.

[0182] Optional, the relationship building module 44 includes:

[0183] The context vocabulary determination unit is used to use the professional vocabulary corpus as the central vocabulary of the context relationship chain, search the Chinese knowledge corpus material, and determine the preceding and following vocabulary according to the predetermined length of the relationship chain and the position of the central vocabulary.

[0184] The relationship chain building unit is used to construct the context relationship chain by taking the central word as the center of the context relationship chain and taking the preceding and following words corresponding to the central word as nodes.

[0185] Optionally, the word segmentation module 45 includes at least one of the following:

[0186] The first word segmentation vocabulary generation unit is used to determine the weight of the nodes in the context relationship chain, determine the segmentation position based on the weight of each node, and merge the professional vocabulary corpus based on the segmentation position to generate word segmentation vocabulary.

[0187] The second word segmentation vocabulary generation unit is used to perform semantic encoding on the nodes in the context relationship chain, construct a dynamic semantic graph based on the encoding results, query the dynamic semantic graph to determine the segmentation position, and merge the professional vocabulary corpus based on the segmentation position to generate word segmentation vocabulary.

[0188] Optionally, the first word segmentation vocabulary generation unit is specifically used for: counting the word frequency of each node in the context relationship chain, wherein the word frequency of each node includes the number of documents and the total number of documents; determining the type of each node, wherein the type of the node is a preceding node or a following node; for each node, determining the proportion of the number of documents and the proportion of the total number of documents based on the word frequency and the corresponding type of the node, and performing a weighted summation of the proportion of the number of documents and the proportion of the total number of documents to obtain the weight of the node.

[0189] Optionally, the second word segmentation and vocabulary generation unit is specifically used for: generating text corpora based on nodes in each context relationship chain; semantically encoding the text corpora based on a pre-trained language model to obtain semantic encoding vectors; calculating the similarity between two semantic encoding vectors for every two semantic encoding vectors; clustering each text corpus based on the similarity to generate a corpus text set; and constructing a dynamic semantic graph based on each corpus text set.

[0190] Optionally, the device further includes at least one of the following:

[0191] The first professional vocabulary group generation module is used to determine the spatial vector coordinates of the central words in the context relationship chain, cluster the spatial vector coordinates, and generate professional vocabulary semantic groups based on the clustering results.

[0192] The second specialized vocabulary group generation module is used to generate specialized vocabulary semantic groups based on the corpus text set corresponding to the context relationship chain.

[0193] The Chinese knowledge text segmentation device provided in this application embodiment can execute the Chinese knowledge text segmentation method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the execution method.

[0194] Example 5

[0195] Figure 7 This is a schematic diagram of a question-and-answer device provided in Embodiment 5 of this application. Figure 7 As shown, the device includes: a question acquisition module 51, a question-and-answer module 52, and a question-and-answer result determination module 53.

[0196] Question acquisition module 51 is used to acquire unanswered questions related to vehicle networking services;

[0197] Question answering module 52 is used to construct prompt text based on the question to be answered, and input the question to be answered and the prompt text into the trained vehicle network big model for question answering. The vehicle network big model is trained by word segmentation using the Chinese knowledge text word segmentation method described in any embodiment of this application.

[0198] The question-and-answer result determination module 53 is used to determine the question-and-answer result based on the output of the vehicle network big model.

[0199] This application provides a question-answering device that solves the problem of inaccurate question-answering results in large-scale models caused by inaccurate word segmentation of Chinese text corpora. By using the Chinese knowledge text word segmentation method provided in any embodiment of this application to segment and train a large-scale vehicle network model, the large-scale vehicle network model can be segmented from the perspective of professional vocabulary, achieving accurate word segmentation, improving the retrieval performance and answer quality of the large-scale vehicle network model; and providing accurate answers to questions, thereby improving the user experience.

[0200] Optional, question-and-answer module 52, including:

[0201] The target semantic group determination unit is used to match the question to be answered with the words in each pre-determined professional vocabulary semantic group, and to take the successfully matched professional vocabulary semantic group as the target professional vocabulary semantic group.

[0202] The first prompt text construction unit is used to construct prompt text based on the semantic prompt words if the target professional vocabulary semantic group has corresponding semantic prompt words;

[0203] The second prompt text construction unit is used to select a word from the target professional vocabulary semantic group to construct prompt text if there is no corresponding semantic prompt word in the target professional vocabulary semantic group.

[0204] Optionally, the device may also include:

[0205] The answer quality assessment module is used to assess the answer quality of the prompt text; analyze the answer quality assessment results of the prompt text corresponding to different words in the target professional vocabulary semantic group, and determine the semantic prompt words corresponding to the target professional vocabulary semantic group.

[0206] The question-and-answer device provided in this application embodiment can execute the question-and-answer method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the execution method.

[0207] Example 6

[0208] Figure 8 This is a schematic diagram of an electronic device according to Embodiment Six of this application, which can be used to implement any embodiment of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the application described and / or claimed herein.

[0209] like Figure 8 As shown, the electronic device 60 includes at least one processor 61 and a memory, such as a read-only memory (ROM) 62 and a random access memory (RAM) 63, communicatively connected to the at least one processor 61. The memory stores computer programs executable by the at least one processor. The processor 61 can perform various appropriate actions and processes based on the computer program stored in the ROM 62 or loaded from storage unit 68 into the RAM 63. The RAM 63 can also store various programs and data required for the operation of the electronic device 60. The processor 61, ROM 62, and RAM 63 are interconnected via a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.

[0210] Multiple components in electronic device 60 are connected to I / O interface 65, including: input unit 66, such as keyboard, mouse, etc.; output unit 67, such as various types of monitors, speakers, etc.; storage unit 68, such as disk, optical disk, etc.; and communication unit 69, such as network card, modem, wireless transceiver, etc. Communication unit 69 allows electronic device 60 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0211] Processor 61 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 61 performs the various methods and processes described above, such as Chinese knowledge text segmentation or question answering methods.

[0212] In some embodiments, the Chinese knowledge text segmentation or question-answering method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 68. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 60 via ROM 62 and / or communication unit 69. When the computer program is loaded into RAM 63 and executed by processor 61, one or more steps of the Chinese knowledge text segmentation or question-answering method described above may be performed. Alternatively, in other embodiments, processor 61 may be configured to perform the Chinese knowledge text segmentation or question-answering method by any other suitable means (e.g., by means of firmware).

[0213] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0214] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0215] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements the Chinese knowledge text segmentation or question-answering method described in any embodiment of this application.

[0216] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0217] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0218] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0219] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0220] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0221] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for segmenting Chinese knowledge text, characterized in that, include: Obtain Chinese knowledge corpus materials corresponding to the business type, segment the Chinese knowledge corpus materials to obtain initial text fragments; For each initial text segment, vocabulary is extracted from the initial text segment based on a pre-generated template vocabulary database to obtain the template vocabulary in the initial text segment; Non-template words are extracted from the template words in the initial text fragment, and the non-template words are analyzed to determine the professional vocabulary corpus; For each of the aforementioned professional vocabulary corpora, a contextual relationship chain is constructed based on the professional vocabulary corpora, and the professional vocabulary corpora are merged according to the nodes in the contextual relationship chain to generate word segmentation vocabulary; The non-template words in the initial text segment are merged according to the segmentation words, and the segmentation result of the initial text segment is determined based on the merging result and the template words.

2. The method according to claim 1, characterized in that, The step of extracting vocabulary from the initial text fragment based on a pre-generated template vocabulary database to obtain template vocabulary from the initial text fragment includes: Determine the vocabulary length corresponding to each template vocabulary in the template vocabulary database; Based on the word length in descending order, the initial text fragment is used to extract the template words in the initial text fragment.

3. The method according to claim 2, characterized in that, The process of extracting vocabulary from the initial text fragment based on the word length in descending order to obtain template vocabulary in the initial text fragment includes: Sort the word lengths in descending order, select the first word length as the current word length to be matched, and take the first word in the initial text segment as the starting point; Based on the current length of the word to be matched and the starting point, determine the word to be matched in the initial text segment, and match the word to be matched with the template words in the template word database; If a match is successful, a template word is extracted from the initial text fragment. The next word of the extracted template word in the initial text fragment is taken as the new starting point, and the length of the first word is taken as the new length of the current word to be matched. The process of determining the word to be matched in the initial text fragment based on the length of the current word to be matched and the starting point, and matching the word to be matched with the template words in the template word database is repeated until all words in the initial text fragment have been matched. If the matching fails and there is a corresponding next word length for the current word length to be matched, then the next word length of the current word length to be matched is taken as the new current word length to be matched, and the process returns to the step of determining the word to be matched in the initial text segment based on the current word length to be matched and the starting point, and matching the word to be matched with the template words in the template word database until all words in the initial text segment have been matched. If the match fails and there is no corresponding next word length for the current word to be matched, the next word of the starting point is taken as the new starting point, the first word length is taken as the new current word to be matched length, and the process returns to the step of determining the word to be matched in the initial text segment based on the current word to be matched length and the starting point, and matching the word to be matched with the template words in the template word database until all words in the initial text segment have been matched.

4. The method according to claim 1, characterized in that, The analysis of the non-template vocabulary to determine the specialized vocabulary corpus includes: Determine the word frequency of each of the non-template words. If the word frequency of the non-template words is greater than a threshold, the non-template words are determined to be high-frequency non-template words. The high-frequency non-template words are vectorized into text to obtain text vector data. Cluster the text vector data as described above, and determine the professional vocabulary corpus based on the clustering results.

5. The method according to claim 1, characterized in that, The construction of contextual relationship chains based on the aforementioned specialized vocabulary corpus includes: Using the professional vocabulary corpus as the central vocabulary of the contextual relationship chain, the Chinese knowledge corpus material is searched, and the preceding and following words are determined according to the predetermined length of the relationship chain and the position of the central vocabulary. The central word is used as the center of the context relationship chain, and the preceding and following words corresponding to the central word are used as nodes to construct the context relationship chain.

6. The method according to claim 1, characterized in that, The step of merging the specialized vocabulary corpus based on the nodes in the context relationship chain to generate segmented vocabulary includes at least one of the following: The weights of the nodes in the context relationship chain are determined, the segmentation positions are determined based on the weights of each node, and the professional vocabulary corpus is merged based on the segmentation positions to generate segmented vocabulary. The nodes in the context relationship chain are semantically encoded, and a dynamic semantic graph is constructed based on the encoding results. The dynamic semantic graph is queried to determine the segmentation position, and the professional vocabulary corpus is merged based on the segmentation position to generate segmented vocabulary.

7. The method according to claim 6, characterized in that, Determining the weights of nodes in the context relationship chain includes: The word frequency of each node in the context relationship chain is counted, and the word frequency of each node includes the number of documents and the total number of times. Determine the type of each node, wherein the node type is either a preceding node or a following node; For each node, the document frequency proportion and the total frequency proportion are determined based on the word frequency and corresponding type of the node. The document frequency proportion and the total frequency proportion are then weighted and summed to obtain the weight of the node.

8. The method according to claim 6, characterized in that, The step of semantically encoding the nodes in the context relationship chain and constructing a dynamic semantic graph based on the encoding results includes: Text corpus is generated based on the nodes in each context relationship chain, and semantic encoding is performed on the text corpus based on a pre-trained language model to obtain a semantic encoding vector; For every two semantic encoding vectors, the similarity between the two semantic encoding vectors is calculated, and the text corpora are clustered according to the similarity to generate a corpus text set; Dynamic semantic graphs are constructed based on the various corpus text sets.

9. The method according to any one of claims 1-8, characterized in that, It also includes at least one of the following: Determine the spatial vector coordinates of the central words in the context relationship chain, cluster the spatial vector coordinates, and generate semantic groups of professional words based on the clustering results; Generate specialized vocabulary semantic groups based on the corpus text set corresponding to the context relationship chain.

10. A question-and-answer method, characterized in that, include: Obtain unanswered questions related to connected vehicle services; Based on the question to be answered, a prompt text is constructed, and the question to be answered and the prompt text are input into the trained vehicle network big model for question answering. The vehicle network big model is trained by word segmentation using the Chinese knowledge text word segmentation method described in any one of claims 1-9. The question-and-answer results are determined based on the output of the aforementioned vehicle-to-everything (V2X) model.

11. The method according to claim 10, characterized in that, The step of constructing prompt text based on the question to be answered includes: The question to be answered is matched with the words in each pre-determined professional term semantic group, and the professional term semantic group that is successfully matched is taken as the target professional term semantic group. If the target professional vocabulary semantic group has a corresponding semantic prompt word, construct a prompt text based on the semantic prompt word; If there is no corresponding semantic prompt word in the target professional vocabulary semantic group, select a word from the target professional vocabulary semantic group to construct the prompt text.

12. The method according to claim 11, characterized in that, After selecting a word from the target professional vocabulary semantic group to construct the prompt text, the method further includes: The quality of the responses to the prompt text is assessed. The results of the response quality assessment of the prompt text corresponding to different words in the target professional vocabulary semantic group are analyzed to determine the semantic prompt words corresponding to the target professional vocabulary semantic group.

13. A Chinese knowledge text segmentation device, characterized in that, include: The material segmentation module is used to acquire Chinese knowledge corpus materials corresponding to business types, segment the Chinese knowledge corpus materials, and obtain initial text fragments; The template vocabulary extraction module is used to extract vocabulary from each initial text fragment based on a pre-generated template vocabulary database to obtain template vocabulary from the initial text fragment. The professional terminology identification module is used to extract non-template words based on template words in the initial text fragment, and analyze the non-template words to identify the professional terminology corpus; The relation chain construction module is used to construct a context relation chain based on each of the professional vocabulary corpora, and merge the professional vocabulary corpora according to the nodes in the context relation chain to generate word segmentation vocabulary; The word segmentation module is used to merge non-template words in the initial text segment according to each of the word segmentation words, and determine the word segmentation result of the initial text segment based on the merging result and the template words.

14. A question-and-answer device, characterized in that, include: The question retrieval module is used to retrieve unanswered questions related to connected vehicle services. The question-answering module is used to construct prompt text based on the question to be answered, and input the question to be answered and the prompt text into the trained vehicle network big model for question answering. The vehicle network big model is trained by word segmentation using the Chinese knowledge text word segmentation method described in any one of claims 1-9. The question-and-answer result determination module is used to determine the question-and-answer result based on the output of the vehicle network big data model.

15. An electronic device, characterized in that, The electronic device includes: At least one processor, and a memory communicatively connected to said at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the Chinese knowledge text segmentation method according to any one of claims 1-9 or the question-and-answer method according to any one of claims 10-12.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the Chinese knowledge text segmentation method of any one of claims 1-9 or the question-answering method of any one of claims 10-12.

17. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the Chinese knowledge text segmentation method according to any one of claims 1-9 or the question-answering method according to any one of claims 10-12.

Images