Dialect intelligent customer service and culture knowledge base system

Abstract

The invention relates to the technical field of agricultural travel services, and provides a dialect intelligent customer service and culture knowledge base system, the system comprises a perception layer, an analysis layer, a knowledge layer and an interaction layer four-dimensional architecture, the perception layer collects and preprocesses dialect voice, noise reduction is performed through spectral subtraction, and Mel frequency cepstrum coefficient features are extracted; the analysis layer identifies dialects based on a fine tuning Wav2Vec2.0 model, converts the dialects into mandarin through a Transform architecture, and completes intention identification and slot filling by using a BERT related model; the knowledge layer constructs a local culture knowledge graph containing non-abandoned, folk and other entities, and supports dynamic updating; and the interaction layer is combined with multiple rounds of dialogue management to generate multiform responses, and can be connected with an external service system. The system also optimizes a feedback module iterative model and knowledge. The system can cover more than ten dialects, realizes dialect interaction, culture interpretation and service conversion closed loop, and assists rural culture revitalizing and rural cultural travel service upgrading.

Description

Technical Field

[0001] This invention relates to the field of ferrous metal smelting technology, specifically to a dialect-based intelligent customer service and cultural knowledge base system. Background Technology

[0002] Current rural tourism service systems suffer from significant shortcomings in dialect support, cultural knowledge integration, interactive coherence, and information timeliness. Most systems only support Mandarin and a few mainstream dialects, exhibiting poor recognition of less common dialects such as Hakka and Minnan, hindering communication for elderly rural users who rely on dialects. Furthermore, customer service can only provide basic service information, failing to connect it to local cultural connotations. Users inquiring about culture-related issues must be transferred to human operators, resulting in low efficiency. Moreover, multi-turn conversations struggle to remember users' dialect habits and interests, leading to a stiff interactive experience. Updates to local cultural knowledge rely on manual input, which is time-consuming and unable to promptly respond to new intangible cultural heritage projects or temporary agricultural festivals, failing to meet users' needs for precise services. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a dialect-based intelligent customer service and cultural knowledge base system, which solves the problems of insufficient dialect support, fragmented cultural knowledge, awkward interaction, and lagging knowledge updates in existing customer service systems.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a dialect-based intelligent customer service and cultural knowledge base system, comprising: The perception layer is used to collect dialect speech signals input by the user, and to perform noise reduction and feature extraction on the dialect speech signals to obtain speech feature vectors; The parsing layer, connected to the perception layer, is used to receive the speech feature vector, convert the speech feature vector into dialect text through a dialect recognition model, convert the dialect text into Mandarin text, and perform intent recognition and slot filling on the Mandarin text to obtain user intent and structured information. The knowledge layer stores a local cultural knowledge graph, which contains a network structure with cultural entities in the agricultural, cultural and tourism fields as nodes and relationships between entities as edges. The parsing layer is connected to the knowledge layer and is used to query the knowledge graph according to the user intent and the structured information to obtain the corresponding cultural knowledge information and service data. The interaction layer, connecting the parsing layer and the knowledge layer, is used to generate and output responses in dialect voice and text form that contain cultural interpretation and service recommendation information, based on the cultural knowledge information and service data obtained from the query and in combination with a multi-turn dialogue management strategy.

[0005] Preferably, the sensing layer includes: The audio input module is compatible with mobile phone microphones, smart speakers, or scenic area audio guide devices, and is used to collect original dialect voice signals. The noise reduction module uses spectral subtraction to estimate and remove environmental noise from the original dialect speech signal; The feature extraction module is used to extract the Mel-frequency cepstral coefficient feature vector from the denoised speech signal as the speech feature vector.

[0006] Preferably, the parsing layer includes: The dialect recognition module, built on a finely tuned Wav2Vec2.0 model, is used to convert the speech feature vector into a dialect pinyin sequence. The dialect conversion module, built on the sequence-to-sequence architecture of Transformer, is used to convert the dialect pinyin sequence into Mandarin text; The semantic understanding module uses the BERT model for intent classification and a combination of a conditional random field model for slot filling to parse out the user intent and the structured information.

[0007] Preferably, the local cultural knowledge graph in the knowledge layer includes node entity types such as intangible cultural heritage projects, folk activities, agricultural seasons, and rural scenic spots; the knowledge graph is constructed and updated in the following ways: Structured data was extracted from local chronicles and official cultural and tourism documents, and entity similarity was calculated using the BERT model for entity alignment. Use a graph database for storage, supporting fuzzy entity queries; By connecting to local cultural and tourism data sources through API interfaces and setting up a manual review backend, the cultural knowledge can be dynamically updated.

[0008] Preferably, the interaction layer includes: The dialogue management module uses a memory network to store the history of multiple rounds of dialogue, which is used to associate user context information in subsequent dialogues and to actively ask follow-up questions in dialect when slot information is missing. The response generation module is used to generate response content that integrates cultural origins, folk significance, and real-time service information based on the query results. The service integration module reserves API interfaces for connecting with external e-commerce or appointment systems, converting users' service appointment requests into orders and updating service data in the knowledge graph in real time.

[0009] Preferably, the system further optimizes the feedback module, which is used to collect user rating feedback on the response, send low-rated samples to the manual review queue, and use the reviewed and corrected data for incremental training and dynamic updating of the dialect recognition model and the local culture knowledge graph.

[0010] Preferably, the dialect recognition module supports at least ten dialects, including Hakka, Minnan, Xiang, Gan, and Wu.

[0011] Preferably, the response format output by the interaction layer includes one or more of the following: dialect speech, Mandarin speech, and mixed text and graphics.

[0012] This invention provides a dialect-based intelligent customer service and cultural knowledge base system. It has the following beneficial effects: This invention effectively eliminates communication barriers for dialect-speaking users, enabling rural populations reliant on dialects to smoothly access agricultural, cultural, and tourism services. It significantly improves the efficiency of cultural consultation and service booking. For businesses, it increases the exposure of intangible cultural heritage workshops and folk experience projects, helping to boost order volume and business efficiency. In terms of cultural inheritance, the system naturally disseminates local culture through interactive interaction, enhancing young people's understanding of intangible cultural heritage, folk customs, and other traditional cultures, promoting the protection and inheritance of niche dialects and local cultures, and facilitating the development of rural agricultural, cultural, and tourism industries and the revitalization and utilization of cultural resources. Detailed Implementation Example

[0013] This invention provides a specific implementation process for a dialect-based intelligent customer service and cultural knowledge base system, using the example of Minnan language users inquiring about Quanzhou Tusundong culture and making experience reservations.

[0014] The system's hardware environment includes audio input devices, computing devices, and storage devices. The audio input devices utilize Huawei's M-precision microphones, compatible with iOS / Android ate60 smartphone microphones, supporting a 16kHz sampling rate and 16-bit depth. Alternatively, Xiaomi AI speakers deployed in scenic areas can be used, supporting far-field sound pickup with a pickup distance of ≤5 meters. The computing devices employ an Intel Xeon Gold 6330 processor with 32 cores and 128GB DDR4 memory, supporting parallel model computing to ensure efficient data processing. Storage devices utilize the Neo4j AuraDB cloud database with a 100GB storage capacity and read / write capabilities. With a throughput of ≥1000 QPS, it meets the data storage and query requirements of knowledge graphs. In terms of software environment, the server uses the Ubuntu 22.04LTS operating system, and the client mobile APP supports iOS 15.0+ and Android 11.0+ versions. Model training and inference use the PyTorch 2.0 framework, and speech processing relies on the Librosa 0.10.1 library for feature extraction and the PyAudio 0.2.13 library for audio acquisition. The client and server communicate using the RESTful API protocol, with data transmission format JSON and a response time of ≤500ms to ensure smooth interaction.

[0015] After the audio input module of the perception layer is activated, the user opens the "Rural Cultural Tourism Service" APP on their mobile phone, clicks the "Dialect Consultation" function button, and speaks the consultation content in Minnan dialect, "I want to know the story of the sea worm jelly and also want to experience making it." The phone's microphone collects the original voice signal in real time, which lasts for 8 seconds, stores it as a WAV format file, and transmits it to the system backend. After receiving the original speech signal, the noise reduction module automatically extracts the first 3 seconds of audio without speech. The background of this period is the sound of waves from the beach in Xunpu Village, Quanzhou. The noise spectrum of this period is estimated by using an improved spectral subtraction method. Then, the noise spectrum is subtracted from the estimated noise spectrum. After processing, the noise energy is reduced by 80%, and the signal-to-noise ratio is increased from 25dB to 40dB, effectively removing environmental interference. The feature extraction module calls the Librosa library to extract a 39-dimensional Mel-frequency cepstral coefficient feature vector from the denoised speech signal. Among them, 13 static coefficients reflect the basic characteristics of speech such as fundamental frequency and energy, and 13 first-order difference coefficients and 13 second-order difference coefficients capture the dynamic change characteristics of speech. After extraction, the feature vector is passed to the parsing layer in NumPy array format.

[0016] The dialect recognition module in the parsing layer receives the 39-dimensional feature vector output by the perception layer and inputs it into the fine-tuned Wav2Vec2.0 model. This model is pre-trained with the LibriSpeech dataset to initialize the parameters, and then fine-tuned using 500,000 dialect conversation data in the agricultural, cultural, and tourism fields. Among them, there are 80,000 Minnan dialect corpora, including local exclusive terms such as "tusundong" and "Xunpu Village". The fine-tuning process iterates 100 rounds, and the connectionist temporal classification loss function is used to optimize the model parameters. Finally, the Minnan dialect pinyin sequence "nguanxiangzaiianntushundongegushi,haixiangtiyenzozuo" is output. After testing, the recognition accuracy of this module for Minnan dialect reaches 96.1%. The dialect conversion module inputs the above dialect pinyin sequence into a sequence-to-sequence architecture model based on Transformer. The encoder and decoder of this model both contain 6 layers of Transformer structures. It is trained with 100,000 dialect-Mandarin parallel corpora, and 2000 mapping relationships of exclusive terms in the agricultural, cultural, and tourism fields are added to the training corpora. After the model processes, the Mandarin text "我想知道土笋冻的故事,还想体验制作" is output, and the conversion accuracy is 98.2%. The semantic understanding module inputs the Mandarin text into the BERT+Softmax model and the BERT+CRF model respectively. The BERT+Softmax model is pre-trained based on the Chinese Wikipedia corpus and fine-tuned with 10,000 intention annotation samples in the agricultural, cultural, and tourism fields, classifying the user's intention as "cultural consultation + service reservation". The BERT+CRF model extracts the structured slot information "cultural entity: tusundong, service type: production experience", and the F1 value of slot filling is 92.5%, without missing key information. After the parsing is completed, the user's intention and structured information are transmitted to the knowledge layer.

[0017] The local cultural knowledge graph in the knowledge layer is stored in the Neo4j graph database. The attributes of the "tusundong" entity include basic attributes, such as origin time: Ming Dynasty, intangible cultural heritage level: Quanzhou-level intangible cultural heritage, production raw materials: sipunculus, associated services, such as experience workshop: Xunpu Village Intangible Cultural Heritage Workshop, opening hours: 9:00 - 17:00, reservation requirements: 1 day in advance, ID number is required. The entity association relationships include "tusundong - belongs to region → Quanzhou", "tusundong - associated experience → production experience". When receiving the user's intention and structured information transmitted by the parsing layer, the system queries the knowledge graph through Cypher statements and obtains the corresponding cultural knowledge information and service data within 200ms. The data update of the knowledge graph is realized by docking the open API of the Quanzhou Cultural and Tourism Bureau, automatically synchronizing new activity information every day, such as the "May 1st Tusundong Cultural Festival". After synchronization, the artificial reviewer verifies the authenticity of the information through the Web background, and the review duration ≤ 1 hour. After passing the verification, it is automatically updated to the knowledge graph to ensure the timeliness of the data, and the update cycle ≤ 24 hours.

[0018] The dialogue management module of the interaction layer uses a memory network to store the current dialogue history, including the dialect type used by the user. Taking Minnan dialect as an example, the inquiry content includes the culture of Tusun jelly and the experience of making it. After analyzing the slot information output by the parsing layer, it is found that "experience time and number of participants" are missing. The system automatically generates a follow-up question in Minnan dialect, "When do you want to experience it? How many people are in your party?" and generates dialect speech based on the Minnan dialect voice library through speech synthesis technology, which is then output to the user. The user replies in Minnan dialect, "We are 3 people, going tomorrow." After processing by the perception layer and parsing layer, the system converts the sentence into Mandarin, "We are 3 people, going tomorrow," and fills in the slot information "Time: tomorrow, Number of people: 3" and transmits it to the interaction layer. The response generation module integrates cultural and service information returned from the knowledge graph to generate a Minnan (Hokkien) text response. It also converts the response into a dialect-based audio and text-image mix, outputting it to the user in multiple formats. The service integration module connects to the Quanzhou Agricultural, Cultural, and Tourism Reservation Platform via a reserved API interface, automatically filling in user and reservation information and generating an order number. After the user clicks the "Reservation" button in the response to submit the order, the system synchronizes the order information to the knowledge graph in real time, updating "Xunpu Village Intangible Cultural Heritage Workshop - Remaining Spots" from 10 to 7, ensuring the accuracy of service data for subsequent user queries.

[0019] After a service is completed, the system's optimization feedback module displays a star rating window via a mobile app. Users select "4 stars" and leave a message like "Satisfied, the story is very detailed, and booking is convenient." The module records this feedback and adds the "Hokkien speech + text + Mandarin text" of the conversation to the training set. Each month, 10,000 similar user dialogue samples are collected for incremental training of the Wav2Vec2.0 model. The training process iterates 50 times, improving the Hokkien recognition accuracy from 96.1% to 96.8%. If a user rating is less than 3 stars, such as feedback like "the health benefits of sea worm jelly were not explained," the module automatically sends the sample to a manual review queue. Reviewers add the knowledge content "Sea worm jelly contains protein and collagen, suitable for middle-aged and elderly people" to the knowledge graph. This addition takes effect within 2 hours, enabling continuous optimization of the system model and knowledge data.

[0020] Tests showed that in this application scenario, the system achieved a 96.1% accuracy rate in recognizing Minnan dialect and a 98.2% accuracy rate in converting dialect to Mandarin, meeting the dialect communication needs of rural users aged 60 and above. During testing, 72% of elderly users reported that they "could accurately understand the user's needs without needing to repeat themselves." The total time from initiating an inquiry to completing an appointment was ≤3 minutes, significantly improving efficiency compared to traditional human customer service. Through system integration, the Xunpu Village Tusun Dong workshop in Quanzhou saw its monthly orders increase from 80 to 250, and the awareness rate of "Tusun Dong Intangible Cultural Heritage" among users aged 18-35 increased from 32% to 89%, fully demonstrating the system's value in user service, merchant operations, and cultural dissemination, aligning with the national rural cultural revitalization strategy.

[0021] During the operation of the system, all 10 supported dialects are trained through the corpus provided by local cultural and tourism departments to ensure that the dialect expressions conform to local language habits. For example, in Hakka, "ngai" corresponds to "I", and in Xiang dialect, "xiá fàn" corresponds to "eat rice". The manual review background of the knowledge graph needs to be equipped with more than 2 local cultural commissioners familiar with agricultural, cultural and tourism knowledge to ensure the authenticity and cultural professionalism of the newly added data. When training the model, the federated learning framework is used to process voice data, and the original voice files of users are not stored. Only the feature vectors and text information are retained to strictly protect user privacy.

[0022] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A dialect-based intelligent customer service and cultural knowledge base system, characterized in that: include: The perception layer is used to collect dialect speech signals input by the user, and to perform noise reduction and feature extraction on the dialect speech signals to obtain speech feature vectors; The parsing layer, connected to the perception layer, is used to receive the speech feature vector, convert the speech feature vector into dialect text through a dialect recognition model, convert the dialect text into Mandarin text, and perform intent recognition and slot filling on the Mandarin text to obtain user intent and structured information. The knowledge layer stores a local cultural knowledge graph, which contains a network structure with cultural entities in the fields of agriculture, culture and tourism as nodes and relationships between entities as edges. The parsing layer is connected to the knowledge layer and is used to query the knowledge graph based on the user intent and the structured information to obtain corresponding cultural knowledge information and service data. The interaction layer, connecting the parsing layer and the knowledge layer, is used to generate and output responses in dialect voice and text form that contain cultural interpretation and service recommendation information, based on the cultural knowledge information and service data obtained from the query and in combination with a multi-turn dialogue management strategy.

2. The dialect intelligent customer service and cultural knowledge base system according to claim 1, characterized in that, The sensing layer includes: The audio input module is compatible with mobile phone microphones, smart speakers, or scenic area audio guide devices, and is used to collect original dialect voice signals. The noise reduction module uses spectral subtraction to estimate and remove environmental noise from the original dialect speech signal; The feature extraction module is used to extract the Mel-frequency cepstral coefficient feature vector from the denoised speech signal as the speech feature vector.

3. The dialect intelligent customer service and cultural knowledge base system according to claim 1, characterized in that, The parsing layer includes: The dialect recognition module, built on a finely tuned Wav2Vec2.0 model, is used to convert the speech feature vector into a dialect pinyin sequence. The dialect conversion module, built on the sequence-to-sequence architecture of Transformer, is used to convert the dialect pinyin sequence into Mandarin text; The semantic understanding module uses the BERT model for intent classification and a combination of a conditional random field model for slot filling to parse out the user intent and the structured information.

4. The dialect intelligent customer service and cultural knowledge base system according to claim 1, characterized in that, The local cultural knowledge graph in the knowledge layer includes node entity types such as intangible cultural heritage projects, folk activities, agricultural seasons, and rural scenic spots; the knowledge graph is constructed and updated in the following ways: Structured data was extracted from local chronicles and official cultural and tourism documents, and entity similarity was calculated using the BERT model for entity alignment. Use a graph database for storage, supporting fuzzy entity queries; By connecting to local cultural and tourism data sources through API interfaces and setting up a manual review backend, the cultural knowledge can be dynamically updated.

5. The dialect intelligent customer service and cultural knowledge base system according to claim 1, characterized in that, The interaction layer includes: The dialogue management module uses a memory network to store the history of multiple rounds of dialogue, which is used to associate user context information in subsequent dialogues and to actively ask follow-up questions in dialect when slot information is missing. The response generation module is used to generate response content that integrates cultural origins, folk significance, and real-time service information based on the query results. The service integration module reserves API interfaces for connecting with external e-commerce or appointment systems, converting users' service appointment requests into orders and updating service data in the knowledge graph in real time.

6. The dialect intelligent customer service and cultural knowledge base system according to claim 1, characterized in that, The system also optimizes the feedback module to collect user ratings of responses, sends low-rated samples to a manual review queue, and uses the reviewed and corrected data for incremental training and dynamic updates of the dialect recognition model and the local cultural knowledge graph.

7. The dialect intelligent customer service and cultural knowledge base system according to claim 3, characterized in that, The dialect recognition module supports at least ten dialects, including Hakka, Minnan, Xiang, Gan, and Wu.

8. The dialect intelligent customer service and cultural knowledge base system according to claim 1, characterized in that, The response format output by the interaction layer includes one or more of the following: dialect speech, Mandarin speech, and mixed text and graphics.