Method, apparatus, device, and medium for loading session data based on structured storage

Abstract

This application discloses a method, apparatus, device, and medium for loading session data based on structured storage, relating to the field of natural language processing technology. The method includes: acquiring raw session data and corresponding metadata; parsing the raw session data; generating first structured session data based on the parsing results and metadata; structurally storing the first structured session data according to data access frequency; and constructing a target index corresponding to the first structured session data based on the parsing results and metadata; receiving a data acquisition request; retrieving corresponding second structured session data from the first structured session data based on the target index; sorting and deduplicating each second structured session data based on the session start time corresponding to each second structured session data; and loading each second structured session data. By structurally storing the session data, the efficiency of data retrieval is improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method, apparatus, device, and medium for loading session data based on structured storage. Background Technology

[0002] With the widespread application of dialogue systems such as intelligent customer service, chatbots, and voice assistants, efficient management of dialogue history has become a core technological challenge.

[0003] Currently, session data management typically involves storing data in an unstructured manner, relying on linear traversal of historical records for retrieval. This approach suffers from low data retrieval efficiency and cannot meet the millisecond-level response requirements of real-time interaction. Therefore, improving the efficiency of session data retrieval and loading has become a pressing technical challenge. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for loading session data based on structured storage, which can improve the efficiency of data retrieval by storing session data in a structured manner. The specific solution is as follows:

[0005] Firstly, this application provides a method for loading session data based on structured storage, including:

[0006] Obtain the original session data and the metadata corresponding to the original session data, parse the original session data using natural language processing technology, and generate the first structured session data corresponding to the original session data based on the corresponding parsing results and the metadata;

[0007] The first structured session data is stored in a structured manner according to the data access frequency corresponding to the first structured session data, and a target index corresponding to the first structured session data is constructed based on the parsing result and the metadata.

[0008] The system receives a data acquisition request from a target user, retrieves second structured session data corresponding to the data acquisition request from the first structured session data based on the target index, sorts and removes duplicates of each second structured session data based on the session start time corresponding to each second structured session data, and loads each second structured session data.

[0009] Optionally, the step of parsing the original conversation data using natural language processing technology includes:

[0010] Natural language processing techniques are used to perform entity recognition on the original session data in order to obtain the target entities in the original session data.

[0011] Natural language processing (NLP) techniques are used to identify the conversation purpose corresponding to each of the original conversation data, and NLP techniques are also used to determine the dependencies between the original conversation data in different rounds.

[0012] Optionally, the step of storing the first structured session data in a structured manner according to the data access frequency corresponding to the first structured session data includes:

[0013] Determine whether the data access frequency corresponding to the current first structured session data is less than a preset access frequency threshold. If the data access frequency corresponding to the current first structured session data is not less than the preset access frequency threshold, then cache the current first structured session data.

[0014] If the data access frequency corresponding to the current first structured session data is less than the preset access frequency threshold, then the current first structured session data is stored in the target distributed file system.

[0015] Obtain new first structured session data, determine the new first structured session data as the current first structured session data, and jump to the step of determining whether the data access frequency corresponding to the current first structured session data is less than a preset access frequency threshold.

[0016] Optionally, constructing the target index corresponding to the first structured session data based on the parsing result and the metadata includes:

[0017] Based on the parsing results and the metadata, a time index, a scene index, an entity index, and a semantic index are constructed corresponding to the first structured session data. The time index is used to retrieve the first structured session data according to the session occurrence time, the scene index is used to retrieve the first structured session data according to the session scene, the entity index is used to retrieve the first structured session data according to the target entity, and the semantic index is used to retrieve the first structured session data according to the session purpose.

[0018] Optionally, retrieving the second structured session data corresponding to the data acquisition request from the first structured session data based on the target index includes:

[0019] The first structured session data is retrieved using the time index to obtain the corresponding first retrieval result, and the first structured session data is retrieved using the scene index to obtain the corresponding second retrieval result;

[0020] The first structured session data is retrieved using the scene index to obtain the corresponding third retrieval result, and the first structured session data is retrieved using the semantic index to obtain the corresponding fourth retrieval result;

[0021] An intersection operation is performed on the first search result, the second search result, the third search result, and the fourth search result to obtain the second structured session data corresponding to the data acquisition request.

[0022] Optionally, the session data loading method based on structured storage further includes:

[0023] Determine the permission level corresponding to the target user, and retrieve the third structured session data corresponding to the permission level from the first structured session data based on the permission level and the target index.

[0024] Optionally, the session data loading method based on structured storage further includes:

[0025] Determine the sensitivity level of the first structured session data. If the sensitivity level of the first structured session data is less than a preset sensitivity level threshold, then directly store the first structured session data.

[0026] If the sensitivity level of the first structured session data is not less than the preset sensitivity level threshold, the first structured session data is encrypted using the target encryption algorithm to obtain the corresponding encrypted session data, and the encrypted session data is stored.

[0027] Secondly, this application provides a session data loading apparatus based on structured storage, comprising:

[0028] The data parsing module is used to obtain the original session data and the metadata corresponding to the original session data, parse the original session data using natural language processing technology, and generate the first structured session data corresponding to the original session data based on the corresponding parsing results and the metadata.

[0029] An index building module is used to store the first structured session data in a structured manner according to the data access frequency corresponding to the first structured session data, and to build a target index corresponding to the first structured session data based on the parsing result and the metadata.

[0030] The data retrieval module is used to receive a data acquisition request from a target user, retrieve second structured session data corresponding to the data acquisition request from the first structured session data based on the target index, sort and deduplicate each second structured session data based on the session start time corresponding to each second structured session data, and load each second structured session data.

[0031] Thirdly, this application provides an electronic device, comprising:

[0032] Memory, used to store computer programs;

[0033] A processor for executing the computer program to implement the aforementioned session data loading method based on structured storage.

[0034] Fourthly, this application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned session data loading method based on structured storage.

[0035] This application first obtains raw session data and its corresponding metadata. It then parses the raw session data using natural language processing (NLP) technology and generates first structured session data based on the parsing results and the metadata. Next, it stores the first structured session data in a structured manner according to the data access frequency. Finally, it constructs a target index for the first structured session data based on the parsing results and the metadata. Finally, it receives a data retrieval request from a target user and retrieves second structured session data corresponding to the request from the first structured session data based on the target index. The second structured session data is then sorted and deduplicated based on the session start time for each data point, and then loaded. Therefore, this application improves the efficiency of data analysis and retrieval by using NLP technology to parse raw session data, generate structured session data, and store the data in a structured manner, resulting in a fixed storage format. Furthermore, by constructing an index for the session data, it enables the retrieval of the session data needed by the target user, avoiding linear traversal of historical session records and significantly improving data retrieval efficiency. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0037] Figure 1 This is a flowchart of a session data loading method based on structured storage disclosed in this application;

[0038] Figure 2 This is a schematic diagram of a session data loading system based on structured storage disclosed in this application;

[0039] Figure 3 This is a flowchart illustrating a session data parsing method disclosed in this application;

[0040] Figure 4 This is a schematic diagram of a session data loading device based on structured storage disclosed in this application;

[0041] Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Current methods for retrieving session data suffer from low retrieval efficiency, failing to meet the millisecond-level response requirements of real-time interaction. To address this, this application provides a session data loading method based on structured storage, which improves data retrieval efficiency by storing session data in a structured manner.

[0044] See Figure 1 As shown, this embodiment of the invention discloses a session data loading method based on structured storage, including:

[0045] Step S11: Obtain the original session data and the metadata corresponding to the original session data, parse the original session data using natural language processing technology, and generate the first structured session data corresponding to the original session data based on the parsing results and the metadata.

[0046] The session data loading system based on structured storage in this embodiment is as follows: Figure 2 As shown, the system adopts a layered architecture, consisting of five main parts: data acquisition layer, structured processing layer, storage layer, index layer, and application layer. The overall workflow in this embodiment, taking "customer service system context loading" as an example, is as follows:

[0047] When a user sends a message to the customer service system, it triggers a context loading request (which includes the conversation ID (dentitydocument, i.e., identity identifier) ​​and the current round).

[0048] The application layer calls the context loading interface and selects either a sliding window or semantic association mode based on business requirements.

[0049] The index layer locates relevant dialogue IDs and message rounds through time indexing or semantic indexing.

[0050] The storage layer retrieves structured dialogue records from main storage or cache and returns them to the application layer.

[0051] The application layer passes context data to the dialogue engine to generate a response.

[0052] In this embodiment, the process of acquiring the original session data and the corresponding metadata is handled by the data acquisition layer described above. The specific functions of the data acquisition layer are as follows:

[0053] The data acquisition layer is responsible for capturing dialogue interaction data in real time, including:

[0054] a) Original dialogue content (i.e., original conversation data): The text / voice interaction content between the user and the system (stored after being converted to text).

[0055] b) Metadata information (i.e., metadata): conversation ID, user ID, session start time, terminal device, business scenario tags (such as "customer service consultation" and "order inquiry"), etc.

[0056] c) Interactive behavior data: message sending time, input duration, message type (text, image, file), etc. Data is accessed through API (Application Programming Interface) or SDK (Software Development Kit), supporting multiple protocols such as HTTP (Hypertext Transfer Protocol) and WebSocket.

[0057] By modeling the association between entities and intents, the system improves the accuracy of contextual understanding in multi-turn dialogues by 40% and reduces response errors caused by missing historical information.

[0058] In this embodiment, the process of parsing the original conversation data using natural language processing technology may specifically include: using natural language processing technology to perform entity recognition on the original conversation data to obtain the target entities in the original conversation data; using natural language processing technology to identify the conversation purpose corresponding to each original conversation data; and using natural language processing technology to determine the dependency relationship between the original conversation data in different rounds.

[0059] The process of parsing the raw conversation data using natural language processing techniques is handled by the structured processing layer described above. Specifically:

[0060] The core function of the structured processing layer is to transform raw dialogue data into a structured metadata model, which includes two key modules:

[0061] a) Dialogue parsing engine:

[0062] The dialogue content is analyzed in depth using NLP (Natural Language Processing) technology.

[0063] Entity recognition: Extract key entities mentioned by the user (such as name, order number, product model) and use pre-trained models such as BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model) in conjunction with a domain dictionary.

[0064] Intent Classification: Identify the core intent of each round of dialogue, i.e. the purpose of the conversation (such as "inquiry", "complaint", "place order"), and use the FastText algorithm (a text classification and word vector representation algorithm) for multi-classification.

[0065] Context dependency analysis: Constructing dependencies between dialogue rounds (such as whether the current dialogue responds to the previous round's question), achieved through dependency parsing and dialogue state tracking techniques.

[0066] Sentiment analysis: Assess user sentiment (positive, negative, neutral) for subsequent service quality analysis.

[0067] b) Metadata Modeling Module:

[0068] Constructing a three-layer metadata model:

[0069] Basic layer: basic information such as dialogue ID, user ID, session time, and business scenario.

[0070] Interaction layer: Interaction features such as message type, sender (user / system), message status (read / unread), and input duration.

[0071] Semantic layer: Semantic features such as entity list, intent label, context dependency, and sentiment label.

[0072] The final structured dialogue record is generated, as shown in the example below:

[0073] {

[0074] "dialogue_id": "20250428_1001",

[0075] "user_id": "U12345",

[0076] "start_time": "2025-04-28 09:00:00",

[0077] "business_scenario": "order_query",

[0078] "messages": [

[0079] {

[0080] "message_id": "1",

[0081] "sender": "user",

[0082] "timestamp": "2025-04-28 09:01:00",

[0083] "content": "What is the status of my orders 123456789?"

[0084] "entities": ["Order No.:123456789"],

[0085] "intent": "order_status_inquiry",

[0086] "dependencies": ["none"], / / No prerequisite dependencies

[0087] "sentiment": "neutral"

[0088] },

[0089] {

[0090] "message_id": "2",

[0091] "sender": "system",

[0092] "timestamp": "2025-04-28 09:01:10",

[0093] "content": "Your order 1234567 is currently being shipped and is expected to arrive today."

[0094] "entities": ["Order No.:123456789"],

[0095] "intent": "order_status_response",

[0096] "dependencies": ["1"], / / Reply to the first message

[0097] "sentiment": "neutral"

[0098] } ]

[0100] }

[0101] Specifically, such as Figure 3 As shown, the parsing process of the raw session data is divided into three stages:

[0102] a) Raw data cleaning: Remove invalid characters, convert to a unified encoding format, and process multimodal data (such as speech-to-text).

[0103] b) In-depth analysis of NLP:

[0104] Word segmentation and part-of-speech tagging: Using the jieba word segmentation tool (a Python library for Chinese text segmentation) in conjunction with a domain dictionary improves the accuracy of entity recognition.

[0105] Named Entity Recognition (NER): Employs the "BERT+CRF (Conditional Random Field)" model, fine-tuned for specific domains (such as e-commerce and finance), achieving an entity recognition F1 score of over 95%.

[0106] Intent classification: Construct a multi-layer neural network model, with input being word vectors of the dialogue text (generated through Word2Vec (a neural network model) or GPT (Generative Pre-Trained Transformer), and output being intent labels (supporting multi-label classification).

[0107] c) Metadata generation: Map the parsing results to a three-layer metadata model to generate a structured JSON (a document format) document.

[0108] The process of obtaining the context dependencies of session data mentioned above includes:

[0109] Define three types of dependencies:

[0110] a) Direct reply: The current message explicitly replies to a historical message (such as "Regarding the order issue you just mentioned..."), which is identified by matching keywords such as "reply" and "just now" combined with syntactic analysis.

[0111] b) Topic continuation: If the current message and historical messages belong to the same topic (such as continuous inquiries about product parameters), it is determined by calculating the cosine similarity of sentence vectors (with a threshold set to 0.7).

[0112] c) State dependency: State transitions in the dialogue state machine (such as from "query order" to "modify address") are tracked through the business scenario state model.

[0113] By establishing a multi-level dialogue metadata model, standardized descriptions of dialogue content, interaction information, and semantic features are achieved, thereby realizing structured storage of conversation data and improving the efficiency of subsequent data retrieval.

[0114] Step S12: Store the first structured session data in a structured manner according to the data access frequency corresponding to the first structured session data, and construct the target index corresponding to the first structured session data based on the parsing result and the metadata.

[0115] In this embodiment, the process of storing the first structured session data in a structured manner according to the data access frequency corresponding to the first structured session data may specifically include: determining whether the data access frequency corresponding to the current first structured session data is less than a preset access frequency threshold; if the data access frequency corresponding to the current first structured session data is not less than the preset access frequency threshold, then caching the current first structured session data; if the data access frequency corresponding to the current first structured session data is less than the preset access frequency threshold, then storing the current first structured session data in the target distributed file system; obtaining new first structured session data, determining the new first structured session data as the current first structured session data, and jumping to the step of determining whether the data access frequency corresponding to the current first structured session data is less than the preset access frequency threshold.

[0116] The process of storing session data described above is handled by the aforementioned storage layer, specifically:

[0117] The storage layer employs a hybrid architecture combining distributed key-value stores (such as Cassandra) and document databases (such as MongoDB):

[0118] a) Primary storage: MongoDB (a distributed document storage database) is used to store complete structured conversation records, supporting flexible schema (the structure used to describe and organize data) design and complex queries.

[0119] b) Hot data caching: Redis stores the recent conversation history of frequently accessed data, reducing database pressure.

[0120] c) Cold data archiving: Regularly archive conversation history exceeding 30 days to a distributed file system to reduce storage costs.

[0121] Data partitioning strategy: Sharding by user ID hash to ensure load balancing; it also supports partitioning by business scenario tags for easy domain management.

[0122] Furthermore, this embodiment employs a hybrid strategy of "hash sharding + range sharding":

[0123] a) Hash Sharding: Hash the user ID and distribute it to different storage nodes to ensure load balancing. Sharding key formula: shard_id = hash(user_id) % node_count.

[0124] b) Scope segmentation: Divide the scope according to business scenario tags (such as "customer service" and "marketing") to facilitate domain management and access control.

[0125] High availability design:

[0126] a) Replication mechanism: Each shard data is stored in 3 replicas, distributed across different racks, and the consensus algorithm is implemented through the Raft protocol, with a failover time of ≤200ms.

[0127] b) Elastic scaling: When the node load exceeds the threshold (e.g., CPU utilization > 80%), data migration is automatically triggered, and the migration process does not affect normal services.

[0128] In this embodiment, the process of constructing a target index corresponding to the first structured session data based on the parsing results and metadata can specifically include: constructing a time index, a scene index, an entity index, and a semantic index corresponding to the first structured session data based on the parsing results and metadata; wherein, the time index is used to retrieve the first structured session data according to the session occurrence time, the scene index is used to retrieve the first structured session data according to the session scene, the entity index is used to retrieve the first structured session data according to the target entity, and the semantic index is used to retrieve the first structured session data according to the session purpose.

[0129] The process of building the above index is handled by the aforementioned index layer, specifically:

[0130] The index layer constructs a multi-dimensional index system to support fast contextual retrieval:

[0131] a) Time Index: A B+ tree index is built based on the conversation timestamp, supporting fast queries by time range.

[0132] b) Scene Index: Build an inverted index for business scene tags to facilitate the aggregation of dialogue history by scene.

[0133] c) Entity Index: Create a global index for key entities (such as order number, user ID) to support "entity-dialogue" related retrieval.

[0134] d) Semantic Indexing: Utilize Elasticsearch to build a full-text search engine, perform word segmentation indexing on dialogue content, intent tags, and entities, and support semantic fuzzy queries.

[0135] Index update mechanism: Asynchronous incremental updates are adopted to ensure that index building does not affect the performance of the main process, and the update latency is controlled within 50ms.

[0136] Step S13: Receive a data acquisition request from the target user, retrieve the second structured session data corresponding to the data acquisition request from the first structured session data based on the target index, sort and deduplicate each second structured session data based on the session start time corresponding to each second structured session data, and load each second structured session data.

[0137] In this embodiment, the process of retrieving the second structured session data corresponding to the data acquisition request from the first structured session data based on the target index may specifically include: retrieving the first structured session data using a time index to obtain a corresponding first retrieval result, and retrieving the first structured session data using a scene index to obtain a corresponding second retrieval result; retrieving the first structured session data using a scene index to obtain a corresponding third retrieval result, and retrieving the first structured session data using a semantic index to obtain a corresponding fourth retrieval result; and performing an intersection operation on the first retrieval result, the second retrieval result, the third retrieval result, and the fourth retrieval result to obtain the second structured session data corresponding to the data acquisition request.

[0138] Specifically, this embodiment adopts a "primary index + secondary index" architecture:

[0139] a) Primary index: The primary key is the dialogue ID, which is stored in a distributed key-value store to ensure primary key queries with a time complexity of O(1).

[0140] b) Secondary index:

[0141] Time range index: Partitioned by year / month / day, with a B+ tree index built in each partition, supporting fast time range queries.

[0142] Semantic Inverted Index: An inverted index is built for each intent tag and entity type, recording all dialogue IDs containing that tag / entity, as shown in the example below:

[0143] Intent label: order_status_inquiry → [dialogue ID1, dialogue ID3, dialogue ID5].

[0144] Entity type: Order number → 123456789 → [Dialogue ID1, Dialogue ID2].

[0145] c) Distributed index synchronization: Cross-node synchronization of index change events is achieved through Apache Kafka (a distributed messaging system), ensuring index consistency and a fault recovery time of less than 30 seconds.

[0146] That is, the indexing process in this embodiment utilizes multiple types of indexes to simultaneously retrieve the first structured session data. In one specific embodiment, if a user needs to query "logistics feedback for order 123456 last week", then this example uses multiple types of indexes to perform data retrieval simultaneously: Entity index:

[0147] Entity index: Lock all messages containing "order123456" → Result set A (i.e., the first search result);

[0148] Time Index: Filter messages from the past 7 days → Result Set B (i.e., the second search result);

[0149] Semantic index: Match the intent of "logistics feedback" → result set C (i.e., the third search result);

[0150] Finally, the intersection of result sets A, B, and C is taken; that is, the overlapping portion of result sets A, B, and C is the session data that the user needs to query. Through the three-layer metadata model, data retrieval efficiency is improved by 300% compared to traditional log storage, and the response time for complex queries (such as queries combining entities and intents) is reduced from seconds to sub-seconds.

[0151] In this embodiment, the process of indexing session data is handled by the aforementioned application layer, specifically:

[0152] The application layer provides a unified API interface that supports the following core functionalities:

[0153] a) Context loading interface: Returns the dialogue history within a specified context range based on the dialogue ID, user ID, and target round (supports both sliding window and semantic association modes).

[0154] b) Dialogue retrieval interface: Supports querying dialogue records by combining conditions such as time, scenario, entity, and intent.

[0155] c) Data analysis interface: Provides statistical data such as dialogue duration, intent distribution, and entity occurrence frequency for business analysis.

[0156] In addition, this embodiment presents two session data loading models to meet the needs of different scenarios:

[0157] a) Sliding window mode (suitable for short dialogue scenarios):

[0158] Extract the most recent N rounds of dialogue in chronological order (e.g., N=50), quickly locate them using time index, and the loading delay is no more than 50ms.

[0159] Algorithm pseudocode:

[0160] def load_sliding_window(dialogue_id, current_round, window_size=50):

[0161] start_round = max(1, current_round - window_size + 1)

[0162] messages = db.query(dialogue_id).where(round >= start_round,round <= current_round)

[0163] return messages;

[0164] b) Semantic association patterns (suitable for long conversations or cross-topic scenarios):

[0165] Based on the entity and intent of the current message, all relevant historical messages (including direct replies, topic continuations, and state-dependent messages) are found through semantic indexing.

[0166] Steps: 1. Extract the entity list (e.g., order number, product ID) and intent tags of the current message. 2. Obtain all historical message IDs containing these entities using the entity index. 3. Filter messages with the same or related intents (e.g., "order_status_inquiry" and "order_modification" are considered related) using the intent index. 4. Sort by time and remove duplicates, returning the Top-K related messages (K=20).

[0167] Algorithm optimization: The Approximate Nearest Neighbor Search (ANNS) algorithm is used to accelerate semantic similarity queries, with a response time of ≤100ms. In sliding window mode, the loading time for 50 rounds of dialogue is ≤50ms; in semantic association mode, the retrieval time for 20 related messages is ≤100ms, meeting the requirements of real-time interaction.

[0168] It should be noted that in this embodiment, the permission level corresponding to the target user can also be determined, and the third structured session data corresponding to the permission level can be retrieved from the first structured session data according to the permission level and the target index.

[0169] Specifically, this embodiment designs role-based access control:

[0170] Based on the RBAC (Role-Based Access Control) model, permissions are set according to the sensitivity level (public / internal / sensitive) of the dialogue data:

[0171] Regular users: can only access their own conversation history.

[0172] Customer service supervisor: Can access all conversation history of their team, but requires secondary authentication (SMS verification code + device fingerprint).

[0173] Administrator: Has full access to all data, but operations must be logged in an audit log.

[0174] c) Audit trail:

[0175] Records all data access activities, with logs including: user ID, operation time, accessed interface, returned data volume, IP address, etc. Supports generating compliance reports by time range, meeting the GDPR (General Data Protection Regulation) requirement to store data access records for at least 6 months.

[0176] In addition, this embodiment can also determine the sensitivity level of the first structured session data. If the sensitivity level of the first structured session data is less than the preset sensitivity level threshold, the first structured session data is directly stored. If the sensitivity level of the first structured session data is not less than the preset sensitivity level threshold, the first structured session data is encrypted using the target encryption algorithm to obtain the corresponding encrypted session data, and the encrypted session data is stored.

[0177] In other words, transmission encryption: TLS (Transport Layer Security) 1.3 protocol is used to encrypt communication between the data acquisition layer and the storage layer, and the key is rotated periodically. Storage encryption: Sensitive fields (i.e., data exceeding a preset sensitivity threshold), such as user IDs and order numbers, are encrypted using AES-256 (the target encryption algorithm), and the encryption key is stored in the Key Management Service (KMS).

[0178] In addition, this embodiment supports high-concurrency access of tens of thousands of QPS (Queries Per Second), and the cluster nodes can be dynamically expanded, with storage capacity and throughput scaling linearly with business growth.

[0179] By using end-to-end encryption and fine-grained access control for sensitive data, audit logs fully record data access history, helping enterprises meet domestic and international data security requirements.

[0180] Therefore, this application utilizes natural language processing technology to parse raw conversation data, generate structured conversation data, and store the conversation data in a structured manner, giving the data a fixed storage format, thereby improving the efficiency of data analysis and retrieval. By constructing an index corresponding to the conversation data, the target user can retrieve the conversation data needed by the target user using the index, avoiding linear traversal of historical conversation records, thus greatly improving the efficiency of data retrieval.

[0181] See Figure 4As shown, this embodiment of the invention discloses a session data loading device based on structured storage, comprising:

[0182] The data parsing module 11 is used to obtain the original session data and the metadata corresponding to the original session data, parse the original session data using natural language processing technology, and generate the first structured session data corresponding to the original session data based on the corresponding parsing results and the metadata.

[0183] The index building module 12 is used to store the first structured session data in a structured manner according to the data access frequency corresponding to the first structured session data, and to build a target index corresponding to the first structured session data based on the parsing result and the metadata.

[0184] The data retrieval module 13 is used to receive a data acquisition request from a target user, retrieve second structured session data corresponding to the data acquisition request from the first structured session data based on the target index, sort and deduplicate each second structured session data based on the session start time corresponding to each second structured session data, and load each second structured session data.

[0185] In some specific embodiments, the data parsing module 11 may specifically include:

[0186] An entity recognition unit is used to perform entity recognition on the original session data using natural language processing technology to obtain the target entity in the original session data.

[0187] The dependency determination unit is used to identify the conversation purpose corresponding to each of the original conversation data using natural language processing technology, and to determine the dependency relationship between the original conversation data in different rounds using natural language processing technology.

[0188] In some specific embodiments, the index building module 12 may specifically include:

[0189] A data caching unit is used to determine whether the data access frequency corresponding to the current first structured session data is less than a preset access frequency threshold. If the data access frequency corresponding to the current first structured session data is not less than the preset access frequency threshold, then the current first structured session data is cached.

[0190] The first data storage unit is configured to store the current first structured session data in the target distributed file system if the data access frequency corresponding to the current first structured session data is less than the preset access frequency threshold.

[0191] The step jump unit is used to obtain new first structured session data, determine the new first structured session data as the current first structured session data, and jump to the step of determining whether the data access frequency corresponding to the current first structured session data is less than a preset access frequency threshold.

[0192] In some specific embodiments, the index building module 12 may specifically include:

[0193] An index building unit is used to build a time index, a scenario index, an entity index, and a semantic index corresponding to the first structured session data based on the parsing results and the metadata; wherein, the time index is used to retrieve the first structured session data according to the session occurrence time, the scenario index is used to retrieve the first structured session data according to the session scenario, the entity index is used to retrieve the first structured session data according to the target entity, and the semantic index is used to retrieve the first structured session data according to the session purpose.

[0194] In some specific embodiments, the data retrieval module 13 may specifically include:

[0195] The first data retrieval unit is used to retrieve the first structured session data using the time index to obtain the corresponding first retrieval result, and to retrieve the first structured session data using the scene index to obtain the corresponding second retrieval result.

[0196] The second data retrieval unit is used to retrieve the first structured session data using the scene index to obtain the corresponding third retrieval result, and to retrieve the first structured session data using the semantic index to obtain the corresponding fourth retrieval result.

[0197] The session data acquisition unit is used to perform an intersection operation on the first search result, the second search result, the third search result, and the fourth search result to obtain the second structured session data corresponding to the data acquisition request.

[0198] In some specific embodiments, the session data loading device based on structured storage further includes:

[0199] The permission determination module is used to determine the permission level corresponding to the target user, and retrieve the third structured session data corresponding to the permission level from the first structured session data according to the permission level and the target index.

[0200] In some specific embodiments, the session data loading device based on structured storage further includes:

[0201] The second data storage unit is used to determine the sensitivity level of the first structured session data. If the sensitivity level of the first structured session data is less than a preset sensitivity level threshold, the first structured session data is stored directly.

[0202] The data encryption unit is used to encrypt the first structured session data using a target encryption algorithm if the sensitivity level of the first structured session data is not less than the preset sensitivity level threshold, so as to obtain the corresponding encrypted session data and store the encrypted session data.

[0203] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0204] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the session data loading method based on structured storage disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0205] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0206] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0207] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the structured storage-based session data loading method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0208] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed session data loading method based on structured storage. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0209] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0210] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0211] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0212] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0213] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for loading session data based on structured storage, characterized by, The method comprises: obtaining original conversation data and metadata corresponding to the original conversation data, parsing the original conversation data by using a natural language processing technology, and generating first structured conversation data corresponding to the original conversation data based on the corresponding parsing result and the metadata; wherein the metadata comprises a dialogue ID, a user ID, a conversation start time, a terminal device, and a business scenario label; structurally storing the first structured conversation data according to the data access frequency of the first structured conversation data, and constructing a target index corresponding to the first structured conversation data based on the parsing result and the metadata; receiving a data acquisition request of a target user, retrieving second structured conversation data corresponding to the data acquisition request from the first structured conversation data based on the target index, sorting and deduplicating each of the second structured conversation data based on the conversation start time corresponding to each of the second structured conversation data, and loading each of the second structured conversation data; wherein the parsing of the original conversation data by using the natural language processing technology comprises: performing entity recognition on the original conversation data by using the natural language processing technology to obtain target entities in the original conversation data; identifying the conversation purposes of each of the original conversation data by using the natural language processing technology, and determining the dependency relationship between the original conversation data of different rounds by using the natural language processing technology; the construction of the target index corresponding to the first structured conversation data based on the parsing result and the metadata comprises: constructing a time index, a scene index, an entity index, and a semantic index corresponding to the first structured conversation data based on the parsing result and the metadata; wherein the time index is used to retrieve the first structured conversation data according to the conversation occurrence time, the scene index is used to retrieve the first structured conversation data according to the conversation scene, the entity index is used to retrieve the first structured conversation data according to the target entity, and the semantic index is used to retrieve the first structured conversation data according to the conversation purpose; the retrieval of the second structured conversation data corresponding to the data acquisition request from the first structured conversation data based on the target index comprises: retrieving the first structured conversation data by using the time index to obtain a corresponding first retrieval result, and retrieving the first structured conversation data by using the scene index to obtain a corresponding second retrieval result; retrieving the first structured conversation data by using the scene index to obtain a corresponding third retrieval result, and retrieving the first structured conversation data by using the semantic index to obtain a corresponding fourth retrieval result; performing intersection operation on the first retrieval result, the second retrieval result, the third retrieval result, and the fourth retrieval result to obtain the second structured conversation data corresponding to the data acquisition request.

2. The method of claim 1, wherein, The first structured conversation data is stored according to the data access frequency corresponding to the first structured conversation data, including: determining whether the data access frequency corresponding to the current first structured conversation data is less than a preset access frequency threshold, if the data access frequency corresponding to the current first structured conversation data is not less than the preset access frequency threshold, the current first structured conversation data is cached; if the data access frequency corresponding to the current first structured conversation data is less than the preset access frequency threshold, the current first structured conversation data is stored in the target distributed file system; obtaining new first structured conversation data, determining the new first structured conversation data as the current first structured conversation data, and jumping to the step of determining whether the data access frequency corresponding to the current first structured conversation data is less than a preset access frequency threshold.

3. The method of claim 1, wherein, Also including: determining the permission level corresponding to the target user, and retrieving third structured conversation data corresponding to the permission level from the first structured conversation data according to the permission level and the target index.

4. The method of claim 1 to 3, wherein, Also including: determining the sensitivity level of the first structured conversation data, if the sensitivity level of the first structured conversation data is less than a preset sensitivity level threshold, the first structured conversation data is directly stored; if the sensitivity level of the first structured conversation data is not less than the preset sensitivity level threshold, the first structured conversation data is encrypted by using a target encryption algorithm to obtain corresponding encrypted conversation data, and the encrypted conversation data is stored.

5. A structured storage based session data loading apparatus, characterized by, Including: a data analysis module, configured to obtain original conversation data and metadata corresponding to the original conversation data, analyze the original conversation data by using natural language processing technology, and generate first structured conversation data corresponding to the original conversation data based on the corresponding analysis result and the metadata; wherein the metadata includes conversation ID, user ID, conversation start time, terminal device and business scenario label; an index construction module, configured to store the first structured conversation data according to the data access frequency corresponding to the first structured conversation data, and construct a target index corresponding to the first structured conversation data based on the analysis result and the metadata; a data retrieval module, configured to receive a data acquisition request of a target user, retrieve second structured conversation data corresponding to the data acquisition request from the first structured conversation data based on the target index, sort and deduplicate each second structured conversation data based on the conversation start time corresponding to each second structured conversation data, and load each second structured conversation data; wherein the data analysis module includes: an entity recognition unit, configured to recognize entities in the original conversation data by using natural language processing technology to obtain target entities in the original conversation data; A dependency relationship determining unit is configured to identify the conversation purposes corresponding to the original conversation data respectively by using a natural language processing technique, and determine the dependency relationship between the original conversation data in different rounds by using the natural language processing technique; The index construction module comprises: An index construction unit is configured to construct a time index, a scene index, an entity index and a semantic index corresponding to the first structured conversation data based on the parsing result and the metadata; the time index is used to retrieve the first structured conversation data according to a conversation occurrence time, the scene index is used to retrieve the first structured conversation data according to a conversation scene, the entity index is used to retrieve the first structured conversation data according to the target entity, and the semantic index is used to retrieve the first structured conversation data according to the conversation purpose; The data retrieval module comprises: A first data retrieval unit is configured to retrieve the first structured conversation data by using the time index to obtain a corresponding first retrieval result, and retrieve the first structured conversation data by using the scene index to obtain a corresponding second retrieval result; A second data retrieval unit is configured to retrieve the first structured conversation data by using the scene index to obtain a corresponding third retrieval result, and retrieve the first structured conversation data by using the semantic index to obtain a corresponding fourth retrieval result; A conversation data obtaining unit is configured to perform an intersection operation on the first retrieval result, the second retrieval result, the third retrieval result and the fourth retrieval result to obtain the second structured conversation data corresponding to the data obtaining request.

6. An electronic device, comprising: comprise: a memory configured to save a computer program; a processor configured to execute the computer program to implement the structured storage based conversation data loading method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, a memory configured to save a computer program; the computer program is executed by a processor to implement the structured storage based conversation data loading method according to any one of claims 1 to 4.

Images