De-duplication method, apparatus and device based on artificial intelligence

Abstract

The application relates to the technical field of artificial intelligence, and discloses a de-duplication method, device and equipment based on artificial intelligence, which comprises the following steps: obtaining preset triple parameters; wherein the preset triple parameters comprise session memory parameters, personal memory parameters and organizational memory parameters; based on the preset triple parameters, tenant data isolation processing, multi-dimensional memory permission configuration processing and session memory range limitation processing are respectively performed on a three-layer memory model to determine a hierarchical memory data set with a tenant permission tag; wherein the three-layer memory model comprises session memory, personal memory and organizational memory; multi-dimensional retrieval processing is performed on the hierarchical memory data set, and a historical target memory content corresponding to to-be-added memory content is determined in combination with a weight fusion sorting and screening mode; and based on the triple parameters and the historical target memory content, progressive de-duplication verification is performed on the to-be-added memory content to obtain a hierarchical de-duplication result. The application can improve de-duplication accuracy.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a deduplication method, apparatus, and device based on artificial intelligence. Background Technology

[0002] In large-scale model dialogue interaction and enterprise multi-tenant intelligent service scenarios, the system needs to continuously accumulate user conversation interaction data to form long-term memory content in order to improve the continuity of intelligent interaction and personalized service capabilities.

[0003] However, as conversations accumulate, a massive amount of data, including conversation facts, user behavior information, and team business rules, is continuously added, which can easily generate a large amount of repetitive, similar, and redundant memory content, leading to problems such as low retrieval efficiency and repeated or disordered intelligent replies.

[0004] In related technologies, a global hash digest (such as MD5) is calculated for the entire text to be added to generate a unique hash value. The hash values ​​of the entire text are directly compared to see if they are completely identical. Only when two texts are exactly the same, they are judged as duplicates and removed to avoid problems such as low retrieval efficiency and repeated or disordered intelligent replies.

[0005] However, hash digest calculation can only identify completely identical text content, which has great limitations. For example, two texts may have the same semantic content and core information, but different sentence structure, word order, and modifiers. Hash digest calculation cannot retrieve duplicate content, resulting in redundant accumulation in the memory and poor deduplication accuracy. Summary of the Invention

[0006] In view of this, this application provides an artificial intelligence-based deduplication method to solve the problem that hash digest calculation cannot accurately deduplicate memory content in related technologies.

[0007] Firstly, this application provides an artificial intelligence-based deduplication method, comprising: obtaining preset triplet parameters; wherein the preset triplet parameters include session memory parameters, personal memory parameters, and organizational memory parameters; based on the preset triplet parameters, performing tenant data isolation processing, multi-dimensional memory permission configuration processing, and session memory range limitation processing on the three-layer memory model respectively, to determine a hierarchical memory dataset with tenant permission labels; wherein the three-layer memory model includes session memory, personal memory, and organizational memory; performing multi-dimensional retrieval processing on the hierarchical memory dataset, and combining a weighted fusion sorting and filtering method to determine the historical target memory content corresponding to the memory content to be added; and performing progressive deduplication verification on the memory content to be added based on the triplet parameters and the historical target memory content to obtain hierarchical deduplication results.

[0008] The AI-based deduplication method provided in this application obtains preset triplet parameters, including session memory parameters, personal memory parameters, and organizational memory parameters. Based on these preset triplet parameters, it performs tenant data isolation processing, multi-dimensional memory permission configuration processing, and session memory range limitation processing on the three-layer memory model composed of session memory, personal memory, and organizational memory, respectively, to determine a hierarchical memory dataset with tenant permission tags. Then, it performs multi-dimensional retrieval processing on the hierarchical memory dataset and combines weighted fusion sorting and filtering to determine the historical target memory content corresponding to the new memory content. Based on the preset triplet parameters and the historical target memory content, it performs progressive deduplication verification on the new memory content. Unlike the single verification method in related technologies that only relies on global hash digest comparison to identify completely identical text, this method can overcome the limitation that hash digest calculation can only identify completely identical text content. It can effectively identify duplicates for memory content that is semantically identical and has the same information but has differences in sentence expression, word order, and modifiers. This effectively solves the technical problems of large recognition limitations, easy accumulation of redundant memory databases, and poor deduplication accuracy of hash deduplication methods.

[0009] In one possible implementation, based on preset triplet parameters, tenant data isolation processing, multi-dimensional memory permission configuration processing, and session memory scope limitation processing are applied to the three-layer memory model respectively to determine a hierarchical memory dataset with tenant permission labels. This includes: based on organizational memory parameters, stripping the memory data of target users to obtain memory stripping results; wherein, the memory stripping results indicate data isolation between different target users; within the same organizational identifier, based on personal memory parameters, configuring private memory permissions for memory data carrying non-empty valid user identifiers to obtain a first permission configuration result; within the same organizational identifier, based on organizational memory parameters, configuring public permissions for memory data with empty user identifiers and no specified bound users to obtain a second permission configuration result; based on session memory parameters, limiting the scope of temporary interactive memory in the current dialogue scenario, defining temporary memory permissions that are only effective for a single session to obtain a session memory limitation result; and based on the memory stripping results, the first permission configuration results, the second permission configuration results, and the session memory limitation results, determining a hierarchical memory dataset with tenant permission labels.

[0010] The AI-based deduplication method provided in this application peels away the memory data of target users based on organizational memory parameters to obtain memory peeling results that indicate data isolation between different target users, thus achieving mutual isolation of memory data of different target users. Then, within the same organizational identifier, private memory permissions are configured for memory data carrying non-empty valid user identifiers based on personal memory parameters to obtain a first permission configuration result. Public permissions are configured for memory data with empty user identifiers and no specified bound users based on organizational memory parameters to obtain a second permission configuration result. Furthermore, the scope of temporary interactive memory in the current dialogue scenario is limited based on session memory parameters, and temporary memory permissions that are only effective for a single session are defined to obtain a session memory limitation result. Finally, the memory peeling results, the first permission configuration results, the second permission configuration results, and the session memory limitation results are combined to determine a hierarchical memory dataset with tenant permission labels. Through multi-dimensional and hierarchical parameter configuration, user-level data isolation, organization-wide private permission configuration, organization-wide public permission configuration, and session temporary permission limitation are completed sequentially, so that memory data of different users, different permission attributes, and different effective scenarios form standardized boundaries, avoiding the problems of various types of memory data being mixed together and permission boundaries being blurred.

[0011] In one possible implementation, based on triplet parameters and historical target memory content, a progressive deduplication check is performed on the new memory content to obtain hierarchical deduplication results, including: performing hash calculation on the new memory content to obtain a first deduplication result based on triplet parameters and historical target memory content; if the first deduplication result indicates that no duplicate content was found, performing vector semantic similarity comparison processing on the new memory content and historical target memory content to obtain a second deduplication result; and fusing the first and second deduplication results to generate a hierarchical deduplication result.

[0012] The AI-based deduplication method provided in this application uses hash calculation to screen for duplicate content. If no duplicate content is found in the first deduplication result, a second deduplication result is obtained by comparing the vector semantic similarity of the new content to be added and the historical target content. Finally, the first and second deduplication results are merged to generate a hierarchical deduplication result. That is, hash calculation achieves accurate and efficient identification of completely duplicate content, while vector semantic similarity comparison compensates for the inability of a single hash calculation to identify semantically similar content. This two-layered differentiated deduplication method progressively completes the deduplication determination, balancing the computational efficiency of accurate deduplication with the comprehensiveness of semantic deduplication identification. It avoids the drawback of a single hash deduplication method, which can only match completely identical text and cannot identify semantically redundant content, effectively improving the completeness and accuracy of deduplication determination for new content to be added.

[0013] In one possible implementation, based on the hierarchical deduplication results, the newly added memory content is processed to obtain compliant memory content, including: if the first deduplication result in the hierarchical deduplication results indicates that the newly added memory content is non-duplicate and the second deduplication result in the hierarchical deduplication results indicates that there is a historical memory entry with semantic similarity to the newly added memory content, the original historical memory content is iteratively updated to obtain compliant memory content; if the first deduplication result in the hierarchical deduplication results indicates that the newly added memory content is non-duplicate and the second deduplication result in the hierarchical deduplication results indicates that there is no historical memory entry with semantic similarity to the newly added memory content, the newly added memory content is determined to be compliant memory content.

[0014] The AI-based deduplication method provided in this application, under the premise that the first deduplication result in the hierarchical deduplication result indicates that the content to be added is not duplicated, executes differentiated memory processing logic according to the different comparison states of the second deduplication result. When the second deduplication result indicates that there is a historical memory entry that is semantically similar to the content to be added, the original historical memory content is iteratively updated to obtain compliant memory content. When the second deduplication result indicates that there is no historical memory entry that is semantically similar to the content to be added, the content to be added is directly determined as compliant memory content. It can accurately distinguish and process the two types of content to be added: those that are not completely duplicated but have semantic similarity and those that are completely dissimilar. This avoids the problem of inaccurate memory data caused by the aging and failure of semantically similar historical memory content and the information lag, while ensuring the normal storage and retention of new memory content. It also eliminates the problem of effective memory loss or redundant memory accumulation caused by uniform processing methods.

[0015] In one possible implementation, the method further includes: if the first deduplication result in the hierarchical deduplication results indicates that the content to be added is duplicated, then delete the content to be added.

[0016] The AI-based deduplication method provided in this application deletes the new memory content when the first deduplication result in the hierarchical deduplication result indicates that the new memory content is duplicated. This method can directly remove new memory content that is determined to be completely duplicated by the first deduplication result, thus eliminating memory data that completely overlaps with historical memory content. This prevents completely duplicated new memory content from being continuously entered into the memory dataset, avoiding the problem of redundant and bloated memory datasets and wasted data storage resources caused by the accumulation of completely duplicated memory content.

[0017] In one possible implementation, based on the first and second plagiarism check results, a hierarchical plagiarism check result is generated by fusing them. This includes: extracting the first and second plagiarism check results corresponding to the same new memory content, and establishing a corresponding binding relationship between the first and second plagiarism check results; performing state matching judgment based on the corresponding binding relationship to determine three memory comparison states: complete repetition, semantic repetition, and no repetition; and encapsulating and generating a hierarchical plagiarism check result based on the three memory comparison states: complete repetition, semantic repetition, and no repetition.

[0018] The AI-based deduplication method provided in this application extracts the first and second deduplication results corresponding to the same memory content to be added, establishes a corresponding binding relationship between the first and second deduplication results, performs state matching judgment based on the corresponding binding relationship, determines three memory comparison states: complete repetition, semantic repetition, and no repetition, and encapsulates and generates hierarchical deduplication results based on the three memory comparison states. This method can associate and unify the two different types of deduplication results corresponding to a single memory content to be added, realize the state linkage judgment of the deduplication dimension and the semantic deduplication dimension, and standardize and comprehensively sort out all the repetition situations of the memory content to be added. This ensures that the generated hierarchical deduplication results can fully reflect the actual repetition state of the memory content to be added, avoiding the problems of one-sided judgment of a single deduplication dimension and the ambiguity of deduplication conclusions and inconsistent judgment standards caused by the fragmentation of multi-dimensional deduplication results.

[0019] In one possible implementation, multi-dimensional retrieval processing is performed on the hierarchical memory dataset, and a weighted fusion ranking and filtering method is combined to determine the historical target memory content corresponding to the new memory content. This includes: configuring a first weight for vector retrieval and a second weight for keyword retrieval based on the hierarchical memory dataset; performing semantic similarity retrieval processing on the memory content in the hierarchical memory dataset based on a preset vector index, and performing keyword retrieval processing on the memory content in the hierarchical memory dataset using a three-root index mechanism for target characters to obtain the original two-dimensional retrieval results; and performing a fusion weighted scoring and ranking of the original two-dimensional retrieval results based on the first and second weights to filter out the historical target memory content corresponding to the new memory content.

[0020] The deduplication method based on artificial intelligence provided in this application configures a first weight corresponding to vector retrieval and a second weight corresponding to keyword retrieval based on the memory dataset. It performs semantic similarity retrieval processing on the memory content in the hierarchical memory dataset according to a preset vector index, and simultaneously uses a three-root index mechanism of target characters to perform keyword retrieval processing on the memory content in the hierarchical memory dataset to obtain two-dimensional retrieval results. Then, the two-dimensional retrieval results are fused, weighted, scored, and sorted by the first and second weights. Finally, the historical target memory content corresponding to the memory content to be added is selected. It can combine the advantages of semantic retrieval and keyword retrieval at the same time. By configuring the weights, it can rationally fuse and correct the two types of retrieval results, avoiding the defects of insufficient matching accuracy of single semantic retrieval and the inability of single keyword retrieval to adapt to semantic fuzzy matching. It enables the retrieval process to take into account both semantic similarity matching and keyword accurate matching capabilities.

[0021] In one possible implementation, the method further includes: based on a preset driving timing, incrementally extracting key facts from the session interaction data corresponding to the session memory to generate initial candidate memory content; deduplicating the initial candidate memory content to remove redundant and duplicate content, obtaining compliant candidate memory content; matching the compliant candidate memory content with the corresponding target session memory, target personal memory, or target organizational memory; writing the compliant candidate memory content into the target session memory, target personal memory, or target organizational memory, and recording the data processing position of the target session memory, target personal memory, or target organizational memory based on a configured incremental sequence pointer; and locking the data processing position through the incremental sequence pointer.

[0022] The AI-based deduplication method provided in this application extracts key facts incrementally from the conversation interaction data corresponding to the conversation memory based on a preset driving timing to generate initial candidate memory content. The initial candidate memory content is then deduplicated, and redundant and duplicate content is removed to obtain compliant candidate memory content. This compliant candidate memory content is matched with the corresponding target conversation memory, target personal memory, or target organizational memory and written into the corresponding target-level memory. A configured incremental sequence pointer records the data processing position of the target conversation memory, target personal memory, or target organizational memory, and then uses the incremental sequence pointer to lock the corresponding data processing position. This method can extract incremental facts only for newly added interaction data in the conversation memory, without needing to perform a full scan of all historical memory data, reducing unnecessary computational overhead. Simultaneously, by removing redundant and duplicate content from the initial candidate memory content through deduplication, it ensures the cleanliness and effectiveness of the compliant candidate memory content. Furthermore, it can adaptively match the compliant candidate memory content to the corresponding target memory level and complete the writing process, avoiding the problem of memory content being stored incorrectly or haphazardly across levels.

[0023] Secondly, this application provides an artificial intelligence-based deduplication device, comprising: an acquisition module for acquiring preset triplet parameters, wherein the preset triplet parameters include session memory parameters, personal memory parameters, and organizational memory parameters; a first processing module for performing tenant data isolation processing, multi-dimensional memory permission configuration processing, and session memory range limitation processing on the three-layer memory model based on the preset triplet parameters, to determine a hierarchical memory dataset with tenant permission tags; wherein the three-layer memory model includes session memory, personal memory, and organizational memory; a determination module for performing multi-dimensional retrieval processing on the hierarchical memory dataset and combining a weighted fusion sorting and filtering method to determine the historical target memory content corresponding to the memory content to be added; a verification module for performing progressive deduplication verification on the memory content to be added based on the triplet parameters and the historical target memory content, to obtain hierarchical deduplication results; and a second processing module for processing the memory content to be added based on the hierarchical deduplication results to obtain compliant memory content.

[0024] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the artificial intelligence-based deduplication method described in the first aspect or any corresponding embodiment.

[0025] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the artificial intelligence-based deduplication method described in the first aspect or any corresponding embodiment thereof.

[0026] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute the artificial intelligence-based deduplication method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating an artificial intelligence-based deduplication method according to an embodiment of this application; Figure 2 This is a flowchart illustrating another AI-based deduplication method according to an embodiment of this application; Figure 3 This is a flowchart illustrating another AI-based deduplication method according to an embodiment of this application; Figure 4 This is a structural block diagram of an artificial intelligence-based deduplication device according to an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] According to an embodiment of this application, an artificial intelligence-based deduplication method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0031] This embodiment provides an artificial intelligence-based deduplication method that can be applied to computer devices such as computers and servers. Figure 1 This is a flowchart illustrating an AI-based deduplication method according to an embodiment of this application, as shown below. Figure 1 As shown, AI-based deduplication methods include: Step S101: Obtain preset triplet parameters; wherein, the preset triplet parameters include conversation memory parameters, personal memory parameters, and organizational memory parameters.

[0032] The preset triplet parameters can indicate a set of preset control parameters used to constrain the data range and permission rules of the three-layer memory model.

[0033] Session memory parameters can indicate control parameters used to limit the scope of temporary memory, permission period, and data retention rules for a single dialogue scenario, and only apply to single-round session interaction data; Personal memory parameters can indicate control parameters used to constrain the data isolation scope, access permissions, and storage rules of a user's private memories, and are only effective for the specified bound user; Organizational memory parameters can indicate control parameters used to constrain the tenant isolation scope, access permissions for all members, and data stripping rules of the organization's common memory, and apply to the entire organizational tenant dimension.

[0034] Specifically, the system reads the preset triplet parameters in the local configuration file. The session memory parameter is configured to be effective only for a single session and cleared temporarily when the session ends. The personal memory parameter is configured to be privately isolated for a single user and readable and writable only by the user. The organization memory parameter is configured to be interconnected within the same tenant and completely separated across tenants. This completes the initialization and acquisition of the triplet parameters.

[0035] In one possible implementation, the system connects to the enterprise tenant management backend and dynamically retrieves customized triplet parameters from cloud storage based on the currently logged-in tenant identifier and user identity permissions. For enterprise customer service scenarios, the conversation memory parameter is configured to retain customer service conversations throughout the entire process and archive them at the end of the shift. The personal memory parameter is configured to store customer service personnel's personal dialogue preferences privately. The organizational memory parameter is configured to share the enterprise's public customer service rules with all employees, thereby achieving dynamic parameter acquisition adapted to business scenarios.

[0036] In one possible implementation, a set of triplet parameter templates corresponding to three levels of permissions—ordinary user, administrator, and super administrator—is preset. Based on the current user's role and permission level, the corresponding triplet parameters are matched. Among them, the personal memory parameters of high-privilege users support batch export of private memories, the organizational memory parameters support editing of public memories, and the session memory parameters support reviewing session memories, thus realizing the parameter acquisition and configuration differentiated by role.

[0037] Step S102: Based on the preset triplet parameters, tenant data isolation processing, multi-dimensional memory permission configuration processing, and session memory scope limitation processing are implemented on the three-layer memory model respectively to determine the hierarchical memory dataset with tenant permission labels; wherein, the three-layer memory model includes session memory, personal memory, and organizational memory.

[0038] The three-layer memory model can indicate a hierarchical memory storage architecture, which can specifically include temporary scenario-level session memory, user-private level personal memory, and organization-public level organizational memory.

[0039] Tenant permission tags can indicate the identification tags used to mark the tenant, user, permission type, and effective scenario of the memory data, so as to achieve data isolation and permission differentiation for multiple tenants and multiple users.

[0040] A hierarchical memory dataset can refer to a standardized set of memory data formed after triplet parameter normalization, data isolation, and permission division.

[0041] Specifically, after determining the preset triplet parameters, the three-layer memory model can be processed to determine the hierarchical memory dataset with tenant permission labels.

[0042] As an example, data stripping and isolation across tenants and users is completed based on organizational memory parameters, permission configuration for private memories of users within the organization is completed based on personal memory parameters, the scope limitation of temporary memories for a single session is completed based on session memory parameters, and finally, tenant permission tags are uniformly attached to all memory data to form a hierarchical memory dataset.

[0043] As an example, for the enterprise office intelligent assistant scenario, the three-layer memory model is classified and processed based on triplet parameters. The conversation memory is organized into temporary data of a single office conversation and attached with temporary conversation tags. The personal memory is organized into private data of user office habits and personal preferences and attached with user private tags. The organizational memory is organized into public data of company rules and regulations and office processes and attached with public organizational tags, thereby realizing the construction of a hierarchical memory dataset exclusive to the office scenario.

[0044] As an example, based on the triplet parameters, a temporal dimension is added to process the conversation memory into segments and layers according to the conversation time and labeled with conversation temporal tags. Personal memory is sorted according to the update time and labeled with private update tags. Organizational memory is classified according to the publication time and labeled with public effective tags. Combined with permission isolation and temporal regularization, a hierarchical memory dataset with both permission differentiation and temporal attributes is generated.

[0045] Step S103: Perform multi-dimensional retrieval processing on the hierarchical memory dataset, and combine weighted fusion sorting and filtering methods to determine the historical target memory content corresponding to the new memory content.

[0046] Historical target memory content can indicate a set of historical memory data that is retrieved from the hierarchical memory dataset, matching the scene and theme of the current memory content to be added.

[0047] Specifically, after determining the hierarchical memory dataset, the content of historical target memory can be further determined.

[0048] As an example, based on a hierarchical memory dataset, vector retrieval is configured as the first weight and keyword retrieval as the second weight. Semantic similarity retrieval is completed through a preset vector index, and keyword retrieval is completed through a three-root index mechanism. The two-dimensional retrieval results are weighted, scored, and sorted to select the memory content with the highest matching degree, which is then determined as the historical target memory content corresponding to the new memory content.

[0049] As an example, for the user's life assistant interaction scenario, based on the tenant permission tags and scenario tags of the hierarchical memory dataset, historical memory data with the same user, the same scenario, and the same permissions are prioritized for screening. Thematically related memories are initially screened through precise keyword matching, and then invalid memories that are irrelevant to the theme or mismatched with the scenario are removed through semantic similarity screening to determine the historical target memory content that is suitable for the life assistant interaction scenario.

[0050] As an example, based on the time-series and permission tags of the hierarchical memory dataset, the latest historical memory content updated within the last 30 days and belonging to the same user and organization is selected first. The new and old memory data are then hierarchically weighted and scored, with priority given to matching recent memories with higher timeliness, while discarding expired and invalid old memory data to determine the target historical memory content.

[0051] Step S104: Based on the triplet parameters and historical target memory content, perform progressive deduplication verification on the new memory content to be added, and obtain hierarchical deduplication results.

[0052] The newly added memory content can indicate the latest user interaction facts, scenario information, business data, and other new memory content generated during the current dialogue interaction process that need to be stored in the database; The hierarchical deduplication results can indicate the fusion of precise deduplication dimensions and semantic deduplication dimensions to determine the redundant attributes of the new memory content.

[0053] Specifically, after determining the triplet parameters and the historical target memory content, the hierarchical deduplication results can be further determined.

[0054] As an example, based on the comparison range limited by the triplet parameters and the historical target memory content, the MD5 hash calculation is performed on the new memory content to obtain the first deduplication result. If the first deduplication result does not find completely duplicate content, the vector semantic similarity comparison is performed between the new memory content and the historical target memory content to obtain the second deduplication result. The two types of deduplication results are bound to determine the three memory states and are then fused to generate a hierarchical deduplication result.

[0055] As an example, the permission isolation rule based on triplet parameters selects only the historical target memory content within the current user's permission range as the comparison benchmark, performs hash-based accurate deduplication on the newly added memory content, filters invalid comparison data across users and tenants, and sets differentiated semantic similarity thresholds for private memory and public memory respectively under the premise of no complete duplication, completes semantic deduplication, and generates hierarchical deduplication results adapted to the permission level.

[0056] As an example, based on the scenario constraint rules of the triplet parameters, different semantic similarity thresholds are configured for session temporary memory, personal private memory, and organizational public memory. First, a unified hash-based accurate deduplication is performed, and then semantic comparison is completed according to the threshold differences of different memory levels to generate hierarchical deduplication results.

[0057] Step S105: Based on the hierarchical deduplication results, process the newly added memory content to obtain compliant memory content.

[0058] Compliant memory content refers to standardized and effective memory content that has undergone deduplication screening and differentiation processing, and has no complete duplication, no invalid semantic redundancy, and can be normally stored in the database.

[0059] Specifically, after determining the results of the hierarchical deduplication check, the newly added content can be processed according to the results to obtain compliant content.

[0060] As an example, if the hierarchical deduplication result indicates that the content to be added is completely duplicated, the content to be added is directly deleted; if there is no complete duplication but there is semantically similar content, the corresponding historical target content is iteratively updated to obtain compliant content; if there is no duplicate content, the content to be added is directly identified as compliant content.

[0061] As an example, for the new memory content corresponding to the session memory, when the semantics are similar, only the temporary session memory field is updated and not persisted; for the new memory content corresponding to the personal memory, when the semantics are similar, the private historical memory content is fully iterated and the user's private data is updated; for the new memory content corresponding to the organizational memory, when the semantics are similar, the organizational public memory is updated and all employee data is synchronized to obtain compliant memory content.

[0062] The AI-based deduplication method provided in this application obtains preset triplet parameters, including session memory parameters, personal memory parameters, and organizational memory parameters. Based on these preset triplet parameters, it performs tenant data isolation processing, multi-dimensional memory permission configuration processing, and session memory range limitation processing on the three-layer memory model composed of session memory, personal memory, and organizational memory, respectively, to determine a hierarchical memory dataset with tenant permission tags. Then, it performs multi-dimensional retrieval processing on the hierarchical memory dataset and uses a weighted fusion sorting and filtering method to determine the historical target memory content corresponding to the new memory content. Based on the preset triplet parameters and the historical target memory content, it performs progressive deduplication verification on the new memory content. Unlike the single verification method in related technologies that only relies on global hash digest comparison to identify completely identical text, this method can overcome the limitation of hash digest calculation that can only identify completely identical text content. It can effectively identify duplicates for memory content that is semantically identical and has the same information but has differences in sentence expression, word order, and modifiers. This effectively solves the technical problems of large recognition limitations, easy accumulation of redundant memory databases, and poor deduplication accuracy of hash deduplication methods.

[0063] like Figure 2 As shown, in one possible implementation, step S102 above, based on preset triplet parameters, performs tenant data isolation processing, multi-dimensional memory permission configuration processing, and session memory scope limitation processing on the three-layer memory model, and determines the hierarchical memory dataset with tenant permission labels, which may include the following steps: Step S2021: Based on the organizational memory parameters, the memory data of the target user is stripped to obtain the memory stripping result; wherein, the memory stripping result indicates the data isolation between different target users.

[0064] The target user can indicate the end user currently participating in intelligent interaction, generating conversational memories, and personal memory data.

[0065] The memory stripping results are used to indicate data isolation between different target users.

[0066] Specifically, after determining the organizational memory parameters, the memory data of the target user can be stripped to obtain the memory stripping result.

[0067] As an example, the system reads the organizational memory parameters, extracts the user unique identifier isolation rules carried in the organizational memory parameters, traverses the memory data, and groups and peels the memory data of different target users according to the user unique identifier bound to the memory data. This cuts the data association relationship between different users, generates a single-user memory data set, and obtains a memory peeling result that can achieve complete isolation of data of different target users.

[0068] As an example, based on the tenant and user two-layer isolation rules built into the organization memory parameters, the memory data of different enterprise tenants is first stripped according to the tenant identifier, and then the data is stripped again according to the user permission identifier for different target users within the same tenant, so as to realize the two-layer memory data isolation across tenants and across users and generate a memory stripping result that combines tenant isolation and user isolation.

[0069] Step S2022: Within the same organization identifier, based on personal memory parameters, configure private memory permissions for memory data carrying non-empty valid user identifiers to obtain the first permission configuration result.

[0070] Organizational identifiers can be used to uniquely distinguish the identities of different companies and teams.

[0071] A non-empty valid user identifier can indicate a unique identity that is real, verifiable, and bound to a specific target user, and is used to locate the owner of private memory data. The first permission configuration result can indicate the permission configuration result obtained after completing the private permission mounting and rule binding of the memory data belonging to a specific user.

[0072] Specifically, within the same organizational identifier, based on the preset private permission rules corresponding to the personal memory parameters, all memory data carrying non-empty valid user identifiers within the current organization are filtered out.

[0073] As an example, the data range with the same organization identifier is locked, the private permission configuration rules of the personal memory parameters are read, the memory content with non-empty valid user identifiers is filtered out, and the user's private read and write permissions are uniformly configured for this part of the memory data, limiting the operation permissions to only the bound target user, and other users in the organization have no access permissions, thus obtaining the first permission configuration result.

[0074] As an example, within the same organizational identity scope, based on the user role hierarchy rules preset by personal memory parameters, differentiated private memory permissions are configured for valid user identities of ordinary employees, administrators, and super administrators. Ordinary users can only view their own private memories, while administrators can manage the private memories of their subordinate users in batches, generating a first permission configuration result with role differentiation.

[0075] Step S2023: Within the same organization identifier, based on the organization memory parameters, configure public permissions for memory data with empty user identifiers and no specified bound user to obtain the second permission configuration result.

[0076] Memory data with an empty user identifier and no specified bound user can indicate general memory data generated within the same organization, not belonging to a single target user, and belonging to the organization's common business.

[0077] Public permissions can indicate open permissions that can be viewed, referenced, and reused by all members within the organization, and are only shared within the current organization's identity scope and are not open to external tenants.

[0078] The second permission configuration result can indicate the permission configuration result obtained after completing the public permission mounting and sharing scope definition of the organization's public memory data.

[0079] Specifically, within the same organizational identifier range, based on the public permission configuration rules carried by the organizational memory parameters, public memory data with empty user identifiers and no specified bound users are filtered out. Then, internal public permissions are configured for this part of the memory data, limiting it to be shared and accessible only to all members of the current organization and not reusable across organizations. This completes the permission regularization of the organizational public memory and generates the second permission configuration result.

[0080] As an example, within the same organizational identity scope, hierarchical public permissions are configured based on organizational memory parameters. General business rules and basic process data are configured with fully open permissions for all employees, while core organizational announcements and confidential business data are configured with restricted public permissions visible to management. This enables hierarchical sharing of public memory data and generates differentiated second permission configuration results.

[0081] Step S2024: Based on the session memory parameters, limit the scope of temporary interactive memory for the current dialogue scenario, define the temporary memory permissions that are only effective for a single session, and obtain the session memory limitation result.

[0082] Temporary interactive memory range can indicate the set boundary of all temporary question-and-answer data and scene context data generated during the current single dialogue interaction.

[0083] Temporary memory permissions can indicate temporary data permissions that are only effective within a single session period and expire or are archived after the session ends.

[0084] The session memory constraint result can indicate the session memory regularization result obtained after completing the definition of the session scenario scope, temporary permission binding, and effective period constraints.

[0085] Specifically, based on the scenario constraint rules and periodic rules preset in the conversation memory parameters, the scope of temporary interactive memory corresponding to the current conversation is determined. The contextual interactive data and temporary question-and-answer data generated in a single conversation are uniformly included in the scope of temporary memory management. At the same time, temporary memory permissions that are only effective for this conversation are defined, limiting this part of the data to be accessible and interactive only within the current conversation period. After the conversation ends, it will no longer participate in regular memory retrieval and reuse, generating a conversation memory limitation result.

[0086] As an example, the temporary control rules for session memory parameters are read, the scope of interactive data in the current single dialogue is locked, the temporary memory access and call permissions that are only effective for this session are defined, the temporary memory permissions are automatically invalidated after the session ends, and the data is only temporarily retained and does not participate in long-term memory updates. This completes the session memory regularization and yields the session memory limitation results.

[0087] As an example, the archiving rules based on the session memory parameters limit the scope of temporary interactive memory in the current dialogue scenario. For each session memory, temporary memory permissions are configured to be readable and writable during the session and automatically archived and locked when the session ends. The archived session memory can only be viewed retrospectively and cannot be modified, forming a traceable session memory limitation result.

[0088] Step S2025: Based on the memory stripping results, the first permission configuration results, the second permission configuration results, and the session memory limitation results, determine the hierarchical memory dataset with tenant permission labels.

[0089] After determining the memory stripping results, the first permission configuration results, the second permission configuration results, and the session memory limitation results, the corresponding tenant labels can be attached to the memory stripping results, the first permission configuration results, the second permission configuration results, and the session memory limitation results to determine the hierarchical memory dataset with tenant permission labels.

[0090] The AI-based deduplication method provided in this application separates the memory data of target users based on organizational memory parameters to obtain memory separation results that indicate data isolation between different target users, thus achieving mutual isolation of memory data of different target users. Then, within the same organizational identifier, private memory permissions are configured for memory data carrying non-empty valid user identifiers based on personal memory parameters to obtain the first permission configuration result. Public permissions are configured for memory data with empty user identifiers and no specified bound users based on organizational memory parameters to obtain the second permission configuration result. Furthermore, the scope of temporary interactive memory in the current dialogue scenario is limited based on session memory parameters, and temporary memory permissions that are effective only in a single session are defined to obtain the session memory limitation result. Finally, the memory separation result, the first permission configuration result, the second permission configuration result, and the session memory limitation result are combined to determine a hierarchical memory dataset with tenant permission labels. Through multi-dimensional and hierarchical parameter configuration, user-level data isolation, organization-wide private permission configuration, organization-wide public permission configuration, and session temporary permission limitation are completed sequentially. This ensures that memory data of different users, different permission attributes, and different effective scenarios form standardized boundaries, avoiding the problems of various types of memory data being mixed together and permission boundaries being blurred.

[0091] like Figure 3 As shown, in one possible implementation, step S103 above, which involves performing progressive deduplication verification on the new memory content based on the triplet parameters and historical target memory content to obtain hierarchical deduplication results, may include the following steps: Step S3031: Based on the triplet parameters and historical target memory content, perform hash calculation on the new memory content to be added to obtain the first deduplication result.

[0092] The first deduplication result can indicate the accurate deduplication result obtained through global hash comparison, which is used to determine whether the new memory content and the historical target memory content are completely identical.

[0093] Specifically, a global hash calculation is performed on the complete text of the content to be added to the memory, generating a corresponding unique hash digest value. At the same time, it is compared one by one with the preset hash value of the historical target memory content. Based on whether the hash value matches completely, a quick screening of completely duplicated content is completed, and the first deduplication result is output to represent the completely duplicated state.

[0094] Step S3032: If the first deduplication result indicates that no duplicate content was found, perform vector semantic similarity comparison processing on the newly added memory content and the historical target memory content to obtain the second deduplication result.

[0095] The second deduplication result can indicate the semantic layer deduplication result obtained through vector semantic comparison, which is used to determine whether there is a semantically similar implicit duplication state between the new memory content and the historical target memory content.

[0096] Specifically, after determining that the first plagiarism check result indicates that no duplicate content was found, a vector semantic similarity comparison can be performed between the newly added memory content and the historical target memory content to obtain the second plagiarism check result.

[0097] As an example, based on the first deduplication result determining that the content to be added does not completely duplicate the content, the content to be added and the historical target memory content corresponding to the permissions and scenarios are uniformly input into the semantic vector model to generate their respective semantic vectors. The cosine similarity algorithm is used to calculate the bidirectional semantic matching degree. A fixed semantic threshold is preset. If the similarity is greater than the threshold, it is determined that there is semantically similar content. If it is less than the threshold, it is determined that there is no semantically similar content, and the corresponding second deduplication result is generated.

[0098] As an example, based on the three-level memory hierarchy corresponding to the triplet parameters, differentiated semantic thresholds are configured. A loose similarity threshold is configured for conversation memory to adapt to the fuzzy semantic matching scenario of temporary dialogue. A high-precision similarity threshold is configured for personal memory and organizational memory. Vector semantic comparison is completed in layers to generate a second deduplication result.

[0099] Step S3033: Based on the first and second plagiarism check results, merge to generate hierarchical plagiarism check results.

[0100] After determining the first and second plagiarism check results, they can be merged to generate hierarchical plagiarism check results. The specific method for generating hierarchical plagiarism check results is described in detail below.

[0101] Specifically, step S3033 above may include the following steps: Step a1: Extract the first and second plagiarism check results corresponding to the same memory content to be added, and establish a corresponding binding relationship between the first and second plagiarism check results.

[0102] The corresponding binding relationship can indicate a data mapping relationship that uniquely associates the precise deduplication dimension results and semantic deduplication dimension results corresponding to the same memory content to be added, and creates a one-to-one correspondence.

[0103] Specifically, taking the single memory content to be added as the only subject, the first deduplication result obtained by hash calculation of the memory content to be added, and the second deduplication result generated corresponding to the memory content to be added are extracted. Using the unique ID of the memory content as the association key, the two types of deduplication results of the same subject are mapped and bound one by one to establish a fixed corresponding binding relationship.

[0104] Step a2: Based on the corresponding binding relationship, perform state matching and determination to identify three memory matching states: complete repetition, semantic repetition, and no repetition.

[0105] The state of complete duplication indicates that the first deduplication result determines that the new memory content is completely identical to the historical target memory content text, and there is a state of memory comparison with complete redundancy in the whole text.

[0106] The semantic duplication status can indicate a memory comparison status where the first duplication check result determines that there is no complete text duplication, but the second duplication check result determines that the content to be added is highly similar in semantics to the historical target content, and there is implicit semantic redundancy.

[0107] The "no duplication" status indicates that the first plagiarism check result is not completely duplicated, the second plagiarism check result is not semantically similar, the content to be added is a completely new and valid memory, and there is no redundant memory comparison status.

[0108] Specifically, based on the corresponding binding relationship between the first and second plagiarism check results, standardized logic judgment is performed: if the bound first plagiarism check result is completely duplicated, it is determined to be a completely duplicated state; if the bound first plagiarism check result is not duplicated and the second plagiarism check result is semantically similar, it is determined to be a semantically duplicated state; if both the bound first and second plagiarism check results are not duplicated, it is determined to be a non-duplicate state.

[0109] Step a3: Based on the three memory comparison states of complete repetition, semantic repetition, and no repetition, encapsulate and generate hierarchical deduplication results.

[0110] After determining the three memory comparison states—complete duplication, semantic duplication, and no duplication—the results can be encapsulated and generated as hierarchical deduplication results.

[0111] As an example, for the three memory comparison states of complete duplication, semantic duplication, and no duplication, the first and second duplication results are integrated and bound together, and uniformly encapsulated according to the standardized data structure. The state type, duplication dimension, and matching result are integrated into one to generate hierarchical duplication results.

[0112] The AI-based deduplication method provided in this application screens for duplicate content through hash calculation. If no duplicate content is found in the first deduplication result, a second deduplication result is obtained by comparing the vector semantic similarity of the new content to be added and the historical target content. Finally, the first and second deduplication results are merged to generate a hierarchical deduplication result. In other words, hash calculation achieves accurate and efficient identification of completely duplicate content, while vector semantic similarity comparison compensates for the inability of a single hash calculation to identify semantically similar content. This two-layered differentiated deduplication method progressively completes the deduplication determination, balancing the computational efficiency of accurate deduplication with the comprehensiveness of semantic deduplication. It avoids the drawback of a single hash deduplication method, which can only match completely identical text and cannot identify semantically redundant content, effectively improving the completeness and accuracy of deduplication determination for new content to be added.

[0113] Furthermore, by extracting the first and second plagiarism check results corresponding to the same new memory content, a corresponding binding relationship between the first and second plagiarism check results is established. Based on the corresponding binding relationship, state matching judgment is carried out to determine three memory comparison states: complete repetition, semantic repetition, and no repetition. Based on the three memory comparison states, hierarchical plagiarism check results are encapsulated and generated. This can associate and unify the two different types of plagiarism check results corresponding to a single new memory content, realize the state linkage judgment of plagiarism check dimension and semantic plagiarism check dimension, and standardize and fully cover all repetition situations of the new memory content. This ensures that the generated hierarchical plagiarism check results can fully reflect the actual repetition state of the new memory content, avoiding the problems of one-sided judgment of a single plagiarism check dimension and the ambiguity of plagiarism check conclusions and inconsistent judgment standards caused by the fragmentation of multi-dimensional plagiarism check results.

[0114] In one possible implementation, the above method also includes: Step b1: If the first deduplication result in the hierarchical deduplication results indicates that the content to be added is not duplicated and the second deduplication result in the hierarchical deduplication results indicates that there is a historical memory entry that is semantically similar to the content to be added, the original historical memory content is iteratively updated to obtain compliant memory content.

[0115] Semantically similar historical memory entries can indicate existing historical memory data entries that differ from the textual descriptions, word order, and modifiers of the memory content to be added, but highly overlap with core business facts, semantic information, and interactive themes.

[0116] Existing historical memory content can indicate historical target memory content that is pre-stored in the hierarchical memory dataset, is semantically similar to the current memory content to be added, and whose information is relatively old and outdated.

[0117] Specifically, if the first deduplication result in the hierarchical deduplication results indicates that the content to be added is not duplicated and the second deduplication result in the hierarchical deduplication results indicates that there are historical memory entries with semantic similarity to the content to be added, it indicates that there is no identical duplicate data in the content to be added, and the original historical memory content retained in the hierarchical memory dataset is consistent with the core semantics of the latest content to be added, but there are problems such as outdated information, lagging expression, and missing details. Therefore, the original historical memory content can be iteratively updated to obtain compliant memory content.

[0118] As an example, when the first deduplication result in the hierarchical deduplication results determines that the memory content to be added is not duplicated, and the second deduplication result determines that there is a historical memory entry that is semantically similar to the memory content to be added, the corresponding original historical memory content in the hierarchical memory dataset is retrieved. Based on the latest factual information of the memory content to be added, the overall information of the original historical memory content is iteratively replaced and updated and optimized. Outdated and invalid information in the original historical memory content is removed, and the latest valid information is integrated to complete the iterative update of the memory content and obtain compliant memory content.

[0119] As an example, semantic comparison is used to locate outdated and missing fields in the original historical memory content, accurately extract the incremental and valid information of the new memory content, and complete the field supplementation, information correction and content update in a targeted manner. The valid and unchanged information in the original historical memory content is retained, and only the lagging and redundant parts are iteratively updated to achieve incremental iteration, retain the valid historical memory data to the greatest extent, and generate compliant memory content.

[0120] Step b2: If the first deduplication result in the hierarchical deduplication results indicates that the content to be added is not duplicated and the second deduplication result in the hierarchical deduplication results indicates that there is no historical memory entry that is semantically similar to the content to be added, the content to be added is determined to be compliant memory content.

[0121] If the first deduplication result in the hierarchical deduplication results indicates that the content to be added is not duplicated and the second deduplication result in the hierarchical deduplication results indicates that there is no historical memory entry with semantic similarity to the content to be added, then it means that the content to be added does not need to be deleted or iterated, and can be directly identified as compliant memory content.

[0122] The AI-based deduplication method provided in this application, under the premise that the first deduplication result in the hierarchical deduplication results all indicate that the content to be added is not duplicated, executes differentiated memory processing logic according to the different comparison states of the second deduplication result. When the second deduplication result indicates that there is a historical memory entry that is semantically similar to the content to be added, the original historical memory content is iteratively updated to obtain compliant memory content. When the second deduplication result indicates that there is no historical memory entry that is semantically similar to the content to be added, the content to be added is directly determined as compliant memory content. It can accurately distinguish and process the two types of content to be added: those that are not completely duplicated but have semantic similarity and those that are completely dissimilar. This avoids the problem of inaccurate memory data caused by the aging and failure of semantically similar historical memory content and the information lag, while ensuring the normal storage and retention of new memory content. It also eliminates the problem of loss of effective memory or accumulation of redundant memory caused by uniform processing methods.

[0123] In one possible implementation, the above method also includes: Step c1: If the first deduplication result in the hierarchical deduplication results indicates that the content to be added is duplicated, delete the content to be added.

[0124] Specifically, the system identifies the status of the first deduplication result in the hierarchical deduplication results. If the first deduplication result clearly indicates that the content to be added is duplicated, it can be determined that the content to be added is completely identical to the historical target memory content text already stored in the hierarchical memory dataset, and does not contain any new factual information, scenario information, or business information, thus constituting invalid and redundant data. Based on this determination, the system directly deletes the content to be added, terminating the subsequent data entry, iteration, and storage processes for that portion of the content.

[0125] The AI-based deduplication method provided in this application deletes the new memory content when the first deduplication result in the hierarchical deduplication results indicates that the new memory content is duplicated. This method can directly remove new memory content that is determined to be completely duplicated by the first deduplication result, thus eliminating memory data that completely overlaps with historical memory content. This prevents completely duplicated new memory content from being continuously entered into the memory dataset, avoiding the problem of redundant and bloated memory datasets and wasted data storage resources caused by the accumulation of completely duplicated memory content.

[0126] In one possible implementation, the step S103 above, which involves performing multi-dimensional retrieval processing on the hierarchical memory dataset and combining it with a weighted fusion sorting and filtering method to determine the historical target memory content corresponding to the new memory content, may include the following steps: Step d1: Based on the hierarchical memory dataset, configure the first weight for vector retrieval and the second weight for keyword retrieval.

[0127] The first weight can be indicated as a pre-configured weighting coefficient for vector retrieval results, used to characterize the contribution ratio and priority of semantic retrieval results in the overall retrieval fusion results.

[0128] The second weight can be a pre-configured weighting coefficient for keyword search results, used to characterize the contribution and priority of keyword search results in the overall search fusion results.

[0129] Specifically, based on the well-organized hierarchical memory dataset, and considering the hierarchical data characteristics of conversational memory, personal memory, and organizational memory within the dataset, as well as business adaptation requirements, the first weight for vector retrieval and the second weight for keyword retrieval are configured differently. The hierarchical memory dataset features clear permissions, distinct levels, and data isolation; different levels of memory data have different requirements for semantic matching and precise keyword matching.

[0130] Step d2 involves performing semantic similarity retrieval on the memory content within the hierarchical memory dataset based on a preset vector index, and using a three-root index mechanism for target characters to perform keyword retrieval on the memory content within the hierarchical memory dataset, resulting in a two-dimensional retrieval raw result.

[0131] Preset vector indexes can indicate semantic vector retrieval indexes pre-built for hierarchical memory datasets, used for quickly matching high-dimensional semantic vectors and calculating text similarity.

[0132] The target character can indicate the feature characters to be extracted from the new memory content. The target character can include characters indicating Chinese characters, characters indicating English characters, etc.

[0133] Specifically, based on the pre-built preset vector index, a full-domain semantic similarity retrieval process is carried out on all compliant memory content within the hierarchical memory dataset. The semantic vector comparison is quickly completed through the vector index, and memory entries that are semantically similar to the memory content to be added are selected in batches to obtain the original matching data of the semantic dimension. At the same time, the target characters of the memory content to be added are extracted, and a dedicated three-root index mechanism is used to complete the keyword index matching to complete the keyword dimension retrieval process.

[0134] Step d3: Based on the first and second weights, the original results of the two-dimensional retrieval are fused, weighted, scored, and sorted to select the historical target memory content corresponding to the new memory content.

[0135] The first and second weights configured are retrieved and assigned to the original semantic search results and the original keyword search results, respectively. Each original dual-dimensional search result is weighted and scored. The semantic matching score and the keyword matching score are integrated to obtain the comprehensive matching score of each memory item. The memory items are sorted from high to low according to the comprehensive matching score. The memory items with the highest comprehensive relevance and the best matching accuracy are selected and identified as the historical target memory content corresponding to the new memory content.

[0136] The AI-based deduplication method provided in this application configures a first weight for vector retrieval and a second weight for keyword retrieval based on the memory dataset. It performs semantic similarity retrieval on the memory content within the hierarchical memory dataset according to a preset vector index, and simultaneously uses a three-root index mechanism for target characters to perform keyword retrieval on the memory content within the hierarchical memory dataset to obtain dual-dimensional retrieval results. Then, the dual-dimensional retrieval results are fused, weighted, scored, and sorted using the first and second weights. Finally, historical target memory content corresponding to the new memory content is selected. This method combines the advantages of both semantic retrieval and keyword retrieval, and rationally integrates and corrects the two types of retrieval results through weight configuration. It avoids the shortcomings of insufficient matching accuracy of single semantic retrieval and the inability of single keyword retrieval to adapt to semantic fuzzy matching, enabling the retrieval process to simultaneously consider the ability of semantic similarity matching and precise keyword matching.

[0137] In one possible implementation, the above method also includes: Step e1: Based on the preset driving timing, extract key facts incrementally from the conversation interaction data corresponding to the conversation memory to generate initial candidate memory content.

[0138] The preset driving timing can indicate the pre-configured timing nodes and event triggering conditions used to trigger incremental memory retrieval, which can include various timings such as session end trigger, polling timer trigger, key behavior trigger, and command trigger.

[0139] Conversational interaction data can indicate the complete raw interaction data generated during the dialogue between the user and the intelligent system, which may include question and answer statements, interaction instructions, operation behaviors, contextual dialogue content, scene parameters, etc.

[0140] Initial candidate memory content can indicate the original candidate memory data that has been preliminarily screened after key fact incremental extraction, has not yet been deduplicated and optimized, and has the potential to be included in the database.

[0141] Specifically, the system reads all raw session interaction data corresponding to the current complete session memory, executes key fact incremental extraction logic, identifies newly added business facts, new user preferences, updated process rules, and newly added interaction command information in the session interaction data, and removes redundant and impurity content such as repeated dialogues, invalid interjections, meaningless greetings, repeated confirmation statements, and blank or invalid fields generated during the session. Through the incremental extraction mechanism, only the newly added valid fact content of this session compared to historical memory is retained to generate initial candidate memory content.

[0142] Step e2 involves deduplicating the initial candidate memory content, removing redundant and duplicate content, and obtaining compliant candidate memory content.

[0143] Redundant and repetitive content can indicate invalid content in the initial candidate memory content, such as local statement repetition, factual fragment superposition, semantic repetition, and redundant expression.

[0144] Compliant candidate memory content can indicate candidate memory data that has undergone internal deduplication and purification and has no local redundancy.

[0145] Specifically, although the generated initial candidate memory content has been cleaned of invalid conversational impurities, it may still contain issues such as repeated segments, semantic overlap, and redundant statements within a single piece of content. For example, there may be local redundancy such as repeated statements of the same fact or segmented recording of the same information. This embodiment performs a dedicated internal deduplication process on the initial candidate memory content. Through text segment comparison, local semantic matching, and fact uniqueness verification, redundant and repeated segments within the candidate content are screened and eliminated to generate compliant candidate memory content.

[0146] Step e3: Match the compliant candidate memory content with the corresponding target session memory, target personal memory, or target organizational memory.

[0147] Based on the hierarchical rules, permission attributes, and content scenarios corresponding to the triplet parameters, the system performs multi-dimensional judgments on the content attributes, timeliness attributes, permission attributes, and ownership entities of compliant candidate memory content to achieve precise hierarchical matching. Specifically, if the candidate content is temporary interaction information for a single session and only effective in the current session, it is matched with the target session memory; if the candidate content is a user's personal preference, private interaction record, or personal setting belonging to a single target user, it is matched with the target personal memory; and if the candidate content is a general business rule, public process, or team-shared information of the organization, without a specific bound user and belonging to the organization as a whole, it is matched with the target organizational memory.

[0148] Step e4: Write the compliant candidate memory content into the target session memory, target personal memory, or target organization memory, and record the data processing location of the target session memory, target personal memory, or target organization memory based on the configured incremental sequence number pointer.

[0149] The incremental sequence pointer can indicate a preset position index pointer with a unique sequence number, used to mark the incremental update site of hierarchical memory data.

[0150] Specifically, based on the matched target memory level, the compliant candidate memory content is written to the corresponding target session memory, target personal memory, or target organizational memory storage unit. At the same time, the pre-configured incremental sequence number pointer is invoked to record the data processing position of the corresponding target memory unit for this memory write operation, and to mark the storage offset, index number, and write position of this incremental data, forming a unique position record for each incremental memory content.

[0151] Step e5: Lock the data processing position using the incremental sequence pointer.

[0152] The data processing location can indicate the write position, data storage offset position, and incremental update index position of the current compliant candidate memory content in the target memory unit.

[0153] Based on the recorded data processing location, a location locking operation is performed using the corresponding incremental sequence number pointer to solidify and lock the storage index, offset position, and sequence number identifier of this incremental memory.

[0154] In one implementation scenario, key facts are incrementally extracted from all conversation interaction data corresponding to the current session. Invalid content such as idle chatter and repetitive phrases is removed, and incremental information such as newly added office preferences, business consultation conclusions, and process operation habits is extracted to generate initial candidate memory content. The initial candidate memory content is internally deduplicated and purified to remove local redundant sentences and semantically repetitive fragments, resulting in refined and compliant candidate memory content. The memory hierarchy is intelligently matched: temporary conversation consultation content is matched with the target conversation memory, employee personal office preference content is matched with the target individual memory, and enterprise general process rules content is matched with the target organizational memory. The compliant candidate memory content is precisely written into the corresponding target memory unit in a hierarchical manner, and the data processing and storage location of each incremental memory is recorded one by one through the incremental sequence number pointer. Finally, all writing positions are fixed and locked by the incremental sequence number pointer to complete the permanent traceability and locking of memory positions.

[0155] The AI-based deduplication method provided in this application extracts key facts incrementally from the conversation interaction data corresponding to the conversation memory based on a preset driving timing to generate initial candidate memory content. This initial candidate memory content is then deduplicated, and redundant and duplicate content is removed to obtain compliant candidate memory content. The compliant candidate memory content is matched with the corresponding target conversation memory, target personal memory, or target organizational memory and written into the corresponding target-level memory. A configured incremental sequence pointer records the data processing position of the target conversation memory, target personal memory, or target organizational memory, and then uses the incremental sequence pointer to lock the corresponding data processing position. This method can extract incremental facts only for newly added interaction data in the conversation memory, without needing to perform a full scan of all historical memory data, reducing unnecessary computational overhead. Simultaneously, by removing redundant and duplicate content from the initial candidate memory content through deduplication, it ensures the cleanliness and effectiveness of the compliant candidate memory content. Furthermore, it can adaptively match the compliant candidate memory content to the corresponding target memory level and complete the writing process, avoiding the problem of memory content being stored incorrectly or haphazardly across levels.

[0156] This embodiment also provides an artificial intelligence-based deduplication device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0157] like Figure 4 As shown, this embodiment provides an artificial intelligence-based deduplication device, which includes: an acquisition module 401 for acquiring preset triplet parameters; wherein the preset triplet parameters include session memory parameters, personal memory parameters, and organizational memory parameters; a first processing module 402 for performing tenant data isolation processing, multi-dimensional memory permission configuration processing, and session memory range limitation processing on the three-layer memory model based on the preset triplet parameters, to determine a hierarchical memory dataset with tenant permission tags; wherein the three-layer memory model includes session memory, personal memory, and organizational memory; a determination module 403 for performing multi-dimensional retrieval processing on the hierarchical memory dataset, and combining a weighted fusion sorting and filtering method to determine the historical target memory content corresponding to the memory content to be added; a verification module 404 for performing progressive deduplication verification on the memory content to be added based on the triplet parameters and the historical target memory content, to obtain hierarchical deduplication results; and a second processing module 405 for processing the memory content to be added based on the hierarchical deduplication results to obtain compliant memory content.

[0158] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0159] In this embodiment, the AI-based deduplication device is presented in the form of a functional unit. Here, a functional unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0160] This application also provides a computer device having the above-described artificial intelligence-based deduplication device.

[0161] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of this application, such as... Figure 5As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 5 Take a processor 10 as an example.

[0162] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0163] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0164] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0165] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0166] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0167] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0168] The computer device also includes a communication interface for communicating with other devices or communication networks.

[0169] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0170] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0171] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A deduplication method based on artificial intelligence, characterized in that, The method includes: Obtain preset triplet parameters; wherein, the preset triplet parameters include session memory parameters, personal memory parameters, and organizational memory parameters; wherein, the session memory parameters are configured to be effective only for a single session and temporarily cleared when the session ends; the personal memory parameters are configured to be privately isolated for a single user and readable and writable only by the user; the organizational memory parameters are configured to allow data sharing within the same tenant and complete data separation across tenants; Based on the preset triplet parameters, tenant data isolation processing, multi-dimensional memory permission configuration processing, and session memory scope limitation processing are respectively implemented on the three-layer memory model to determine the hierarchical memory dataset with tenant permission labels; wherein, the three-layer memory model includes session memory, personal memory, and organizational memory; The hierarchical memory dataset is subjected to multi-dimensional retrieval processing, and combined with a weighted fusion sorting and filtering method, the historical target memory content corresponding to the new memory content is determined. Based on the triplet parameters and the historical target memory content, a progressive deduplication check is performed on the new memory content to obtain the hierarchical deduplication results; Based on the hierarchical deduplication results, the newly added memory content is processed to obtain compliant memory content; Specifically, based on the preset triplet parameters, tenant data isolation processing, multi-dimensional memory permission configuration processing, and session memory scope limitation processing are implemented on the three-layer memory model respectively, to determine the hierarchical memory dataset with tenant permission labels, including: Based on the aforementioned organizational memory parameters, the memory data of the target user is stripped to obtain a memory stripping result; wherein, the memory stripping result indicates data isolation between different target users; Within the same organization identifier, based on the personal memory parameters, private memory permissions are configured for memory data carrying a non-empty valid user identifier to obtain the first permission configuration result; Within the same organization identifier, based on the organization memory parameters, public permissions are configured for memory data with empty user identifiers and no specified bound user, resulting in a second permission configuration result; Based on the session memory parameters, the scope of temporary interactive memory for the current dialogue scenario is limited, and temporary memory permissions that are effective only for a single session are defined to obtain the session memory limitation result. Based on the memory stripping results, the first permission configuration results, the second permission configuration results, and the session memory limitation results, a hierarchical memory dataset with tenant permission labels is determined.

2. The deduplication method based on artificial intelligence according to claim 1, characterized in that, Based on the triplet parameters and the historical target memory content, a progressive deduplication check is performed on the new memory content to obtain hierarchical deduplication results, including: Based on the triplet parameters and the historical target memory content, a hash calculation is performed on the new memory content to obtain the first deduplication result; If the first plagiarism check result indicates that no duplicate content was found, the vector semantic similarity comparison is performed between the newly added memory content and the historical target memory content to obtain the second plagiarism check result; Based on the first and second plagiarism detection results, the hierarchical plagiarism detection results are generated by merging them.

3. The deduplication method based on artificial intelligence according to claim 1, characterized in that, Based on the aforementioned hierarchical deduplication results, the newly added memory content is processed to obtain compliant memory content, including: If the first deduplication result in the hierarchical deduplication result indicates that the new memory content is not duplicated and the second deduplication result in the hierarchical deduplication result indicates that there is a historical memory entry that is semantically similar to the new memory content, the original historical memory content is iteratively updated to obtain compliant memory content. If the first deduplication result in the hierarchical deduplication results indicates that the content to be added is not duplicated and the second deduplication result in the hierarchical deduplication results indicates that there is no historical memory entry that is semantically similar to the content to be added, then the content to be added is determined to be compliant memory content.

4. The deduplication method based on artificial intelligence according to claim 3, characterized in that, The method further includes: If the first deduplication result in the hierarchical deduplication results indicates that the content to be added is duplicated, then the content to be added is deleted.

5. The deduplication method based on artificial intelligence according to claim 2, characterized in that, The process of fusing the first and second plagiarism detection results to generate the hierarchical plagiarism detection results includes: Extract the first and second plagiarism check results corresponding to the same new memory content, and establish a corresponding binding relationship between the first and second plagiarism check results; Based on the corresponding binding relationship, state matching and determination are performed to identify three memory comparison states: complete duplication, semantic duplication, and no duplication. Based on three memory comparison states—complete repetition, semantic repetition, and no repetition—a hierarchical deduplication result is generated.

6. The deduplication method based on artificial intelligence according to claim 1, characterized in that, The hierarchical memory dataset is subjected to multi-dimensional retrieval processing, and combined with a weighted fusion sorting and filtering method, the historical target memory content corresponding to the new memory content is determined, including: Based on the hierarchical memory dataset, configure the first weight for vector retrieval and the second weight for keyword retrieval; Based on the preset vector index, semantic similarity retrieval processing is performed on the memory content in the hierarchical memory dataset, and keyword retrieval processing is performed on the memory content in the hierarchical memory dataset using the three-root index mechanism of target characters, to obtain the original two-dimensional retrieval results; Based on the first and second weights, the original results of the two-dimensional retrieval are fused, weighted, scored, and sorted to select the historical target memory content corresponding to the new memory content.

7. The deduplication method based on artificial intelligence according to claim 1, characterized in that, The method further includes: Based on a preset driving timing, key facts are incrementally extracted from the conversation interaction data corresponding to the conversation memory to generate initial candidate memory content. The initial candidate memory content is deduplicated to remove redundant and duplicate content, resulting in compliant candidate memory content. The compliant candidate memory content is matched with the corresponding target session memory, target personal memory, or target organizational memory; Write the compliant candidate memory content into the target session memory, target personal memory, or target organization memory, and record the data processing location of the target session memory, target personal memory, or target organization memory based on the configured incremental sequence number pointer; The data processing position is locked by the incremental sequence number pointer.

8. A deduplication device based on artificial intelligence, characterized in that, The device includes: The acquisition module is used to acquire preset triplet parameters; wherein, the preset triplet parameters include session memory parameters, personal memory parameters, and organizational memory parameters; wherein, the session memory parameters are configured to be effective only for a single session and temporarily cleared when the session ends; the personal memory parameters are configured to be privately isolated for a single user and readable and writable only by the user; the organizational memory parameters are configured to allow data interoperability within the same tenant and complete data separation across tenants; The first processing module is used to perform tenant data isolation processing, multi-dimensional memory permission configuration processing, and session memory scope limitation processing on the three-layer memory model based on the preset triplet parameters, to determine a hierarchical memory dataset with tenant permission labels; wherein, the three-layer memory model includes session memory, personal memory, and organizational memory; wherein, based on the preset triplet parameters, the three-layer memory model is subjected to tenant data isolation processing, multi-dimensional memory permission configuration processing, and session memory scope limitation processing, to determine a hierarchical memory dataset with tenant permission labels, including: Based on the aforementioned organizational memory parameters, the memory data of the target user is stripped to obtain a memory stripping result; wherein, the memory stripping result indicates data isolation between different target users; Within the same organization identifier, based on the personal memory parameters, private memory permissions are configured for memory data carrying a non-empty valid user identifier to obtain the first permission configuration result; Within the same organization identifier, based on the organization memory parameters, public permissions are configured for memory data with empty user identifiers and no specified bound user, resulting in a second permission configuration result; Based on the session memory parameters, the scope of temporary interactive memory for the current dialogue scenario is limited, and temporary memory permissions that are effective only for a single session are defined to obtain the session memory limitation result. Based on the memory stripping results, the first permission configuration results, the second permission configuration results, and the session memory limitation results, a hierarchical memory dataset with tenant permission labels is determined; The determination module is used to perform multi-dimensional retrieval processing on the hierarchical memory dataset and, in combination with a weighted fusion sorting and filtering method, determine the historical target memory content corresponding to the new memory content to be added. The verification module is used to perform progressive deduplication verification on the new memory content based on the triplet parameters and the historical target memory content, and obtain the hierarchical deduplication results. The second processing module is used to process the newly added memory content based on the hierarchical deduplication results to obtain compliant memory content.

9. An electronic device, characterized in that, It includes a memory and a processor, which are interconnected and communicate with each other. The memory stores computer instructions, and the processor executes the computer instructions to implement the deduplication method based on artificial intelligence as described in any one of claims 1-7.

Images