A privacy data processing method and device, a storage medium and an electronic device

Abstract

The embodiment of the specification discloses a privacy data processing method and device, a storage medium and an electronic equipment, wherein the method comprises the following steps: in response to a data write instruction for target privacy data, a master node controls the target privacy data as a target temporary file to perform a write processing operation and triggers a node delay loading mechanism; when the write processing operation is completed, the target temporary file is updated to a target formal file; under the node delay loading mechanism, a slave node monitors the write processing operation of the target privacy data; and when the write processing operation is completed, the target formal file corresponding to the target privacy data is subjected to a memory loading processing.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and electronic device for processing privacy data. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, the demand for processing unstructured data (such as text, images, voice, and video) in privacy-preserving scenarios is increasing. To support efficient data storage and retrieval, vectorization technology is widely used to transform unstructured data into high-dimensional vectors, enabling storage and retrieval using databases. Currently, with the continuous expansion of data scale and the widespread application of distributed storage technologies, traditional relational data storage systems struggle to efficiently handle the storage and retrieval of large-scale vector data. Therefore, the storage and processing of large-scale privacy-preserving data has become a hot topic in the field. Summary of the Invention

[0003] This specification provides a privacy data processing method, apparatus, storage medium, and electronic device, the technical solutions of which are as follows:

[0004] In a first aspect, embodiments of this specification provide a privacy data processing method, characterized in that it is applied to a distributed data system, the distributed data system including master nodes and slave nodes, the method comprising:

[0005] In response to a data write command for target privacy data, the master node is controlled to treat the target privacy data as a target temporary file for write processing and trigger a node delayed loading mechanism. When the write processing operation is completed, the target temporary file is updated to the target official file.

[0006] Under the node delayed loading mechanism, the slave node is controlled to monitor the write processing operation of the target privacy data. When the write processing operation is completed, the target formal file corresponding to the target privacy data is loaded into memory.

[0007] In one feasible implementation, controlling the master node to treat the target privacy data as a target temporary file for write processing and triggering a node delayed loading mechanism includes:

[0008] Determine the file data type of the target privacy data, and determine the target node delayed loading mechanism for the target privacy data based on the file data type;

[0009] The master node is controlled to write the target privacy data as a target temporary file and trigger the target node's delayed loading mechanism.

[0010] In one feasible implementation, the target privacy data includes target vector data, target forward index data, and target inverted index data.

[0011] Under the node delayed loading mechanism, the slave node is controlled to monitor the write processing operation of the target privacy data. When the write processing operation is completed, the target official file corresponding to the target privacy data is loaded into memory, including:

[0012] If the file data type is a vector data type, then under the node delayed loading vector data mechanism, the slave node is controlled to monitor the first file update time of the target file. When the first file update time changes, the write processing operation is determined to be completed, and the target vector data is loaded into the local memory and / or hardware acceleration device of the slave node.

[0013] If the file data type is a forward data type, then under the node delayed loading forward data mechanism, the slave node is controlled to monitor the file completion mark of the target file, and based on the file completion mark, the write processing operation is determined to be completed, and the target forward data is loaded into the local memory of the slave node in read-only mode;

[0014] If the file data type is inverted index data, then under the node delayed loading index data mechanism, the slave node is controlled to monitor the second file update time of the target formal file. When the second file update time changes with time update, the write processing operation is determined to be completed, and the target inverted index data is loaded into the local memory of the slave node.

[0015] In one feasible implementation, loading the target vector data into the slave node's local memory and / or hardware acceleration device includes: loading the target vector data into the slave node's local memory and / or hardware acceleration device using a full loading method or an on-demand loading method; and / or,

[0016] The step of loading the target forward sorting data into the local memory of the slave node in read-only mode includes: determining the target incremental forward sorting data corresponding to the target forward sorting data using an incremental loading method, and loading the target incremental forward sorting data into the local memory of the slave node in read-only mode; and / or,

[0017] The step of loading the target inverted index data into the local memory of the slave node includes: determining the incremental index mapping relationship corresponding to the target inverted index data using an incremental loading method, and loading the incremental index mapping relationship into the local memory of the slave node.

[0018] In one feasible implementation, the method further includes:

[0019] In response to a privacy data retrieval request, the file data retrieval type is determined based on the privacy data retrieval request, and the privacy data retrieval processing is performed using the data retrieval method corresponding to the file data type.

[0020] In one feasible implementation, the privacy data retrieval processing using the data retrieval method corresponding to the file data retrieval type includes:

[0021] If the data type of the retrieved file is vector data retrieval type, then the vector similarity retrieval method is adopted, the target vector data in the target formal file is loaded from the slave node, the similarity between the target query vector corresponding to the privacy data retrieval request and the target vector data is calculated to obtain the vector similarity calculation result, and the vector retrieval data processing is performed based on the vector similarity calculation result;

[0022] If the data type of the searched file is a forward data retrieval type, then the forward data pre-retrieval method is adopted, and the slave node is controlled to perform forward data pre-retrieval on the target file. The first target data set is selected according to the attribute conditions corresponding to the privacy data retrieval request, and the first target data set is transmitted to the vector retrieval engine. The first target data set is used to assist the vector retrieval engine in narrowing the retrieval scope.

[0023] If the data type of the retrieved file is an inverted index data type, then an index data-assisted retrieval method is adopted. The slave node is controlled to perform inverted index keyword matching on the target official file. The second target data set is retrieved according to the keywords corresponding to the privacy data retrieval request. A candidate data set is generated based on the second target data set and the first target data set corresponding to the forward data pre-retrieval method. The candidate data set is transmitted to the vector retrieval engine. The candidate data set is used to assist the vector retrieval engine in narrowing the retrieval scope.

[0024] In one feasible implementation, the file data retrieval type includes a fusion data retrieval type targeting the vector data retrieval type, the forward index data retrieval type, and the inverted index data type.

[0025] The process of performing privacy data retrieval using the data retrieval method corresponding to the file data retrieval type includes:

[0026] Obtain the second target data set generated through the index data-assisted retrieval method;

[0027] Determine the attribute conditions corresponding to the privacy data retrieval request, and use a forward sorting data pre-retrieval method to perform forward sorting data attribute filtering on the second target data set based on the attribute conditions to obtain a filtered candidate data set;

[0028] The target query vector corresponding to the privacy data retrieval request is determined using the vector similarity retrieval method. The similarity between the target query vector and the candidate vector data in the candidate data set is calculated to obtain the vector similarity calculation result. Based on the vector similarity calculation result, vector retrieval data processing is performed to obtain the target retrieval result.

[0029] In one feasible implementation, the target privacy data includes target vector data, target forward index data, and target inverted index data.

[0030] The method further includes:

[0031] In the write processing operation, a vector perturbation method is used to perform vector encryption based on a fixed value by performing an addition operation on each dimension of the target vector data;

[0032] In the write processing operation, a symmetric encryption algorithm is used to encrypt the target forward-sorted data.

[0033] In the write processing operation, keyword encryption is used to encrypt the target inverted index data.

[0034] Secondly, embodiments of this specification provide a privacy data processing apparatus applied to a distributed data system, the distributed data system including a master node and slave nodes, the apparatus comprising:

[0035] The writing module is used to respond to a data writing command for target privacy data, control the master node to treat the target privacy data as a target temporary file for writing processing operation and trigger the node delayed loading mechanism, and update the target temporary file to the target formal file when the writing processing operation is completed;

[0036] The loading module is used to control the slave node to monitor the write processing operation of the target privacy data under the node delayed loading mechanism, and to perform memory loading processing on the target formal file corresponding to the target privacy data when the write processing operation is completed.

[0037] In one feasible implementation, the writing module is configured to:

[0038] Determine the file data type of the target privacy data, and determine the target node delayed loading mechanism for the target privacy data based on the file data type;

[0039] The master node is controlled to write the target privacy data as a target temporary file and trigger the target node's delayed loading mechanism.

[0040] In one feasible implementation, the target privacy data includes target vector data, target forward index data, and target inverted index data.

[0041] The writing module is used for:

[0042] If the file data type is a vector data type, then under the node delayed loading vector data mechanism, the slave node is controlled to monitor the first file update time of the target file. When the first file update time changes, the write processing operation is determined to be completed, and the target vector data is loaded into the local memory and / or hardware acceleration device of the slave node.

[0043] If the file data type is a forward data type, then under the node delayed loading forward data mechanism, the slave node is controlled to monitor the file completion mark of the target file, and based on the file completion mark, the write processing operation is determined to be completed, and the target forward data is loaded into the local memory of the slave node in read-only mode;

[0044] If the file data type is inverted index data, then under the node delayed loading index data mechanism, the slave node is controlled to monitor the second file update time of the target formal file. When the second file update time changes with time update, the write processing operation is determined to be completed, and the target inverted index data is loaded into the local memory of the slave node.

[0045] In one feasible implementation, the loading module is configured to: load the target vector data into the local memory and / or hardware acceleration device of the slave node using a full loading method or an on-demand loading method; and / or,

[0046] The loading module is configured to: determine the target incremental forward data corresponding to the target forward data using an incremental loading method, and load the target incremental forward data into the local memory of the slave node in read-only mode; and / or,

[0047] The loading module is configured to: determine the incremental index mapping relationship corresponding to the target inverted index data using an incremental loading method, and load the incremental index mapping relationship into the local memory of the slave node.

[0048] In one feasible implementation, the device is further used for:

[0049] In response to a privacy data retrieval request, the file data retrieval type is determined based on the privacy data retrieval request, and the privacy data retrieval processing is performed using the data retrieval method corresponding to the file data type.

[0050] In one feasible implementation, the device is further configured to: if the data type of the retrieved file is a vector data retrieval type, then adopt a vector similarity retrieval method, control the loading of target vector data from the target formal file by the slave node, perform similarity calculation between the target query vector corresponding to the privacy data retrieval request and the target vector data to obtain a vector similarity calculation result, and perform vector retrieval data processing based on the vector similarity calculation result;

[0051] If the data type of the searched file is a forward data retrieval type, then the forward data pre-retrieval method is adopted, and the slave node is controlled to perform forward data pre-retrieval on the target file. The first target data set is selected according to the attribute conditions corresponding to the privacy data retrieval request, and the first target data set is transmitted to the vector retrieval engine. The first target data set is used to assist the vector retrieval engine in narrowing the retrieval scope.

[0052] If the data type of the retrieved file is an inverted index data type, then an index data-assisted retrieval method is adopted. The slave node is controlled to perform inverted index keyword matching on the target official file. The second target data set is retrieved according to the keywords corresponding to the privacy data retrieval request. A candidate data set is generated based on the second target data set and the first target data set corresponding to the forward data pre-retrieval method. The candidate data set is transmitted to the vector retrieval engine. The candidate data set is used to assist the vector retrieval engine in narrowing the retrieval scope.

[0053] In one feasible implementation, the file data retrieval type includes a fusion data retrieval type targeting the vector data retrieval type, the forward index data retrieval type, and the inverted index data type.

[0054] The device is also used for:

[0055] Obtain the second target data set generated through the index data-assisted retrieval method;

[0056] Determine the attribute conditions corresponding to the privacy data retrieval request, and use a forward sorting data pre-retrieval method to perform forward sorting data attribute filtering on the second target data set based on the attribute conditions to obtain a filtered candidate data set;

[0057] The target query vector corresponding to the privacy data retrieval request is determined using the vector similarity retrieval method. The similarity between the target query vector and the candidate vector data in the candidate data set is calculated to obtain the vector similarity calculation result. Based on the vector similarity calculation result, vector retrieval data processing is performed to obtain the target retrieval result.

[0058] In one feasible implementation, the target privacy data includes target vector data, target forward index data, and target inverted index data.

[0059] The device is also used for:

[0060] In the write processing operation, a vector perturbation method is used to perform vector encryption based on a fixed value by performing an addition operation on each dimension of the target vector data;

[0061] In the write processing operation, a symmetric encryption algorithm is used to encrypt the target forward-sorted data.

[0062] In the write processing operation, keyword encryption is used to encrypt the target inverted index data.

[0063] Thirdly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0064] Fourthly, embodiments of this specification provide an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0065] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0066] In one or more embodiments of this specification, in response to a data write instruction for target privacy data, the master node is controlled to treat the target privacy data as a target temporary file for write processing and trigger a node delayed loading mechanism. When the write processing is complete, the target temporary file is updated to the target formal file. Under the node delayed loading mechanism, the slave nodes are controlled to monitor the write processing of the target privacy data. When the write processing is complete, the target formal file corresponding to the target privacy data is loaded into memory. By introducing a temporary file mechanism and a delayed loading mechanism, key limitations of data consistency and system complexity in a distributed environment are avoided, simplifying the implementation of traditional distributed systems (e.g., eliminating the need for complex real-time synchronization and metadata management). This provides high reliability for privacy data processing. Delayed loading avoids reading temporary data, ensuring data consistency and is applicable to various data types (vectors, ascending indexes, inverted indexes). Simultaneously, the delayed loading mechanism is dynamically triggered based on the update status of the target file, and slave nodes load the formal file on demand, reducing unnecessary resource consumption and improving system operating efficiency. Furthermore, it ensures the security of privacy data, preventing data errors and privacy leaks caused by incomplete data or improper synchronization in a distributed storage environment. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a flowchart illustrating a privacy data processing method provided in an embodiment of this specification;

[0069] Figure 2 This is a flowchart illustrating a data loading method provided in an embodiment of this specification;

[0070] Figure 3 This is a flowchart illustrating a data retrieval method provided in an embodiment of this specification;

[0071] Figure 4 This is a flowchart illustrating a data retrieval method provided in an embodiment of this specification;

[0072] Figure 5 This is a flowchart illustrating another data retrieval method provided in the embodiments of this specification;

[0073] Figure 6 This is a schematic diagram of the structure of a privacy data processing device provided in the embodiments of this specification;

[0074] Figure 7 This is a schematic diagram of the structure of an electronic device provided in the embodiments of this specification;

[0075] Figure 8 This is a schematic diagram of the operating system and user space structure provided in the embodiments of this specification;

[0076] Figure 9 yes Figure 8 Architecture diagram of the Android operating system in China;

[0077] Figure 10 yes Figure 8 Architecture diagram of the iOS operating system. Detailed Implementation

[0078] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0079] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0080] In related technologies, such as those used in healthcare, finance, and education, large amounts of private data (e.g., medical records, transaction records, user behavior data) are involved. This data has the following characteristics:

[0081] Sensitivity: Privacy must be strictly protected to prevent data breaches.

[0082] High dimensionality: Data is usually represented in the form of feature vectors, which have high dimensionality and need to support efficient retrieval.

[0083] Distribution: Due to the wide range of data sources, the data may be stored in a distributed environment, requiring the design of a highly reliable and low-complexity distributed system.

[0084] Vector databases are database systems specifically designed for storing, managing, and retrieving high-dimensional vector data. In recent years, with the development of artificial intelligence and big data technologies, the demand for vector databases has been continuously growing. Main application scenarios include:

[0085] Similarity retrieval in search engines (such as content-based image search).

[0086] Similarity calculation for personalized recommendations in a recommendation system.

[0087] Facial recognition and fingerprint recognition in the security field.

[0088] Processing privacy-preserving data in a distributed environment presents numerous challenges: communication delays or incomplete writes between nodes can lead to data errors, and the need to satisfy complex retrieval requests for privacy-preserving data across multiple nodes is also a significant issue.

[0089] Currently, privacy data processing has the following limitations:

[0090] It is necessary to ensure data consistency and avoid data errors caused by communication delays between nodes or incomplete writes; it is necessary to dynamically and efficiently manage privacy data stored on multiple nodes to meet complex retrieval needs; and it is necessary to ensure the security of privacy data and avoid the leakage of sensitive information during storage and processing.

[0091] The privacy data processing methods described in one or more embodiments of this specification can at least solve all or some of the limitations listed above.

[0092] The present specification will now be described in detail with reference to specific embodiments.

[0093] In one embodiment, such as Figure 1 As shown, a privacy-preserving data processing method is proposed. This method can be implemented using a computer program and can run on a privacy-preserving data processing device based on the von Neumann architecture. This computer program can be integrated into applications or run as a standalone utility application. The privacy-preserving data processing device can be a distributed data system, comprising one or more master nodes and one or more slave nodes; the number of master and slave nodes can be flexibly adjusted according to specific scenarios, while ensuring high availability and dynamic scalability of the system.

[0094] Distributed data systems are built on server equipment, such as rack-mount, blade, tower, or cabinet-type server equipment, or using workstations, mainframes, or other hardware devices with strong computing capabilities; they can also be server clusters composed of multiple servers. The servers in the service cluster can be composed in a symmetrical manner, where each server is functionally and hierarchically equivalent in the transaction chain, and each server can provide services independently. The independent provision of services can be understood as not requiring the assistance of other servers.

[0095] Specifically, the privacy data processing method includes:

[0096] S102: In response to the data write instruction for the target privacy data, control the master node to treat the target privacy data as a target temporary file for write processing operation and trigger the node delayed loading mechanism. When the write processing operation is completed, update the target temporary file to the target formal file.

[0097] Target privacy data refers to sensitive data that needs to be stored and processed, such as user information, medical records, and financial transactions. It typically includes various data types, including vector data, forward indexes, and inverted indexes. This type of privacy data must meet privacy protection requirements during processing to prevent data leakage or unauthorized access.

[0098] Furthermore, in this specification, the distributed data system is a vector database built on a distributed architecture. A vector database is specifically designed for storing and querying vectors. Vectorized privacy data typically originates from feature representations of unstructured data (such as text, speech, images, and videos). These are widely used in machine learning, deep learning, and other similar scenarios.

[0099] Optionally, in this specification, privacy data in distributed data systems involves data types such as vector data, forward index data, and inverted indexes. For vector storage in distributed data systems, Faiss, a high-performance vector search library, can be used as the vector storage engine. Faiss is particularly suitable for large-scale, efficient vector nearest neighbor searches, responsible for storing and quickly retrieving high-dimensional vector data. RocksDB, a high-performance key-value store database suitable for handling large-scale, high-frequency read / write scenarios, can also be used as the forward index storage engine. Forward index storage preserves the mapping relationship between raw data and vectors for precise retrieval. Xapian, an efficient full-text search library used to build inverted indexes, can also be used as the inverted index retrieval engine. Inverted index retrieval enables efficient querying of text data, working in conjunction with forward index data to achieve precise matching.

[0100] This specification utilizes a distributed storage system for privacy data processing, ensuring balanced data distribution and high availability. Different database types (vector, forward, and inverted indexes) are stored in independent directories. It supports hierarchical management of forward, inverted, and vector data to meet the storage needs of different data types. High-precision vector similarity retrieval (such as KNN retrieval) is supported, making it suitable for recommendation systems, search engines, and other scenarios. Large-scale distributed deployment is supported; innovative distributed storage and delayed loading mechanisms reduce implementation complexity and improve system reliability.

[0101] Target temporary file: An intermediate state file for data writing. The master node first writes the target privacy data to the temporary file to prevent incomplete data from being exposed to slave nodes before the writing process is complete. The target temporary file ensures the integrity and consistency of the written data, preventing slave nodes from loading incomplete or erroneous data. Setting the target temporary file is also a crucial part of implementing the node delayed loading mechanism. When writing target privacy data, it is set to be written to a temporary file, and slave nodes are configured not to load the temporary file; slave nodes can only access the actual file.

[0102] Target final file: After completing the write operation to the target temporary file, the master node updates the temporary file to a final file through file movement or renaming operations. The final file represents the final state after data writing is complete, and slave nodes can only access the final file.

[0103] Delayed loading mechanism: This specification creatively proposes a dynamic data loading mechanism for distributed storage systems. The slave node triggers the data loading operation based on the file update status of the master node (such as file update time or completion marker). The purpose of the delayed loading mechanism is to ensure that the slave node only loads the officially written files.

[0104] In a schematic representation, a user or system issues a data write command for target privacy data. Upon receiving this command, the master node generates a temporary file and performs the write operation on the target privacy data, ensuring data integrity and correctness during the write process. Simultaneously, a node delayed loading mechanism is triggered. The master node can broadcast notifications or update file metadata (such as update time) to slave nodes while writing the target temporary file, indicating to the slave nodes that the target file is being written. After the master node completes the write operation on the target privacy data, it updates the target temporary file to the target official file via file movement (mv) or renaming. At the same time, the master node records file status update information (such as completion markers or update time).

[0105] For example, the target privacy data includes target vector data, target forward index data, and target inverted index data, and the method further includes:

[0106] Step A2: In the write processing operation, a vector perturbation method is used to perform vector encryption processing based on a fixed value by performing an addition operation on each dimension of the target vector data;

[0107] Target vector data refers to unstructured data (such as text, images, audio, and video) transformed into a high-dimensional numerical representation through feature extraction. Vector data is used to support similarity calculations, such as user feature vectors in recommendation systems or feature vectors in image retrieval.

[0108] Vector perturbation: This method achieves simple encryption by adding or subtracting a fixed value from each dimension of the vector data. The perturbation value is a fixed value, often a small number such as ±0.1 or ±0.2.

[0109] To illustrate, during the write operation, a fixed perturbation value is set for each vector dimension. For example, the perturbation value for the vector [1.2, 3.4, 5.6] is [+0.1, -0.1, +0.2]. Then, an addition operation is performed on each dimension of the target vector data; for example, the encrypted vector becomes [1.3, 3.3, 5.8]. The encrypted vector data is written to the target temporary file, and the original vector is restored using the corresponding descrambling operation when loaded from the slave node later.

[0110] Encryption using vector perturbation is simple and has little impact on vector similarity calculation.

[0111] It ensures data privacy while supporting a balance between search accuracy.

[0112] Step A4: In the write processing operation, a symmetric encryption algorithm is used to encrypt the fields of the target forward-sorted data;

[0113] Target forward sorting data: This is the stored result of attribute information or metadata related to data items. For example, the mapping relationship between document ID and its specific descriptive attributes (such as timestamp, category, user information, etc.).

[0114] Encryption processing of forward-sorted data: Field encryption is used; the characteristics of field encryption are fast encryption and decryption speed, which is suitable for the encryption needs of large-scale field data.

[0115] For example, a symmetric encryption algorithm can be used: use a symmetric key (such as the RC4 algorithm) to encrypt each field of the sorted data.

[0116] As an illustration, the master node dynamically generates a symmetric key to encrypt the forward sorting fields in the target forward sorting data. For each forward sorting field, a symmetric encryption algorithm is used to encrypt its value, and the encrypted forward sorting data is written to the target temporary file. The master node distributes the encryption key to the slave nodes for decryption when the slave nodes load data subsequently.

[0117] The above method ensures that every field of the forward sorted data is protected and that privacy information is not leaked during storage or loading. The dynamic key mechanism improves security and prevents keys from being misused over a long period of time.

[0118] Step A6: In the write processing operation, keyword encryption is used to encrypt the target inverted index data.

[0119] Target inverted index data: This is a mapping structure from keywords to data items, often used to support fast keyword retrieval. For example, the keyword "privacy" is mapped to a list of document IDs [ID1, ID3, ID5].

[0120] Inverted index encryption: Keyword encryption is used; Keyword encryption method: Irreversible encryption algorithms based on hash functions (such as MD5, SHA-256) are used to encrypt the keywords. The encrypted keywords cannot be directly reverse-parsed from the original content, making it suitable for protecting sensitive words.

[0121] To illustrate, first determine the hash algorithm (e.g., SHA-256) used for keyword encryption, then execute the keyword encryption algorithm to encrypt each keyword, generating an irreversible encrypted value. For example, the hash value of the keyword "privacy" might be 2d3e4f567890abc. Establish a mapping relationship between the encrypted keywords and a list of document IDs, for example: encrypted keyword 2d3e4f567890abc -> [ID1, ID3, ID5]. Write the encrypted inverted index to the target temporary file.

[0122] The keyword encryption method described above avoids the direct exposure of the original keywords, ensuring privacy during the search process. The encrypted keywords still support the search function, enabling privacy-protected keyword retrieval through encrypted matching.

[0123] In this specification, the encryption methods for the three types of target privacy data during the write operation are as follows:

[0124] Target vector data: Vector perturbation is used, and simple encryption is performed through addition operations to balance privacy and retrieval accuracy.

[0125] Target forward sorting data: Fields are encrypted using a symmetric encryption algorithm, and dynamic keys enhance security.

[0126] Target inverted index data: Keyword encryption is used, and irreversible encryption of keywords is achieved through a hash function to prevent the leakage of sensitive words.

[0127] These encryption methods work together to ensure the security of target privacy data in a distributed storage system, while supporting efficient retrieval operations.

[0128] S104: Under the node delayed loading mechanism, control the slave node to monitor the write processing operation of the target privacy data, and when the write processing operation is completed, perform memory loading processing on the target formal file corresponding to the target privacy data.

[0129] Slave node: An auxiliary node in a distributed storage system, primarily responsible for reading data and processing user requests (such as retrieval tasks). In some implementations, slave nodes are not responsible for data writing; they only load the actual file into local memory or hardware-accelerated devices after the master node has completed the data writing process.

[0130] As an illustration, under the node delayed loading mechanism, the write processing operation of the target privacy data is monitored. For example, the slave node determines whether the master node has completed the write operation by monitoring the metadata of the target file (such as update time, completion marker, etc.). The monitoring method can be: periodic polling or listening for file update notifications.

[0131] Memory loading process: Once the slave node determines that the target official file has been written, it loads the official file corresponding to the target privacy data into local memory, or directly loads it into a hardware acceleration device (such as a GPU or NPU) to support subsequent high-performance retrieval operations.

[0132] In a schematic manner, under the node delayed loading mechanism, the slave node periodically checks the target file's metadata, such as the file's last update time, completion flag, or status flag, to determine whether the write operation has been completed. When the file status is detected as completed, the slave node confirms that the target file is ready. Based on the loading trigger conditions of the delayed loading mechanism (such as the completion of the write operation), the slave node begins loading the content of the target file from the master node. On one hand, the slave node loads the target file into its local memory, which can be used for regular retrieval tasks. On the other hand, it loads the target file into hardware acceleration devices (such as GPU memory or NPU storage), which can be used for high-performance vector retrieval or computation tasks. After the slave node completes loading the target file, the file data can be called by subsequent retrieval modules, thereby supporting the efficient retrieval needs of the distributed storage system.

[0133] In one or more embodiments of this specification, in response to a data write instruction for target privacy data, the master node is controlled to treat the target privacy data as a target temporary file for write processing and trigger a node delayed loading mechanism. When the write processing is complete, the target temporary file is updated to the target formal file. Under the node delayed loading mechanism, the slave nodes are controlled to monitor the write processing of the target privacy data. When the write processing is complete, the target formal file corresponding to the target privacy data is loaded into memory. By introducing a temporary file mechanism and a delayed loading mechanism, key limitations of data consistency and system complexity in a distributed environment are avoided, simplifying the implementation of traditional distributed systems (e.g., eliminating the need for complex real-time synchronization and metadata management). This provides high reliability for privacy data processing. Delayed loading avoids reading temporary data, ensuring data consistency and is applicable to various data types (vectors, ascending indexes, inverted indexes). Simultaneously, the delayed loading mechanism is dynamically triggered based on the update status of the target file, and slave nodes load the formal file on demand, reducing unnecessary resource consumption and improving system operating efficiency. Furthermore, it ensures the security of privacy data, preventing data errors and privacy leaks caused by incomplete data or improper synchronization in a distributed storage environment.

[0134] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data loading method proposed in this specification. Specifically:

[0135] S202: Determine the file data type of the target privacy data, and determine the target node delayed loading mechanism for the target privacy data based on the file data type;

[0136] The target privacy data includes target vector data, target forward index data, and target inverted index data;

[0137] The file data types include vector data types, forward index data types, and inverted index data types;

[0138] Vector data type: Used to store high-dimensional feature vectors of unstructured data (such as text, images, etc.).

[0139] Forward data type: Stores data-related attributes or metadata (such as timestamps, category tags, etc.).

[0140] Inverted index data type: The mapping relationship between keywords and file IDs, used to support keyword retrieval.

[0141] Target node delayed loading mechanism: A dynamic loading mechanism designed in advance for different file data types, used to control the way and timing of loading data from nodes.

[0142] Furthermore, if the file data type is a vector data type, then the target node delayed loading mechanism is a node delayed loading vector data mechanism;

[0143] If the file data type is a forward sorted data type, then the target node delayed loading mechanism is a node delayed loading forward sorted data mechanism;

[0144] If the file data type is an inverted index data type, then the target node delayed loading mechanism is a node delayed loading index data mechanism;

[0145] S204: Control the master node to write the target privacy data as a target temporary file and trigger the target node's delayed loading mechanism;

[0146] As an example, the master node writes the target privacy data to a temporary file, ensuring that the data is not loaded by the slave node before the writing process is complete. Simultaneously with writing to the temporary file, the master node triggers a corresponding delayed loading mechanism, notifying the slave nodes to monitor the update status of the target file.

[0147] S206: If the file data type is a vector data type, then under the node delayed loading vector data mechanism, the slave node is controlled to monitor the first file update time of the target formal file. When the first file update time changes with time update, the write processing operation is determined to be completed, and the target vector data is loaded into the local memory and / or hardware acceleration device of the slave node.

[0148] First file update time: This indicates the last update time of the official file corresponding to the target vector data, used to determine whether the data has been completely written.

[0149] Hardware acceleration devices: including GPUs or NPUs, used for high-performance computing (such as vector similarity retrieval).

[0150] As an illustration, the slave node periodically checks the first file update time of the target file to confirm whether it has changed. Once the file update time is updated, the slave node determines that the master node has completed writing the target vector data. The target vector data is then loaded into the slave node's local memory or directly into hardware acceleration devices such as GPUs / NPUs to support subsequent high-performance retrieval tasks.

[0151] In one feasible implementation, loading the target vector data into the local memory and / or hardware acceleration device of the slave node includes: loading the target vector data into the local memory and / or hardware acceleration device of the slave node using a full loading method or an on-demand loading method.

[0152] Local memory: The system memory of the slave node, used to store the loaded vector data.

[0153] Full load method: Loads all the contents of the target vector data into the memory or hardware device of the slave node at once. After loading using the full load method, complete retrieval can be supported directly, avoiding the complexity of dynamic loading.

[0154] On-demand loading: Load only a portion of the target vector data (such as specific shards or retrieval-related vectors) according to actual needs. This saves memory and hardware resources and is suitable for scenarios with large amounts of data and locality of retrieval requests.

[0155] As an illustration, once the node detects that the formal file corresponding to the target vector data has been written, a loading operation is triggered. The system decides whether to use a full loading method or an on-demand loading method based on the configuration or runtime requirements.

[0156] Memory loading: Loads data into the system memory of the slave node for use by regular computing tasks.

[0157] Hardware loading: Loads data into the video memory of hardware devices such as GPUs / NPUs to support high-performance parallel computing (such as vector similarity calculation).

[0158] Furthermore, after ensuring that the loaded data is available, the system marks the loading completion status and provides data access capabilities to the retrieval module.

[0159] By adopting the above approach, the system can achieve a balance between performance and resources, while providing flexible solutions for vector retrieval scenarios of different scales and needs.

[0160] S208: If the file data type is a forward data type, then under the node delayed loading forward data mechanism, the slave node is controlled to monitor the file completion mark of the target file, and the write processing operation is determined to be completed based on the file completion mark, and the target forward data is loaded into the local memory of the slave node in read-only mode.

[0161] File completion flag: A flag added by the master node when writing target forward data to indicate whether the write operation has been completed.

[0162] Read-only loading: Loads the target forward sorting data from the node in read-only mode, avoiding further modification of the forward sorting file.

[0163] As an illustration, the slave node detects the completion marker of the target file to confirm whether the writing is complete. Based on the change in the file completion marker's status, the slave node confirms that the master node has completed writing the forward index data. The target forward index data is then loaded into the slave node's memory in read-only mode for subsequent retrieval operations.

[0164] In one feasible implementation, loading the target forward data into the local memory of the slave node in read-only mode includes: determining the target incremental forward data corresponding to the target forward data using an incremental loading method, and loading the target incremental forward data into the local memory of the slave node in read-only mode.

[0165] Target forward-ranked data: Stores attribute information or metadata related to data items, such as document ID, timestamp, category tags, etc. Forward-ranked data allows for quick location of data items through attribute filtering and is an important part of the retrieval process.

[0166] Read-only mode: After data is loaded into the slave node, it is accessed in read-only mode to prevent modification of the original data during loading or retrieval tasks. This ensures data integrity and consistency and is suitable for operations such as attribute queries.

[0167] Incremental loading: Only the portion of the data that has changed since the last load (i.e., the incremental portion) is loaded, avoiding the repeated loading of the full data. This reduces data transfer volume and loading time, thus lowering system resource consumption.

[0168] Target incremental forward sorting data: The newly added or updated data portion extracted from the target forward sorting data of the master node. The range of incremental data is determined by the file status or change marker.

[0169] As an illustration, the slave node triggers a load operation when it detects that the official file corresponding to the target forward sorting data has been written. Based on the status changes of the target official file (such as file completion markers or incremental update flags), it determines the incremental portion of the forward sorting data to be loaded. By comparing the currently loaded data version with the latest data version, it extracts the newly added or updated forward sorting data records. The target incremental forward sorting data is loaded into the slave node's local memory in read-only mode. After the incremental forward sorting data is loaded, it is merged or replaced with existing data, enabling the slave node to directly use the latest forward sorting data to perform retrieval tasks.

[0170] Example:

[0171] The original forward sorting data version V1 includes: {ID1: attribute 1, ID2: attribute 2}.

[0172] Incremental data version V2 has updated a new record: {ID3:attribute3}.

[0173] Only load {ID3:attribute3} from the node to avoid reloading the content of V1.

[0174] By employing the above method, combining incremental loading with read-only data mode, the system can efficiently and securely load the target ascending data into the slave node's memory. Incremental loading reduces unnecessary duplicate transmissions, while read-only data mode ensures data consistency, making it suitable for large-scale distributed storage and retrieval systems that require frequent updates to the ascending data.

[0175] S210: If the file data type is an inverted index data type, then under the node delayed loading index data mechanism, the slave node is controlled to monitor the second file update time of the target formal file. When the second file update time changes with time update, the write processing operation is determined to be completed, and the target inverted index data is loaded into the local memory of the slave node.

[0176] Second file update time: This indicates the last update time of the official file corresponding to the target inverted index data, used to determine whether the index data has been updated.

[0177] As an illustration, the slave node periodically checks the update time of the second file of the target official file to confirm whether it has changed. Once the file update time is updated, the slave node confirms that the master node has completed writing the target inverted index data. The target inverted index data is then loaded into the slave node's memory to support subsequent keyword searches.

[0178] In one feasible implementation, loading the target inverted index data into the local memory of the slave node includes: determining the incremental index mapping relationship corresponding to the target inverted index data using an incremental loading method, and loading the incremental index mapping relationship into the local memory of the slave node;

[0179] Incremental index mapping relationship: The newly added or updated mapping relationship portion extracted from the target inverted index data of the master node.

[0180] Incremental loading: Only newly added or updated inverted index mappings are loaded, avoiding the reloading of existing complete indexes. This reduces data transfer and loading time, optimizing system performance.

[0181] Indicatively, when a slave node detects that the file corresponding to the target inverted index data has been updated (e.g., a change in file update time or an incremental update flag), an incremental loading operation is triggered. Then, the incremental index mapping relationship is determined: 1. By comparing the current index state loaded by the slave node with the latest index state of the master node, the incremental portion to be loaded is identified; 2. The timestamp or version number of the keyword update is recorded. The keywords of the incremental portion and their newly added mapping relationships are extracted. The identified incremental mapping relationships are then loaded into the slave node's local memory. During loading, only the mapping of the incremental portion is updated, while the existing complete index is retained. After the incremental loading is complete, the slave node updates its local index state to ensure consistency with the latest index state of the master node.

[0182] Understandably, dynamically updating the target inverted index data through incremental loading can significantly improve loading efficiency, reduce system resource overhead, and enhance real-time performance. The determination and loading process of the incremental index mapping relationship ensures efficient management of the inverted index by the slave nodes, making it suitable for large-scale distributed storage and retrieval systems where keywords change frequently.

[0183] In the embodiments described in this specification, a dedicated delayed loading mechanism is designed for different file data types (vector data, forward index data, and inverted index data) of the target privacy data. Combined with incremental loading, this achieves efficient data management and dynamic updates. By using status markers for temporary and formal files, slave nodes can accurately monitor data writing completion before loading, avoiding the loading of incomplete data and ensuring data consistency and integrity. Simultaneously, the on-demand loading and incremental loading mechanisms significantly reduce system resource consumption, improve loading efficiency and real-time performance, and are suitable for diverse retrieval needs in large-scale distributed storage systems.

[0184] Please see Figure 3 , Figure 3 This is a flowchart illustrating a data retrieval method proposed in this specification. Specifically:

[0185] S302: In response to a privacy data retrieval request, determine the file data retrieval type based on the privacy data retrieval request;

[0186] Privacy data retrieval request: A query task initiated by a user or system, which includes the conditions or range of data to be queried.

[0187] Examples: Attribute criteria: Filtering data based on a specific time period or category. Keyword criteria: Searching for data containing certain keywords. Vector criteria: Retrieval tasks based on vector similarity.

[0188] File data retrieval types: These indicate the retrieval method for the target privacy data, with the specific retrieval type determined based on the query content. These include: vector data retrieval type (processing vector similarity queries), forward index data retrieval type (queries based on attribute filtering), and inverted index data type (keyword matching queries);

[0189] This illustrates how a search request is parsed, user needs are analyzed based on the content of the search request, and search criteria are extracted. These search criteria are then mapped to the corresponding file data search types.

[0190] For example: Querying documents containing the keyword "privacy" → inverted index data retrieval type; filtering data with a time range of 2023 → forward index data retrieval type; finding feature data most similar to the target vector → vector data retrieval type.

[0191] S304: Perform privacy data retrieval processing using the data retrieval method corresponding to the data type of the retrieved file.

[0192] Data retrieval methods: Retrieval methods designed for different file data types. These include:

[0193] Vector similarity retrieval: Calculates similarity based on vector data and returns the most similar data items.

[0194] Forward sorting data pre-retrieval: Quickly filter data items that meet the criteria based on attribute conditions.

[0195] Inverted index-assisted retrieval: Locating relevant data items through keyword matching.

[0196] As an example, based on the file data retrieval type determined in S302, select the corresponding data retrieval method and execute the retrieval operation;

[0197] For example: Vector similarity retrieval: Load target vector data. Calculate the similarity between the query vector and the target vector, and return the top k most similar results.

[0198] For example: Forward sorted data pre-retrieval: Load the forward sorted data file. Filter data items that meet the attribute conditions and return candidate datasets.

[0199] For example: Inverted index-assisted retrieval: Load inverted index data. Locate the mapping of data items based on keyword queries and return keyword matching results.

[0200] In one or more embodiments of this specification, by parsing privacy data retrieval requests, the corresponding file data retrieval type is accurately determined, and targeted data retrieval methods are employed, achieving efficient processing of diverse query needs. Based on the classification design of file data types, the system can flexibly invoke methods such as vector similarity retrieval, forward data pre-retrieval, and inverted index-assisted retrieval according to the query content, improving retrieval efficiency and result accuracy. Simultaneously, this dynamic selection and execution mechanism optimizes resource utilization and adapts to complex privacy data retrieval scenarios.

[0201] Please see Figure 4 , Figure 4 This is a flowchart illustrating a data retrieval method proposed in this specification. Specifically:

[0202] S402: In response to a privacy data retrieval request, determine the file data retrieval type based on the privacy data retrieval request;

[0203] Privacy data retrieval request: A query request initiated by a user or system, which may include multiple conditions, such as attribute conditions (e.g., time range), keyword conditions (e.g., matching keywords), vector similarity conditions, etc.

[0204] File data retrieval types: Based on the content of the privacy data retrieval request, determine the matching retrieval method, which mainly includes: vector data retrieval type, forward index data retrieval type, and inverted index data type.

[0205] This example illustrates how to parse the content of a privacy data retrieval request and determine the corresponding file data retrieval type based on the query conditions (vector, attribute, keyword).

[0206] In some embodiments, the privacy data retrieval process performed using the data retrieval method corresponding to the file data retrieval type can be performed by referring to the following steps.

[0207] S404: If the data type of the retrieved file is vector data retrieval type, then the vector similarity retrieval method is adopted, the target vector data in the target formal file is loaded from the slave node, the similarity between the target query vector corresponding to the privacy data retrieval request and the target vector data is calculated to obtain the vector similarity calculation result, and the vector retrieval data processing is performed based on the vector similarity calculation result;

[0208] Vector data retrieval type: This indicates that the query content of the privacy data retrieval request needs to be processed through vector similarity calculation. It is commonly used for similarity matching of unstructured data (such as images, voice, text embeddings, etc.).

[0209] Target vector data: A high-dimensional vector feature representation extracted from target privacy data, used to store and compute similarity between data.

[0210] Target query vector: A vectorized representation of the query content input by the user or system, used as input for retrieval, and used to calculate similarity with the target vector data.

[0211] Similarity calculation: This measures the degree of similarity between two vectors. Commonly used calculation methods include:

[0212] Euclidean distance: Calculates the geometric distance between two vectors. Cosine similarity: Calculates the consistency of the directions of two vectors. Dot product: Measures the magnitude of the projections of two vectors onto each other in the same direction.

[0213] Vector similarity calculation results: The ranking results are calculated based on the similarity between the target query vector and the target vector data. Usually, the top k most similar vector data are selected.

[0214] This example illustrates how to load target vector data from a target file using a node, calculate the similarity between the target query vector and the loaded target vector data one by one, sort the data by similarity from highest to lowest, and return the k most similar data items and their associated information (such as data IDs). Based on the similarity calculation results, the system processes and returns the search results. Example search results: [ID1, ID2, ID3], representing the three data items most similar to the target query vector.

[0215] In addition, acceleration optimization methods can be adopted in the process of calculating vector similarity: using hardware acceleration devices (such as GPUs / NPUs) for parallel computation; and utilizing approximate nearest neighbor search (such as Faiss or Annoy) to optimize large-scale vector computation.

[0216] Understandably, by controlling the loading of target vector data from nodes and performing retrieval based on vector similarity calculations, the problem of large-scale retrieval of unstructured data is solved. Leveraging dynamic loading and efficient similarity calculation methods, retrieval efficiency and accuracy are significantly improved, making it suitable for recommendation, search, and matching scenarios, meeting the requirements for both real-time performance and accuracy.

[0217] S406: If the data type of the searched file is a forward data search type, then the forward data pre-search method is adopted to control the slave node to perform forward data pre-search on the target file, filter out the first target data set according to the attribute conditions corresponding to the privacy data search request, and transmit the first target data set to the vector search engine. The first target data set is used to assist the vector search engine in narrowing the search scope.

[0218] Forward sorting data retrieval type: A retrieval method in privacy data retrieval requests that filters based on data attribute conditions (such as time, category, tags, etc.). Suitable for quickly and initially filtering out target data that meets the attribute conditions.

[0219] Forward sorting data pre-retrieval method: This method involves pre-filtering the forward sorting data in the target document based on attribute conditions to generate a candidate data set.

[0220] The first target dataset: The dataset that meets the search criteria and is selected from the forward-sorted data. It usually contains the identification information of the data items (such as data IDs).

[0221] Vector retrieval engine (also known as vector retrieval module): A module specifically designed to handle retrieval tasks based on vector similarity. By receiving candidate datasets as input, it reduces the computational scope and improves retrieval efficiency.

[0222] Indicatively, the process involves: 1) Loading forward data: The slave node loads the forward data from the target file into local memory. The forward data stores attribute information for data items, such as timestamps and category tags. 2) Parsing attribute conditions: Attribute conditions are extracted from the privacy data retrieval request. 3) Performing forward data pre-retrieval: The slave node filters the forward data, searching for data items that meet the attribute conditions. 4) Transmitting the first target data set: The filtered first target data set is transmitted to the vector retrieval engine. 5) Assisting vector retrieval: The vector retrieval engine uses the first target data set as its search scope, performing vector similarity calculations only on candidate data. By narrowing the search scope, the computational complexity of vector retrieval is reduced, improving retrieval efficiency.

[0223] Understandably, by using forward sorting data pre-retrieval, the first target data set that meets the attribute conditions is quickly filtered out and transmitted to the vector retrieval engine, effectively narrowing the scope of vector retrieval. This step fully utilizes the structured characteristics of forward sorting data, improving retrieval efficiency and the utilization of computing resources. It is particularly suitable for complex retrieval scenarios that combine attribute filtering and similarity matching, meeting the optimization goal of multi-dimensional query needs.

[0224] S408: If the data type of the retrieved file is an inverted index data type, then the index data-assisted retrieval method is adopted to control the slave node to perform inverted index keyword matching on the target formal file, retrieve the second target data set according to the keywords corresponding to the privacy data retrieval request, generate a candidate data set based on the second target data set and the first target data set corresponding to the forward data pre-retrieval method, and transmit the candidate data set to the vector retrieval engine. The candidate data set is used to assist the vector retrieval engine in narrowing the retrieval scope.

[0225] Inverted index data type: For keyword retrieval scenarios, the inverted index stores the mapping relationship between keywords and document IDs to support fast keyword matching.

[0226] Index-assisted retrieval method: Based on inverted index, it quickly locates the set of data items containing the target keyword through keyword matching.

[0227] The second target data set: the set of search results matching the inverted index keywords, that is, the identifiers of data items containing the search keywords.

[0228] The first target dataset: the dataset that meets the attribute conditions and is pre-selected through forward sorting.

[0229] Candidate data set: generated by the intersection of the second target data set and the first target data set, further optimized into the input range for vector retrieval.

[0230] Vector retrieval engine: A module specifically designed to perform vector similarity calculations for high-precision similarity matching.

[0231] This example illustrates how control is used to perform keyword matching on the inverted index data of the target official file from the control node. Based on the keyword conditions in the privacy data retrieval request, a list of document IDs containing the keywords is located to generate a second target dataset.

[0232] Furthermore, a forward sorting data pre-retrieval method is used to filter the data set based on attribute conditions. A candidate data set is generated by the intersection operation of the second target data set and the first target data set.

[0233] Example:

[0234] The second target data set is [ID1, ID3, ID5].

[0235] The first target data set is [ID2, ID3, ID4].

[0236] The candidate dataset is [ID3].

[0237] Furthermore, the candidate data set is passed to the vector retrieval engine as the input range for vector similarity calculation, thereby further optimizing retrieval efficiency.

[0238] Understandably, by generating a second target data set through inverted index keyword matching and combining it with the first target data set generated through forward index pre-retrieval, a candidate data set is dynamically generated, significantly narrowing the scope of vector retrieval. Through multi-level collaborative retrieval, the retrieval efficiency and accuracy in complex query scenarios are improved, especially suitable for mixed query requirements involving keywords, attributes, and vector similarity, meeting the performance requirements of real-time and large-scale data retrieval.

[0239] In one or more embodiments of this specification, by parsing privacy data retrieval requests, the file data retrieval type is dynamically determined. Combined with multi-level collaborative optimization of vector similarity retrieval, forward data pre-retrieval, and inverted index-assisted retrieval, efficient processing of complex retrieval scenarios is achieved. Attribute filtering of forward data and keyword matching of the inverted index effectively narrow the range of candidate data, reduce the computational burden of vector similarity calculation, and significantly improve retrieval efficiency and accuracy. This hierarchical retrieval method based on data type is suitable for mixed query requirements of attributes, keywords, and vector similarity, meeting the high requirements for real-time performance and accuracy in large-scale distributed storage systems.

[0240] Optionally, the file data retrieval types include a fusion data retrieval type for the vector data retrieval type, the forward index data retrieval type, and the inverted index data type. Please refer to [link to relevant documentation]. Figure 5 , Figure 5 This is a flowchart illustrating another data retrieval method proposed in this specification. Specifically, the privacy data retrieval processing performed using the data retrieval method corresponding to the file data retrieval type can be carried out in the following ways:

[0241] S502: Obtain the second target data set generated by the index data-assisted retrieval method;

[0242] Inverted index-assisted retrieval: Based on the keyword conditions in the privacy data retrieval request, the inverted index is used to quickly locate the set of document IDs containing the specified keywords, so as to generate a second target dataset for merging as the result of keyword matching.

[0243] This example illustrates how the inverted index data from the target official file is loaded into local memory from the node, controlling the loading of this data. Keyword conditions are parsed from the privacy data retrieval request, a keyword matching operation is performed, the set of data items matching the keywords is searched in the inverted index, and the document IDs containing the target keywords are extracted, generating a second target data set.

[0244] Then, the second target data set generated by keyword matching is passed to subsequent retrieval steps to further optimize the candidate data range.

[0245] It's understandable that using indexed data to assist in retrieval involves leveraging the inverted index to quickly match keywords and generate a preliminary second target dataset. This step fully utilizes the efficiency of the inverted index, providing a precise initial range of candidate data for subsequent forward data filtering and vector similarity calculation, thereby improving overall retrieval efficiency and making it suitable for handling large-scale data and complex query needs.

[0246] S504: Determine the attribute conditions corresponding to the privacy data retrieval request, and use a forward sorting data pre-retrieval method to perform forward sorting data attribute filtering on the second target data set based on the attribute conditions to obtain a filtered candidate data set;

[0247] Attribute conditions: Query constraints extracted from privacy data retrieval requests, used to filter data items that meet specific attribute requirements. Common attribute conditions include: time range (e.g., "data from 2023"), category tags (e.g., "healthcare", "finance", etc.), region, or other metadata conditions;

[0248] Forward sorting data pre-retrieval method: By using the structured storage of forward sorting data (such as the mapping relationship between attribute fields and data items), attribute conditions are quickly matched to filter out the data set that meets the conditions.

[0249] Second target dataset: A preliminary candidate dataset generated from the inverted index retrieval, containing data items that meet the keyword criteria.

[0250] Filtered candidate dataset: A more accurate dataset generated by further filtering based on attribute conditions on the second target dataset.

[0251] The illustration shows how control loads the forward-sorted data from the target official file into local memory. Attribute conditions are extracted from the privacy data retrieval request. Based on the second target data set, attribute condition matching is performed on the corresponding forward-sorted data. The filtered results that meet the attribute conditions are used as a candidate data set and passed to subsequent steps.

[0252] By employing a forward-sorting data pre-retrieval method, attribute-based filtering is introduced on top of the second target dataset, effectively narrowing down the candidate data range and reducing unnecessary data items for subsequent vector similarity calculations. This hierarchical optimization mechanism leverages the structured storage characteristics of forward-sorting data, improving both filtering efficiency and the accuracy of search results, making it particularly suitable for scenarios involving mixed queries using keywords, attribute conditions, and vector similarity.

[0253] S506: The target query vector corresponding to the privacy data retrieval request is determined by the vector similarity retrieval method, and the similarity is calculated between the target query vector and the candidate vector data in the candidate data set to obtain the vector similarity calculation result. The target retrieval result is obtained by vector retrieval data processing based on the vector similarity calculation result.

[0254] Vector similarity retrieval method: Based on the similarity calculation between the target query vector and the candidate vector data, the retrieval method determines the most relevant data item.

[0255] Commonly used similarity calculation methods include:

[0256] Euclidean distance: measures the geometric distance between two vectors; the smaller the value, the more similar they are.

[0257] Cosine similarity: measures the similarity of the directions of two vectors; the closer the value is to 1, the more similar they are.

[0258] Dot product: measures the correlation between two vectors in the same direction.

[0259] Target query vector: This transforms the query conditions in a privacy data retrieval request into a vectorized representation, used for similarity comparison with target data. Example: The user input text "privacy protection" is vectorized as [0.2, 0.5, 0.8].

[0260] Candidate dataset: A set of candidate datasets generated by filtering data through forward sorting attributes, which includes target vector data that needs further matching.

[0261] Vector similarity calculation results: Ranking results generated based on the similarity between the target query vector and the candidate vector data, used to return the final search results.

[0262] Target search results: Based on the similarity calculation results, select the k most similar data items as the search results.

[0263] This example illustrates the vectorization of query conditions for privacy data retrieval requests. It loads candidate vector data, i.e., loads corresponding vector data from a candidate dataset for similarity calculation. It calculates the similarity between the target query vector and each candidate vector data, generating a similarity score. Based on the similarity score, it sorts the data from highest to lowest and returns the top k data items as the retrieval results.

[0264] Understandably, vector similarity retrieval generates accurate search results by calculating the similarity between the target query vector and candidate vector data. Combining this with the candidate data set filtered by positive ranking attributes significantly reduces the computational scope while maintaining high search accuracy. Utilizing dynamic range optimization and hardware acceleration, this step meets the requirements of real-time performance, scalability, and high accuracy when handling complex retrieval tasks (such as recommendation, search, and matching), making it a core component for achieving efficient unstructured data retrieval.

[0265] In one or more embodiments of this specification, a hierarchical collaborative mechanism integrating inverted indexes, forward data filtering, and vector similarity retrieval achieves efficient processing of complex retrieval tasks. The inverted index assists in quickly generating a preliminary candidate data set, the forward data pre-retrieval further filters and optimizes using attribute conditions, and vector similarity retrieval calculates precise similarities within the candidate data set, ultimately generating the target retrieval result. This hierarchical optimization strategy effectively narrows the computational scope, reduces resource consumption, and significantly improves retrieval efficiency and accuracy. It is suitable for multi-condition mixed query scenarios and meets the real-time and high-accuracy requirements of large-scale distributed storage systems.

[0266] The following will combine Figure 6 This specification provides a detailed description of the privacy data processing apparatus provided in the embodiments. It should be noted that... Figure 6 The privacy data processing device shown is used to perform the functions described in this specification. Figures 1-5 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts related to the embodiments of this specification. For specific technical details not disclosed, please refer to this specification. Figures 1-5 The example shown.

[0267] Please see Figure 6 This diagram illustrates the structure of a privacy data processing device according to an embodiment of this specification. The privacy data processing device 1 can be implemented as all or part of a user terminal through software, hardware, or a combination of both. According to some embodiments, the privacy data processing device 1 includes a writing module 11 and a loading module 12, specifically used for:

[0268] The writing module 11 is used to respond to a data writing instruction for target privacy data, control the master node to use the target privacy data as a target temporary file for writing processing operation and trigger the node delayed loading mechanism, and update the target temporary file to the target formal file when the writing processing operation is completed;

[0269] The loading module 12 is used to control the slave node to monitor the write processing operation of the target privacy data under the node delayed loading mechanism, and to perform memory loading processing on the target formal file corresponding to the target privacy data when the write processing operation is completed.

[0270] In one feasible implementation, the writing module 11 is used for:

[0271] Determine the file data type of the target privacy data, and determine the target node delayed loading mechanism for the target privacy data based on the file data type;

[0272] The master node is controlled to write the target privacy data as a target temporary file and trigger the target node's delayed loading mechanism.

[0273] In one feasible implementation, the target privacy data includes target vector data, target forward index data, and target inverted index data.

[0274] The writing module 11 is used for:

[0275] If the file data type is a vector data type, then under the node delayed loading vector data mechanism, the slave node is controlled to monitor the first file update time of the target file. When the first file update time changes, the write processing operation is determined to be completed, and the target vector data is loaded into the local memory and / or hardware acceleration device of the slave node.

[0276] If the file data type is a forward data type, then under the node delayed loading forward data mechanism, the slave node is controlled to monitor the file completion mark of the target file, and based on the file completion mark, the write processing operation is determined to be completed, and the target forward data is loaded into the local memory of the slave node in read-only mode;

[0277] If the file data type is inverted index data, then under the node delayed loading index data mechanism, the slave node is controlled to monitor the second file update time of the target formal file. When the second file update time changes with time update, the write processing operation is determined to be completed, and the target inverted index data is loaded into the local memory of the slave node.

[0278] In one feasible implementation, the loading module 12 is configured to: load the target vector data into the local memory and / or hardware acceleration device of the slave node using a full loading method or an on-demand loading method; and / or,

[0279] The loading module is configured to: determine the target incremental forward data corresponding to the target forward data using an incremental loading method, and load the target incremental forward data into the local memory of the slave node in read-only mode; and / or,

[0280] The loading module is configured to: determine the incremental index mapping relationship corresponding to the target inverted index data using an incremental loading method, and load the incremental index mapping relationship into the local memory of the slave node.

[0281] In one feasible implementation, the device 1 is further used for:

[0282] In response to a privacy data retrieval request, the file data retrieval type is determined based on the privacy data retrieval request, and the privacy data retrieval processing is performed using the data retrieval method corresponding to the file data type.

[0283] In one feasible implementation, the device is further configured to: if the data type of the retrieved file is a vector data retrieval type, then adopt a vector similarity retrieval method, control the loading of target vector data from the target formal file by the slave node, perform similarity calculation between the target query vector corresponding to the privacy data retrieval request and the target vector data to obtain a vector similarity calculation result, and perform vector retrieval data processing based on the vector similarity calculation result;

[0284] If the data type of the searched file is a forward data retrieval type, then the forward data pre-retrieval method is adopted, and the slave node is controlled to perform forward data pre-retrieval on the target file. The first target data set is selected according to the attribute conditions corresponding to the privacy data retrieval request, and the first target data set is transmitted to the vector retrieval engine. The first target data set is used to assist the vector retrieval engine in narrowing the retrieval scope.

[0285] If the data type of the retrieved file is an inverted index data type, then an index data-assisted retrieval method is adopted. The slave node is controlled to perform inverted index keyword matching on the target official file. The second target data set is retrieved according to the keywords corresponding to the privacy data retrieval request. A candidate data set is generated based on the second target data set and the first target data set corresponding to the forward data pre-retrieval method. The candidate data set is transmitted to the vector retrieval engine. The candidate data set is used to assist the vector retrieval engine in narrowing the retrieval scope.

[0286] In one feasible implementation, the file data retrieval type includes a fusion data retrieval type targeting the vector data retrieval type, the forward index data retrieval type, and the inverted index data type.

[0287] The device 1 is also used for:

[0288] Obtain the second target data set generated through the index data-assisted retrieval method;

[0289] Determine the attribute conditions corresponding to the privacy data retrieval request, and use a forward sorting data pre-retrieval method to perform forward sorting data attribute filtering on the second target data set based on the attribute conditions to obtain a filtered candidate data set;

[0290] The target query vector corresponding to the privacy data retrieval request is determined using the vector similarity retrieval method. The similarity between the target query vector and the candidate vector data in the candidate data set is calculated to obtain the vector similarity calculation result. Based on the vector similarity calculation result, vector retrieval data processing is performed to obtain the target retrieval result.

[0291] In one feasible implementation, the target privacy data includes target vector data, target forward index data, and target inverted index data.

[0292] The device 1 is also used for:

[0293] In the write processing operation, a vector perturbation method is used to perform vector encryption based on a fixed value by performing an addition operation on each dimension of the target vector data;

[0294] In the write processing operation, a symmetric encryption algorithm is used to encrypt the target forward-sorted data.

[0295] In the write processing operation, keyword encryption is used to encrypt the target inverted index data.

[0296] It should be noted that the privacy data processing device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the privacy data processing method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the privacy data processing device and the privacy data processing method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0297] The example numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the examples.

[0298] In the embodiments described in this specification, in response to a data write instruction for target privacy data, the master node is controlled to treat the target privacy data as a target temporary file for write processing and trigger a node delayed loading mechanism. When the write processing operation is complete, the target temporary file is updated to the target formal file. Under the node delayed loading mechanism, the slave nodes are controlled to monitor the write processing operation of the target privacy data. When the write processing operation is complete, the target formal file corresponding to the target privacy data is loaded into memory. By introducing a temporary file mechanism and a delayed loading mechanism, the key limitations of data consistency and system complexity in a distributed environment are avoided, simplifying the implementation of traditional distributed systems (e.g., eliminating the need for complex real-time synchronization and metadata management). This provides high reliability for privacy data processing. Delayed loading avoids reading temporary data, ensuring data consistency and is applicable to various data types (vectors, ascending indexes, inverted indexes). Simultaneously, the delayed loading mechanism is dynamically triggered based on the update status of the target file, and slave nodes load the formal file on demand, reducing unnecessary resource consumption and improving system operating efficiency. Furthermore, it ensures the security of privacy data, preventing data errors and privacy leaks caused by incomplete data or improper synchronization in a distributed storage environment.

[0299] This specification also provides a computer storage medium that can store multiple instructions adapted to be loaded and executed by a processor as described above. Figures 1-5 The privacy data processing method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-5 The specific details of the illustrated embodiments will not be elaborated here.

[0300] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-5 The privacy data processing method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-5 The specific details of the illustrated embodiments will not be elaborated here.

[0301] Please refer to Figure 7 This diagram illustrates a structural block diagram of an electronic device provided in an exemplary embodiment of this specification. The electronic device in this specification may include one or more components such as a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, memory 120, input device 130, and output device 140 may be connected via the bus 150.

[0302] Processor 110 may include one or more processing cores. Processor 110 connects to various parts of the electronic device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 110 may integrate one or more of the following: central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.

[0303] The memory 120 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 120 may include a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described below, etc. The operating system may be the Android system, including systems deeply developed based on the Android system, the iOS system developed by Apple Inc., including systems deeply developed based on the iOS system, or other systems. The data storage area may also store data created by the electronic device during use, such as phonebook data, audio and video data, chat log data, etc.

[0304] See Figure 8As shown, the memory 120 can be divided into operating system space and user space. The operating system runs in the operating system space, while native and third-party applications run in the user space. To ensure that different third-party applications can achieve good running performance, the operating system allocates corresponding system resources for each application. However, different application scenarios within the same third-party application have different requirements for system resources. For example, in local resource loading scenarios, third-party applications have high requirements for disk read speed; in animation rendering scenarios, third-party applications have high requirements for GPU performance. Since the operating system and third-party applications are independent of each other, the operating system often cannot promptly perceive the current application scenario of a third-party application, resulting in the operating system's inability to adapt system resources accordingly to the specific application scenario of the third-party application.

[0305] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to establish data communication between the third-party applications and the operating system. This would allow the operating system to obtain the current scenario information of the third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0306] Taking the Android operating system as an example, the programs and data stored in memory 120 are as follows: Figure 9As shown, the memory 120 can store the Linux kernel layer 320, the system runtime library layer 340, the application framework layer 360, and the application layer 380. The Linux kernel layer 320, system runtime library layer 340, and application framework layer 360 belong to the operating system space, while the application layer 380 belongs to the user space. The Linux kernel layer 320 provides low-level drivers for various hardware components of the electronic device, such as display drivers, audio drivers, camera drivers, Bluetooth drivers, Wi-Fi drivers, and power management. The system runtime library layer 340 provides support for key features of the Android system through several C / C++ libraries. For example, the SQLite library provides database support, the OpenGL / ES library provides 3D graphics support, and the Webkit library provides browser kernel support. The system runtime library layer 340 also provides the Android runtime library, which mainly provides core libraries that allow developers to write Android applications using the Java language. The Application Framework Layer 360 provides various APIs that may be used when building applications. Developers can also use these APIs to build their own applications, such as activity management, window management, view management, notification management, content provider, package management, call management, resource management, and location management. At least one application runs in the Application Layer 380. These applications can be native applications that come with the operating system, such as contacts, SMS, clock, and camera apps; or third-party applications developed by third-party developers, such as games, instant messaging, and photo editing apps.

[0307] Taking the operating system as an example (iOS), the programs and data stored in memory 120 are as follows: Figure 10As shown, the iOS system includes: Core OS layer 420, Core Services layer 440, Media layer 460, and Cocoa Touch layer 480. Core OS layer 420 includes the operating system kernel, drivers, and low-level program frameworks. These low-level program frameworks provide hardware-level functionality for use by the program frameworks located in Core Services layer 440. Core Services layer 440 provides system services and / or program frameworks required by applications, such as Foundation framework, account framework, advertising framework, data storage framework, network connectivity framework, geolocation framework, motion framework, etc. Media layer 460 provides applications with audiovisual interfaces, such as interfaces related to graphics and images, audio technology, video technology, and AirPlay (wireless playback of audio and video transmission technologies). Cocoa Touch layer 480 provides various commonly used interface-related frameworks for application development and is responsible for user touch interaction on electronic devices. Examples include local notification services, remote push services, advertising frameworks, game tool frameworks, message user interface (UI) frameworks, UIKit frameworks, map frameworks, and so on.

[0308] exist Figure 10 The framework shown includes, but is not limited to, the base framework in the core service layer 440 and the UIKit framework in the touchable layer 480. The base framework provides many basic object classes and data types, offering the most basic system services to all applications, and is independent of the UI. The UIKit framework, on the other hand, provides a basic UI class library for creating touch-based user interfaces. iOS applications can use the UIKit framework to provide their UI, thus providing the application's infrastructure for building user interfaces, drawing, handling user interaction events, responding to gestures, and so on.

[0309] The methods and principles for implementing data communication between third-party applications and the operating system in the iOS system can be found in the Android system, and will not be repeated here.

[0310] The input device 130 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 140 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined into a touch screen, which is used to receive touch operations from the user using a finger, stylus, or any suitable object on or near it, and to display the user interface of various applications. The touch screen is usually located on the front panel of the electronic device. The touch screen can be designed as a full-screen, curved screen, or irregularly shaped screen. The touch screen can also be designed as a combination of a full-screen and a curved screen, or a combination of an irregularly shaped screen and a curved screen; this specification does not limit this aspect.

[0311] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WiFi) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0312] In the embodiments of this specification, the executing entity for each step can be the electronic device described above. Optionally, the executing entity for each step can be the operating system of the electronic device. The operating system can be Android, iOS, or other operating systems; this specification does not limit this.

[0313] The electronic device described in this specification can also be equipped with a display device. This display device can be any device capable of displaying information, such as a cathode ray tube display (CR), a light-emitting diode display (LED), an e-ink screen, a liquid crystal display (LCD), or a plasma display panel (PDP). Users can use the display device on the electronic device to view displayed text, images, videos, and other information. The electronic device can be a smartphone, tablet computer, gaming device, AR (Augmented Reality) device, automobile, data storage device, audio playback device, video playback device, laptop, desktop computing device, or wearable device such as a smartwatch, smart glasses, smart helmet, smart bracelet, smart necklace, or smart clothing.

[0314] exist Figure 7 In the illustrated electronic device, the processor 110 can be used to call the application stored in the memory 120 and specifically execute the method steps of one or more embodiments of this specification.

[0315] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0316] The above-disclosed embodiments are merely preferred embodiments of this specification and should not be construed as limiting the scope of this specification. Therefore, any equivalent variations made in accordance with the claims of this specification shall still fall within the scope of this specification.

Claims

1. A method for processing privacy data, characterized in that, Applied to a distributed data system, the distributed data system including master nodes and slave nodes, the method includes: In response to a data write command for target privacy data, the master node is controlled to treat the target privacy data as a target temporary file for write processing and trigger a node delayed loading mechanism. When the write processing operation is completed, the target temporary file is updated to the target official file. Under the node delayed loading mechanism, the slave node is controlled to monitor the write processing operation of the target privacy data. When the write processing operation is completed, the target formal file corresponding to the target privacy data is loaded into memory.

2. The method according to claim 1, characterized in that, The control mechanism for the master node to write the target privacy data as a target temporary file and trigger a node delayed loading mechanism includes: Determine the file data type of the target privacy data, and determine the target node delayed loading mechanism for the target privacy data based on the file data type; The master node is controlled to write the target privacy data as a target temporary file and trigger the target node's delayed loading mechanism.

3. The method according to claim 2, characterized in that, The target privacy data includes target vector data, target forward index data, and target inverted index data. Under the node delayed loading mechanism, the slave node is controlled to monitor the write processing operation of the target privacy data. When the write processing operation is completed, the target official file corresponding to the target privacy data is loaded into memory, including: If the file data type is a vector data type, then under the node delayed loading vector data mechanism, the slave node is controlled to monitor the first file update time of the target file. When the first file update time changes, the write processing operation is determined to be completed, and the target vector data is loaded into the local memory and / or hardware acceleration device of the slave node. If the file data type is a forward data type, then under the node delayed loading forward data mechanism, the slave node is controlled to monitor the file completion mark of the target file, and based on the file completion mark, the write processing operation is determined to be completed, and the target forward data is loaded into the local memory of the slave node in read-only mode; If the file data type is an inverted index data type, then under the node delayed loading index data mechanism, the slave node is controlled to monitor the second file update time of the target file. When the second file update time changes with time update, the write processing operation is determined to be completed, and the target inverted index data is loaded into the local memory of the slave node.

4. The method according to claim 3, characterized in that, The step of loading the target vector data into the local memory and / or hardware acceleration device of the slave node includes: loading the target vector data into the local memory and / or hardware acceleration device of the slave node using a full loading method or an on-demand loading method; and / or, The step of loading the target forward sorting data into the local memory of the slave node in read-only mode includes: determining the target incremental forward sorting data corresponding to the target forward sorting data using an incremental loading method, and loading the target incremental forward sorting data into the local memory of the slave node in read-only mode; and / or, The step of loading the target inverted index data into the local memory of the slave node includes: determining the incremental index mapping relationship corresponding to the target inverted index data using an incremental loading method, and loading the incremental index mapping relationship into the local memory of the slave node.

5. The method according to claim 1, characterized in that, The method further includes: In response to a privacy data retrieval request, the file data retrieval type is determined based on the privacy data retrieval request, and the privacy data retrieval processing is performed using the data retrieval method corresponding to the file data type.

6. The method according to claim 5, characterized in that, The process of performing privacy data retrieval using the data retrieval method corresponding to the file data retrieval type includes: If the data type of the retrieved file is vector data retrieval type, then the vector similarity retrieval method is adopted, the target vector data in the target formal file is loaded from the slave node, the similarity between the target query vector corresponding to the privacy data retrieval request and the target vector data is calculated to obtain the vector similarity calculation result, and the vector retrieval data processing is performed based on the vector similarity calculation result; If the data type of the searched file is a forward data retrieval type, then the forward data pre-retrieval method is adopted, and the slave node is controlled to perform forward data pre-retrieval on the target file. The first target data set is selected according to the attribute conditions corresponding to the privacy data retrieval request, and the first target data set is transmitted to the vector retrieval engine. The first target data set is used to assist the vector retrieval engine in narrowing the retrieval scope. If the data type of the retrieved file is an inverted index data type, then an index data-assisted retrieval method is adopted. The slave node is controlled to perform inverted index keyword matching on the target official file. The second target data set is retrieved according to the keywords corresponding to the privacy data retrieval request. A candidate data set is generated based on the second target data set and the first target data set corresponding to the forward data pre-retrieval method. The candidate data set is transmitted to the vector retrieval engine. The candidate data set is used to assist the vector retrieval engine in narrowing the retrieval scope.

7. The method according to claim 6, characterized in that, The file data retrieval types include a fusion data retrieval type targeting the vector data retrieval type, the forward index data retrieval type, and the inverted index data type. The process of performing privacy data retrieval using the data retrieval method corresponding to the file data retrieval type includes: Obtain the second target data set generated through the index data-assisted retrieval method; Determine the attribute conditions corresponding to the privacy data retrieval request, and use a forward sorting data pre-retrieval method to perform forward sorting data attribute filtering on the second target data set based on the attribute conditions to obtain a filtered candidate data set; The target query vector corresponding to the privacy data retrieval request is determined using the vector similarity retrieval method. The similarity between the target query vector and the candidate vector data in the candidate data set is calculated to obtain the vector similarity calculation result. Based on the vector similarity calculation result, vector retrieval data processing is performed to obtain the target retrieval result.

8. The method according to claim 1, characterized in that, The target privacy data includes target vector data, target forward index data, and target inverted index data. The method further includes: In the write processing operation, a vector perturbation method is used to perform vector encryption based on a fixed value by performing an addition operation on each dimension of the target vector data; In the write processing operation, a symmetric encryption algorithm is used to encrypt the target forward-sorted data. In the write processing operation, keyword encryption is used to encrypt the target inverted index data.

9. A privacy data processing device, characterized in that, The device is applied to a distributed data system, which includes master nodes and slave nodes, and includes: The writing module is used to respond to a data writing command for target privacy data, control the master node to treat the target privacy data as a target temporary file for writing processing operation and trigger the node delayed loading mechanism, and update the target temporary file to the target formal file when the writing processing operation is completed; The loading module is used to control the slave node to monitor the write processing operation of the target privacy data under the node delayed loading mechanism, and to perform memory loading processing on the target formal file corresponding to the target privacy data when the write processing operation is completed.

10. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed the method steps as claimed in any one of claims 1 to 8.

Images