Data storage method, device, storage medium, and program product

Abstract

The application discloses a data storage method and device, a storage medium and a program product. All attribute data of a document set is subjected to data deduplication to obtain a plurality of target attribute data, and all target attribute data in the document set is continuously stored in a first storage space. According to the storage position of each attribute data in the first storage space, a first index value corresponding to each attribute data is determined, and all first index values are continuously stored in a second storage space. According to the storage position of the first index value in the second storage space, a second index value corresponding to each non-empty attribute in each document is determined, and all second index values are stored in a third storage space. The application can effectively reduce the data storage amount and improve the data search performance, and can avoid any memory fragmentation. The application can be widely applied to storage technologies in scenes such as cloud technology, artificial intelligence, intelligent transportation, and auxiliary driving.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and in particular to a data storage method, apparatus, storage medium, and program product. Background Technology

[0002] For recall systems used in search and recommendation scenarios, a forward-ordered storage design is often required for the materials in the recall system. Typically, the materials in a recall system are called documents, and each document includes multiple attributes (e.g., a news article includes a title, author, and body text). Some attributes in a document have data, while others do not, resulting in sparse data types. Faced with the massive amounts of documents in a recall system, the industry generally adopts a document list array-based storage design. Each document is assigned a storage container that can encapsulate dynamic arrays. The storage container includes arrays corresponding to each attribute. When a document has more than one instance of data for a certain attribute, the data is inserted into the corresponding array. However, because many attributes in a document lack data, and these attributes also occupy storage space, the storage overhead for document data is relatively high. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0004] This invention provides a data storage method, apparatus, storage medium, and program product that can optimize the data storage structure for sparse data and save data storage overhead.

[0005] On one hand, embodiments of the present invention provide a data storage method, including the following steps:

[0006] Obtain a document collection, the document collection including multiple documents, each document including multiple attributes, at least one of the multiple attributes being a non-empty attribute, the non-empty attribute including at least one attribute data;

[0007] Deduplication is performed on all attribute data in the document collection to obtain multiple target attribute data.

[0008] All the target attribute data are stored contiguously in the first storage space;

[0009] Based on the storage location of the target attribute data corresponding to each attribute data in the first storage space, a first index value corresponding to each attribute data is determined, and the first index value is used to index the attribute data in the first storage space;

[0010] All the first index values ​​are stored consecutively in the second storage space;

[0011] Based on the storage location of the first index value in the second storage space, a second index value corresponding to each non-empty attribute in each document is determined, and the second index value is used to index the first index value in the second storage space;

[0012] All the second index values ​​are stored in the third storage space.

[0013] On the other hand, embodiments of the present invention also provide a data storage device, including:

[0014] The acquisition module is used to acquire a document collection, the document collection including multiple documents, each document including multiple attributes, at least one of the multiple attributes being a non-empty attribute, the non-empty attribute including at least one attribute data;

[0015] The data deduplication module is used to deduplicat all attribute data in the document collection to obtain multiple target attribute data.

[0016] The first storage module is used to continuously store all the target attribute data in the first storage space;

[0017] The first indexing module is used to determine a first index value corresponding to each attribute data according to the storage location of the target attribute data corresponding to each attribute data in the first storage space. The first index value is used to index the attribute data in the first storage space.

[0018] The second storage module is used to continuously store all the first index values ​​in the second storage space;

[0019] The second indexing module is used to determine the second index value corresponding to each non-empty attribute in each document according to the storage location of the first index value in the second storage space. The second index value is used to index the first index value in the second storage space.

[0020] The third storage module is used to store all the second index values ​​in the third storage space.

[0021] Optionally, the second index value includes a third index value and a fourth index value; the second index module specifically includes:

[0022] The first determining unit is used to traverse each document in the document set, and for the currently traversed document, obtain the third index value corresponding to each non-empty attribute in the document according to the document tag corresponding to the document and the attribute tag corresponding to each non-empty attribute in the document, wherein the third index value is used to index the non-empty attributes of the document.

[0023] The second determining unit is configured to determine, in the second storage space, the first index value corresponding to each of the third index values;

[0024] The third determining unit is used to determine the fourth index value corresponding to each of the third index values ​​based on the storage location of the first index value corresponding to each of the third index values ​​in the second storage space.

[0025] Optionally, the data storage device further includes a tag determination module, the tag determination module being used for:

[0026] Set the document tags for each of the aforementioned documents;

[0027] Obtain the attribute type of each of the attributes included in each of the documents;

[0028] The attribute label is set for each attribute according to the attribute type, so that the attribute labels corresponding to the attributes with the same attribute type are the same.

[0029] Optionally, the second determining unit is specifically used for:

[0030] Based on the attribute data corresponding to each of the first index values ​​in the second storage space, determine the document and the non-empty attribute corresponding to the first index value;

[0031] The first index value corresponding to the third index value is determined based on the document corresponding to the first index value and the non-empty attribute.

[0032] Optionally, the third determining unit is specifically used for:

[0033] The fourth index value corresponding to the third index value is determined based on the first starting position and the first data length of the first index value corresponding to the third index value in the second storage space;

[0034] or,

[0035] The fourth index value corresponding to the third index value is determined based on the first start position and the first end position of the first index value corresponding to the third index value in the second storage space;

[0036] or,

[0037] The fourth index value corresponding to the third index value is determined based on the first end position of the first index value corresponding to the third index value in the second storage space and the first data length.

[0038] Optionally, the third storage space includes a fourth storage space and a fifth storage space; the third storage module specifically includes:

[0039] The first storage unit is used to continuously store all the third index values ​​in the fourth storage space;

[0040] The second storage unit is used to store all the fourth index values ​​consecutively in the fifth storage space;

[0041] The order in which the third index value is stored in the fourth storage space is consistent with the order in which the fourth index value is stored in the fifth storage space.

[0042] Optionally, the third index value includes the document tag and the attribute tag, and there is a sequential order between the document tags and between the attribute tags;

[0043] The first storage unit is specifically used for:

[0044] Sort all the third index values ​​according to the order of the document tags and the order of the attribute tags to obtain a sequence of third index values;

[0045] According to the third index value sequence, each of the third index values ​​is stored in the fourth storage space in sequence, and all the third index values ​​are arranged consecutively in the fourth storage space.

[0046] Optionally, the second storage unit is specifically used for:

[0047] The fourth index value is sorted according to the third index value sequence to obtain the fourth index value sequence;

[0048] According to the fourth index value sequence, each of the fourth index values ​​is stored in the fifth storage space in sequence, and all the fourth index values ​​are arranged consecutively in the fifth storage space.

[0049] Optionally, the document has document tags, the attributes have attribute tags, and there is a sequence between the document tags and between the attribute tags;

[0050] The second storage module is specifically used for:

[0051] Sort all the first index values ​​according to the order of the document tags and the order of the attribute tags to obtain the first index value sequence;

[0052] Based on the first index value sequence, each of the first index values ​​is stored sequentially in the second storage space, and all the first index values ​​are arranged consecutively in the second storage space.

[0053] Optionally, the first index module specifically includes:

[0054] The fourth determining unit is used to traverse each attribute data in the document collection, search for the target attribute data that matches the currently traversed attribute data in the first storage space, and obtain the current target attribute data.

[0055] The fifth determining unit is used to determine the first index value corresponding to the attribute data currently being traversed, based on the storage location of the current target attribute data in the first storage space.

[0056] Optionally, the fifth determining unit is specifically used for:

[0057] Based on the second starting position and the second data length of the current target attribute data in the first storage space, determine the first index value corresponding to each attribute data;

[0058] or,

[0059] Based on the second start position and the second end position of the current target attribute data in the first storage space, determine the first index value corresponding to each attribute data;

[0060] or,

[0061] Based on the second end position and the second data length of the current target attribute data in the first storage space, the first index value corresponding to each attribute data is determined.

[0062] On the other hand, embodiments of the present invention also provide a data storage device, including:

[0063] At least one processor;

[0064] At least one memory for storing at least one program;

[0065] The data storage method described above is implemented when at least one of the programs is executed by at least one of the processors.

[0066] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to implement the data storage method as described above.

[0067] On the other hand, embodiments of the present invention also provide a computer program product, including a computer program or computer instructions, the computer program or computer instructions being stored in a computer-readable storage medium, a processor of a computer device reading the computer program or computer instructions from the computer-readable storage medium, and the processor executing the computer program or computer instructions to cause the computer device to perform the data storage method as described above.

[0068] The embodiments of the present invention include at least the following beneficial effects: First, a document set is obtained, comprising multiple documents, each document comprising multiple attributes, at least one of which is a non-empty attribute, and the non-empty attribute includes at least one attribute data. Then, all attribute data in the document set is deduplicated to obtain multiple target attribute data, and all target attribute data are stored contiguously in a first storage space. This effectively achieves deduplication of data storage and reduces data storage volume. Next, based on the storage location of the target attribute data corresponding to each attribute data in the first storage space, a first index value corresponding to each attribute data is determined, and all first index values ​​are stored contiguously in a second storage space. In the first storage space, attribute data can be indexed using the first index value. Then, based on the storage location of the first index value in the second storage space, the second index value corresponding to each non-empty attribute in each document is determined, and all second index values ​​are stored in the third storage space. The first index value can be indexed using the second index value in the second storage space. Since the second index value is established for non-empty attributes in the document, when indexing the attribute data of the document, the non-empty attributes with data in the document can be quickly located first using the second index value, and then the attribute data corresponding to the non-empty attributes can be obtained according to the first index value corresponding to the second index value, effectively improving the performance of data retrieval. In addition, the contiguous storage method adopted in this embodiment of the invention can avoid generating any memory fragmentation, greatly improving the utilization rate of storage space. Attached Figure Description

[0069] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0070] Figure 1a This is a schematic diagram of an implementation environment to which the data storage method provided in this embodiment of the invention is applicable;

[0071] Figure 1b This is a schematic diagram of another implementation environment to which the data storage method provided in this embodiment of the invention is applicable;

[0072] Figure 2 This is a flowchart illustrating the data storage method provided in an embodiment of the present invention;

[0073] Figure 3 This is a schematic diagram of the storage structure of the target attribute data in the first storage space according to an embodiment of the present invention;

[0074] Figure 4 This is a schematic diagram of the storage structure of the first index value in the second storage space according to an embodiment of the present invention;

[0075] Figure 5a This is a schematic diagram of the storage structure of the first index value in the second storage space in a specific example of the present invention;

[0076] Figure 5b This is a schematic diagram of the storage structure of the first index value in the second storage space in another specific example of the present invention;

[0077] Figure 5c This is a schematic diagram of the storage structure of the first index value in the second storage space in another specific example of the present invention;

[0078] Figure 6 This is a schematic diagram of the storage structure of the third index value in the fourth storage space according to an embodiment of the present invention;

[0079] Figure 7 This is a schematic diagram of the storage structure of the fourth index value in the fifth storage space according to an embodiment of the present invention;

[0080] Figure 8a This is a schematic diagram of the storage structure of the fourth index value in the fifth storage space in a specific example of the present invention;

[0081] Figure 8b This is a schematic diagram of the storage structure of the fourth index value in the fifth storage space in another specific example of the present invention;

[0082] Figure 8c This is a schematic diagram of the storage structure of the fourth index value in the fifth storage space in another specific example of the present invention;

[0083] Figure 9 This is a schematic diagram of the structure of a data storage device provided in an embodiment of the present invention;

[0084] Figure 10 This is a schematic diagram of another data storage device provided in an embodiment of the present invention. Detailed Implementation

[0085] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The described embodiments should not be considered as limitations on the present invention, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present invention.

[0086] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0087] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0088] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.

[0089] 1) Recall (match) refers to retrieving as much relevant information as possible from the full dataset, primarily used in search and recommendation scenarios. For example, search recall is based on the search terms input by the target audience, retrieving a set of candidate documents related to those terms from the document library. Another example is recommendation recall, which matches based on the target audience's attributes and content profile; when the target audience opens recommendations, a set of highly relevant content is retrieved from the content library according to a recall strategy.

[0090] 2) Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, e-commerce platforms, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0091] 3) Big data refers to data sets that cannot be captured, managed, and processed within a certain timeframe using conventional software tools. It represents massive, rapidly growing, and diverse information assets that require new processing models to achieve stronger decision-making, insightful discovery, and process optimization capabilities. With the advent of the cloud era, big data has attracted increasing attention. Big data requires specialized technologies to effectively process large amounts of data within a tolerable timeframe. Technologies suitable for big data include massively parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems.

[0092] 4) Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and cryptographic algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying platform, a platform product service layer, and an application service layer.

[0093] 5) Intelligent Traffic System (ITS), also known as Intelligent Transportation System, effectively integrates advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control, and vehicle manufacturing, strengthening the connection between vehicles, roads, and users, thereby forming a comprehensive transportation system that ensures safety, improves efficiency, improves the environment, and saves energy.

[0094] 6) Intelligent Vehicle Infrastructure Cooperative Systems (IVICS), or vehicle-road cooperative systems for short, represent a development direction for Intelligent Transportation Systems (ITS). IVICS utilizes advanced wireless communication and next-generation Internet technologies to implement comprehensive, real-time dynamic information exchange between vehicles and infrastructure. Based on the collection and fusion of dynamic traffic information across all times and spaces, it conducts active vehicle safety control and cooperative road management, fully realizing effective collaboration between people, vehicles, and roads. This ensures traffic safety, improves traffic efficiency, and ultimately forms a safe, efficient, and environmentally friendly road traffic system.

[0095] Currently, both search engines and recommendation engines have basically formed a structure combining recall and ranking. The goal of recall is to extract a number of materials that may be suitable for display from a massive amount of material, while ranking is mainly responsible for sorting the materials according to the interests of the target audience before displaying them to them. For recall systems used in search and recommendation applications, it is often necessary to design the storage of materials (which can be called documents) in a forward order on the backend server so that the content of the document can be retrieved based on the document tags during the recall process. Generally speaking, each document includes multiple attributes (for example, a news article includes multiple attributes such as title, author, and body text).

[0096] For example, the recall system includes multiple documents, named doc0, doc1, doc2, ... Please refer to Table 1, which shows the data structure of document doc0. doc0 has 200 attributes divided according to the fields to which the data belongs, named field0, field1, ..., field199. Among them, the attributes with data are field0, field10, and field38, and the remaining attributes have no data. Attributes with data can be called non-null attributes, and attributes without data can be called null attributes. It is worth noting that non-null attributes can have multiple values ​​(representing multi-valued attributes) or only one value (representing single-valued attributes). In doc0, all non-null attributes have multiple values.

[0097] Table 1

[0098]

[0099] Please refer to Table 2, which shows the data structure of document doc1. doc1 has the same number and names of attributes as doc0. In doc1, there are four non-empty attributes: field0, field2, field15, and field68. Among them, field0 is a single value, and the other non-empty attributes are multi-valued.

[0100] Table 2

[0101]

[0102] As can be seen from the document data structure shown in Tables 1 and 2, some attributes in the document have data, while others do not, thus the document data has the characteristic of being sparse.

[0103] Faced with the massive amounts of documents retrieved by search and recommendation engines, the industry typically stores document data using an array-based approach. Each document is encapsulated in a dynamic array container, which includes arrays corresponding to various attributes. When an attribute has more than one value, the value is inserted into the corresponding array. However, because many attributes in a document lack data, and these empty attributes also occupy storage space, the storage overhead is significant. Another technique involves classifying and storing sparse data differently based on its sparsity. This approach is primarily designed for single-value scenarios and cannot effectively utilize the compression introduced by value duplication in multi-value scenarios, thus also resulting in high storage overhead.

[0104] In view of this, embodiments of the present invention provide a data storage method, a data storage device, a computer-readable storage medium, and a computer program product, which can optimize the data storage structure for the characteristics of sparse data and save data storage overhead.

[0105] In the data storage method provided in this embodiment of the invention, a document set is first obtained, which includes multiple documents. Each document includes multiple attributes, at least one of which is a non-empty attribute, and the non-empty attribute includes at least one attribute data. Then, all attribute data in the document set is deduplicated to obtain multiple target attribute data, and all target attribute data are stored contiguously in a first storage space. Next, based on the storage location of the target attribute data corresponding to each attribute data in the first storage space, a first index value corresponding to each attribute data is determined. This first index value is used to index the attribute data in the first storage space, and all first index values ​​are stored contiguously in a second storage space. Then, based on the storage location of the first index values ​​in the second storage space, a second index value corresponding to each non-empty attribute in each document is determined. This second index value is used to index the first index value in the second storage space, and all second index values ​​are stored in a third storage space.

[0106] The aforementioned data storage method effectively deduplicates data, reducing the amount of data stored. The contiguous storage approach avoids memory fragmentation, significantly improving storage space utilization. Furthermore, this method establishes a second index value for non-empty attributes in the document, enabling rapid location of attributes containing data during indexing. It also allows retrieval of the attribute data corresponding to the non-empty attribute based on the first index value corresponding to the second index value, effectively enhancing data retrieval performance.

[0107] It should be understood that the data storage method provided in this embodiment of the invention can be applied to the backend server of a target application. This server can be used to manage the data of recalled documents used by the target application to implement search or recommendation functions, and supports data writing and reading. In practical applications, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud storage services. Furthermore, the server can be a node on a blockchain.

[0108] It should be noted that the target application in this embodiment of the invention can be any application with search or content recommendation functions, such as a navigation application, video application, social application, music application, shopping application, or news application. This embodiment of the invention does not impose any limitation on the target application.

[0109] To facilitate understanding of the data storage method provided in the embodiments of the present invention, the implementation environment of the data storage method is described below.

[0110] Please see Figure 1a , Figure 1a A schematic diagram illustrating an implementation environment suitable for the data storage method provided in this embodiment of the invention is shown. Figure 1a As shown, the implementation environment includes terminal 101 and server 102. Terminal 101 and server 102 are directly or indirectly connected via wired or wireless communication. Terminal 101 and server 102 can be nodes in a blockchain, and the number of terminals 101 and servers 102 can be one or at least two, without specific limitations here.

[0111] Terminal 101 may include, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc.

[0112] It should be understood that terminal 101 has at least a display function for displaying search results or recommended content. For example, terminal 101 may run a target application, such as a navigation application. In response to a route query term entered by the target user using the navigation application, terminal 101 sends a route search request carrying the query term to server 102. After receiving the route search results from server 102 based on the route search request, terminal 101 can display the route search results to the target user. As another example, if the target application is a video application, terminal 101 sends a video retrieval request carrying the target user's attribute information to server 102. After receiving recommended content from server 102 based on the video retrieval request, terminal 101 displays the recommended content to the target user.

[0113] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0114] Server 102 provides background services for the target application, and its memory stores document data. For example, when the target application is a navigation application, after receiving route query terms input by the target user from terminal 101, server 102 recalls candidate routes related to the route query terms based on the document data stored in memory, sorts the candidate routes according to a preset recommendation algorithm to obtain recommended route search results, and sends the recommended route search results to terminal 101 for display. As another example, when the target application is a video application, after receiving a video retrieval request from terminal 101, server 102 recalls candidate video content that the target user may be interested in based on the document data stored in memory, sorts the candidate video content according to a preset recommendation algorithm to obtain the final recommended video content, and then pushes the recommended video content to terminal 101. In this process, when storing document data, server 102 first deduplicates the attribute data of multiple documents to obtain the target attribute data, and then stores all the target attribute data in the first storage space of memory. Next, based on the storage location of the target attribute data in the first storage space, a first index value is created for each attribute data in each document. This first index value is used to index the target attribute data in the first storage space to obtain the specific value of the attribute data, and all first index values ​​are stored in the second storage space of memory. Then, a second index value is created for each non-empty attribute in each document. This second index value is used to index the first index value corresponding to the non-empty attribute in the document in the second storage space, and all second index values ​​are stored in the third storage space of memory. When server 102 recalls a document, it can obtain the first index value based on the second index value corresponding to the non-empty attribute of the document, and then obtain the target attribute data based on the first index value.

[0115] Understandably, to improve the efficiency of document data analysis, processing, and storage, at least one analysis server can be used to analyze and process the document data, and multiple data servers can be used for collaborative storage of the document data. Please see [link to relevant documentation]. Figure 1b , Figure 1b This diagram illustrates another implementation environment suitable for the data storage method provided in this embodiment of the invention. For example... Figure 1bAs shown, the implementation environment includes a terminal 201, an analysis server 202, and a data server. The data server includes a first data server 203a, a second data server 203b, and a third data server 203c. The terminal 201 is connected to the analysis server 202, and the analysis server 202 is connected to the first data server 203a, the second data server 203b, and the third data server 203c, respectively. The terminal 201 is used to display search results or recommended content pushed by the analysis server 202. The analysis server 202 provides background services for the target application installed on the terminal 201. Based on the search terms or target object attribute information sent by the terminal 201, it retrieves document data from the data server and then returns search results or recommended content to the terminal 201, enabling the terminal 201 to display search results or recommended content to the target object.

[0116] Specifically, in the process of storing document data, the analysis server 202 first performs deduplication on the attribute data of multiple documents to obtain the target attribute data; all the target attribute data is stored on the first data server 203a; then, based on the storage location of the target attribute data on the first data server 203a, the analysis server 202 establishes a first index value for each attribute data in each document. This first index value is used to index the target attribute data on the first data server 203a to obtain the specific value of the attribute data; all the first index values ​​are stored on the second data server 203b; then, the analysis server 202 establishes a second index value for each non-empty attribute in each document. This second index value is used to index the first index value corresponding to the non-empty attribute in the document on the second data server 203b; all the second index values ​​are stored on the third data server 203c. When the analysis server 202 recalls a document, it can first obtain the second index value corresponding to the non-empty attribute of the document from the third data server 203c, then obtain the first index value from the second data server 203b based on the second index value, and finally obtain the target attribute data from the first data server 203a based on the first index value.

[0117] It should be noted that in various specific embodiments of the present invention, when processing is required based on data related to the characteristics of an object (such as a user's attribute information or a set of attribute information), the permission or consent of the corresponding object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when an embodiment of the present invention needs to obtain the attribute information of an object, separate permission or consent from the corresponding object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the separate permission or consent of the corresponding object will the necessary object-related data for the normal operation of the embodiments of the present invention be obtained.

[0118] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data storage method provided in an embodiment of the present invention. For ease of description, the following embodiments use a server as an example to illustrate the execution of this data storage method. Figure 2 As shown, the data storage method includes steps S310 to S370, which are described in detail below:

[0119] Step S310: Obtain a document collection, which includes multiple documents. Each document includes multiple attributes, at least one of which is a non-empty attribute, and the non-empty attribute includes at least one attribute data.

[0120] It is understandable that the document collection is the material collection of the recall system, and the documents in the document collection are the materials that can be recalled in the recall system. For example, the candidate materials of a search engine, which selects materials related to the search terms entered by the target object from the candidate material collection and pushes them; or the recommended materials of a recommendation engine, which selects materials that the target object may be interested in from the recommended material collection based on the attributes of the target object and pushes them.

[0121] It's also understandable that each document in a document collection includes multiple attributes. Based on whether an attribute contains data in the document, the attributes in the document can be divided into non-empty attributes and empty attributes. Non-empty attributes are those with at least one attribute data, while empty attributes are those without attribute data. For example, in document doc0 shown in Table 1, field0, field10, and field38 are non-empty attributes, and the remaining attributes are empty attributes; in document doc1 shown in Table 2, field0, field2, field15, and field68 are non-empty attributes, and the remaining attributes are empty attributes.

[0122] In some possible embodiments, attributes in a document represent field attributes, and corresponding attribute data represents the strings corresponding to those field attributes. These strings can include one or more of the following: numbers 0 to 9, uppercase letters, lowercase letters, punctuation marks, and special characters. For example, a document may contain 200 fields, which can be of type string. Some fields may not have corresponding strings, while others may have one or more strings. Fields without strings are called empty attributes, and fields with strings are called non-empty attributes. For fields with corresponding strings, fields with only one string can be called single-valued fields, and fields with at least two strings can be called multi-valued fields. Those skilled in the art should understand that attributes in a document can also represent other types of attributes besides fields; this is not specifically limited here.

[0123] Optionally, after obtaining the document collection, the data storage method provided in this embodiment of the invention further includes the following steps:

[0124] (1) Set document tags for each document;

[0125] (2) Obtain the attribute types of the attributes included in each document;

[0126] (3) Set attribute tags for each attribute according to the attribute type, so that the attribute tags corresponding to attributes with the same attribute type are the same.

[0127] It is understandable that after obtaining the document collection, a unique document tag is set for each document in the document collection. The document tag is used to identify the document and can consist of at least one of numbers, letters, and text.

[0128] It should be noted that the document tags for each document have a specific order. The sorting rules can be based on tag values ​​or document creation time. For example, assuming the system stores N documents, and each document's tag is a number, with doc0 to docN-1 corresponding to tags 0 to N-1, then the order of the document tags can be determined by ascending order of their numerical values.

[0129] It is understandable that each attribute in a document has a corresponding attribute tag, which can consist of at least one of numbers, letters, and text. The attribute tag corresponding to an attribute can be determined according to the attribute type, and each attribute type has a unique corresponding attribute tag. For example, assuming that attributes field0 to field199 in Table 1 and Table 2 represent different attribute types, and the attribute tags corresponding to each attribute are numbers, then the attribute tags corresponding to attributes field0 to field199 can be set to 0 to 199.

[0130] It should be noted that the attribute tags corresponding to each attribute have a specific order. The specific sorting rule can be to sort the attribute tags according to the tag value. For example, the attribute tags corresponding to field0 to field199 are numbered 0 to 199, so the order of each attribute tag can be determined by sorting the tags in ascending order of the numbers.

[0131] Step S320: Perform data deduplication on all attribute data in the document collection to obtain multiple target attribute data.

[0132] For example, after obtaining the document collection, for each document in the document collection, each attribute data in the document is traversed, and the currently traversed attribute data is compared with the existing target attribute data to determine whether the currently traversed attribute data is duplicated with the existing target attribute data. If they are not duplicated, the currently traversed attribute data is determined as the new target attribute data; if they are duplicated, the currently traversed attribute data is determined as non-target attribute data.

[0133] The following explanation uses document doc0 (as shown in Table 1) and document doc1 (as shown in Table 2) as examples.

[0134] In practice, we can first iterate through each non-empty attribute in doc0 in ascending order of attribute number. When iterating through the first non-empty attribute field0 of doc0, since the attribute data "123", "456" and "789" in field0 do not duplicate existing target attribute data, all attribute data in field0 are identified as new target attribute data. When iterating through the second non-empty attribute field10 of doc0, since the attribute data "123" and "789" in field10 duplicate existing target attribute data, all attribute data in field10 are identified as non-target attribute data. When iterating through the third non-empty attribute field38 of doc0, since the attribute data "456" in field38 duplicates existing target attribute data, the attribute data "456" in field38 is identified as non-target attribute data, while the attribute data "4567" and "999" in field38 do not duplicate existing target attribute data, so "4567" and "999" are identified as new target attribute data.

[0135] Similarly, we can iterate through each non-empty attribute in doc1 in ascending order of attribute number. When iterating through the first non-empty attribute field0, since the attribute data "456" in field0 is a duplicate of the existing target attribute data, the attribute data "456" in field0 is determined to be non-target attribute data. When iterating through the second non-empty attribute field2, since the attribute data "123", "456", and "789" in field2 are all duplicates of the existing target attribute data, all attribute data in field2 are determined to be non-target attribute data. When iterating through the third non-empty attribute field... At d15, since the attribute data "xyz" and "wox" in field15 do not duplicate existing target attribute data, all attribute data in field15 are identified as new target attribute data. When traversing the fourth non-empty attribute field68 of doc1, since the attribute data "456" and "4567" in field68 duplicate existing target attribute data, the attribute data "456" and "4567" in field68 are identified as non-target attribute data, while the attribute data "xx999" in field68 does not duplicate existing target attribute data, so "xx999" is identified as new target attribute data.

[0136] It is understood that, in this embodiment of the invention, the target attribute data is obtained by deduplicating all attribute data included in the document collection. Subsequently, only the target attribute data needs to be stored, thereby avoiding the repeated storage of the same attribute data and achieving the purpose of reducing data storage volume and reducing storage overhead.

[0137] Step S330: Store all target attribute data continuously in the first storage space.

[0138] It is understood that the first storage space described in this embodiment of the invention is a contiguous storage space in the server's memory. After deduplicating all attribute data to obtain the target attribute data, each target attribute data is sequentially stored in the first storage space.

[0139] It should be noted that the target attribute data in this embodiment of the invention is stored in the first storage space in a continuous storage manner. Continuous storage, as described here, means that the target attribute data is arranged continuously and without interruption within the first storage space. In other words, there are no bit intervals between adjacent target attribute data points, nor between multiple characters included in the target attribute data, within the first storage space.

[0140] In practice, whenever a new target attribute data is determined, that new target attribute data is stored in the first storage space. The new target attribute data is immediately adjacent to the previous target attribute data in the first storage space, with no bit intervals between them. When the target attribute data is a string, the characters of the string are also consecutive in the first storage space, with no bit intervals between characters.

[0141] Please see Figure 3 , Figure 3 A schematic diagram of the storage structure of the target attribute data in the first storage space is shown. Based on the target attribute data obtained after deduplicating all attribute data in the document collection, such as “123”, “456”, “789”, “4567”, “999”, “xyz”, “wox”, and “xx999”, the first determined target attribute data “123” is first stored in the first storage space, and the target attribute data “123” occupies the interval [0, 2] in the first storage space. Next, the target attribute data “456” is stored in the first storage space, and the target attribute data “456” is next to the target attribute data “123” in the first storage space, occupying the interval [3, 5]. And so on, the target attribute data “789”, “4567”, “999”, “xyz”, “wox”, and “xx999” are stored in the first storage space, and their corresponding intervals in the first storage space are [6, 8], [9, 12], [13, 15], [16, 18], [19, 21], and [22, 26].

[0142] It should be noted that, Figure 3 The target attribute data in the examples shown are all string type data. To facilitate understanding of the solutions in this embodiment of the invention, in Figure 3 In the standard, target attribute data in the first storage space is represented directly by the characters themselves. However, in practical applications, character data is stored in memory in ASCII code form. Therefore, each character of the target attribute data must be converted into ASCII code before it can be stored in the first storage space. Each character, after being converted into the binary form of ASCII code, is a one-byte data. For example, the ASCII codes for the characters "1", "2", and "3" in the target attribute data "123" are "49", "50", and "51". After converting the ASCII codes "49", "50", and "51" into binary, they correspond to "00110001", "00110010", and "00110011". Therefore, the actual storage form of "123" in the first storage space is "001100010011001000110011".

[0143] It is understandable that the intervals described above are actually byte intervals. For example, the interval [0, 2] described above represents the storage space occupied by bytes 0 to byte 2, and the data length of the target attribute data "456" in the interval [0, 2] in the first storage space is 3 bytes; as another example, the interval [9, 12] described above represents the storage space occupied by bytes 9 to byte 12, and the data length of the target attribute data "4567" in the interval [9, 12] in the first storage space is 4 bytes.

[0144] from Figure 3 As shown in the example, the multiple target attribute data are arranged consecutively in the first storage space, with no bit intervals between adjacent target attribute data and no bit intervals between the multiple characters included in the target attribute data. Therefore, no memory fragmentation is generated, thereby reducing the storage space occupied by the target attribute data and improving the storage space utilization.

[0145] Step S340: Determine the first index value corresponding to each attribute data according to the storage location of the target attribute data in the first storage space. The first index value is used to index the attribute data in the first storage space.

[0146] Understandably, in order to facilitate the retrieval of the document's corresponding attribute data during the retrieval process, it is necessary to establish a first index value for each attribute data. This first index value indicates the storage location of the document's attribute data in the first storage space. Based on the storage location indicated by the first index value, the document's corresponding attribute data can be indexed in the first storage space.

[0147] For example, step S340 can be implemented as follows: traverse each attribute data of each document in the document collection, and perform the following processing on the attribute data currently being traversed: search for target attribute data that matches the attribute data currently being traversed in the first storage space to obtain the current target attribute data; determine the first index value corresponding to the attribute data currently being traversed based on the storage location of the current target attribute data in the first storage space.

[0148] For example, during the process of traversing the attribute data in document doc0, when the attribute data "123" in the non-empty attribute field0 of doc0 is encountered, the target attribute data matching "123" is searched in the first storage space, thus determining that "123" is... Figure 3 The storage location in the first storage space shown is in the range [0, 2]. Therefore, based on the determined storage location, the first index value corresponding to the attribute data "123" in the non-empty attribute field0 of doc0 can be determined.

[0149] Specifically, determining the first index value corresponding to the currently traversed attribute data based on the storage location of the current target attribute data in the first storage space can include the following three implementation methods:

[0150] The first method involves determining the first index value corresponding to each attribute data based on the second starting position and the second data length of the current target attribute data in the first storage space.

[0151] For example, assuming the current target attribute data is "123", and the starting position of "123" in the first storage space is byte 0, and the data length is 3 bytes, we can determine that the second starting position is 0 and the second data length is 3. Then, the first index value corresponding to the attribute data "123" in the non-empty attribute field0 of doc0 can be represented by "0--3". Then, by encoding "0--3", we can obtain the first index value corresponding to "123".

[0152] For example, suppose the current target attribute data is "456". The starting position of "456" in the first storage space is byte 3 and the data length is 3 bytes. That is, the second starting position is determined to be 3 and the second data length is 3. Then, the first index value corresponding to the attribute data "456" in the non-empty attribute field0 of doc0 can be represented by "3--3". Then, by encoding "3--3", the first index value corresponding to "456" can be obtained.

[0153] The second method involves determining the first index value corresponding to each attribute data based on the second start position and the second end position of the current target attribute data in the first storage space.

[0154] For example, assuming the current target attribute data is "123", and the starting position of "123" in the first storage space is byte 0 and the ending position is byte 2, we can determine that the second starting position is 0 and the second ending position is 2. Then, the first index value corresponding to the attribute data "123" in the non-empty attribute field0 of doc0 can be represented by "0--2". Then, by encoding "0--2", we can obtain the first index value corresponding to "123".

[0155] For example, suppose the current target attribute data is "456". The starting position of "456" in the first storage space is byte 3 and the ending position is byte 5. We can determine that the second starting position is 3 and the second ending position is 5. Then, the first index value corresponding to the attribute data "456" in the non-empty attribute field0 of doc0 can be represented by "3--5". Then, by encoding "3--5", we can get the first index value corresponding to "456".

[0156] The third method involves determining the first index value corresponding to each attribute data based on the second end position and the second data length of the current target attribute data in the first storage space.

[0157] For example, assuming the current target attribute data is "123", the end position of "123" in the first storage space is byte 2, and the data length is 3 bytes, we can determine that the second end position is 2 and the second data length is 3. Then, the first index value corresponding to the attribute data "123" in the non-empty attribute field0 of doc0 can be represented by "2--3". Then, by encoding "2--3", we can obtain the first index value corresponding to "123".

[0158] For example, suppose the current target attribute data is "456". The end position of "456" in the first storage space is byte 5 and the data length is 3 bytes. We can determine that the second end position is 5 and the second data length is 3. Then, the first index value corresponding to the attribute data "123" in the non-empty attribute field0 of doc0 can be represented by "5--3". Then, by encoding "5--3", we can get the first index value corresponding to "123".

[0159] It is understandable that any of the first index value representations provided above can determine the storage location of the attribute data in the first storage space. For example, in the first method, a first index value is represented as "0--3", meaning that the value "123" of the attribute data to be searched is composed of the first byte 0 and the next 3 bytes. In the second method, a first index value is represented as "0--2", meaning that the value "123" of the attribute data to be searched is composed of the first byte 0 to the second byte. In the third method, a first index value is represented as "2--3", meaning that the value "123" of the attribute data to be searched is composed of the first byte 2 and the next 3 bytes.

[0160] Step S350: Store all first index values ​​consecutively in the second storage space.

[0161] It is understood that, in this embodiment of the invention, the first index value corresponding to each attribute data in each document is stored in a second storage space, which is a contiguous storage space in the server's memory. The first index values ​​described in this embodiment are stored contiguously in the second storage space. This contiguous storage means that the first index values ​​are arranged continuously and without interruption in the second storage space. That is, there is no bit interval between adjacent first index values ​​in the second storage space, nor is there a bit interval between the multiple characters included in a first index value.

[0162] For example, storing all first index values ​​consecutively in the second storage space can be achieved in the following way: sort all first index values ​​according to the order of document tags and attribute tags to obtain a sequence of first index values; according to the sequence of first index values, store each first index value in the second storage space in turn, and arrange all first index values ​​consecutively in the second storage space.

[0163] It is understandable that documents have document tags, attributes have attribute tags, and there is a specific order between document tags and attribute tags. In practice, all first index values ​​can be sorted firstly according to the ascending order of document tags and attribute tags to obtain a sequence of first index values. Then, based on the order of the first index values ​​in the sequence, each first index value is stored sequentially in the second storage space, ensuring that the first index values ​​are stored continuously and without interruption, thereby reducing the storage space occupied by the first index values ​​and avoiding memory fragmentation.

[0164] Please see Figure 4 , Figure 4 A schematic diagram of the storage structure of the first index value in the second storage space is shown. It should be noted that... Figure 4 Information such as "doc0 / field0 / 0" appearing in the text is only for assisting in understanding the scheme of the embodiments of the present invention. In actual implementation, the second storage space does not store this information. Figure 4 As shown, the first index values ​​corresponding to documents doc0, doc1, ... are arranged sequentially in the second storage space according to the document tags in ascending order. That is, the second storage space is divided into multiple intervals according to document tags, and each interval is used to store all the first index values ​​for a single document. In the interval corresponding to doc0, the multiple first index values ​​corresponding to doc0 are arranged sequentially in the second storage space according to the attribute tags in ascending order. Similarly, in the intervals corresponding to documents such as doc1, the first index values ​​are also arranged sequentially in the second storage space according to the attribute tags in ascending order.

[0165] It is understandable that each first index value is binary data formed by encoding the attribute data according to the storage location information in the first storage space.

[0166] In one possible implementation, the storage location information includes a second starting position and a second data length. In this case, each first index value can be represented as "second starting position -- second data length". See also... Figure 5a , Figure 5a A schematic diagram of the storage structure of the first index value in the second storage space is shown in a specific example of the present invention. It should be noted that... Figure 5a Information such as "doc0 / field0 / 0" appearing in the text is only for the purpose of assisting in understanding the scheme of the embodiments of the present invention. In actual implementation, the second storage space does not store this information. Figure 5a In the example, the first index value, obtained from the second starting position and the second data length of the target attribute data in the first storage space, is stored continuously in the second storage space.

[0167] Assuming an 8-byte storage space is allocated for each first index value, 7 bytes can be used to store the encoded information corresponding to the second starting position, and the remaining 1 byte can store the encoded information corresponding to the second data length. For example, field0 of doc0 has 3 attribute data, namely "123", "456" and "789". According to the representation of "second starting position -- second data length", the corresponding first index values ​​are "0--3", "3--3", and "6--3". The second starting position is encoded with 56 bits, occupying 7 bytes; the second data length is encoded with 8 bits, occupying 1 byte. Finally, the encoded information of the second starting position and the second data length are concatenated to obtain an 8-byte first index value. In specific implementation, the second starting position can be placed in the lower 7 bytes and the second data length can be placed in the higher 1 byte; alternatively, the second data length can be placed in the lower 1 byte and the second starting position can be placed in the higher 7 bytes. In addition, the overall byte length of the first index value, the byte length occupied by the second starting position, and the second data length can all be adjusted appropriately according to actual needs, and this embodiment of the invention does not impose any restrictions on this.

[0168] In another possible implementation, the storage location information includes a second start position and a second end position. In this case, each first index value can be represented as "second start position -- second end position". See also Figure 5b , Figure 5b A schematic diagram of the storage structure of the first index value in the second storage space is shown in a specific example of the present invention. It should be noted that... Figure 5bInformation such as "doc0 / field0 / 0" appearing in the text is only for the purpose of assisting in understanding the scheme of the embodiments of the present invention. In actual implementation, the second storage space does not store this information. Figure 5b In the example, the first index value, obtained from the second start position and the second end position of the target attribute data in the first storage space, is stored continuously in the second storage space.

[0169] Assuming an 8-byte storage space is allocated for each first index value, 4 bytes can be used to store the encoded information corresponding to the second start position, and the remaining 4 bytes can be used to store the encoded information corresponding to the second end position. For example, field0 of doc0 has 3 attribute data: "123", "456", and "789". According to the representation of "second start position -- second data length", the corresponding first index values ​​are "0--2", "3--5", and "6--8". The second start position is encoded with 32 bits, occupying 4 bytes; the second end position is encoded with 32 bits, occupying 4 bytes. Finally, the encoded information of the second start position and the second end position are concatenated to obtain an 8-byte first index value. In specific implementation, the second start position can be placed in the lower 4 bytes and the second end position in the higher 4 bytes; alternatively, the second end position can be placed in the lower 4 bytes and the second start position in the higher 4 bytes. Furthermore, the overall byte length of the first index value, and the byte lengths occupied by the second start position and the second end position can be adjusted appropriately according to actual needs, and the embodiments of the present invention do not impose any restrictions on this.

[0170] In another possible implementation, the storage location information includes a second end position and a second data length. In this case, each first index value can be represented as "second end position -- second data length". See also Figure 5c , Figure 5c This diagram illustrates the storage structure of the first index value in the second storage space in yet another specific example of the present invention. It should be noted that... Figure 5c Information such as "doc0 / field0 / 0" appearing in the text is only for the purpose of assisting in understanding the scheme of the embodiments of the present invention. In actual implementation, the second storage space does not store this information. Figure 5c In the example, the first index value, obtained from the second end position of the target attribute data in the first storage space and the second data length, is stored continuously in the second storage space.

[0171] Assuming an 8-byte storage space is allocated for each first index value, 7 bytes can be used to store the encoded information corresponding to the second end position, and the remaining 1 byte can store the encoded information corresponding to the second data length. For example, field0 of doc0 has 3 attribute data, namely "123", "456" and "789". According to the representation of "second end position -- second data length", the corresponding first index values ​​are "2--3", "5--3", and "8--3". The second end position is encoded with 56 bits, occupying 7 bytes; the second data length is encoded with 8 bits, occupying 1 byte. Finally, the encoded information of the second end position and the second data length are concatenated to obtain an 8-byte first index value. In specific implementation, the second end position can be placed in the lower 7 bytes and the second data length can be placed in the higher 1 byte; alternatively, the second data length can be placed in the lower 1 byte and the second end position can be placed in the higher 7 bytes. In addition, the overall byte length of the first index value, the byte length occupied by the second end position, and the second data length can all be adjusted appropriately according to actual needs, and this embodiment of the invention does not impose any restrictions on this.

[0172] Step S360: Based on the storage location of the first index value in the second storage space, determine the second index value corresponding to each non-empty attribute in each document. The second index value is used to index the first index value in the second storage space.

[0173] In this embodiment of the invention, a second index value is established for each non-empty attribute in each document. This second index value is used to index the first index value in the second storage space. Since the second index value is established for the non-empty attributes in the document, when indexing the attribute data of the document, the non-empty attributes with data in the document can be quickly located first through the second index value, and then the attribute data corresponding to the non-empty attribute can be obtained according to the first index value corresponding to the second index value, effectively improving the performance of data retrieval.

[0174] In one possible implementation, the second index value includes a third index value and a fourth index value. The third index value is used to indicate the document and non-empty attributes, and the fourth index value is used to indicate the storage location of the first index value in the second storage space. The mapping relationship between the document and non-empty attributes indicated by the third index value and the first index value in the second storage space can be determined through the fourth index value.

[0175] For example, determining the second index value corresponding to each non-empty attribute in each document based on the storage location of the first index value in the second storage space can specifically include the following steps: traversing each document in the document collection, for the currently traversed document, obtaining the third index value corresponding to each non-empty attribute in the document based on the document tag corresponding to the document and the attribute tag corresponding to each non-empty attribute in the document, the third index value being used to index the non-empty attributes of the document; determining the first index value corresponding to each third index value in the second storage space; and determining the fourth index value corresponding to each third index value based on the storage location of the first index value corresponding to each third index value in the second storage space.

[0176] In this embodiment of the invention, it is necessary to first establish a corresponding third index value for each non-empty attribute in each document. This third index value is determined by "document tag + attribute tag".

[0177] Taking the non-empty attribute field0 in document doc0 as an example, its corresponding third index value is composed of the document tag of doc0 and the attribute tag of field0, which can be represented as "document tag + attribute tag". Assuming that the document tag of doc0 is 0 and the attribute tag of field0 is 0, then the corresponding third index value is "0--0".

[0178] With the third index value set to "0--0", the storage location of the first index value corresponding to doc0 and field0 is found in the second storage space. Assuming each first index value has a length of 8 bytes, the storage location information of the first index value corresponding to doc0 and field0 can be obtained as [0, 3]. Here, [0, 3] indicates that starting from the 0th 8-byte, three consecutive 8-byte bytes in the second storage space belong to the first index value corresponding to doc0 and field0. That is, the 0th to 2nd first index values ​​in the second storage space are the first index values ​​corresponding to doc0 and field0. Based on the storage location information [0, 3], the fourth index value corresponding to doc0 and field0 can be represented as "0--3".

[0179] It is understandable that the second index value corresponding to the non-empty attribute field0 in doc0 is obtained based on the corresponding third index value "0--0" and fourth index value "0--3".

[0180] For example, determining the first index value corresponding to the third index value in the second storage space may specifically include the following steps: determining the document and non-empty attributes corresponding to the first index value based on the attribute data corresponding to each first index value in the second storage space; and determining the first index value corresponding to the third index value based on the document and non-empty attributes corresponding to the first index value.

[0181] It should be understood that all first index values ​​in the second storage space correspond one-to-one with all attribute data included in the document collection. That is, each first index value in the second storage space has a corresponding attribute data. Based on the document to which the corresponding attribute data belongs and the non-empty attribute, the mapping relationship between the first index value and the third index value can be determined, and then the fourth index value corresponding to the third index value can be determined based on the mapping relationship.

[0182] For example, determining the fourth index value corresponding to the third index value based on the storage location of the first index value corresponding to the third index value in the second storage space can include the following three implementation methods:

[0183] The first method involves determining the fourth index value corresponding to the third index value based on the first starting position of the first index value in the second storage space and the first data length. In other words, the fourth index value can be represented as "first starting position + first data length".

[0184] For example, the current third index value indicates the non-empty attribute field0 in document doc0. There are 3 first index values ​​corresponding to the current third index value in the second storage space. Starting from the 0th 8-byte in the second storage space, the next 3 consecutive 8-byte bytes belong to the first index values ​​corresponding to doc0 and field0. Thus, the first starting position is 0, the first data length is 3, and the fourth index value corresponding to the current third index value can be represented by "0--3".

[0185] For example, the current third index value indicates the non-empty attribute field0 in document doc1. The current third index value corresponds to one first index value in the second storage space, which is in the 8th 8-byte of the second storage space. Thus, the first starting position is 8 and the first data length is 1. The fourth index value corresponding to the current third index value can be represented as "8--1".

[0186] The second method involves determining the fourth index value corresponding to the third index value based on the first start position and the first end position of the first index value in the second storage space. In other words, the fourth index value can be represented as "first start position + first end position".

[0187] For example, the current third index value indicates the non-empty attribute field0 in document doc0. The current third index value corresponds to three first index values ​​in the second storage space, starting from the 0th 8-byte and ending at the 2nd 8-byte. Thus, the first starting position is 0 and the first ending position is 2. The fourth index value corresponding to the current third index value can be represented by "0--2".

[0188] For example, the current third index value indicates the non-empty attribute field0 in document doc1. The current third index value corresponds to one first index value in the second storage space, which is in the 8th 8-byte of the second storage space. Therefore, the first start position is 8 and the first end position is 8. The fourth index value corresponding to the current third index value can be represented as "8--8".

[0189] The third method involves determining the fourth index value based on the first end position and the first data length of the first index value in the second storage space. In other words, the fourth index value can be represented as "first end position + first data length".

[0190] For example, the current third index value indicates the non-empty attribute field0 in document doc0. The current third index value corresponds to three first index values ​​in the second storage space. Starting from the 0th 8-byte in the second storage space and ending at the 2nd 8-byte, there are three consecutive 8-byte sequences. Therefore, the first end position is 2 and the first data length is 3. The fourth index value corresponding to the current third index value can be represented as "2--3".

[0191] For example, the current third index value indicates the non-empty attribute field0 in document doc1. The current third index value corresponds to one first index value in the second storage space, which is in the 8th 8-byte of the second storage space. Therefore, the first start position is 8, the first end position is 8, and the data length is 1 byte. The fourth index value corresponding to the current third index value can be represented by "8--1".

[0192] It is understandable that any of the fourth index value representations provided above can yield the storage location of the first index value corresponding to the third index value in the second storage space. For example, in the first method, a first index value is represented as "0--3", meaning that three consecutive bytes starting from byte 0 in the second storage space are the first index values ​​corresponding to the current third index value; in the second method, a first index value is represented as "0--2", meaning that bytes 0 to byte 2 in the second storage space are the first index values ​​corresponding to the current third index value; and in the third method, a first index value is represented as "2--3", meaning that the three bytes preceding byte 2 in the second storage space are the first index values ​​corresponding to the current third index value.

[0193] It is also understandable that, since the first index values ​​are arranged in the second storage space according to the order of document tags and attribute tags, the first index values ​​corresponding to the same non-empty attribute are arranged consecutively in the second storage space. Therefore, when describing the fourth index value, any two of the first start position, the first end position, and the first data length can be used, which is sufficient to describe all the first index values ​​corresponding to a certain non-empty attribute in the second storage space. This reduces the data length of the fourth index value and saves storage overhead.

[0194] It should be noted that the third index value described in the embodiments of the present invention is optional. That is, the second index value may only include the fourth index value. In specific implementation, the mapping relationship between non-empty attributes and the fourth index value can be established by preset rules. For example, a pointer is preset to point to the fourth index value corresponding to each non-empty attribute in the document, and the storage location of the fourth index value corresponding to the non-empty attribute is calculated by the pointer.

[0195] Step S370: Store all second index values ​​in the third storage space.

[0196] It is understood that, in this embodiment of the invention, the second index value corresponding to each non-empty attribute in each document is stored in the third storage space. When the second index value includes both the third and fourth index values, the third storage space may include a fourth and a fifth storage space, where the fourth and fifth storage spaces are contiguous storage spaces within the server's memory.

[0197] In one possible implementation, storing all second index values ​​in a third storage space can specifically include: storing all third index values ​​contiguously in a fourth storage space, and storing all fourth index values ​​contiguously in a fifth storage space, wherein the order in which the third index values ​​are stored in the fourth storage space is consistent with the order in which the fourth index values ​​are stored in the fifth storage space. That is, the third and fourth index values ​​are stored separately, and there is a one-to-one correspondence between the third index values ​​in the fourth storage space and the fourth index values ​​in the fifth storage space; third index values ​​and fourth index values ​​at the same storage location are mutually corresponding. For example, the third index value stored in the 0th 8-byte of the fourth storage space and the fourth index value stored in the 0th 8-byte of the fifth storage space are mutually corresponding.

[0198] For example, storing all third index values ​​consecutively in the fourth storage space can be achieved through the following steps: sorting all third index values ​​according to the order of document tags and attribute tags to obtain a sequence of third index values; and storing each third index value sequentially in the fourth storage space according to the sequence of third index values, with all third index values ​​arranged consecutively in the fourth storage space.

[0199] It is understandable that the third index value includes document tags and attribute tags, and there is a sequential order between the document tags and the attribute tags. In specific implementation, all third index values ​​can be sorted in ascending order of document tags and ascending order of attribute tags to obtain a sequence of third index values. Then, based on the order of each third index value in the sequence, each third index value is stored sequentially in the fourth storage space, ensuring that the third index values ​​are arranged continuously and without interruption in the fourth storage space, thereby reducing the storage space occupied by the third index values ​​and avoiding memory fragmentation.

[0200] Please see Figure 6 , Figure 6 A schematic diagram of the storage structure of the third index value in the fourth storage space is shown. (For example...) Figure 6 As shown, the fourth storage space is divided into multiple intervals. Each interval stores the third index value of a document's non-empty attributes. The third index values ​​in the fourth storage space are arranged in ascending order of document tags and attribute tags.

[0201] It is understandable that each third index value is binary data encoded based on the corresponding document tags and attribute tags. In other words, the third index value is stored in binary form in the fourth storage space. For example, if the document corresponding to the current third index value is doc1 and the corresponding non-empty attribute is field2, then the third index value can be represented as "1--2", and "1--2" is stored in the fourth storage space after being encoded in binary.

[0202] In specific implementation, the document tag and attribute tag in the third index value are encoded separately, and each third index value can be represented as "document tag -- attribute tag". Assuming an 8-byte storage space is allocated for each third index value, 4 bytes can be used to store the encoded information corresponding to the document tag, and the remaining 4 bytes can be used to store the encoded information corresponding to the attribute tag. For example, for field2 of doc1, its third index value can be represented as "1--2". 32 bits are used to encode the document tag "1" in the third index value, occupying 4 bytes; 32 bits are used to encode the attribute tag "2" in the third index value, occupying 4 bytes. Finally, the encoded information of the document tag and attribute tag are concatenated to obtain an 8-byte third index value. In specific implementation, the document tag can be placed in the lower 4 bytes and the attribute tag in the higher 4 bytes; alternatively, the document tag can be placed in the higher 4 bytes and the attribute tag in the lower 4 bytes. Furthermore, the overall byte length of the third index value, and the byte lengths occupied by the document tag and attribute tag respectively, can be appropriately adjusted according to actual needs, and this embodiment of the invention does not impose any limitations on this.

[0203] For example, storing all fourth index values ​​consecutively in the fifth storage space can be achieved through the following steps: sorting the fourth index values ​​according to the third index value sequence to obtain a fourth index value sequence; storing each fourth index value sequentially in the fifth storage space according to the fourth index value sequence, with all fourth index values ​​arranged consecutively in the fifth storage space.

[0204] It is understandable that there is a one-to-one correspondence between all fourth index values ​​and all third index values. Therefore, the order of the fourth index values ​​in the fifth storage space can be made consistent with the order of the third index values ​​in the fourth storage space, so as to facilitate the search for the fourth index value corresponding to the third index value in the fifth storage space. In specific implementation, the fourth index values ​​can be sorted according to the third index value sequence obtained in the previous steps to obtain a fourth index value sequence. The order of the fourth index values ​​in the fourth index value sequence is consistent with the order of the third index values ​​in the third index value sequence, both following the order of document tags from smallest to largest and attribute tags from smallest to largest. Then, based on the order of each fourth index value in the fourth index value sequence, each fourth index value is stored in the fifth storage space in sequence, so that the fourth index values ​​are arranged continuously and without interruption in the fifth storage space, in order to reduce the storage space occupied by the fourth index values ​​and avoid memory fragmentation.

[0205] Please see Figure 7 , Figure 7 A schematic diagram of the storage structure of the fourth index value in the fifth storage space is shown. It should be noted that... Figure 7Information such as "doc0 / field0" appearing in the text is only for the purpose of assisting in understanding the scheme of the embodiments of the present invention. In actual implementation, the fifth storage space does not store this information. Figure 7 As shown, the fourth index values ​​corresponding to each third index value are arranged sequentially in the fifth storage space. Figure 6 The order in which the third index value is stored in the fourth storage space is the same as... Figure 7 The fourth index value is stored in the same order as the fifth storage space.

[0206] It is understandable that the fourth index value corresponding to each third index value is binary data formed by encoding the storage location information of the first index value corresponding to the third index value in the second storage space. In other words, the fourth index value is determined based on the storage location information of the first index value.

[0207] In one possible implementation, the storage location information includes a first starting position and a first data length. In this case, each fourth index value can be represented as "first starting position -- first data length". See also Figure 8a , Figure 8a A schematic diagram of the storage structure of the fourth index value in the fifth storage space is shown in a specific example. Figure 8a In the example, the fourth index value, obtained by using the first index value corresponding to the third index value at the first starting position in the second storage space and the first data length, is stored continuously in the fifth storage space.

[0208] Assuming an 8-byte storage space is allocated for each fourth index value, 7 bytes can be used to store the encoding information corresponding to the first starting position, and the remaining 1 byte can store the encoding information corresponding to the first data length. For example, if the current third index value indicates the non-empty attribute field0 in document doc0, and there are 3 corresponding first index values ​​in the second storage space, starting from the 0th 8-byte and ending at the 2nd 8-byte, then the first starting position is 0, the first data length is 3, and the fourth index value corresponding to the current third index value can be represented as "0-3". 56 bits are used to encode the first starting position "0" in the fourth index value, occupying 7 bytes; 8 bits are used to encode the first data length "3" in the fourth index value, occupying 1 byte. Finally, the encoding information of the first starting position and the first data length are concatenated to obtain an 8-byte fourth index value. In practice, the first starting position can be placed in the lower 7 bytes and the first data length in the higher 1 byte; alternatively, the first starting position can be placed in the higher 7 bytes and the first data length in the lower 1 byte. In addition, the overall byte length of the fourth index value, the byte length occupied by the first starting position and the first data length can all be adjusted appropriately according to actual needs, and the embodiments of the present invention do not impose any restrictions on this.

[0209] In another possible implementation, the storage location information includes a first start position and a first end position. In this case, each fourth index value can be represented as "first start position -- first end position". See also Figure 8b , Figure 8b A schematic diagram of the storage structure of the fourth index value in the fifth storage space is shown in a specific example. Figure 8b In the example, the fourth index value, obtained by using the first index value corresponding to the third index value at the first start position and the first end position in the second storage space, is stored consecutively in the fifth storage space.

[0210] Assuming an 8-byte storage space is allocated for each first index value, 4 bytes can be used to store the encoding information corresponding to the first start position, and the remaining 4 bytes can be used to store the encoding information corresponding to the first end position. For example, if the current third index value indicates the non-empty attribute field0 in document doc0, there are 3 corresponding first index values ​​in the second storage space. Starting from the 0th 8-byte and ending at the 2nd 8-byte, the first start position is 0 and the first end position is 2. The fourth index value corresponding to the current third index value can be represented by "0-2". 32 bits are used to encode the first start position "0" in the fourth index value, occupying 4 bytes; 32 bits are used to encode the first end position "2" in the fourth index value, occupying 4 bytes; finally, the encoding information of the first start position and the first end position are concatenated to obtain an 8-byte fourth index value. In practice, the first start position can be placed in the lower 4 bytes and the first end position in the higher 4 bytes; alternatively, the first start position can be placed in the higher 4 bytes and the first end position in the lower 4 bytes. In addition, the overall byte length of the fourth index value, and the byte lengths occupied by the first start position and the first end position can be adjusted appropriately according to actual needs, and the embodiments of the present invention do not impose any restrictions on this.

[0211] In another possible implementation, the storage location information includes a first end position and a first data length. In this case, each fourth index value can be represented as "first end position -- first data length". See also Figure 8c , Figure 8c A schematic diagram of the storage structure of the fourth index value in the fifth storage space is shown in a specific example. Figure 8c In the example, the fourth index value, obtained by combining the first index value corresponding to the third index value at the first end position of the second storage space and the first data length, is stored consecutively in the fifth storage space.

[0212] Assuming an 8-byte storage space is allocated for each fourth index value, 7 bytes can be used to store the encoding information corresponding to the first end position, and the remaining 1 byte can store the encoding information corresponding to the first data length. For example, if the current third index value indicates the non-empty attribute field0 in document doc0, and there are 3 corresponding first index values ​​in the second storage space, starting from the 0th 8-byte and ending at the 2nd 8-byte, then the first end position is 2 and the first data length is 3. The fourth index value corresponding to the current third index value can be represented as "2--3". 56 bits are used to encode the first end position "2" in the fourth index value, occupying 7 bytes; 8 bits are used to encode the first data length "3" in the fourth index value, occupying 1 byte. Finally, the encoding information of the first end position and the first data length are concatenated to obtain an 8-byte fourth index value. In practice, the first end position can be placed in the lower 7 bytes and the first data length in the higher 1 byte; alternatively, the first end position can be placed in the higher 7 bytes and the first data length in the lower 1 byte. In addition, the overall byte length of the fourth index value, the byte length occupied by the first starting position and the first data length can all be adjusted appropriately according to actual needs, and the embodiments of the present invention do not impose any restrictions on this.

[0213] In another possible implementation, all second index values ​​are stored in a third storage space. Specifically, this can include concatenating the third and fourth index values ​​to obtain a second index value, and then storing this second index value in the third storage space. That is, a set of corresponding third and fourth index values ​​are concatenated to obtain a second index value, which is then stored in the third storage space. Multiple second index values ​​are arranged consecutively in the third storage space. Here, the third storage space is a contiguous storage space in the server's memory.

[0214] The data storage method provided in the embodiments of the present invention will be described below through specific application examples.

[0215] Suppose a document collection contains 10,000,000 documents (doc), each document has 200 fields (i.e., attributes), each document has an average of 30 fields with data, each field with data has an average of 3 data items (i.e., attribute data), each data item is a 10-byte string, and the repetition rate of these string values ​​is 80%. The storage space required to store the data from the 200 fields of these 10,000,000 documents can be calculated as follows:

[0216] The total storage cost = storage cost of field data + storage cost of the first index value + storage cost of the third index value + storage cost of the fourth index value = 30*3*10*0.2*10000000 + 30*3*8*10000000 + 30*8*10000000 + 30*8*10000000 = 1.8G + 7.2G + 2.4G + 2.4G = 13.8G.

[0217] Compared to the traditional method of using non-deduplicated storage, which allocates an 8-byte placeholder for each document, this method uses std::vector. <std::string>If mounted in this way, the required memory overhead is as follows:

[0218] cost = 10,000,000 * 200 * 8 (placeholder pointer storage overhead) + 10,000,000 * 30 * 24 (std::vector shell memory overhead) + 10,000,000 * 30 * 3 * 32 (std::string shell memory overhead) + 10,000,000 * 30 * 3 * 10 (actual string content overhead) + 10,000,000 * 30 * 3 * std::string memory doubled at the tail (averaging 4 bytes) = 16G + 7.2G + 3.2G + 9G + 900M * 4 = 35.4G + 3.6G = 39G

[0219] By comparison, it can be found that, under the same conditions, the memory overhead for storing 10,000,000 documents is 13.8 / 39 = 35.3% of that of the traditional method, saving 64.7% of memory. Furthermore, the memory in the embodiment of the present invention is compactly distributed, without adding any memory fragmentation to the system, which brings great advantages to both computing and storage.

[0220] The following examples illustrate the application scenarios of the embodiments of the present invention.

[0221] Scene 1:

[0222] The data storage method provided in this embodiment of the invention can be applied to navigation scenarios. Specifically, the navigation system generates multiple route documents based on collected road network data. The data storage method provided in this embodiment of the invention deduplicates the attribute data included in all route documents and stores them consecutively in a first storage space. Then, a first index value is determined for each attribute data in each document, and all first index values ​​are consecutively stored in a second storage space. A second index value is also determined for each non-empty attribute in each document, and all second index values ​​are consecutively stored in a third storage space. Here, the attributes of the route document may include estimated travel time, travel distance, toll, number of traffic lights along the route, etc. After receiving a route retrieval request from a smartphone or in-vehicle terminal, the server retrieves the relevant route documents based on the departure and destination locations input by the driver in the route retrieval request. During the recall process, the first index value corresponding to the non-empty attribute in each route document can be obtained through the second index value. Then, the attribute data included in each non-empty attribute can be obtained using the first index value, thereby obtaining the attribute data corresponding to each route document. Next, based on the attribute data, departure location, and destination location of each route document, a set of candidate route documents is determined. Then, the set of candidate route documents is sorted by a preset recommendation algorithm to determine the target route document to be recalled.

[0223] Scene 2:

[0224] The data storage method provided in this embodiment of the invention can be applied to video recommendation scenarios. Specifically, the navigation system generates multiple video documents based on preset video content. The data storage method provided in this embodiment of the invention deduplicates the attribute data included in all video documents and stores them consecutively in a first storage space. Then, a first index value is determined for each attribute data in each document, and all first index values ​​are consecutively stored in a second storage space. A second index value is also determined for each non-empty attribute in each document, and all second index values ​​are consecutively stored in a third storage space. Here, the attributes of the video document may include the video title, video duration, content type, video creator, etc. After receiving a video retrieval request from the user device, the server recalls relevant video documents based on the object attribute information in the video retrieval request to recommend video content that the user may be interested in. During the recall process, the first index value corresponding to the non-empty attribute in each video document can be obtained through the second index value. Then, the attribute data included in each non-empty attribute can be obtained using the first index value, thereby obtaining the attribute data corresponding to each video document. Next, a set of candidate video documents is determined based on the attribute data and object attribute information of each video document. Then, the set of candidate video documents is sorted by a preset recommendation algorithm to determine the target video document to be recalled.

[0225] Please see Figure 9 This invention also provides a data storage device, comprising:

[0226] The acquisition module 801 is used to acquire a document collection, which includes multiple documents. Each document includes multiple attributes, at least one of which is a non-empty attribute, and the non-empty attribute includes at least one attribute data.

[0227] The data deduplication module 802 is used to deduplicat all attribute data in the document collection to obtain multiple target attribute data.

[0228] The first storage module 803 is used to continuously store all target attribute data in the first storage space;

[0229] The first index module 804 is used to determine the first index value corresponding to each attribute data according to the storage location of the target attribute data corresponding to each attribute data in the first storage space. The first index value is used to index the attribute data in the first storage space.

[0230] The second storage module 805 is used to continuously store all the first index values ​​in the second storage space;

[0231] The second index module 806 is used to determine the second index value corresponding to each non-empty attribute in each document according to the storage location of the first index value in the second storage space. The second index value is used to index the first index value in the second storage space.

[0232] The third storage module 807 is used to store all the second index values ​​in the third storage space.

[0233] In one embodiment, the second index value includes a third index value and a fourth index value; the second index module 806 specifically includes:

[0234] The first determining unit is used to traverse each document in the document collection. For the currently traversed document, it obtains the third index value corresponding to each non-empty attribute in the document based on the document tag corresponding to the document and the attribute tag corresponding to each non-empty attribute in the document. The third index value is used to index the non-empty attributes of the document.

[0235] The second determining unit is used to determine the first index value corresponding to each third index value in the second storage space;

[0236] The third determining unit is used to determine the fourth index value corresponding to each third index value in the second storage space according to the storage location of the first index value corresponding to each third index value.

[0237] In one embodiment, the data storage device further includes a tag determination module, which is used to:

[0238] Set document tags for each document;

[0239] Get the attribute types of the attributes included in each document;

[0240] Set attribute tags for each attribute based on its attribute type, so that attributes with the same attribute type have the same attribute tags.

[0241] In one embodiment, the second determining unit is specifically used for:

[0242] Based on the attribute data corresponding to each first index value in the second storage space, determine the document and non-empty attributes corresponding to the first index value;

[0243] Based on the document and non-empty attributes corresponding to the first index value, determine the first index value corresponding to the third index value.

[0244] In one embodiment, the third determining unit is specifically used for:

[0245] The fourth index value corresponding to the third index value is determined based on the first starting position of the first index value in the second storage space and the first data length.

[0246] or,

[0247] Based on the first start position and the first end position of the first index value corresponding to the third index value in the second storage space, determine the fourth index value corresponding to the third index value;

[0248] or,

[0249] Based on the first end position of the first index value corresponding to the third index value in the second storage space and the first data length, determine the fourth index value corresponding to the third index value.

[0250] In one embodiment, the third storage space includes a fourth storage space and a fifth storage space; the third storage module 807 specifically includes:

[0251] The first storage unit is used to store all the third index values ​​consecutively in the fourth storage space;

[0252] The second storage unit is used to store all the fourth index values ​​consecutively in the fifth storage space;

[0253] The order in which the third index value is stored in the fourth storage space is consistent with the order in which the fourth index value is stored in the fifth storage space.

[0254] In one embodiment, the third index value includes document tags and attribute tags, and there is a sequential order between the document tags and between the attribute tags;

[0255] The first storage unit is specifically used for:

[0256] Sort all third index values ​​according to the order of document tags and attribute tags to obtain a sequence of third index values;

[0257] Based on the sequence of third index values, each third index value is stored in the fourth storage space in turn, and all third index values ​​are arranged consecutively in the fourth storage space.

[0258] In one embodiment, the second storage unit is specifically used for:

[0259] Sort the fourth index value according to the third index value sequence to obtain the fourth index value sequence;

[0260] Based on the sequence of fourth index values, each fourth index value is stored sequentially in the fifth storage space, and all fourth index values ​​are arranged consecutively in the fifth storage space.

[0261] In one embodiment, a document has a document tag, attributes have attribute tags, and there is a sequence between the document tags and between the attribute tags;

[0262] The second storage module 805 is specifically used for:

[0263] Sort all first index values ​​according to the order of document tags and attribute tags to obtain the first index value sequence;

[0264] Based on the sequence of first index values, each first index value is stored sequentially in the second storage space, and all first index values ​​are arranged consecutively in the second storage space.

[0265] In one embodiment, the first index module 804 specifically includes:

[0266] The fourth determining unit is used to traverse each attribute data in the document collection, search for the target attribute data that matches the currently traversed attribute data in the first storage space, and obtain the current target attribute data.

[0267] The fifth determining unit is used to determine the first index value corresponding to the attribute data being traversed, based on the storage location of the current target attribute data in the first storage space.

[0268] In one embodiment, the fifth determining unit is specifically used for:

[0269] Based on the second starting position and second data length of the current target attribute data in the first storage space, determine the first index value corresponding to each attribute data;

[0270] or,

[0271] Based on the second start position and the second end position of the current target attribute data in the first storage space, determine the first index value corresponding to each attribute data;

[0272] or,

[0273] Based on the second end position and the second data length of the current target attribute data in the first storage space, determine the first index value corresponding to each attribute data.

[0274] It should be noted that since the data storage device of this embodiment can implement the data storage method of the previous embodiment, the data storage device of this embodiment has the same technical principle and the same beneficial effect as the data storage method of the previous embodiment. To avoid repetition, it will not be described again here.

[0275] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0276] Please see Figure 10 The present invention also discloses a data storage device, the data storage device 900 comprising:

[0277] At least one processor 901;

[0278] At least one memory 902 is used to store at least one program;

[0279] When at least one program is executed by at least one processor 901, the data storage method as described in any of the preceding embodiments is implemented.

[0280] This invention also discloses a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, implements the data storage method as described in any of the preceding embodiments.

[0281] This invention also discloses a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, causing the computer device to perform the data storage method as described in any of the preceding embodiments.

[0282] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0283] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than" refers to two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0284] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, apparatuses, or units, and may be electrical, mechanical, or other forms.

[0285] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0286] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0287] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0288] The step numbers in the above method embodiments are set only for ease of explanation and do not limit the order between steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.< / std::string>

Claims

1. A data storage method, characterized in that, Includes the following steps: Obtain a document collection, the document collection including multiple documents, each document including multiple attributes, at least one of the multiple attributes being a non-empty attribute, the non-empty attribute including at least one attribute data; Deduplication is performed on all attribute data in the document collection to obtain multiple target attribute data; All the target attribute data are stored contiguously in the first storage space; Based on the storage location of the target attribute data corresponding to each attribute data in the first storage space, a first index value corresponding to each attribute data is determined, and the first index value is used to index the attribute data in the first storage space; All the first index values ​​are stored consecutively in the second storage space; Based on the storage location of the first index value in the second storage space, a second index value corresponding to each non-empty attribute in each document is determined, and the second index value is used to index the first index value in the second storage space; All the second index values ​​are stored in the third storage space.

2. The data storage method according to claim 1, characterized in that, The second index value includes a third index value and a fourth index value; determining the second index value corresponding to each non-empty attribute in each document based on the storage location of the first index value in the second storage space includes: Traverse each document in the document collection. For the currently traversed document, obtain the third index value corresponding to each non-empty attribute in the document based on the document tag corresponding to the document and the attribute tag corresponding to each non-empty attribute in the document. The third index value is used to index the non-empty attributes of the document. In the second storage space, determine the first index value corresponding to each of the third index values; The fourth index value corresponding to each of the third index values ​​is determined based on the storage location of the first index value corresponding to each of the third index values ​​in the second storage space.

3. The data storage method according to claim 2, characterized in that, After obtaining the document collection, the method further includes: Set the document tags for each of the aforementioned documents; Obtain the attribute type of each of the attributes included in each of the documents; The attribute label is set for each attribute according to the attribute type, so that the attribute labels corresponding to the attributes with the same attribute type are the same.

4. The data storage method according to claim 2, characterized in that, Determining the first index value corresponding to the third index value in the second storage space includes: Based on the attribute data corresponding to each of the first index values ​​in the second storage space, determine the document and the non-empty attribute corresponding to the first index value; The first index value corresponding to the third index value is determined based on the document corresponding to the first index value and the non-empty attribute.

5. The data storage method according to claim 2, characterized in that, Determining the fourth index value corresponding to the third index value based on the storage location of the first index value corresponding to the third index value in the second storage space includes: The fourth index value corresponding to the third index value is determined based on the first starting position and the first data length of the first index value corresponding to the third index value in the second storage space; or, The fourth index value corresponding to the third index value is determined based on the first start position and the first end position of the first index value corresponding to the third index value in the second storage space; or, The fourth index value corresponding to the third index value is determined based on the first end position of the first index value corresponding to the third index value in the second storage space and the first data length.

6. The data storage method according to claim 2, characterized in that, The third storage space includes a fourth storage space and a fifth storage space; The step of storing all the second index values ​​in the third storage space includes: All the third index values ​​are stored consecutively in the fourth storage space; All the fourth index values ​​are stored consecutively in the fifth storage space; The order in which the third index value is stored in the fourth storage space is consistent with the order in which the fourth index value is stored in the fifth storage space.

7. The data storage method according to claim 6, characterized in that, The third index value includes the document tags and the attribute tags, and there is a sequential order between the document tags and between the attribute tags; The step of storing all the third index values ​​contiguously in the fourth storage space includes: Sort all the third index values ​​according to the order of the document tags and the order of the attribute tags to obtain a sequence of third index values; According to the third index value sequence, each of the third index values ​​is stored in the fourth storage space in sequence, and all the third index values ​​are arranged consecutively in the fourth storage space.

8. The data storage method according to claim 7, characterized in that, The step of storing all the fourth index values ​​contiguously in the fifth storage space includes: The fourth index value is sorted according to the third index value sequence to obtain the fourth index value sequence; According to the fourth index value sequence, each of the fourth index values ​​is stored in the fifth storage space in sequence, and all the fourth index values ​​are arranged consecutively in the fifth storage space.

9. The data storage method according to claim 1, characterized in that, The document has document tags, and the attributes have attribute tags. There is a sequence between the document tags and between the attribute tags. The step of storing all the first index values ​​consecutively in the second storage space includes: Sort all the first index values ​​according to the order of the document tags and the order of the attribute tags to obtain the first index value sequence; Based on the first index value sequence, each of the first index values ​​is stored sequentially in the second storage space, and all the first index values ​​are arranged consecutively in the second storage space.

10. The data storage method according to claim 1, characterized in that, The step of determining the first index value corresponding to each attribute data according to the storage location of the target attribute data corresponding to each attribute data in the first storage space includes: Traverse each attribute data of each document in the document collection, and search in the first storage space for the target attribute data that matches the currently traversed attribute data to obtain the current target attribute data; Based on the storage location of the current target attribute data in the first storage space, determine the first index value corresponding to the attribute data currently being traversed.

11. The data storage method according to claim 10, characterized in that, Determining the first index value corresponding to the currently traversed attribute data based on the storage location of the current target attribute data in the first storage space includes: Based on the second starting position and the second data length of the current target attribute data in the first storage space, determine the first index value corresponding to each attribute data; or, Based on the second start position and the second end position of the current target attribute data in the first storage space, determine the first index value corresponding to each attribute data; or, Based on the second end position and the second data length of the current target attribute data in the first storage space, the first index value corresponding to each attribute data is determined.

12. A data storage device, characterized in that, include: The acquisition module is used to acquire a document collection, the document collection including multiple documents, each document including multiple attributes, at least one of the multiple attributes being a non-empty attribute, the non-empty attribute including at least one attribute data; The data deduplication module is used to deduplicat all attribute data in the document collection to obtain multiple target attribute data. The first storage module is used to continuously store all the target attribute data in the first storage space; The first indexing module is used to determine a first index value corresponding to each attribute data according to the storage location of the target attribute data corresponding to each attribute data in the first storage space. The first index value is used to index the attribute data in the first storage space. The second storage module is used to continuously store all the first index values ​​in the second storage space; The second indexing module is used to determine the second index value corresponding to each non-empty attribute in each document according to the storage location of the first index value in the second storage space. The second index value is used to index the first index value in the second storage space. The third storage module is used to store all the second index values ​​in the third storage space.

13. A data storage device, characterized in that, include: At least one processor; At least one memory for storing at least one program; The data storage method as described in any one of claims 1 to 11 is implemented when at least one of the programs is executed by at least one of the processors.

14. A computer-readable storage medium, characterized in that, It stores a processor-executable program, which, when executed by a processor, is used to implement the data storage method as described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program or computer instructions, characterized in that, The computer program or the computer instructions are stored in a computer-readable storage medium, the processor of the computer device reads the computer program or the computer instructions from the computer-readable storage medium, and the processor executes the computer program or the computer instructions, causing the computer device to perform the data storage method as described in any one of claims 1 to 11.

Images