Feature data storage method and target object analysis method based on feature data

Abstract

The application relates to the technical fields of smart cities and smart travel, and provides a feature data storage method and a target object analysis method based on feature data. The feature data storage method comprises the following steps: for a target data processing scene, determining feature categories to which feature data required for data processing of a target object belongs; respectively determining respective data structure conditions of each feature category under the target data processing scene; determining respective memory container types matched with each data structure condition, so as to obtain a corresponding relationship between the feature categories and the memory container types; and based on the corresponding relationship, storing feature data of the target object into a target memory container of a corresponding memory container type according to the feature category to which the feature data belongs. In this way, when data processing of the target object is performed, the feature data of the target object can be directly read based on the memory container, online data analysis is avoided, and the feature data acquisition efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, storage medium, and computer program product for storing feature data, as well as a method, apparatus, computer device, storage medium, and computer program product for analyzing target objects based on feature data. Background Technology

[0002] With the development of computer technology, data storage has become a crucial part of data analysis. For example, WeChat's search function requires a large amount of feature data to score articles, and this feature data generally needs to be stored in advance. Common storage formats for feature data include JSON (JavaScript Object Notation, a lightweight data exchange format), Protocol Buffers (a data description language), and raw data storage, which is then parsed online to obtain the feature data used for data analysis.

[0003] However, currently used feature data storage methods require the raw data to be parsed into a specific format that facilitates retrieval and querying in practical data analysis applications. This data parsing process consumes significant data processing resources, resulting in low efficiency in feature data acquisition. Summary of the Invention

[0004] Therefore, it is necessary to provide a feature data storage method, apparatus, computer equipment, storage medium, and computer program product that can improve the efficiency of feature data acquisition, as well as a target object analysis method, apparatus, computer equipment, storage medium, and computer program product based on feature data, to address the above-mentioned technical problems.

[0005] A method for storing feature data, the method comprising:

[0006] For the target data processing scenario, determine the feature category to which the feature data required for data processing of the target object belongs;

[0007] Determine the data structure conditions corresponding to each of the aforementioned feature categories in the target data processing scenario;

[0008] Determine the memory container type that matches each of the data structure conditions to obtain the correspondence between the feature categories and memory container types;

[0009] Based on the correspondence, the feature data of the target object are stored in the target memory container of the corresponding memory container type according to the feature category to which it belongs.

[0010] A feature data storage device, the device comprising:

[0011] The category determination module is used to determine the feature category to which the feature data required for data processing of the target object belongs, based on the target data processing scenario.

[0012] The structural condition determination module is used to determine the data structure conditions corresponding to each of the aforementioned feature categories in the target data processing scenario.

[0013] The container type determination module is used to determine the memory container type that matches each of the data structure conditions, and to obtain the correspondence between the feature category and the memory container type;

[0014] The data storage module is used to store the feature data of the target object into the target memory container of the corresponding memory container type according to the corresponding feature category, based on the correspondence.

[0015] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0016] For the target data processing scenario, determine the feature category to which the feature data required for data processing of the target object belongs;

[0017] Determine the data structure conditions corresponding to each of the aforementioned feature categories in the target data processing scenario;

[0018] Determine the memory container type that matches each of the data structure conditions to obtain the correspondence between the feature categories and memory container types;

[0019] Based on the correspondence, the feature data of the target object are stored in the target memory container of the corresponding memory container type according to the feature category to which it belongs.

[0020] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0021] For the target data processing scenario, determine the feature category to which the feature data required for data processing of the target object belongs;

[0022] Determine the data structure conditions corresponding to each of the aforementioned feature categories in the target data processing scenario;

[0023] Determine the memory container type that matches each of the data structure conditions to obtain the correspondence between the feature categories and memory container types;

[0024] Based on the correspondence, the feature data of the target object are stored in the target memory container of the corresponding memory container type according to the feature category to which it belongs.

[0025] A computer program product includes a computer program, characterized in that, when the computer program is executed by a processor, it performs the following steps:

[0026] For the target data processing scenario, determine the feature category to which the feature data required for data processing of the target object belongs;

[0027] Determine the data structure conditions corresponding to each of the aforementioned feature categories in the target data processing scenario;

[0028] Determine the memory container type that matches each of the data structure conditions to obtain the correspondence between the feature categories and memory container types;

[0029] Based on the correspondence, the feature data of the target object are stored in the target memory container of the corresponding memory container type according to the feature category to which it belongs.

[0030] The aforementioned feature data storage method, apparatus, computer equipment, storage medium, and computer program product, for the target data processing scenario, determine the feature category to which the feature data required for data processing of the target object belongs, determine the corresponding data structure conditions for each feature category in the target data processing scenario, determine the memory container type that matches each data structure condition, obtain the correspondence between feature categories and memory container types, and based on the correspondence, store the feature data of the target object into the target memory container of the corresponding memory container type according to its feature category. Based on the data structure conditions of the feature data in the target data processing scenario, the feature data is directly stored according to the memory container corresponding to the data structure conditions, so that when processing the target object, the feature data of the target object can be directly read from the memory container, avoiding online data parsing and improving the efficiency of feature data acquisition.

[0031] A target object analysis method based on feature data, the method comprising:

[0032] Based on the feature categories of the feature data required in the target data processing scenario, and according to the correspondence between the feature categories and memory container types, find the target memory container corresponding to each feature category of the target object;

[0033] From each of the target memory containers, read the feature data that meets the data structure conditions corresponding to the target data processing scenario;

[0034] The target object is analyzed based on the feature data to obtain the data analysis results of the target object in the target data processing scenario.

[0035] A target object analysis device based on feature data, the device comprising:

[0036] The container lookup module is used to find the target memory container corresponding to each feature category of the target object based on the feature categories of the feature data required in the target data processing scenario and according to the correspondence between the feature categories and memory container types.

[0037] The feature reading module is used to read feature data that meets the data structure conditions corresponding to the target data processing scenario from each of the target memory containers;

[0038] The data analysis module is used to analyze the target object based on the feature data to obtain the data analysis results of the target object in the target data processing scenario.

[0039] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0040] Based on the feature categories of the feature data required in the target data processing scenario, and according to the correspondence between the feature categories and memory container types, find the target memory container corresponding to each feature category of the target object;

[0041] From each of the target memory containers, read the feature data that meets the data structure conditions corresponding to the target data processing scenario;

[0042] The target object is analyzed based on the feature data to obtain the data analysis results of the target object in the target data processing scenario.

[0043] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0044] Based on the feature categories of the feature data required in the target data processing scenario, and according to the correspondence between the feature categories and memory container types, find the target memory container corresponding to each feature category of the target object;

[0045] From each of the target memory containers, read the feature data that meets the data structure conditions corresponding to the target data processing scenario;

[0046] The target object is analyzed based on the feature data to obtain the data analysis results of the target object in the target data processing scenario.

[0047] A computer program product includes a computer program, characterized in that, when the computer program is executed by a processor, it performs the following steps:

[0048] Based on the feature categories of the feature data required in the target data processing scenario, and according to the correspondence between the feature categories and memory container types, find the target memory container corresponding to each feature category of the target object;

[0049] From each of the target memory containers, read the feature data that meets the data structure conditions corresponding to the target data processing scenario;

[0050] The target object is analyzed based on the feature data to obtain the data analysis results of the target object in the target data processing scenario.

[0051] The aforementioned target object analysis method, apparatus, computer equipment, storage medium, and computer program product based on feature data, based on the feature categories of the feature data required in the target data processing scenario, and according to the correspondence between feature categories and memory container types, finds the target memory container corresponding to each feature category of the target object, reads feature data that meets the data structure conditions corresponding to the target data processing scenario from each target memory container, analyzes the target object based on the feature data, and obtains the data analysis results of the target object in the target data processing scenario. In the target memory container, feature data that meets the data structure conditions corresponding to the target data processing scenario is stored according to feature categories, so that when processing data for the target object, feature data corresponding to each feature category and meeting the data structure conditions can be directly read from the target memory container without online data parsing, thus improving the efficiency of feature data acquisition. Attached Figure Description

[0052] Figure 1 This is an application environment diagram of a feature data storage method and a target object analysis method based on feature data in one embodiment;

[0053] Figure 2 This is a flowchart illustrating a feature data storage method in one embodiment;

[0054] Figure 3 This is a schematic diagram of the structure of a map container in one embodiment;

[0055] Figure 4 This is a schematic diagram of the structure of a vector container in one embodiment;

[0056] Figure 5 This is a flowchart illustrating a target object analysis method based on feature data in one embodiment;

[0057] Figure 6 This is a flowchart illustrating a feature data storage method in another embodiment;

[0058] Figure 7 This is a flowchart illustrating a target object analysis method based on feature data in one embodiment;

[0059] Figure 8 This is a structural block diagram of a feature data storage device in one embodiment;

[0060] Figure 9 This is a structural block diagram of a target object analysis device based on feature data in one embodiment;

[0061] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] The feature data storage method and the target object analysis method based on feature data provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. During the storage of feature data, server 104 determines the feature category of the feature data required by the target object based on the target data processing scenario corresponding to terminal 102. Server 104 determines the data structure conditions corresponding to each feature category in the target data processing scenario, determines the memory container type that matches each data structure condition, obtains the correspondence between feature categories and memory container types, and stores the feature data of the target object into the target memory container of the corresponding memory container type according to its feature category.

[0064] In the process of target object analysis based on feature data, server 104 determines the target object to be searched based on the data processing request triggered by terminal 102. Based on the feature categories of the feature data required in the target data processing scenario, server 104 searches for the target memory container corresponding to each feature category of the target object according to the correspondence between feature categories and memory container types. From each target memory container, feature data that meets the data structure conditions corresponding to the target data processing scenario is read. Based on the feature data, the target object is analyzed to obtain the data analysis results of the target object in the target data processing scenario.

[0065] For example, before a user travels, they can check the weather via mobile terminal 102 or in-vehicle terminal 102. In response to the user's search operation via the search bar on terminal 102, server 104 reads feature data that matches each feature category of each article from the target memory container corresponding to weather-related articles, analyzes the feature data, filters out recommended articles, and displays the recommended articles on mobile terminal 102 or in-vehicle terminal 102.

[0066] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, portable wearable devices, smart voice interaction devices, smart home appliances and vehicle terminals, and the server 104 can be implemented by a standalone server or a server cluster composed of multiple servers.

[0067] In one embodiment, such as Figure 2 As shown, a feature data storage method is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0068] Step 202: For the target data processing scenario, determine the feature category to which the feature data required for data processing of the target object belongs.

[0069] In this context, the target object is the object that requires data processing, and the target data processing scenario distinguishes different data processing procedures for the target object. In a target data processing scenario, the target object requiring data processing can be a single object, multiple objects of the same type, or multiple objects of different types. For example, in a scenario of object search and recommendation display, the target object can be an article, multiple articles matching the same keyword, or articles, images, videos, etc., matching the same keyword. The target data processing scenario can be determined based on specific business needs, such as a scenario where object search and recommendation display is performed by inputting keywords. In a specific application, in a search and recommendation scenario, the target object can be all objects matching the keyword. Data processing for the target object includes analyzing its feature data and filtering recommended objects based on the analysis results.

[0070] Feature data refers to the data required for data processing. Feature categories characterize the data type of feature data. The feature categories of the feature data required for data processing of a target object correspond to the target data processing scenario. Specifically, for the same data processing scenario, if the corresponding target object includes multiple categories, the feature categories of the feature data for each category of target object can be different. For the same target object, the feature categories of the required feature data can also be different in different data processing scenarios. Taking scoring articles to filter recommended articles as an example, the feature categories of the article feature data needed for scoring articles include entity words, word weights, tags, categories, titles, and embedded features. Taking scoring images to filter recommended images as an example, the feature categories of the image feature data needed for image scoring include color features, texture features, shape features, and spatial relationship features.

[0071] Specifically, the server can store corresponding feature data for each data processing scenario. This stored feature data is used to perform data analysis for that specific scenario. For example, in a search and recommendation scenario, the search content can include articles, videos, images, audio, etc. In this scenario, because the search results need to be filtered and recommended, a layered filtering approach is generally used. For instance, the first layer searches 10,000 articles as target objects, obtains the corresponding feature data for each target object, and scores and filters them according to the rules of the first layer. This filters out 3,000 articles as target objects for the second layer, scores and filters them according to the rules of the second layer, filters out 500 articles as target objects for the third layer, scores and filters them according to the rules of the third layer, filters out 100 articles as target objects for the fourth layer, scores and filters them according to the rules of the fourth layer, and finally filters out 200 articles as target objects for the fifth layer and recommends them to the user.

[0072] Step 204: Determine the corresponding data structure conditions for each feature category in the target data processing scenario.

[0073] Among these, data structure conditions are the format conditions that feature data must meet for data processing in the target data processing scenario. Data structure conditions define both the data storage method and the data type. When processing the target object in the target data processing scenario, the required feature data can be quickly and easily obtained from the stored data that meets the data structure conditions, enabling appropriate data processing. Data format conditions correspond to feature categories; different feature categories may have the same or different data format conditions.

[0074] Data structure conditions include data storage type conditions and data format conditions. Taking articles as the target object as an example, the data structure conditions corresponding to the two feature categories of article word weights and article tags are both map.<string,uint32_t> The data storage type is specified as map, and the data format is specified as follows:<string,uint32_t> In this context, `map` represents key-value pair storage, `string` indicates the data type of the key in the key-value pair, and `uint32_t` indicates the data type of the value in the key-value pair (where `u` represents `unsigned`, meaning the variable cannot be negative; `int` represents the type `int`; `32` represents four bytes, i.e., `int` type; `_t` represents a macro definition using `typedef`; and `uint32_t` represents an unsigned `int` macro definition defined using `typedef`). The data structure condition corresponding to the feature category of "embedding feature" in the article is `vector`.<uint32_t> In this context, vector represents the data storage method as element-value storage, and uint32_t represents the data type of the element-value.

[0075] Specifically, for the target data processing scenario, after determining the feature categories of the feature data required for data processing of the target object, the server determines the corresponding data structure conditions for each feature category under the target data processing scenario. Furthermore, the data structure conditions corresponding to each feature category can be configured according to the needs of the data processing scenario. The server obtains the data structure conditions corresponding to each feature category for the target data processing scenario by searching for data structure condition configuration data that matches the target data processing scenario.

[0076] Step 206: Determine the memory container type that matches each data structure condition, and obtain the correspondence between feature categories and memory container types.

[0077] The memory container type is used to distinguish memory containers with different memory space structures, and the memory container type is matched with the data structure condition. The data structure condition corresponds to the feature type and matches the memory container type. Using the data structure condition as intermediate data, the correspondence between the feature type and the content container type can be obtained.

[0078] Specifically, the server constructs a correspondence between feature categories and memory container types based on the data structure conditions corresponding to each feature category in the target data processing scenario, as well as the memory container type that matches each data structure condition. This determines which type of memory container the feature data of each feature category needs to be stored in, thereby facilitating the rapid and convenient acquisition of feature data in the target data processing scenario.

[0079] In a specific application, the server determines the type of memory container that meets the data storage type conditions based on the data storage type and data format conditions in the data structure conditions, and establishes a correspondence between feature categories and memory container types. For example, the data structure condition map<string,uint32_t> If the data storage type is specified as a map, then the memory container type is a map memory container. For example, if the data structure is specified as a vector...<uint32_t> If the data storage type is vector, then the memory container type is vector memory container.

[0080] Step 208: Based on the correspondence, store the feature data of the target object into the target memory container of the corresponding memory container type according to the feature category to which it belongs.

[0081] A target memory container is a collection of objects of a specific type, used for storing characteristic data according to data structure conditions. The target memory container is a read-only container in a planar memory model. Planar memory model is a memory address space management method where the application sees a single, contiguous address space. The CPU (central processing unit) can directly address all available memory locations without resorting to any memory segmentation or paging mechanisms. A read-only container is a container that stores characteristic data that can only be read, not modified. Using a planar memory model read-only container as the target memory container ensures that the stored data is stored contiguously and compactly in memory space, thereby effectively improving storage space utilization and reducing storage space occupation. Memory containers can be divided into two categories: sequential containers and associative containers.

[0082] Furthermore, based on the correspondence between feature categories and memory container types, the server extracts feature data of the target object according to the feature category, and stores the extracted feature data into the target memory container of the corresponding memory container type according to the feature category. This achieves classified storage of feature data, enabling accurate and rapid acquisition of feature data based on the feature category to which the required feature data belongs in the target data processing scenario.

[0083] In the above embodiments, by determining the feature category of the feature data required for data processing of the target object for the target data processing scenario, determining the corresponding data structure conditions for each feature category in the target data processing scenario, determining the memory container type that matches each data structure condition, and obtaining the correspondence between feature categories and memory container types, based on the correspondence, the feature data of the target object is stored in the target memory container of the corresponding memory container type according to its feature category. Based on the data structure conditions of the feature data in the target data processing scenario, the feature data is directly stored according to the memory container corresponding to the data structure conditions, so that when processing the target object, the feature data of the target object can be directly read from the memory container, avoiding online data parsing and improving the efficiency of feature data acquisition.

[0084] In one embodiment, based on the correspondence, the feature data of the target object is stored in the target memory container of the corresponding memory container type according to the feature category to which it belongs, including: for the feature data of each feature category of the target object, determining the target memory container of the memory container type based on the memory container type corresponding to the feature category; and storing the feature data of the target feature category in the target memory container according to the memory space structure of the target memory container.

[0085] The memory space structure refers to the data storage space within a memory container. The memory space structure corresponds to the memory container type and conforms to the data structure conditions corresponding to that type. Different memory container types can have different memory space structure compositions, and even containers with the same memory space structure can have different data storage methods. For example, memory container types include associative containers and sequential containers. Sequential containers store and access elements sequentially based on their position within the container, while associative containers store and retrieve elements by key. Taking a map container as an example of an associative container, it provides a one-to-one data storage capability based on a key-value relationship. The keys in a map container are unique and arranged in a specific order. A sequential container is a linear list where elements have an ordered relationship; it is a linearly structured ordered cluster. The memory space structure of an associative container includes a data storage area, an index area, a metadata area, and a footer area. The memory space structure of a sequential container also includes a data storage area, a metadata area, and a footer area. The data storage formats in the data storage areas of associative and sequential containers differ.

[0086] Specifically, the server extracts feature data for each feature category of each target object according to feature categories. For the feature data of each feature category of the same target object, based on the memory container type corresponding to the feature category, it determines the target memory container of that type. According to the memory space structure of the target memory container, the feature data is stored in the target memory container. One target memory container is used to store the feature data of one feature category of one target object. For example, if there are 100 target objects, and each target object has feature data of 5 different feature categories, then the number of target memory containers required to store the feature data of these 100 target objects is 500.

[0087] In this embodiment, feature data of the same feature category are stored according to the memory space structure of the target memory container. The memory space structure conforms to the data structure conditions corresponding to the memory container type, which enables the feature data of the target object to be obtained quickly and accurately according to the feature category in the target data processing scenario.

[0088] In one embodiment, storing feature data of the target feature category into the target memory container according to the memory space structure of the target memory container includes: constructing data to be stored for each block based on the feature data of the same feature category and according to the data conditions corresponding to each block in the memory space structure of the target memory container; and writing the data to be stored into each block of the target memory container.

[0089] The memory space structure comprises multiple contiguous blocks, each used to store different types of data. The specific block composition of the memory space structure is related to the type of memory container the target memory container belongs to. Target memory containers of the same type have the same specific block composition in their memory space structure.

[0090] The data to be stored is data determined based on the feature data of the target object and used to store it in the corresponding blocks. The blocks in the memory space structure include blocks for storing feature data and blocks for storing associated data of the feature data. Among them, the associated data of the feature data includes metadata of the feature data, etc.

[0091] Specifically, the server obtains the data conditions corresponding to each block in the memory space structure of the target memory container, constructs the data to be stored for each block according to the data conditions corresponding to each block based on the feature data of the same feature category, and then writes the data to be stored into each block of the target memory container according to the block corresponding to the data to be stored.

[0092] In a specific application, taking the target memory container corresponding to the first feature type of the target object as an associative container as an example, the memory space structure of the associative container includes a data storage area, an index information area, a metadata area, and a footer information area. Based on the feature data of the first feature type of the target object, the server constructs the data to be stored in the data storage area, index information area, metadata area, and footer information area respectively. Then, according to the blocks corresponding to the data to be stored, the constructed data to be stored is written into each block of the target memory container. Similarly, taking the target memory container corresponding to the second feature type of the target object as a sequential container as an example, the memory space structure of the sequential container includes a data storage area, a metadata area, and a footer information area. Based on the feature data of the first feature type of the target object, the server constructs the data to be stored in the data storage area, metadata area, and footer information area respectively. Then, according to the blocks corresponding to the data to be stored, the constructed data to be stored is written into each block of the target memory container.

[0093] In this embodiment, the server constructs the data to be stored for each block according to the data conditions corresponding to different blocks. This ensures that the feature data is stored contiguously and compactly in memory. Furthermore, it enables the feature data to be stored according to data structure conditions, facilitating subsequent data retrieval and acquisition.

[0094] In one embodiment, determining the memory container type that matches each data structure condition includes: determining the feature data lookup method represented by the data structure condition; when the feature data lookup method is binary search, determining the memory container type that matches the data structure condition as an associated container that stores feature data according to key-value pairs.

[0095] Associative containers are memory containers that store feature data according to key-value pairs. They provide the ability to quickly store and access data elements using keys. Specifically, associative containers include sets, multisets, maps, and multimaps. Taking the map container as an example, its implementation principle is as follows: The map internally builds a red-black tree. A red-black tree is a strictly balanced binary tree that automatically sorts data. Within the map, all feature data is ordered, enabling binary search on the feature data to improve search efficiency.

[0096] Specifically, when the server determines that the feature data search method represented by the data structure condition is binary search based on the data storage type condition in the data structure condition, it determines that the memory container type matching the data structure condition is an associated container that stores feature data according to key-value pairs.

[0097] In a specific application, when the data storage type condition in the data structure condition is map storage, it can be determined that the feature data search method represented by the data structure condition is binary search, and the memory container type that matches the data structure condition is map container.

[0098] In this embodiment, the server determines the type of memory container that matches the data structure conditions to store the feature data by using the data search method characterized by the data structure conditions. This enables the server to quickly and accurately obtain feature data of a specified feature category in subsequent data processing, thereby improving the efficiency of feature data acquisition.

[0099] In one embodiment, the target memory container is an associative container, and its memory space structure includes a data storage area, an index information area, a metadata area, and a footer information area. Based on feature data of the same feature category, and according to the data conditions corresponding to each block in the memory space structure of the target memory container, the data to be stored for each block is constructed, including: using each feature data belonging to the same feature category as the value in a key-value pair to construct key-value pairs as the data to be stored in the data storage area; constructing index information as the data to be stored in the index information area based on the key in each key-value pair and the storage location information corresponding to each key; constructing metadata as the data to be stored in the metadata area based on the data type information of the key-value pairs; and constructing footer information as the data to be stored in the footer information area based on the legal identification information of the associative container, the first storage location information corresponding to the metadata area, and the second storage location information corresponding to the index information area.

[0100] Among them, such as Figure 3 As shown, the memory space structure of an associative container includes a data block, an index block, a metadata area, and a footer. The footer is a storage area for storing footer information, which consists of a magic number and two handles. The magic number is fixed and used to determine the validity of the memory area corresponding to this associative container during memory loading. The two handles are the metadata handle and the index info handle. The metadata handle points to the original location of the metadata block, and the index info handle points to the original location of the index block. The original locations of both the index block and the metadata area can be represented by offset and size.

[0101] The index block of the associative container is a storage area used to store the key in each key-value pair and the storage location information corresponding to each key. The index block consists of a series of fixed 16-bit key_ids and fixed 16-bit key_pos, where key_id = hash(key); key_pos points to the position of the original key; another feature of the index block is that it is sorted in ascending order by key_id so that the corresponding key can be found by binary search.

[0102] The metadata block of an associative container is a storage area used to store metadata, including data type information for key-value pairs from the data storage area. The data type information for key-value pairs includes string→string, string→int, and int→int, representing the data types of the key and value in the pair. (See reference...) Figure 3 If the data type information of the key-value pairs recorded in the metadata is string→string, then the key-value pairs in the data storage area include the key, the length of the key, the value, and the length of the value. If the data type information of the key-value pairs recorded in the metadata is string→int, then the key-value pairs in the data storage area include the key, the length of the key, and the value. If the data type information of the key-value pairs recorded in the metadata includes int→int, then the key-value pairs in the data storage area include the key and the value. In addition, the metadata area can also store metadata such as the data version information of the target object and the memory container type. Expansion space is also reserved in the metadata area for extending the metadata.

[0103] The data block of the associated container is a storage area that stores data in key-value pairs, used to store the original key-value pairs. In one embodiment, the key and value in the key-value pairs in the data storage area are stored using variable-length storage to save space.

[0104] Specifically, when the associated container uses the above memory space structure to store features, no encoding or decoding operations are needed during feature data acquisition and processing. Nor is it necessary to store the data in an additional memory container that conforms to the data format. The only memory required to access the data is a buffer, requiring zero additional allocation, and it supports binary search, effectively solving the performance issues caused by existing Protobuf formats. This memory space structure trades space for time, allocating an additional 4 bytes (fixed16 + fixed16) of space for each key to support binary search. However, the additional space allocated for the key is far less than the space required to store the decoded data in an additional memory container that conforms to the data format.

[0105] In this embodiment, feature data is stored in an associated container consisting of a data storage area, an index information area, a metadata area, and a footer information area. This enables the required feature data to be retrieved quickly and conveniently using a binary search method when acquiring and processing the feature data, thereby improving data acquisition efficiency.

[0106] In one embodiment, determining the memory container type that matches each data structure condition includes: determining the feature data lookup method represented by the data structure condition; when the feature data lookup method is sequential traversal, determining the memory container type that matches the data structure condition as a sequential container that stores feature data according to element values.

[0107] In sequential containers, the arrangement of elements is not related to the size of the element values, but rather to the order in which the elements are added to the container. Types of sequential containers include vector, list, and deque (double-ended queue). Taking vector as an example, the characteristic data is used as storage elements, stored sequentially. Each element has a unique corresponding position number, and all operations are performed according to the number or position. For example, traversing elements involves accessing them one by one in positional order.

[0108] Specifically, when the server determines that the feature data lookup method represented by the data structure condition is sequential traversal based on the data storage type condition in the data structure condition, it determines that the memory container type matching the data structure condition is a sequential container that stores feature data according to element values.

[0109] In a specific application, when the data storage type condition in the data structure condition is vector storage, it can be determined that the characteristic data search method represented by the data structure condition is sequential traversal, and the memory container type that matches the data structure condition is vector container.

[0110] In this embodiment, the server determines the type of memory container that matches the data structure conditions to store the feature data by using the data search method characterized by the data structure conditions. This enables the server to quickly and accurately obtain feature data of a specified feature category in subsequent data processing, thereby improving the efficiency of feature data acquisition.

[0111] In one embodiment, the target memory container is a sequential container, and its memory space structure includes a data storage area, a metadata area, and a footer information area. Based on feature data of the same feature category, and according to the data conditions corresponding to each block in the memory space structure of the target memory container, the data to be stored for each block is constructed, including: constructing element values ​​as data to be stored in the data storage area based on each feature data belonging to the same feature category; constructing metadata as data to be stored in the metadata area based on the data type information of the element values; and constructing footer information as data to be stored in the footer information area based on the legal identification information of the sequential container and the storage location information corresponding to the metadata area.

[0112] Among them, such as Figure 4 As shown, the memory space structure of a sequential container includes a data block, a metadata block, and a footer. The footer is a storage area for storing footer information, which consists of a magic number and a metadata handle. The magic number is fixed and used to determine the validity of the memory region corresponding to this sequential container during memory loading. The metadata handle points to the original location of the metadata block.

[0113] The metadata block of a sequential container is a storage area used to store metadata, including data type information of the element values ​​in the data storage area. The data type information of the element values ​​includes string, struct{int, int}, int, etc. (See reference...) Figure 4 If the data type information of the element values ​​recorded in the metadata is string, then the element values ​​in the data storage area include the element value and its length. If the data type information of the element values ​​recorded in the metadata is struct{int, int}, then the key-value pairs in the data storage area include each element value in the element value group. If the data type information of the key-value pairs recorded in the metadata is int, then the key-value pairs in the data storage area include the element values. In addition, the metadata area can also store metadata such as the target object's data version information and memory container type. Expansion space is also reserved in the metadata area for extending the metadata.

[0114] The data block of a sequential container is a storage area that stores data in the form of element values, used to store raw feature data. In one embodiment, the storage method corresponding to the element values ​​in the data storage area is variable-length storage to save space.

[0115] Specifically, when loading feature data from a vector container online, a String with contiguous storage space is obtained. The parser then parses the footer portion to obtain the actual offset position of the data area. After that, it iterates sequentially to obtain the relevant element values. Throughout the feature data process, no decoding is required. Based on a high-performance and low-memory data structure that requires no decoding and uses ZeroCopy, this meets the high-performance and low-memory requirements of online feature retrieval, enabling quick and convenient acquisition of the required feature data and improving data acquisition efficiency.

[0116] In one embodiment, such as Figure 5 As shown, a target object analysis method based on feature data is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0117] Step 502: Based on the feature categories of the feature data required in the target data processing scenario, find the target memory container corresponding to each feature category of the target object according to the correspondence between feature categories and memory container types.

[0118] The "target data processing scenario" distinguishes different data processing procedures for target objects, which are the objects that require data processing. In a target data processing scenario, the target objects requiring data processing can be a single object, multiple objects of the same type, or multiple objects of different types. For example, in a scenario of object search and recommendation display, the target object can be an article, multiple articles matching the same keyword, or articles, images, videos, etc., matching the same keyword. The target data processing scenario can be determined based on specific business needs, such as a scenario where object search and recommendation display is performed by inputting keywords. In a specific application, in a search and recommendation scenario, the target object can be all objects matching the keyword. Data processing for the target object includes analyzing its feature data and filtering recommended objects based on the analysis results.

[0119] Feature categories are used to characterize the data categories of feature data. The feature categories of the feature data required for data processing of a target object correspond to the target data processing scenario. Feature data corresponds to the target object and is the data required for data processing of the target object. Specifically, for the same data processing scenario, if the corresponding target object includes multiple categories, the feature categories of the feature data for each category of target object can be different. For the same target object, the feature categories of the required feature data can also be different in different data processing scenarios. Taking scoring articles to filter recommended articles as an example, the feature categories of the article feature data used for scoring articles include entity words, word weights, tags, categories, titles, and embedded features. Taking scoring images to filter recommended images as an example, the feature categories of the image feature data used for image scoring include color features, texture features, shape features, and spatial relationship features.

[0120] Memory container types are used to distinguish memory containers with different memory space structures. The memory container type is matched with the data structure condition. The data structure condition corresponds to the feature type and matches the memory container type. Using the data structure condition as intermediate data, the correspondence between the feature type and the content container type can be obtained. Memory container types can be divided into two categories: sequential containers and associative containers.

[0121] A target memory container is a collection of objects of a specific type, used for storing characteristic data according to data structure conditions. The target memory container is a read-only container in a planar memory model. Planar memory model is a memory address space management method where the application sees a single, contiguous address space. The CPU can directly address all available memory locations without resorting to any memory segmentation or paging mechanisms. A read-only container is one that stores characteristic data that can only be read, not modified. Using a planar memory model read-only container as the target memory container ensures that the stored data is stored contiguously and compactly in memory, thereby effectively improving storage space utilization and reducing storage space usage.

[0122] Specifically, in response to a user-triggered data processing request, the server determines the target object and the feature categories of the required feature data based on the target data processing scenario corresponding to the request. The server then searches for the corresponding target memory containers according to the mapping between feature categories and memory container types. For example, if there are 100 target objects, and each target object requires feature data of 5 different feature categories, then it is necessary to search for the 5 target memory containers required for the feature data of all 100 target objects, resulting in a total of 500 target feature containers to be searched.

[0123] Step 504: Read the feature data that meets the data structure conditions corresponding to the target data processing scenario from each target memory container.

[0124] Among these, data structure conditions are the format conditions that feature data must meet for data processing in the target data processing scenario. Data structure conditions define both the data storage method and the data type. When processing the target object in the target data processing scenario, the required feature data can be quickly and easily obtained from the stored data that meets the data structure conditions, enabling appropriate data processing. Data format conditions correspond to feature categories; different feature categories may have the same or different data format conditions.

[0125] Data structure conditions include data storage type conditions and data format conditions. Taking articles as the target object as an example, the data structure conditions corresponding to the two feature categories of article word weights and article tags are both map.<string,uint32_t> The data storage type is specified as map, and the data format is specified as follows:<string,uint32_t> Here, `map` represents the data storage method as key-value pairs, `string` indicates the data type of the key in the key-value pair, and `uint32_t` indicates the data type of the value in the key-value pair. The data structure condition corresponding to the feature category of article embedding features is `vector`.<uint32_t> In this context, vector represents the data storage method as element-value storage, and uint32_t represents the data type of the element-value.

[0126] Specifically, based on each feature category required by the target object in the target data processing scenario, the server reads feature data that conforms to the data structure conditions corresponding to the target data processing scenario from the target memory container corresponding to each feature category. The feature data in the target memory container is stored according to the data structure conditions corresponding to the target data processing scenario, thus enabling direct acquisition of feature data for each feature category of the target object. Furthermore, the acquired feature data all conforms to the data structure conditions corresponding to the target data processing scenario. Since the data format of the feature data is constructed offline during storage, online use only requires loading the byte stream into memory. In other words, the online feature data resides in a contiguous memory space; this contiguous space is directly used as a map or vector container, eliminating the need to parse the contents of the map or vector containers to obtain the feature data for analysis of the target object.

[0127] Step 506: Analyze the target object based on the feature data to obtain the data analysis results of the target object in the target data processing scenario.

[0128] The analysis method for the target object is determined based on the target data processing scenario, and the results obtained correspond to the target data processing scenario. For example, in a search and recommendation scenario, the analysis method for the target object is to score the target object based on feature data, and then filter out the recommended target objects. As another example, in a precise matching scenario, the analysis method for the target object is to match the target object based on feature data, and then filter out the target object with the highest matching degree.

[0129] In this embodiment, based on the feature categories of the feature data required in the target data processing scenario, and according to the correspondence between feature categories and memory container types, the target memory container corresponding to each feature category of the target object is found. From each target memory container, feature data that meets the data structure conditions corresponding to the target data processing scenario is read. Based on the feature data, the target object is analyzed to obtain the data analysis results of the target object in the target data processing scenario. In the target memory container, feature data that meets the data structure conditions corresponding to the target data processing scenario is stored according to feature categories. This allows feature data corresponding to each feature category and meeting the data structure conditions to be read directly from the memory container when processing data for the target object, without the need for online data parsing, thus improving the efficiency of feature data acquisition.

[0130] In one embodiment, the target memory container is an associated container that stores feature data according to key-value pairs. From each target memory container, feature data that meets the data structure conditions corresponding to the target data processing scenario is read, including: for each associated container, when the legal identifier information in the footer of the associated container indicates that the memory data is legal, the storage location information of the index information area is obtained from the footer information; based on the storage location information of the index information area, a binary search is performed in the index information area to obtain the target keyword corresponding to the feature data to be obtained; the keyword location information matching the target keyword is obtained from the index information area; based on the keyword location information, the target key-value pair is searched in the data storage area of ​​the associated container; based on the key-value pair, the feature data of the target object under the feature category corresponding to the associated container is obtained.

[0131] Associative containers are memory containers that store feature data according to key-value pairs. They provide the ability to quickly store and access data elements using keys. Specifically, associative containers include sets, multisets, maps, and multimaps. Taking the map container as an example, its implementation principle is as follows: The map internally builds a red-black tree. A red-black tree is a strictly balanced binary tree with automatic data sorting capabilities. Within the map, all feature data is ordered, enabling binary search on the feature data to improve search efficiency.

[0132] The memory space structure of an associative container includes a data storage area, an index information area, a metadata area, and a footer information area. The footer area is used to store footer information, which consists of a magic number and two handles. The magic number is fixed and used to determine the validity of the memory region corresponding to this associative container during memory loading. The two handles are the metadata info handle and the index info handle. The metadata info handle points to the original location of the metadata block, and the index info handle points to the original location of the index block.

[0133] The index block of the associated container is a storage area used to store the key in each key-value pair and the storage location information corresponding to each key. The index block consists of a series of fixed 16-bit key_id and fixed 16-bit key_pos, where key_id = hash(key); key_pos points to the position of the original key; another feature of the index block is that it is sorted in ascending order of key_id, so that the corresponding key can be found by binary search.

[0134] The metadata block of an associated container is a storage area used to store metadata, including data type information for key-value pairs from the data storage area. Additionally, the metadata block can also store metadata such as the target object's version information and the memory container type.

[0135] The data block of the associated container is a storage area that stores data in key-value pairs, used to store the original key-value pairs. In one embodiment, the key and value in the key-value pairs in the data storage area are stored using variable-length storage to save space.

[0136] Specifically, when the server retrieves feature data from the associated containers, it uses a binary search method. Further, for each associated container, if the valid identifier information in the container's footer indicates that the memory data is valid, the server retrieves the storage location information of the index information area from the footer information. Based on this storage location information, the server performs a binary search within the index information area to obtain the target keyword corresponding to the feature data to be retrieved. Then, the server retrieves the keyword location information matching the target keyword from the index information area. Based on this keyword location information, the server searches for the target key-value pair in the associated container's data storage area. Finally, based on the key-value pair, the server retrieves the feature data of the target object under the corresponding feature category of the associated container and analyzes the target object.

[0137] In this embodiment, the associated container uses the memory space structure described above to store features. When retrieving feature data from the associated container for data processing, no encoding or decoding operations are required, nor is it necessary to put the data into an additional memory container that meets the data format requirements. The only memory required to access the data is a buffer, which does not require additional allocation and supports binary search. This effectively solves the performance problems caused by existing Protobuf formats, thereby saving the total memory space required for feature data storage and retrieval, and avoiding the data parsing process, thus improving the efficiency of feature data retrieval.

[0138] In one embodiment, the target memory container is a sequential container that stores feature data according to element values; from each target memory container, feature data that meets the data structure conditions corresponding to the target data processing scenario is read, including: for each sequential container, when the legal identification information in the footer of the sequential container indicates that the memory data is legal, determining the offset position of the feature data to be acquired in the data storage area; performing a traversal search based on the offset position, and using the found element value as the feature data of the target object under the feature category corresponding to the sequential container.

[0139] In sequential containers, the arrangement of elements is not related to the size of the element values, but rather to the order in which the elements are added to the container. Types of sequential containers include vector, list, and deque (double-ended queue). Taking vector as an example, the characteristic data is used as storage elements, stored sequentially. Each element has a unique corresponding position number, and all operations are performed according to the number or position. For example, traversing elements involves accessing them one by one in positional order.

[0140] The memory space structure of a sequential container includes a data storage area, a metadata area, and a footer information area. The footer area of ​​a sequential container is a storage area used to store footer information, which consists of a magic number and a metadata handle. The magic number is fixed and used to determine the validity of the memory region corresponding to this sequential container when loading from memory. The metadata handle points to the original location of the metadata block.

[0141] The metadata block of a sequential container is a storage area used to store metadata, such as data type information of the element values ​​in the data storage area. In addition, the metadata block can also store metadata such as the data version information of the target object and the type of the memory container.

[0142] The data block of a sequential container is a storage area that stores data as element values ​​and is used to store raw feature data. The element values ​​in this data block are stored using variable-length storage to save space.

[0143] Specifically, when the server retrieves feature data from the associated containers, the feature data is retrieved through sequential traversal. Further, for each sequential container, if the valid identifier information in the container's footer indicates that the memory data is valid, the server determines the offset position of the feature data to be retrieved in the data storage area; based on the offset position, a traversal search is performed, and the found element value is used as the feature data of the target object under the corresponding feature category of the sequential container.

[0144] In this embodiment, feature data is stored using a sequential container. When loading feature data from the vector container online, a contiguous memory space is obtained. The parser parses the footer information area at the end of the vector container to obtain the actual offset position of the data area, and obtains the content of the target element value through sequential traversal. No decoding is required throughout the feature data process. Based on a high-performance and low-memory data structure that requires no decoding and uses ZeroCopy, this approach meets the high-performance and low-memory requirements of online feature retrieval, enabling quick and convenient acquisition of the necessary feature data and improving the efficiency of feature data acquisition.

[0145] In one embodiment, such as Figure 6 As shown, a feature data storage method is provided, including the following steps:

[0146] Step 602: For the target data processing scenario, determine the feature category to which the feature data required for data processing of the target object belongs.

[0147] Step 604: Determine the corresponding data structure conditions for each feature category in the target data processing scenario.

[0148] Step 606: When the feature data search method represented by the data structure condition is binary search, determine the memory container type that matches the data structure condition as an associative container.

[0149] Step 608: When the feature data search method represented by the data structure condition is sequential traversal, determine that the memory container type matching the data structure condition is a sequential container.

[0150] Step 610: Based on the memory container type matched for each data structure condition, obtain the correspondence between feature categories and memory container types.

[0151] Step 612: For the feature data of each feature category of the target object, determine the target memory container that belongs to the memory container type based on the memory container type corresponding to the feature category.

[0152] Step 614: Based on the feature data of the same feature category, determine the data type to be stored for each block in the memory space structure of the associated container.

[0153] Step 616: Use each feature data belonging to the same feature category as the value in the key-value pair to construct the key-value pair as the data to be stored in the data storage area.

[0154] Step 618: Based on the key in each key-value pair and the storage location information corresponding to each key, construct index information as the data to be stored in the index information area.

[0155] Step 620: Based on the data type information of the key-value pairs, construct metadata as the data to be stored in the metadata area.

[0156] Step 622: Based on the legal identification information of the associated container, the first storage location information corresponding to the metadata area, and the second storage location information corresponding to the index information area, construct the footer information as the data to be stored in the footer information area.

[0157] Step 624: Write the data to be stored into each block of the associated container.

[0158] Step 626: Based on the feature data of the same feature category, store the data types corresponding to each block in the memory space structure of the sequential container.

[0159] Step 628: Based on each feature data belonging to the same feature category, construct element values ​​as data to be stored in the data storage area.

[0160] Step 630: Based on the data type information of the element values, construct metadata as the data to be stored in the metadata area.

[0161] Step 632: Based on the valid identification information of the sequential container and the storage location information corresponding to the metadata area, construct the footer information as the data to be stored in the footer information area.

[0162] Step 634: Write the data to be stored into each block of the sequential container.

[0163] In one embodiment, such as Figure 7 As shown, a target object analysis method based on feature data is provided, including the following steps:

[0164] Step 702: Based on the feature categories of the feature data required in the target data processing scenario, find the target memory container corresponding to each feature category of the target object according to the correspondence between feature categories and memory container types.

[0165] Step 704: When the target memory container is an associated container that stores feature data according to key-value pairs, and the valid identifier information in the footer information of the associated container indicates that the memory data is valid, obtain the storage location information of the index information area from the footer information.

[0166] Step 706: Based on the storage location information of the index information area, perform a binary search in the index information area to obtain the target keyword corresponding to the feature data to be acquired.

[0167] Step 708: Obtain the keyword position information that matches the target keyword from the index information area.

[0168] Step 710: Based on the keyword location information, locate the target key-value pair in the data storage area of ​​the associated container.

[0169] Step 712: Based on key-value pairs, obtain the feature data of the target object under the feature category corresponding to the associated container.

[0170] Step 714: When the target memory container is a sequential container that stores feature data according to element values, and the legal identifier information in the footer of the sequential container indicates that the memory data is legal, determine the offset position of the feature data to be acquired in the data storage area.

[0171] Step 716: Perform a traversal search based on the offset position, and use the found element value as the feature data of the target object under the feature category corresponding to the sequential container.

[0172] Step 718: Analyze the target object based on the feature data to obtain the data analysis results of the target object in the target data processing scenario.

[0173] Currently, there are three main commonly used feature data storage formats: JSON, Protocol Buffers, and online parsing of raw data. Taking the "Term Weight" feature as an example, the space overhead (in bytes) and performance overhead (in microseconds) of storing the feature using the above structures are shown in Table 1:

[0174] Table 1: Space overhead data (in bytes) and performance overhead (in microseconds)

[0175]

[0176] Here, parsing performance refers to parsing the original data into a map.<string,int> The time taken for the structural process (the structure ultimately used by the business side) is calculated by averaging over 10,000 iterations. Performance testing was conducted in a single-threaded environment, eliminating memory allocation contention.

[0177] The test data above shows that, thanks to Protocol Buffers' compact binary serialization format and varint encoding for integer types (a method of encoding positive integers using variable-length bytes), the proto format reduces space usage by 73% compared to the json format, making it a good feature storage method. However, in terms of performance overhead, even with the least time-consuming proto format, acquiring features from the 500 articles selected at the fifth layer takes a total of 8.5 milliseconds, which is acceptable. But acquiring features from the 10,000 articles selected at the second layer takes a total of 170 milliseconds. It should be noted that for CPU-intensive tasks, although parallel feature acquisition can reduce the time spent per transaction, the total CPU time remains unchanged. The increase in total CPU time will ultimately affect overall performance. With a total time of 170 milliseconds, taking a 60-core CPU and a QPS (Queries per second) of 200 as an example, CPU utilization will increase by 56%. Therefore, neither protobuf nor json formats can simultaneously achieve high retrieval performance and low memory usage for feature data storage.

[0178] This application provides an application data processing scenario that utilizes the aforementioned feature data storage method and feature data-based target object analysis method. Specifically, the application of the feature data storage method and feature data-based target object analysis method in this application data processing scenario is as follows:

[0179] Taking search recommendation as an example, a large amount of feature data is needed to score the search results, and then filter them layer by layer until the desired results are obtained and recommended to the user. Specifically, taking articles as the search object, each level of filtering requires scoring the filtered articles. Scoring articles depends on the feature data of each article, therefore, this feature data needs to be stored.

[0180] In search recommendation scenarios, the data structure corresponding to the feature data required for article scoring is shown in Table 2:

[0181] Table 2: Data Structures Corresponding to Feature Data Required for Article Scoring

[0182] Feature categories Required data structure Term Weight map<string,uint32_t> Entity map<string,vector<struct{uint32_t,uint32_t}>> Tag map<string,uint32_t> Classification vector<{uint32_t,uint32_t}> Topic map<uint32_t,uint32_t> Embedding vector<uint32_t>

[0183] The above feature categories are based on reading-only flat RO containers, which are read-only containers with flat space. Flat RO containers include two types of containers: flat RO map and flat RO vector. The memory space corresponding to these two types of containers is contiguous and compact.

[0184] Specifically, the memory space structure of an associative container includes a data storage area, an index information area, a metadata area, and a footer information area. The footer area stores footer information, which consists of a magic number and two handles. The magic number is fixed and used to determine the validity of the memory area corresponding to this associative container during memory loading. The two handles are the metadata info handle and the index info handle. The metadata info handle points to the original location of the metadata block, and the index info handle points to the original location of the index block. The index block stores the key and its corresponding storage location information for each key-value pair. The index block consists of a series of fixed 16-bit key_ids and fixed 16-bit key_pos, where key_id = hash(key); key_pos points to the original key's location. A key feature of the index block is that it is sorted in ascending order of key_id, allowing for binary search to find the corresponding key. The metadata block stores metadata, including the data type information of the key-value pairs in the data storage area. In addition, the metadata area can store metadata such as the data version information of the target object and the type of the memory container. The data storage area (data block) stores data in key-value pairs and is used to store the original key-value pairs. The key and value pairs in the data storage area are stored using variable-length storage.

[0185] Using a flat RO map container to store feature data, no encoding or decoding operations are required when retrieving feature data, nor is it necessary to put the data into the map container again. The only memory required to access the data is a buffer, which does not need to be allocated separately. Furthermore, feature data retrieval supports binary search, which can effectively solve the performance problems caused by Protobuf format, etc.

[0186] The memory space structure of a sequential container includes a data storage area, a metadata area, and a footer area. The footer area stores footer information, which consists of a magic number and a metadata handle. The magic number is fixed and used to determine the validity of the memory region corresponding to the sequential container during memory loading. The metadata handle points to the original location of the metadata block. The metadata block stores metadata, including data type information of the element values ​​in the data storage area. It can also store version information of the target object and the type of the memory container. The data storage area stores raw characteristic data as element values. The element values ​​in the data storage area are stored using variable-length storage to save space.

[0187] When loading feature data stored in a flat RO vector container online, the result is a string with contiguous storage space. The parser parses the footer part at the end of the vector container to obtain the actual offset position of the data area. Then, it iterates sequentially to obtain the feature data content related to the article feature category. In the entire process of obtaining feature data, no decoding is required. Like the flat RO map container, the whole process is also ZeroCopy, which effectively improves the efficiency of feature data acquisition.

[0188] It should be understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or stages, which are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0189] In one embodiment, such as Figure 8 As shown, a feature data storage device 800 is provided. This device can be a software module, a hardware module, or a combination of both, integrated into a computer device. Specifically, the device includes: a category determination module, a structure condition determination module, a container type determination module, and a data storage module, wherein:

[0190] The category determination module 802 is used to determine the feature category to which the feature data required for data processing of the target object belongs, based on the target data processing scenario.

[0191] The structural condition determination module 804 is used to determine the data structure conditions corresponding to each of the feature categories in the target data processing scenario.

[0192] The container type determination module 806 is used to determine the memory container type that matches each of the data structure conditions, and to obtain the correspondence between the feature category and the memory container type.

[0193] The data storage module 808 is used to store the feature data of the target object into the target memory container of the corresponding memory container type according to the corresponding feature category, based on the correspondence.

[0194] In one embodiment, the data storage module includes a target memory container module and a feature data storage module; wherein:

[0195] The target memory container module is used to determine the target memory container belonging to the memory container type based on the feature data of each feature category of the target object and the memory container type corresponding to the feature category.

[0196] The feature data storage module is used to store the feature data of the target feature category into the target memory container according to the memory space structure of the target memory container.

[0197] In one embodiment, the feature data storage module includes a data construction module and a data writing module, wherein:

[0198] The data to be stored construction module is used to construct the data to be stored for each block based on the feature data of the same feature category and according to the data conditions corresponding to each block in the memory space structure of the target memory container.

[0199] The data writing module is used to write the data to be stored into each block of the target memory container.

[0200] In one embodiment, the container type determination module is further configured to determine that the memory container type matching the data structure condition is an associative container when the feature data search method represented by the data structure condition is a binary search; the feature data in the associative container is stored according to key-value pairs.

[0201] In one embodiment, the memory space structure of the associated container includes a data storage area, an index information area, a metadata area, and a footer information area. The data to be stored construction module is further configured to construct key-value pairs as data to be stored in the data storage area, using each feature data belonging to the same feature category as a value in a key-value pair; construct index information as data to be stored in the index information area based on the keyword in each key-value pair and the storage location information corresponding to each keyword; construct metadata as data to be stored in the metadata area based on the data type information of the key-value pairs; and construct footer information as data to be stored in the footer information area based on the legal identification information of the associated container, the first storage location information corresponding to the metadata area, and the second storage location information corresponding to the index information area.

[0202] In one embodiment, the container type determination module is further configured to determine that the memory container type matching the data structure condition is a sequential container when the feature data search method represented by the data structure condition is sequential traversal; the feature data in the sequential container is stored according to the element value.

[0203] In one embodiment, the memory space structure of the sequential container includes a data storage area, a metadata area, and a footer information area; the data to be stored construction module is further configured to construct element values ​​as data to be stored in the data storage area based on each feature data belonging to the same feature category; construct metadata as data to be stored in the metadata area based on the data type information of the element values; and construct footer information as data to be stored in the footer information area based on the legal identification information of the sequential container and the storage location information corresponding to the metadata area.

[0204] In one embodiment, such as Figure 9 As shown, a target object analysis device 900 based on feature data is provided. This device can be a software module, a hardware module, or a combination of both integrated into a computer device. Specifically, the device includes: a container lookup module 902, a feature reading module 904, and a data analysis module 906, wherein:

[0205] The container lookup module 902 is used to look up the target memory container corresponding to each feature category of the target object based on the feature categories of the feature data required in the target data processing scenario and according to the correspondence between the feature categories and memory container types.

[0206] The feature reading module 904 is used to read feature data that meets the data structure conditions corresponding to the target data processing scenario from each of the target memory containers;

[0207] The data analysis module 906 is used to analyze the target object based on the feature data to obtain the data analysis results of the target object in the target data processing scenario.

[0208] In one embodiment, the target memory container is an associated container that stores feature data according to key-value pairs; the feature reading module is further configured to, for each associated container, when the legal identifier information in the footer information of the associated container indicates that the memory data is legal, obtain the storage location information of the index information area from the footer information; based on the storage location information of the index information area, perform a binary search in the index information area to obtain the target keyword corresponding to the feature data to be obtained; obtain the keyword location information matching the target keyword from the index information area; based on the keyword location information, search for the target key-value pair in the data storage area of ​​the associated container; and based on the key-value pair, obtain the feature data of the target object under the feature category corresponding to the associated container.

[0209] In one embodiment, the target memory container is a sequential container that stores feature data according to element values; the feature reading module is further configured to, for each sequential container, determine the offset position of the feature data to be acquired in the data storage area when the legal identification information in the footer information of the sequential container indicates that the memory data is legal; perform a traversal search based on the offset position, and use the found element value as the feature data of the target object under the feature category corresponding to the sequential container.

[0210] Specific embodiments of the feature data storage device and the target object analysis device based on feature data can be found in the embodiments of the feature data storage method and the target object analysis method based on feature data described above, and will not be repeated here. Each module in the above-mentioned feature data storage device and target object analysis device based on feature data can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or it can be stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0211] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database provides a memory container for storing feature data. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a feature data storage method and a target object analysis method based on the feature data.

[0212] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0213] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0214] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0215] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0216] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0217] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0218] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for storing feature data, characterized in that, The method includes: For the target data processing scenario, determine the feature category to which the feature data required for data processing of the target object belongs; Determine the data structure conditions corresponding to each of the aforementioned feature categories in the target data processing scenario; Determine the memory container type that matches each of the data structure conditions to obtain the correspondence between the feature categories and memory container types; Based on the correspondence, the feature data of the target object are stored in the target memory container of the corresponding memory container type according to the feature category to which it belongs; In this context, each block of the target memory container is written with data to be stored constructed for that block; when the memory container type is a sequential container, the feature data in the sequential container is stored according to element values; the memory space structure of the sequential container includes a data storage area, a metadata area, and a footer information area; the data to be stored in the data storage area is an element value constructed based on each feature data belonging to the same feature category; the data to be stored in the metadata area is metadata constructed based on the data type information of the element values; the data to be stored in the footer information area is footer information constructed based on the legal identification information of the sequential container and the storage location information corresponding to the metadata area.

2. The method according to claim 1, characterized in that, Based on the correspondence, storing the feature data of the target object into target memory containers of corresponding memory container types according to their feature categories includes: For each feature category of the target object, based on the memory container type corresponding to the feature category, a target memory container belonging to the memory container type is determined; According to the memory space structure of the target memory container, the feature data of the target feature category is stored in the target memory container.

3. The method according to claim 2, characterized in that, The step of storing the feature data of the target feature category into the target memory container according to the memory space structure of the target memory container includes: Based on feature data of the same feature category, and according to the data conditions corresponding to each block in the memory space structure of the target memory container, the data to be stored for each block is constructed respectively. The data to be stored is written into each block of the target memory container.

4. The method according to claim 3, characterized in that, Determining the memory container type that matches each of the data structure conditions includes: When the feature data search method represented by the data structure condition is binary search, the memory container type that matches the data structure condition is determined to be an associative container; The feature data in the associated container is stored in key-value pairs.

5. The method according to claim 4, characterized in that, The memory space structure of the associated container includes a data storage area, an index information area, a metadata area, and a footer information area; The feature data based on the same feature category, according to the data conditions corresponding to each block in the memory space structure of the target memory container, respectively constructs the data to be stored for each block, including: Each feature data belonging to the same feature category is used as the value in a key-value pair to construct a key-value pair as the data to be stored in the data storage area. Based on the key in each key-value pair and the storage location information corresponding to each key, index information is constructed as the data to be stored in the index information area; Based on the data type information of the key-value pairs, metadata is constructed as the data to be stored in the metadata area; Based on the legal identification information of the associated container, the first storage location information corresponding to the metadata area, and the second storage location information corresponding to the index information area, footer information is constructed as the data to be stored in the footer information area.

6. The method according to claim 3, characterized in that, Determining the memory container type that matches each of the data structure conditions includes: When the feature data search method represented by the data structure condition is sequential traversal, the memory container type that matches the data structure condition is determined to be a sequential container.

7. A target object analysis method based on feature data, characterized in that, The method includes: Based on the feature categories of the feature data required in the target data processing scenario, and according to the correspondence between the feature categories and memory container types, find the target memory container corresponding to each feature category of the target object; When the target memory container is a sequential container that stores feature data according to element values, for each sequential container, when the legal identifier information in the footer information of the sequential container indicates that the memory data is legal, the offset position of the feature data to be acquired in the data storage area is determined. Based on the offset position, a traversal search is performed, and the found element value is used as the feature data of the target object under the feature category corresponding to the sequential container; From each of the target memory containers, read the feature data that meets the data structure conditions corresponding to the target data processing scenario; The target object is analyzed based on the feature data to obtain the data analysis results of the target object in the target data processing scenario.

8. The method according to claim 7, characterized in that, The method further includes: When the target memory container is an associated container that stores feature data according to key-value pairs, for each associated container, when the valid identifier information in the footer information of the associated container indicates that the memory data is valid, the storage location information of the index information area is obtained from the footer information. Based on the storage location information of the index information area, a binary search is performed in the index information area to obtain the target keyword corresponding to the feature data to be acquired; Obtain the keyword location information that matches the target keyword from the index information area; Based on the keyword location information, the target key-value pair is searched from the data storage area of ​​the associated container; Based on the key-value pairs, the feature data of the target object under the feature category corresponding to the associated container is obtained.

9. A feature data storage device, characterized in that, The device includes: The category determination module is used to determine the feature category to which the feature data required for data processing of the target object belongs, based on the target data processing scenario. The structural condition determination module is used to determine the data structure conditions corresponding to each of the aforementioned feature categories in the target data processing scenario. The container type determination module is used to determine the memory container type that matches each of the data structure conditions, and to obtain the correspondence between the feature category and the memory container type; The data storage module is used to store the feature data of the target object into target memory containers of corresponding memory container types according to their respective feature categories, based on the correspondence relationship. Each block of the target memory container is written with data to be stored constructed for that block. When the memory container type is a sequential container, the feature data in the sequential container is stored according to element values. The memory space structure of the sequential container includes a data storage area, a metadata area, and a footer information area. The data to be stored in the data storage area is an element value constructed based on each feature data belonging to the same feature category. The data to be stored in the metadata area is metadata constructed based on the data type information of the element values. The data to be stored in the footer information area is footer information constructed based on the legal identification information of the sequential container and the storage location information corresponding to the metadata area.

10. The feature data storage device according to claim 9, characterized in that, The data storage module includes a target memory container module and a feature data storage module; wherein: The target memory container module is used to determine the target memory container belonging to the memory container type based on the feature data of each feature category of the target object and the memory container type corresponding to the feature category. The feature data storage module is used to store the feature data of the target feature category into the target memory container according to the memory space structure of the target memory container.

11. The feature data storage device according to claim 10, characterized in that, The feature data storage module includes a data construction module and a data writing module, wherein: The data to be stored construction module is used to construct the data to be stored for each block based on the feature data of the same feature category and according to the data conditions corresponding to each block in the memory space structure of the target memory container. The data writing module is used to write the data to be stored into each block of the target memory container.

12. The feature data storage device according to claim 11, characterized in that, The container type determination module is further configured to determine the memory container type matching the data structure condition as an associative container when the feature data search method represented by the data structure condition is a binary search; the feature data in the associative container is stored according to key-value pairs.

13. The feature data storage device according to claim 12, characterized in that, The memory space structure of the associated container includes a data storage area, an index information area, a metadata area, and a footer information area; the data to be stored construction module is also used to construct key-value pairs as the data to be stored in the data storage area by taking each feature data belonging to the same feature category as the value in the key-value pair; Based on the key in each key-value pair and the storage location information corresponding to each key, index information is constructed as the data to be stored in the index information area; Based on the data type information of the key-value pairs, metadata is constructed as the data to be stored in the metadata area; Based on the legal identification information of the associated container, the first storage location information corresponding to the metadata area, and the second storage location information corresponding to the index information area, footer information is constructed as the data to be stored in the footer information area.

14. The feature data storage device according to claim 11, characterized in that, The container type determination module is further configured to determine that the memory container type matching the data structure condition is a sequential container when the feature data search method represented by the data structure condition is sequential traversal.

15. A target object analysis device based on feature data, characterized in that, The device includes: The container lookup module is used to find the target memory container corresponding to each feature category of the target object based on the feature categories of the feature data required in the target data processing scenario and according to the correspondence between the feature categories and memory container types. The feature reading module is used to determine the offset position of the feature data to be acquired in the data storage area when the target memory container is a sequential container that stores feature data according to element values. For each sequential container, if the legal identifier information in the footer information of the sequential container indicates that the memory data is legal, the module performs a traversal search based on the offset position and uses the found element value as the feature data of the target object under the feature category corresponding to the sequential container. It also reads feature data that meets the data structure conditions corresponding to the target data processing scenario from each target memory container. The data analysis module is used to analyze the target object based on the feature data to obtain the data analysis results of the target object in the target data processing scenario.

16. The target object analysis device based on feature data according to claim 15, characterized in that, The feature reading module is also used to obtain the storage location information of the index information area from the footer information when the target memory container is an associated container that stores feature data according to key-value pairs. Based on the storage location information of the index information area, a binary search is performed in the index information area to obtain the target keyword corresponding to the feature data to be acquired; Obtain the keyword location information that matches the target keyword from the index information area; Based on the keyword location information, the target key-value pair is searched from the data storage area of ​​the associated container; Based on the key-value pairs, the feature data of the target object under the feature category corresponding to the associated container is obtained.

17. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

18. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images