Data reading method and device and data storage method and device

Abstract

The invention provides a data reading method and device, and a data storage method and device, which are applied to various scenes such as cloud technology, and the method comprises the following steps: in response to a service reading request carrying target reading protocol data, obtaining at least two pieces of preset writing protocol data; determining first target write protocol data to which each piece of target feature data belongs; determining a database corresponding to the first target write protocol data based on preset mapping information to obtain a first target database corresponding to each piece of target feature data; obtaining a reading result corresponding to each piece of target feature data from a first target database corresponding to each piece of target feature data; and generating a feature data reading result according to the reading result corresponding to each piece of target feature data. According to the embodiment of the invention, the feature data can be pulled as required, and the consumption of network broadband resources from a database to a business reading service is reduced; and different databases are adopted for storage according to the characteristics of the feature data, so that the consumption of memory resources and the cost of accurate recommendation are reduced.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and specifically relates to a data reading method, a data storage method and a device. Background Art

[0002] In the recommendation scenario, the ranking database usually refers to the key-value database. A key-value database is a database that stores data in key-value pairs, where key refers to the keyword and value refers to the value.

[0003] The read-write service in the related art usually uses the identity document (ID) of the positive row data as the key and the full positive row fields of the positive row data as the value to request the full positive row fields of the positive row database. Then the read-write service selects the necessary fields from the requested full positive row fields. This reading method causes the positive row database and the read-write service to consume more network bandwidth resources. In addition, the positive row database in the related art usually adopts a storage database. In order to meet the massive positive row reading and writing performance requirements of the business, the related art usually adopts a pure memory remote dictionary service (Remote dictionary server, Redis) storage database. This storage method does not have the feature of adaptively distinguishing positive row features, resulting in more memory resource consumption, thereby increasing the data reading and writing costs. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a data reading method, a data storage method and a device.

[0005] On the one hand, the present application proposes a data reading method, the method comprising:

[0006] In response to a service read request carrying target read protocol data, at least two preset write protocol data are obtained; each of the preset write protocol data includes preset feature data; the target read protocol data includes at least one target feature data, and the at least one target feature data is obtained from the preset feature data included in each of the preset write protocol data;

[0007] Determine, from the at least two preset write protocol data, first target write protocol data to which each target characteristic data belongs;

[0008] Determine the database corresponding to the first target write protocol data based on the preset mapping information, and obtain the first target database corresponding to each of the target feature data; the preset mapping information is used to characterize the mapping relationship between each of the preset write protocol data and the database written by the included preset feature data; the database written by the included preset feature data is determined based on the generation method and / or timeliness information of the included preset feature data;

[0009] Acquire a reading result corresponding to each target feature data from a first target database corresponding to each target feature data;

[0010] According to the reading result corresponding to each of the target characteristic data, a characteristic data reading result corresponding to the service reading request is generated.

[0011] In one aspect, the present application proposes a data storage method, the method comprising:

[0012] In response to a first remote procedure call request carrying characteristic data to be written, obtaining second target write protocol data including the characteristic data to be written from at least two preset write protocol data; each of the preset write protocol data includes preset characteristic data; the preset characteristic data includes the characteristic data to be written;

[0013] Based on the second target write protocol data, the data format of the feature data to be written is processed into data in a write protocol format; and based on the preset mapping information, a second target database corresponding to the second target write protocol data is determined; the preset mapping information is used to characterize the mapping relationship between each of the preset write protocol data and the database into which the included preset feature data is written; the database into which the included preset feature data is written is determined based on the generation method and / or timeliness information of the included preset feature data;

[0014] A second remote procedure call request is sent to the second target database, wherein the second remote procedure call request carries the data in the write protocol format, so that the second target database responds to the second remote procedure call request and writes the data in the write protocol format into the second target database.

[0015] On the other hand, the present application provides a data reading device, the device comprising:

[0016] A business read request response module, configured to obtain at least two preset write protocol data in response to a business read request carrying target read protocol data; each of the preset write protocol data includes preset feature data; the target read protocol data includes at least one target feature data, and the at least one target feature data is obtained from the preset feature data included in each of the preset write protocol data;

[0017] A first target write protocol data determination module, used to determine the first target write protocol data to which each of the target characteristic data belongs from the at least two preset write protocol data;

[0018] A first target database determination module is used to determine the database corresponding to the first target write protocol data based on preset mapping information, and obtain the first target database corresponding to each of the target feature data; the preset mapping information is used to characterize the mapping relationship between each of the preset write protocol data and the database written by the included preset feature data; the database written by the included preset feature data is determined based on the generation method and / or timeliness information of the included preset feature data;

[0019] A reading result acquisition module, used for acquiring a reading result corresponding to each target feature data from a first target database corresponding to each target feature data;

[0020] The feature data reading result generating module is used to generate the feature data reading result corresponding to the service reading request according to the reading result corresponding to each target feature data.

[0021] On the other hand, the present application provides a data storage device, the device comprising:

[0022] A second target write protocol data acquisition module is used to respond to a first remote procedure call request carrying characteristic data to be written, and to acquire second target write protocol data including the characteristic data to be written from at least two preset write protocol data; each of the preset write protocol data includes preset characteristic data; and the preset characteristic data includes the characteristic data to be written;

[0023] A format processing module, used for processing the data format of the feature data to be written into data in a write protocol format based on the second target write protocol data; and determining a second target database corresponding to the second target write protocol data based on preset mapping information; the preset mapping information is used to characterize the mapping relationship between each of the preset write protocol data and the database into which the included preset feature data is written; the database into which the included preset feature data is written is determined based on the generation method and / or timeliness information of the included preset feature data;

[0024] A call request sending module is used to send a second remote procedure call request to the second target database, wherein the second remote procedure call request carries data in the write protocol format, so that the second target database responds to the second remote procedure call request and writes the data in the write protocol format into the second target database.

[0025] On the other hand, the present application proposes an electronic device for data reading or data storage, the electronic device comprising a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the data reading method or data storage method as described above.

[0026] On the other hand, the present application proposes a computer-readable storage medium, which stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the data reading method or data storage method as described above.

[0027] On the other hand, the present application proposes a computer program product, including a computer program, which implements the data reading method or data storage method as described above when executed by a processor.

[0028] The data reading method proposed in the embodiment of the present application obtains at least two preset write protocol data by responding to a business read request carrying target read protocol data; determines the first target write protocol data to which each target feature data belongs from the at least two preset write protocol data; determines the database corresponding to the first target write protocol data based on the preset mapping information, and obtains the first target database corresponding to each target feature data; obtains the reading result corresponding to each target feature data from the first target database corresponding to each target feature data; generates the feature data reading result corresponding to the business read request according to the reading result corresponding to each target feature data, thereby realizing the isolation of the read business and the underlying heterogeneous storage, reducing the access and use cost of the read business, and being able to manage the read protocol data of each business read and write service, and realizing the acquisition of the reading result corresponding to each target feature data from the first target database corresponding to each target feature data, thereby realizing the on-demand pulling of feature data from different databases, and greatly reducing the consumption of network bandwidth resources between the database and the business read service.

[0029] The data storage method proposed in the embodiment of the present application uses different databases to store feature data according to the generation method and / or timeliness information of the feature data, that is, during the writing process, the second target database corresponding to the second target write protocol data is determined based on the preset mapping information; the preset mapping information is used to characterize the mapping relationship between each preset write protocol data and the database into which the included preset feature data is written; the database into which the included preset feature data is written is determined based on the generation method and / or timeliness information of the included preset feature data, thereby realizing adaptive distinction of the generation method and / or timeliness information of the feature data in the data storage stage, reducing the consumption of memory resources in the data storage process, and thereby reducing the cost of data storage and the cost of accurate recommendation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 The invention is a system framework diagram showing a data reading and data storage system according to an exemplary embodiment.

[0032] Figure 2 The figure is a flowchart of a data storage method according to an exemplary embodiment.

[0033] Figure 3 The figure is a schematic diagram of a characteristic data writing process according to an exemplary embodiment.

[0034] Figure 4 is a flow chart of a data reading method according to an exemplary embodiment. Figure 1 .

[0035] Figure 5 is a flow chart of a data reading method according to an exemplary embodiment. Figure 2 .

[0036] Figure 6 is a flow chart of a data reading method according to an exemplary embodiment. Figure 3 .

[0037] Figure 7 is a block diagram of a data reading device according to an exemplary embodiment.

[0038] Figure 8 is a block diagram of a data reading device according to an exemplary embodiment.

[0039] Fig. 9 A hardware structure block diagram of a server provided according to an exemplary embodiment. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0042] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing. Cloud technology can mainly include security, big data, database, industry applications, network, storage, management tools and computing.

[0043] Big data refers to a collection of data that cannot be captured, managed, and processed by conventional software tools within a certain time frame. It is a massive, high-growth, and diverse information asset that requires new processing models to have stronger decision-making power, insight discovery, and process optimization capabilities. With the advent of the cloud era, big data has also attracted more and more attention. Big data requires special technologies to effectively process large amounts of data within a tolerable time frame. Technologies applicable to big data include large-scale parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems.

[0044] Figure 1 FIG. 1 is a system framework diagram of data reading and data storage according to an exemplary embodiment. Figure 1As shown, the data reading and data storage system may include at least a business reading service, a unified access layer, a database, and a business writing service. The business reading service, the unified access layer, the database, and the business writing service may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0045] Specifically, the service reading service is used to send a service reading request to the unified access layer, and receive a characteristic data reading result sent by the unified access layer.

[0046] Specifically, the unified access layer is used to respond to the service read request, pull fields from the database as needed, and obtain the characteristic data reading result corresponding to the service read request.

[0047] Specifically, the business write service is used to send a data write request to the database.

[0048] Specifically, the database is used to write feature data in response to the data write request. Exemplarily, the database may include a memory database based on a double-buffered object (a FasterKV database based on a double-buffered pure memory), a database based on a hybrid storage of memory and a solid-state drive (SSD) (a YottaDB database based on a two-level cache of memory + SSD), and a memory-based key-value database (a Redis-based database). Redis refers to a remote dictionary service, and its full English name is Remote Dictionary Server.

[0049] The memory database based on the double cache object is used to store batch update feature data, which may include statistical data, other dictionary data, etc. The statistical data refers to the data obtained by the distributed timing statistics of the spark platform, and the other dictionary data refers to the data obtained by the distributed machine learning model of the spark platform. Spark is a fast and general computing engine designed for large-scale data processing.

[0050] The database based on hybrid storage of memory and solid state drive is used to store streaming update feature data. The streaming update feature data may include basic audit data, other streaming update fields, etc.

[0051] The memory-based key-value database is used to store real-time feature data, which may include real-time statistical data, other real-time data, etc.

[0052] Specifically, the business writing service may include a writing component, and the writing component may include a streaming writing central control, a distributed task platform component, and the like.

[0053] Exemplarily, the business reading service and business writing service may include: smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, etc., but are not limited thereto. The business reading service and business writing service may also be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0054] It should be noted that Figure 1 This is just an example. In other scenarios, other system framework diagrams may also be included.

[0055] Figure 2 is a flow chart of a data storage method according to an exemplary embodiment. The method can be used for Figure 1 In the system framework diagram in . This specification provides the method operation steps as described in the embodiments or flowcharts, but may include more or fewer operation steps based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or server product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 2 As shown, the method may include:

[0056] S101. In response to a first remote procedure call request carrying characteristic data to be written, obtain second target write protocol data including the characteristic data to be written from at least two preset write protocol data; each of the preset write protocol data includes preset characteristic data; the preset characteristic data includes the characteristic data to be written.

[0057] It should be noted that the execution entity of steps S101 to S105 can be Figure 1 The write component in the business write service.

[0058] Optionally, the first remote procedure call request may be sent by a business write service. The full name of the remote procedure call request is Remote Procedure Call, abbreviated as RPC. Exemplarily, the business write service may be various types of services, which may include but are not limited to: business streaming data processing services, object storage services (COS services), distributed file systems (Hadoop Distributed File System, HDFS) services, etc.

[0059] Optionally, the feature data to be written and the preset feature data can be forward-row feature data. The forward-row feature data is a concept relative to the reverse-row feature data, and the forward-row feature data stores various attribute characteristics of the material (i.e., forward-row data). Exemplarily, the feature data to be written and the preset feature data can be the forward-row data itself generated by the business streaming data processing service, COS service, HDFS service, etc., such as the title, author, release time, etc. Alternatively, the feature data to be written and the preset feature data can be the identification information of the forward-row data generated by the business streaming data processing service, COS service, HDFS, etc., such as the id of the forward-row data (video_id, etc.), the length of the forward-row data (title_length), etc. In other embodiments, in some scenarios, the feature data to be written and the preset feature data can also be reverse-row feature data.

[0060] Optionally, the at least two preset write protocol data are pre-registered by the client account on the regular scheduling management platform, which is a business protocol management platform that manages the full preset write protocol data and read-write service preset write protocol data of the business. Among them, the full preset write protocol data and the read-write service preset write protocol data are facing different objects and processes. The full preset write protocol is that the data occurs in the regular scheduling production link of the business. The business can have multiple regular scheduling field production services. One production service corresponds to one write protocol data, and the field names between different write protocol data will not be repeated; the read-write service preset write protocol data occurs in the business read service link. Similarly, the business has multiple read services, and one business read service corresponds to one read protocol data, wherein the fields in the read protocol data can come from multiple preset write protocol data.

[0061] Optionally, each preset write protocol data includes a field and a field type, the field may refer to the preset feature data, and the field type may refer to the type of the preset feature data. Exemplarily, the format of the preset write protocol data may be as follows, which may include 6 fields and types:

[0062]

[0063] Among them, video_id refers to the video document id, that is, the material id, title_length refers to the title length, run_time refers to the running time, publish_time refers to the publishing time, weishi_hq_mini_video refers to the satellite TV cloud broadcast, media_id refers to the media number, int64 refers to the signed 64-bit integer data type, int refers to the integer type, and string refers to the string type. It should be noted that the format of the above-mentioned preset write protocol data is only an example. In other implementations, the preset write protocol data may also include other numbers of fields and types.

[0064] Since each preset write protocol data includes preset characteristic data and its corresponding type; each preset write protocol data includes the characteristic data to be written in the preset characteristic data, the write component can obtain the second target write protocol data including the characteristic data to be written from at least two preset write protocol data in response to the first remote procedure call request. For example, at least two preset write protocol data include preset write protocol data 1, preset write protocol data 2, and preset write protocol data 3, the preset characteristic data is field A, and preset write protocol data 2 includes field A and its type, then preset write protocol data 2 is considered to be the second target write protocol data.

[0065] S103. Based on the second target write protocol data, the data format of the feature data to be written is processed into data in a write protocol format; and based on the preset mapping information, a second target database corresponding to the second target write protocol data is determined; the preset mapping information is used to characterize the mapping relationship between each preset write protocol data and the database into which the included preset feature data is written; the database into which the included preset feature data is written is determined based on the generation method and / or timeliness information of the included preset feature data.

[0066] Optionally, when a client account registers preset write protocol data on the forwarding management platform, it can indicate the forwarding database corresponding to the preset write protocol data, that is, establish mapping information between each preset write protocol data and the database into which the included preset feature data is written.

[0067] For various feature data (e.g., front row feature data) of materials in information flow and small world business scenarios (e.g., small world short video business), there is no requirement for updating timeliness. Before establishing mapping information, the database to which each preset feature data is written can be determined according to the generation method and / or timeliness information of the preset feature data. The following is an introduction to the process of determining the database to which each preset feature data is written:

[0068] First, various preset feature data of materials in the recommendation scenario have different production methods and timeliness information. According to the production method and / or timeliness information of the preset feature data, the preset feature data can be divided into batch update feature data, streaming update feature data, real-time feature data, etc. The embodiment of the present application can select the underlying storage database according to the characteristics of the production method and / or timeliness information of the preset feature data.

[0069] Batch update of feature data: This type of data is generally data generated regularly by a preset platform. For example, a machine learning model is regularly trained to produce output. By deploying the model on the spark framework, the full amount of output material positive row data can be generated, which may include but is not limited to features such as labels and semantic relevance. For batch updates of this type of data, the underlying storage selection of the embodiment of the present application is based on a dual-buffer pure memory FasterKV database. Taking the batch update of feature data as an example of batch update of positive row feature data, the positive row data is managed by the version number of the positive row feature data currently stored in FasterKV. During the version switching process, the old and new versions will coexist. The old version of the data will not be taken offline until the new version is successfully loaded into the memory. It should be noted that FasterKV generally handles positive row read requests. Under normal circumstances, only one cache (buffer) object is required. When switching the index version, two buffer objects need to be prepared. Since FasterKV generally processes positive read requests, while existing pure memory methods process both positive read and write requests, existing solutions require more central processing unit (CPU) resources, where the unit CPU resource cost is more than 10 times that of memory. Therefore, the FasterKV database based on double-buffer pure memory can reduce the consumption of memory resources during data storage.

[0070] Streaming updated feature data: This type of data is generally the positive feature data with low update frequency and low timeliness in the business scenario, that is, data with timeliness information less than the preset timeliness threshold, such as title, author, publication time, etc. For this type of data, the underlying storage selection of the embodiment of the present application is a YottaDB database based on a two-level cache of memory + SSD. When reading feature data from the database, it is first determined whether the memory database is hit. If it is hit, the result is returned directly, otherwise the SSD database is continued to be requested. The design of this database greatly reduces memory overhead and reduces machine costs.

[0071] Real-time characteristic data: This type of data is generally generated by the click flow log of items on the recommendation link, that is, data with time-effectiveness information greater than or equal to the preset time-effectiveness threshold, such as the number of clicks, the number of likes, the number of user comments, etc. This type of data has high real-time requirements, and there are massive flow logs in browsers and small-world services, so this type of data requires high update and write forward storage requirements. For this type of high real-time data, the underlying storage selection of the embodiment of the present application is based on a Redis database. The storage needs to handle the high read query rate per second (Queries-per-second, QPS) of the recommendation system and the high write QPS of material updates at the same time. The storage is a read-and-write scenario, so high-performance read-write storage such as Redis is required.

[0072] In this way, the preset feature data can be classified and stored according to the production method and / or timeliness information of the preset feature data, avoiding the use of memory for unified storage, and effectively reducing the consumption of memory resources in the feature data storage process.

[0073] After determining the database into which each preset characteristic data is written, mapping information between each preset writing protocol data and the database into which the included preset characteristic data is written may be established.

[0074] For example, the preset write protocol data A: (corresponding to database A)

[0075]

[0076] The preset characteristic data included in the preset write protocol data A are a_1, a_2, and a_3, and mapping information between the preset write protocol data and the database A written by a_1, a_2, and a_3 can be established.

[0077] Default write protocol B: (corresponding to database B)

[0078]

[0079] The preset characteristic data included in the preset write protocol data B are b_1, b_2, and b_3, and mapping information between the preset write protocol data and the database B written by b_1, b_2, and b_3 can be established.

[0080] It should be noted that the aforementioned FasterKV and streaming storage YottaDB can also be replaced by other open source Key-Value and NoSQL databases, such as Rocksdb, LevelDB and other open source solutions. Among them, NoSQL database refers to a non-relational database, Rocksdb is a storage engine library of a key-value storage interface, and LevelDB is an efficient key-value embedded database.

[0081] Accordingly, on the one hand, in the above step S103, since the pre-mapping information for characterizing the mapping relationship between each preset write protocol data and the database into which the preset characteristic data included is written is pre-established, the write component can determine the database into which the preset characteristic data included in the second target write protocol data is written according to the preset mapping information, and obtain the second target database. For example, if the second target write protocol data is the write protocol YottaDB, then the second target database is YottaDB. On the other hand, the write component can process the data format of the characteristic data to be written into data in the write protocol format based on the second target write protocol data, that is, process it into a format corresponding to the second target write protocol data.

[0082] S105. Send a second remote procedure call request to the second target database, the second remote procedure call request carrying data in a write protocol format, so that the second target database responds to the second remote procedure call request and writes the data in the write protocol format into the second target database.

[0083] In this embodiment, after obtaining the data in the write protocol format, the write component can send an RPC request to the corresponding second target database for the output data, and the second target database responds to the second remote procedure call request and writes the data in the write protocol format to the second target database.

[0084] Therefore, different databases can be used to store feature data according to the generation method and / or timeliness information of the feature data, that is, during the writing process, the second target database corresponding to the second target write protocol data is determined based on the preset mapping information; the preset mapping information is used to characterize the mapping relationship between each preset write protocol data and the database into which the included preset feature data is written; the database into which the included preset feature data is written is determined based on the generation method and / or timeliness information of the included preset feature data, thereby realizing adaptive distinction of the generation method and / or timeliness information of the feature data in the data storage stage, reducing the consumption of memory resources in the data storage process, and thereby reducing the cost of data storage and the cost of accurate recommendation.

[0085] Optionally, the above step S101 may be implemented in a variety of ways, which are not specifically limited here. In one embodiment, in the above step S101, the above-mentioned obtaining the second target write protocol data including the positive row feature data to be written from at least two preset write protocol data may include:

[0086] The preset characteristic data included in each preset write protocol data is determined.

[0087] From the preset characteristic data included in each write protocol data, the preset characteristic data corresponding to the positive row characteristic data to be written is obtained.

[0088] The preset write protocol data to which the preset characteristic data corresponding to the positive row characteristic data to be written belongs is determined as the second target write protocol data.

[0089] In this embodiment, the writing component may first determine at least one preset characteristic data included in each preset write protocol data, and the writing component may determine the preset characteristic data identical to the preset characteristic data from the at least one preset characteristic data included in each preset write protocol data, and determine the preset write protocol data to which the preset characteristic data identical to the positive row characteristic data to be written belongs as the second target write protocol data. That is, the writing component determines the preset write protocol data including the preset characteristic data from each preset write protocol data, and obtains the second target write protocol data. Since the preset write protocol data includes the preset characteristic data and its type, the second target write protocol data of the positive row characteristic data to be written is determined according to the preset characteristic data included in the preset write protocol data, which can improve the accuracy of determining the second target write protocol data.

[0090] In another embodiment, the writing component can also extract feature information of the positive row feature data to be written, and extract feature information of preset feature data included in each preset write protocol data, calculate the characteristic similarity between the feature information of the positive row feature data to be written and the feature information of the preset feature data included in each preset write protocol data, and use the preset write protocol data to which the preset feature data belongs whose feature similarity with the feature information of the positive row feature data to be written meets the preset similarity threshold as the second target write protocol data.

[0091] Optionally, the above step S103 of "processing the data format of the to-be-written characteristic data into data in a write protocol format based on the second target write protocol data" can be implemented in a variety of ways, which are not specifically limited here. In one embodiment, the above-mentioned processing the data format of the to-be-written characteristic data into data in a write protocol format based on the second target write protocol data includes:

[0092] Based on the second target write protocol data, the data format of the feature data to be written is serialized and processed into binary data to obtain data in a write protocol format.

[0093] Exemplarily, the feature data to be written can be serialized into buffer binary data in a flatbuffer format according to the second target write protocol data to obtain data in a write protocol format, wherein a flatbuffer is a binary buffer that uses an offset address to organize nested objects.

[0094] For example, the second target write protocol data is {"video_id":123,"title_length":10,"run_time":1800,"publish_time":20230619,"weishi_hq_mini_video":1,"media_id":"a123"}, and the write control can serialize the feature data to be written into buffer binary data in flatbuffer format according to the second target write protocol data.

[0095] Since the network protocol is based on binary, based on the second target writing protocol data, the data format of the feature data to be written is serialized into binary data, which can improve the reading and writing convenience and efficiency of the feature data to be written.

[0096] In another embodiment, a data format conversion model may be pre-trained, and the data format conversion model is used to convert the data format obtained from the feature data into data in a write protocol format in the case of a known write protocol data. After the data format conversion model is trained, the write component may use the second target write protocol data as the known write protocol data, and input the second target write protocol data and the feature data to be written into the data format conversion model, so as to serialize the data format of the feature data to be written into binary data through the data format conversion model, and obtain data in a write protocol format.

[0097] Optionally, the "determining the second target database corresponding to the second target write protocol data based on the preset mapping information" in the above step S103 can be implemented in a variety of ways, which are not specifically limited here. In one embodiment, the above determination of the second target database corresponding to the second target write protocol data based on the preset mapping information includes:

[0098] Determine the preset positive row data and at least two databases included in the second target write protocol data; the at least two databases are determined based on the generation method and / or timeliness information of the preset feature data included in each preset write protocol data.

[0099] Based on the preset mapping information, the database into which the preset positive row data included in the second target write protocol data is written is obtained from at least two databases to obtain a second target database.

[0100] In this embodiment, at least two databases can be determined based on the generation method and / or timeliness information of the preset characteristic data included in each preset write protocol data, and the at least two databases include business streaming data processing services, COS services, HDFS services, etc.

[0101] Since the preset mapping information is used to characterize the mapping relationship between each preset write protocol data and the database into which the preset characteristic data is written, after determining the preset positive row data included in the second target write protocol data, the database into which the preset positive row data included in the second target write protocol data is written can be obtained from the at least two databases according to the preset mapping information to obtain the second target database. This improves the accuracy of determining the second target database.

[0102] In another embodiment, the preset mapping information can also be directly used to characterize the mapping relationship between each preset write protocol data and the corresponding database, that is, the mapping relationship between each preset write protocol data and the corresponding database is pre-established to obtain the preset mapping information. The write component can directly obtain the second target database corresponding to the second target write protocol data according to the preset mapping information.

[0103] In an optional embodiment, the feature data to be written is streaming update feature data, the second target database is a database based on hybrid storage of memory and solid state drive, and the method further includes:

[0104] Based on the streaming write control, in response to the first remote procedure call request sent by the business streaming data processing service, the second target write protocol data is obtained from at least two preset write protocol data.

[0105] Based on the streaming write central control, the data format of the feature data to be written is processed into data in a write protocol format through the second target write protocol data; and based on the preset mapping information, a database based on hybrid storage of memory and solid state drive is determined.

[0106] Based on streaming write, the central control sends a second remote procedure call request to the database based on hybrid storage of memory and solid-state drive, so that the database based on hybrid storage of memory and solid-state drive responds to the second remote procedure call request and writes the data in the write protocol format to the database based on hybrid storage of memory and solid-state drive.

[0107] Figure 3 FIG. 1 is a schematic diagram of a characteristic data writing process according to an exemplary embodiment. Figure 3As shown, for streaming update feature data, the front row configuration management central control service can pull at least two preset write protocol data pre-registered by the client account from the front row management platform. The business streaming data processing service sends an RPC request carrying streaming update feature data to the streaming write central control, and the streaming write central control responds to the RPC to obtain the second target write protocol data of the feature data to be written from at least two preset write protocol data of the front row configuration management central control. Among them, for the process of obtaining at least two preset write protocol data and the second target write protocol data, please refer to the above step S101, which will not be repeated here. Based on the second target write protocol data, the streaming write central control processes the data format of the feature data to be written into data in the write protocol format, and determines the database based on the hybrid storage of memory and solid-state hard disk corresponding to the second target write protocol data based on the preset mapping information. Among them, for the data format processing and the determination process of the database based on the hybrid storage of memory and solid-state hard disk, please refer to the above step S103, which will not be repeated here. The streaming write control sends an RPC to the database based on the hybrid storage of memory and solid-state hard disk, so that the database based on the hybrid storage of memory and solid-state hard disk responds to the RPC and writes the data in the write protocol format to the database based on the hybrid storage of memory and solid-state hard disk. Among them, please refer to the above-mentioned step S105 for the data writing process, which will not be repeated here. Therefore, the second target write protocol data of the streaming update feature data can be accurately obtained from the positive row configuration management control through the streaming write control, and the streaming update feature data can be accurately written into the database corresponding to the second target write protocol data. According to the production method and timeliness information of the streaming update feature data, the streaming update feature data can be written into the database based on the hybrid storage of memory and solid-state hard disk, thereby realizing the classification and storage of feature data according to the characteristics of the feature data, avoiding the defect of large memory resource consumption caused by single pure memory storage of feature data.

[0108] In other embodiments, the streaming write central control may also directly obtain the second target write protocol data of the streaming update characteristic data from the forward management platform.

[0109] In some optional embodiments, the feature data to be written is real-time feature data, the second target database is a memory-based key-value database, and the method further includes:

[0110] Based on the streaming write central control responding to the first remote procedure call request sent by the object storage service, obtaining second target write protocol data from at least two preset write protocol data;

[0111] Based on the streaming write central control, the data format of the feature data to be written is processed into data in a write protocol format through the second target write protocol data; and based on the preset mapping information, a memory-based key-value database is determined;

[0112] Based on the streaming write, the central control sends a second remote procedure call request to the key-value database based on the memory, so that the key-value database based on the memory responds to the second remote procedure call request and writes the data in the write protocol format into the key-value database based on the memory.

[0113] Continue as Figure 3 As shown, for the real-time characteristic data, the forward configuration management central control service can pull at least two preset write protocol data pre-registered by the client account from the forward management platform. The object storage service sends an RPC request carrying the real-time characteristic data to the streaming write central control, and the streaming write central control responds to the RPC to obtain the second target write protocol data of the real-time characteristic data from the at least two preset write protocol data of the forward configuration management central control. Among them, for the process of obtaining at least two preset write protocol data and the second target write protocol data, please refer to the above step S101, which will not be repeated here. Based on the second target write protocol data, the streaming write central control processes the data format of the real-time characteristic data into data in the write protocol format, and determines the memory-based key-value database corresponding to the second target write protocol data based on the preset mapping information. Among them, for the data format processing and the determination process of the database based on the hybrid storage of memory and solid-state hard disk, please refer to the above step S103, which will not be repeated here. The streaming write control sends an RPC to the memory-based key-value database, so that the memory-based key-value database responds to the RPC and writes the data in the write protocol format to the memory-based key-value database. For the data writing process, please refer to the above step S105, which will not be repeated here. Thus, the second target write protocol data of the real-time feature data can be accurately obtained from the positive row configuration management control through the streaming write control, and the real-time feature data can be accurately written into the memory-based key-value database corresponding to the second target write protocol data. The real-time feature data can be written into the memory-based key-value database according to the production method and timeliness information of the real-time feature data, thereby realizing the classification and storage of the feature data according to the characteristics of the feature data, avoiding the defect of large memory resource consumption caused by the single memory storage of the positive row data.

[0114] Continue as Figure 3 As shown, the object storage service can subscribe to the COS service for the main row. The main row subscription COS service can regularly pull feature data from the business COS service, and the business COS service can pull data from the full business data.

[0115] In other embodiments, the streaming write central control can also directly obtain the second target write protocol data of the real-time characteristic data from the forward management platform.

[0116] In some optional embodiments, the feature data to be written is batch update feature data, the second target database is a memory database based on a double cache object, and the method further includes:

[0117] Based on the distributed task platform component responding to the first remote procedure call request sent by the distributed file system service, obtaining the second target write protocol data from at least two preset write protocol data controlled in the positive row configuration management center;

[0118] Based on the distributed task platform component, the data format of the feature data to be written is processed into data in a write protocol format through the second target write protocol data; and based on the preset mapping information, a memory database based on a double cache object corresponding to the second target write protocol data is determined;

[0119] The distributed task platform component sends a second remote procedure call request to the memory database based on the double cache object, so that the memory database based on the double cache object responds to the second remote procedure call request and writes the data in the write protocol format into the memory database based on the double cache object.

[0120] Continue as Figure 3As shown, for batch update feature data, the front row configuration management central control service can pull at least two preset write protocol data pre-registered by the client account from the front row management platform. HDFS can send an RPC request carrying batch update feature data to the distributed task platform component (Venus batch build component), and the distributed task platform component responds to the RPC to obtain the second target write protocol data of the batch update feature data from at least two preset write protocol data of the front row configuration management central control. Among them, for the process of obtaining at least two preset write protocol data and the second target write protocol data, please refer to the above step S101, which will not be repeated here. Based on the second target write protocol data, the distributed task platform component processes the data format of the batch update feature data into data in the write protocol format, and determines the memory database based on the double cache object corresponding to the second target write protocol data based on the preset mapping information. Among them, for the data format processing and the determination process of the database based on the hybrid storage of memory and solid-state hard disk, please refer to the above step S103, which will not be repeated here. The distributed task platform component sends an RPC to the memory database based on the double-buffered object, so that the memory database based on the double-buffered object responds to the RPC and writes the data in the write protocol format to the memory database based on the double-buffered object. Among them, please refer to the above step S105 for the data writing process, which will not be repeated here. Therefore, the distributed task platform component can accurately obtain the second target write protocol data of the batch update feature data from the positive row configuration management central control, and accurately write the batch update feature data to the memory database based on the double-buffered object corresponding to the second target write protocol data. The batch update feature data can be written to the memory database based on the double-buffered object according to the production method and timeliness information of the batch update feature data, thereby realizing the classified storage of feature data and avoiding the defect of large memory resource consumption caused by the single memory storage of the positive row data. Among them, the Venus batch construction component is a one-stop machine learning platform that provides users with the ability to manage processing, analysis and exploration around data and AI, and opens up the whole process of exploration experiment → feature engineering → model training → model launch.

[0121] In other embodiments, the distributed task platform component may also directly obtain the second target write protocol data of the real-time characteristic data from the forward management platform.

[0122] The following describes the feature data reading process. During the feature data reading process, the embodiment of the present application proposes a unified access layer. The unified access layer can mainly isolate the business and the underlying heterogeneous storage, reduce the cost of business access, and manage the business reading protocols of each service through the unified access layer. It can realize on-demand pulling of positive-ranking fields, which greatly reduces the network bandwidth between the underlying heterogeneous storage and the business service. Figure 4 is a flow chart of a data reading method according to an exemplary embodiment. Figure 1 ,like Figure 4 As shown, the data reading method may include:

[0123] S201. In response to a business read request carrying target read protocol data, obtain at least two preset write protocol data; each preset write protocol data includes preset feature data; the target read protocol data includes at least one target feature data, and the at least one target feature data is obtained from the preset feature data included in each preset write protocol data.

[0124] Figure 5 is a flow chart of a data reading method according to an exemplary embodiment. Figure 2 .like Figure 5 As shown, the business read service can send a business read request carrying target read protocol data to the unified access layer. The unified access layer responds to the business read request and obtains at least two preset write protocol data pre-registered by the client account on the regular management platform. Each preset write protocol data includes preset feature data and the type of preset feature data.

[0125] Among them, one business read service corresponds to one read protocol data. Different read protocol data often have different characteristic data, and the number of characteristic data that each business read service needs to obtain is also different, that is, the target read protocol data may include at least one target characteristic data. Specifically, at least one target characteristic data is obtained from the preset characteristic data included in each preset write protocol data, that is, at least one target characteristic data included in a target read protocol data can be derived from at least one preset write protocol data including preset characteristic data. For example,

[0126] The preset write protocol data A is as follows:

[0127]

[0128] The preset write protocol data B is as follows:

[0129]

[0130] The target read protocol data is:

[0131]

[0132]

[0133] It can be seen that at least one target characteristic data (a_1, b_2) included in the target read protocol data is derived from the preset write protocol data A and the preset write protocol data B respectively.

[0134] It should be noted that the target feature data may be forward-row feature data, and the target feature data may be forward-row data itself generated by a business streaming data processing service, COS service, HDFS service, etc., such as title, author, release time, etc. Alternatively, the target feature data may be identification information of forward-row data generated by a business streaming data processing service, COS service, HDFS, etc., such as the id of the forward-row data (video_id, etc.), the length of the forward-row data (title_length), etc. In other implementations, in certain scenarios, the target feature data may also be reverse-row feature data.

[0135] S203. Determine the first target write protocol data to which each target characteristic data belongs from at least two preset write protocol data.

[0136] Since each target characteristic data included in the target read protocol data may be derived from at least one preset write protocol data, the unified access layer may determine the first target write protocol data to which each target characteristic data belongs from at least two preset write protocol data.

[0137] For example, the above-mentioned target read protocol data includes at least one target characteristic data (a_1, b_2), a_1 comes from the preset write protocol data A, and b_2 comes from the preset write protocol data B. Then the first target write protocol data belonging to a_1 is the preset write protocol data A, and the first target write protocol data belonging to b_2 is the preset write protocol data B.

[0138] S205. Determine the database corresponding to the first target write protocol data based on the preset mapping information, and obtain the first target database corresponding to each target feature data; the preset mapping information is used to characterize the mapping relationship between each preset write protocol data and the database into which the included preset feature data is written; the database into which the included preset feature data is written is determined based on the generation method and / or timeliness information of the included preset feature data.

[0139] In an embodiment of the present application, since pre-mapping information is pre-established to characterize the mapping relationship between each preset write protocol data and the database into which the included preset feature data is written, the unified access layer can first determine the preset feature data included in the first target write protocol data, and then the unified access layer determines the database into which the preset feature data included in the first target write protocol data is written according to the preset mapping information, and obtains the first target database corresponding to each target feature data.

[0140] For the process of establishing the above-mentioned preset mapping information and determining the database into which each preset characteristic data is written, please refer to the relevant introduction in the above-mentioned data storage method, which will not be repeated here.

[0141] For example, the target read protocol data includes at least one target characteristic data (a_1, b_2). According to the preset mapping information, the first target database corresponding to a_1 is determined to be database A, and the second target database corresponding to b_2 is determined to be preset database B.

[0142] S207. Obtain the reading result corresponding to each target feature data from the first target database corresponding to each target feature data.

[0143] S209. Generate a characteristic data reading result corresponding to the business read request according to the reading result corresponding to each target characteristic data.

[0144] Continue as Figure 5 As shown, the data reading method provided by the embodiment of the present application can provide field routing capabilities. Since the preset write protocol data and the target read protocol data are independent, the feature data included in the target read protocol data is required by the business read service. It is possible that the feature data included in the target read protocol data comes from multiple databases. Therefore, the unified access layer can have the ability of field routing, wherein the field routing capability refers to: from which databases can at least one target feature data included in the target read protocol data be obtained. In addition, the data reading method provided by the embodiment of the present application can also provide on-demand return capabilities, wherein the on-demand return capability refers to: since the required feature data of different business read services are different, the unified access layer selects the required feature data according to the preset read protocol data, and then splices and returns it to the business read service.

[0145] Optionally, in the above steps S207 to S209, based on the above field routing capability and on-demand return capability, the unified access layer can obtain the reading result corresponding to each target characteristic data from the first target database corresponding to each target characteristic data after determining the first target database corresponding to each target characteristic data. And process the reading result corresponding to each target characteristic data to generate the characteristic data reading result corresponding to the business read request. For example, the above target read protocol data includes at least one target characteristic data (a_1, b_2), and according to the above preset mapping information, it is determined that the first target database corresponding to a_1 is database A, and the first target database corresponding to b_2 is the preset database B. The access layer can obtain the reading result corresponding to a_1 from database A, obtain the reading result corresponding to b_1 from database B, and generate the characteristic data reading result corresponding to the business read request according to the reading result corresponding to a_1 and the reading result corresponding to b_1.

[0146] Therefore, by responding to a business read request carrying target read protocol data, at least two preset write protocol data are obtained; from the at least two preset write protocol data, the first target write protocol data to which each target feature data belongs is determined; based on the preset mapping information, the database corresponding to the first target write protocol data is determined to obtain the first target database corresponding to each target feature data; from the first target database corresponding to each target feature data, the read result corresponding to each target feature data is obtained; according to the read result corresponding to each target feature data, the feature data read result corresponding to the business read request is generated, thereby realizing isolation of business read services and underlying heterogeneous storage, reducing the cost of accessing and using read services, and being able to manage the read protocol data of each read and write service, realizing isolation of read services and underlying heterogeneous storage, reducing the cost of accessing and using read services, and being able to manage the read protocol data of each business read and write service, realizing acquisition of the read result corresponding to each target feature data from the first target database corresponding to each target feature data, thereby realizing on-demand pulling of feature data from different databases, greatly reducing the consumption of network broadband resources between the database and the business read service.

[0147] Optionally, the above step S203 can be implemented in a variety of ways, which are not specifically limited here. In one embodiment, the above step of determining the first target write protocol data to which each target feature data belongs from at least two preset write protocol data may include:

[0148] The preset characteristic data included in each preset write protocol data is determined.

[0149] The preset characteristic data corresponding to each target characteristic data is acquired from the preset characteristic data included in each preset write protocol data.

[0150] The preset write protocol data to which the preset characteristic data corresponding to each target characteristic data belongs is determined as the first target write protocol data.

[0151] In this embodiment, the unified access layer may first determine at least one preset characteristic data included in each preset write protocol data, and the unified access layer may determine the preset characteristic data identical to each target characteristic data from the at least one preset characteristic data included in each preset write protocol data, and determine the preset write protocol data to which the preset characteristic data identical to each target characteristic data belongs as the first target write protocol data. That is, the unified access layer determines the preset write protocol data including each target characteristic data from each preset write protocol data to obtain the first target write protocol data. Since the preset write protocol data includes the preset characteristic data and the type of the preset characteristic data, the first target write protocol data to which each target characteristic data belongs is determined according to the preset characteristic data included in the preset write protocol data, which can improve the determination accuracy of the first target write protocol data, thereby improving the recommendation accuracy.

[0152] In another embodiment, the unified access layer can also extract feature information of the target feature data, and extract feature information of the preset feature data included in each preset write protocol data, calculate the characteristic similarity between the feature information of the target feature data and the feature information of the preset feature data included in each preset write protocol data, and use the preset write protocol data to which the preset feature data whose feature similarity with the feature information of the target feature data satisfies a preset similarity threshold as the first target write protocol data.

[0153] Optionally, the above step S205 can be implemented in a variety of ways, which are not specifically limited here. In one embodiment, in the above step S205, the above-mentioned determination of the database corresponding to the first target write protocol data based on the preset mapping information to obtain the first target database corresponding to each target feature data may include:

[0154] The preset characteristic data and at least two databases included in the first target write protocol data are obtained; the at least two databases are determined based on the generation method and / or timeliness information of the preset characteristic data included in each preset write protocol data.

[0155] Based on the preset mapping information, the database into which the preset characteristic data included in the first target write protocol data is written is obtained from at least two databases to obtain a first target database corresponding to each target characteristic data.

[0156] In this embodiment, at least two databases can be determined in advance based on the generation method and / or timeliness information of the preset characteristic data included in each preset write protocol data. The at least two databases may include: a memory database based on double cache objects, a database based on hybrid storage of memory and solid-state drive, and a memory-based key-value database.

[0157] Since the preset mapping information is used to characterize the mapping relationship between each preset write protocol data and the database into which the preset feature data included is written. After determining the preset feature data included in the first target write protocol data, the database into which the preset feature data included in the first target write protocol data is written can be obtained from the at least two databases according to the preset mapping information to obtain the first target database corresponding to each target feature data. Specifically, it can be: in the case where the target feature data is batch update feature data, the first target database corresponding to the target feature data is a memory database based on a double cache object; in the case where the target feature data is streaming update feature data, the first target database corresponding to the target feature data is a database based on a hybrid storage of memory and solid-state hard disk; in the case where the target feature data is real-time feature data, the first target database corresponding to the target feature data is a memory-based key-value database.

[0158] In this way, the first target database corresponding to each target feature data can be accurately determined based on the pre-established preset mapping information and at least two databases determined according to the generation method and / or timeliness information of the preset feature data, thereby effectively improving the determination accuracy of the first target database and thus improving the recommendation accuracy.

[0159] In another embodiment, the preset mapping information can also be directly used to characterize the mapping relationship between each preset write protocol data and the corresponding database, that is, the mapping relationship between each preset write protocol data and the corresponding database is pre-established to obtain the preset mapping information. In the above step S205, the unified access layer can directly obtain the first target database corresponding to the first target write protocol data according to the preset mapping information.

[0160] It should be noted that if Figure 5 As described above, each database can be represented by a proxy server (eg, a kv proxy).

[0161] Optionally, the above step S207 can be implemented in a variety of ways, which are not specifically limited here. In one embodiment, the unified access layer can extract the feature information of the target feature data, and obtain data whose similarity between the feature information and the feature information of the target feature data meets a preset similarity threshold from the data contained in the first target database, and obtain a reading result corresponding to the target feature data.

[0162] In another embodiment, Figure 6 is a flow chart of a data reading method according to an exemplary embodiment. Figure 3 ,like Figure 6As shown, in the above step S207, the above step of obtaining the reading result corresponding to each target feature data from the first target database corresponding to each target feature data may also include:

[0163] S2071. Determine the field position information of each target characteristic data in the first target write protocol data to which it belongs.

[0164] S2073. Determine the offset address information of each target characteristic data according to the field position information; the offset address information is used to represent the number of bytes between each target characteristic data and the starting data in the first target write protocol data to which it belongs.

[0165] S2075. According to the offset address information of each target feature data, obtain the binary data of each target feature data from the first target database corresponding to each target feature data.

[0166] S2077. Generate a reading result corresponding to each target feature data according to the binary data of each target feature data.

[0167] Since the data stored in the preset database is binary data during the data storage stage. Therefore, in this embodiment, the unified access layer can parse and obtain the binary data through the addressing characteristics of the offset address (offset) of the flatbuffer. Specifically, in the above step S2071, the unified access layer can take the starting data of the first target write protocol data as the starting point, count the field number of each target feature data in the first target write protocol data to which it belongs, and obtain the field position information of each target feature data in the first target write protocol data to which it belongs, that is, the field position information refers to the field number of each target feature data in the first target write protocol to which it belongs. It should be noted that there can be multiple statistical methods, which are not specifically limited here.

[0168] In the above step S2073, as an example, the unified access layer determines the number of bytes between each target feature data and the starting data in the first target write protocol data to which it belongs, based on the field position information of each target feature data in the first target write protocol data to which it belongs, and obtains the offset address information of each target feature data. For example, the field position information of each target feature data in the first target write protocol data to which it belongs is the Ath field, and the number of bytes between the Ath field and the starting data is B, then the offset address information of each target feature data is B. As another example, in order to improve the accuracy of determining the offset address information, the unified access layer can also take the field position information as the starting point, take the first preset number of fields forward, obtain the first field position information, and take the second preset number of fields backward, obtain the second field position information. The number of bytes between the first field position information and the starting data is calculated as the first byte number, the number of bytes between the second field position information and the starting data is calculated as the second byte number, and the byte number range between the first byte number and the second byte number is used as the offset address information.

[0169] In the above steps S2075-S2077, the unified access layer obtains data corresponding to the offset address information from the first target database corresponding to each target feature data according to the offset address information of each target feature data, and obtains the binary data of each target feature data (i.e., buffer binary). The access layer uses the binary data of each target feature data as the reading result corresponding to each target feature data.

[0170] As an example, the offset address information may be determined by the following formula:

[0171] offset = (x + 2) * 2;

[0172] Here, x is the field position information, that is, the field number of the target feature data in the first target write protocol to which it belongs.

[0173] For example, the above-mentioned target read protocol data includes at least one target characteristic data (a_1, b_2), and according to the above-mentioned preset mapping information, it is determined that the first target database corresponding to a_1 is database A, and the first target database corresponding to b_2 is the preset database B. a_1 is the first field in the preset write protocol data A, and the corresponding offset = (1 + 2) * 2 = 6, that is, 6 is the offset address information of a_1 in database A, and the unified access layer finds the buffer binary corresponding to the offset address information in the database A, and obtains the binary data corresponding to a_1, that is, obtains the reading result corresponding to a_1. Similarly, the first target database corresponding to b_2 is database B, and b_2 is the second field in the preset write protocol data B, and the corresponding offset = (1 + 2) * 2 = 8, that is, 8 is the offset address information of b_2 in database B, and the unified access layer finds the buffer binary corresponding to the offset address information in the database B, and obtains the binary data corresponding to b_2, that is, obtains the reading result corresponding to b_2.

[0174] Therefore, binary data can be parsed through the addressing feature of the offset address (offset) of the flatbuffer, thereby improving the generation accuracy and efficiency of the reading results corresponding to each target feature data; in addition, the business reading service and the underlying heterogeneous storage are isolated, the access and use cost of the business reading service is reduced, and the read protocol data of each business reading service can be managed, so that the reading result corresponding to each target feature data can be obtained on demand from the first target database corresponding to each target feature data, thereby realizing on-demand pulling of feature data, which greatly reduces the consumption of network bandwidth resources between the database and the business service.

[0175] In an optional embodiment, the target feature data is streaming update feature data, and the first target database is a database based on a hybrid storage of memory and solid state drive. In the above step S2075, the above method of obtaining binary data of each target feature data from the first target database corresponding to each target feature data according to the offset address information of each target feature data includes:

[0176] According to the offset address information of each target feature data, the binary data of each target feature data is searched from the memory of the first target database corresponding to each target feature data.

[0177] When the binary data of each target feature data is not found in the memory, the binary data of each target feature data is searched from the solid state hard disk of the first target database corresponding to each target feature data.

[0178] In other embodiments, when the binary data of each target feature data is found in the memory, the operation of searching the binary data of each target feature data from the solid state disk of the first target database corresponding to each target feature data is not performed.

[0179] In this embodiment, when the target feature data is streaming update feature data, since the corresponding first target database is a database based on mixed storage of memory and solid state disk, the unified access layer can first check in the memory database to determine whether the binary data of each target feature data can be found in the memory database. If so, the found binary data is directly returned. If not, the unified access layer continues to request the solid state disk database to find the binary data of each target feature data from the solid state disk database. As a result, the memory resource overhead is greatly reduced, and the machine cost and accurate recommendation cost are reduced.

[0180] In an optional embodiment, in the above step S209, generating the characteristic data reading result corresponding to the service reading request according to the reading result corresponding to each target characteristic data may include:

[0181] The reading results corresponding to each target feature data are spliced ​​together to obtain the feature data reading results.

[0182] As an example, the unified access layer can concatenate the reading results corresponding to each target feature data to obtain the feature data reading results. As another example, the unified access layer can concatenate the reading results corresponding to each target feature data into a buffer in the format of a flat buffer to obtain the feature data reading results corresponding to the business read request. Among them, flatbuffer is an open source protocol, and the unified access layer can use the protocol stub code to automatically convert the reading results corresponding to each target feature data into buffer binary data. As a result, the feature data pulled on demand from different databases can be concatenated into binary data in a buffer format, which improves the accuracy and efficiency of obtaining the feature data reading results corresponding to the business read request, thereby improving the recommendation accuracy and efficiency.

[0183] In an optional embodiment, in the above step S201, the above response to the service read request carrying the target read protocol data may include:

[0184] In response to the business read request sent by the business read service; the number of business read requests sent by the business read service per second is less than a preset number threshold.

[0185] In this embodiment, continue as Figure 5As shown, the above data reading method can also realize the multi-tenant flow limiting function. In order to ensure the stability of the entire positive row database, a maximum query rate per second QPS can be set for each business reading service, that is, the number of business reading requests that the business reading service can send per second.

[0186] In one implementation, in order to prevent abnormal traffic of the business reading service from causing database downtime, the QPS of the business reading service can be set to be less than a preset quantity threshold. The preset quantity threshold can be set according to actual business needs and is not specifically limited to this. For example, it can be set to 1000 / s.

[0187] In some embodiments, the number of service read requests sent by the service read service per second may also be greater than or equal to a preset number threshold.

[0188] Accordingly, after generating the characteristic data reading result corresponding to the service reading request in the above step, the above method further includes:

[0189] Send the feature data reading results to the business reading service.

[0190] In this embodiment, after the unified access layer obtains the characteristic data reading result, the characteristic data reading result can be returned to the business reading service, thereby realizing the on-demand return capability of the characteristic data.

[0191] It should be noted that the second target database in the storage process may be the same as or different from the first target database in the reading process.

[0192] Figure 7 is a block diagram of a data reading device according to an exemplary embodiment. Figure 7 As shown, the data reading device comprises:

[0193] The service read request response module 301 is used to obtain at least two preset write protocol data in response to the service read request carrying the target read protocol data; each of the preset write protocol data includes preset feature data; the target read protocol data includes at least one target feature data, and the at least one target feature data is obtained from the preset feature data included in each of the preset write protocol data;

[0194] A first target write protocol data determining module 303, configured to determine, from the at least two preset write protocol data, the first target write protocol data to which each of the target characteristic data belongs;

[0195] The first target database determination module 305 is used to determine the database corresponding to the first target write protocol data based on preset mapping information, and obtain the first target database corresponding to each of the target feature data; the preset mapping information is used to characterize the mapping relationship between each of the preset write protocol data and the database written by the included preset feature data; the database written by the included preset feature data is determined based on the generation method and / or timeliness information of the included preset feature data;

[0196] A reading result acquisition module 307 is used to acquire a reading result corresponding to each target feature data from a first target database corresponding to each target feature data;

[0197] The characteristic data reading result generating module 309 is used to generate the characteristic data reading result corresponding to the service reading request according to the reading result corresponding to each target characteristic data.

[0198] In an optional embodiment, the first target write protocol data determination module includes:

[0199] A preset characteristic data determining unit, used to determine the preset characteristic data included in each of the preset write protocol data;

[0200] A corresponding characteristic data determining unit, configured to obtain preset characteristic data corresponding to each of the target characteristic data from the preset characteristic data included in each of the preset write protocol data;

[0201] The write protocol data determination module is used to determine the preset write protocol data to which the preset characteristic data corresponding to each of the target characteristic data belongs as the first target write protocol data.

[0202] In an optional embodiment, the first target database determination module includes:

[0203] A feature database acquisition unit, used to acquire the preset feature data included in the first target write protocol data and at least two databases; the at least two databases are determined based on the generation method and / or timeliness information of the preset feature data included in each of the preset write protocol data;

[0204] The database processing unit is used to obtain the database into which the preset characteristic data included in the first target write protocol data is written from the at least two databases based on the preset mapping information, and obtain the first target database corresponding to each of the target characteristic data.

[0205] In an optional embodiment, the reading result acquisition module includes:

[0206] A field position information determining unit, used to determine the field position information of each target characteristic data in the first target write protocol data to which it belongs;

[0207] An offset address determination unit, used to determine the offset address information of each of the target characteristic data according to the field position information; the offset address information is used to characterize the number of bytes between each of the target characteristic data and the starting data in the first target write protocol data to which it belongs;

[0208] A binary data determining unit, configured to obtain binary data of each target feature data from a first target database corresponding to each target feature data according to offset address information of each target feature data;

[0209] The reading result generating unit is used to generate the reading result corresponding to each target feature data according to the binary data of each target feature data.

[0210] In an optional embodiment, the target characteristic data is at least one of batch update characteristic data, streaming update characteristic data, and real-time characteristic data; the batch update characteristic data is data generated regularly by a preset platform; the streaming update characteristic data is data whose timeliness information is less than a preset timeliness threshold; the real-time characteristic data is data whose timeliness information is greater than or equal to the preset timeliness threshold; the first target database is at least one of a memory database based on dual cache objects, a database based on hybrid storage of memory and solid-state hard disk, and a key-value database based on memory;

[0211] In the case where the target feature data is batch update feature data, the first database corresponding to the target feature data is a memory database based on a double buffer object;

[0212] In the case where the target characteristic data is streaming updated characteristic data, the first database corresponding to the target characteristic data is a database based on hybrid storage of memory and solid state drive;

[0213] In the case where the target characteristic data is real-time characteristic data, the first target database corresponding to the target characteristic data is a memory-based key-value database.

[0214] In an optional embodiment, the target characteristic data is the streaming update characteristic data, the first target database is a database based on hybrid storage of memory and solid state drive, and the binary data determination unit includes:

[0215] A first search subunit, configured to search for binary data of each target feature data from a memory of a first target database corresponding to each target feature data according to offset address information of each target feature data;

[0216] The second search subunit is used to search for the binary data of each target feature data from the solid state disk of the first target database corresponding to each target feature data when the binary data of each target feature data is not found in the memory.

[0217] In an optional embodiment, the feature data reading result generating module includes:

[0218] The splicing unit is used to splice the reading results corresponding to each of the target characteristic data to obtain the characteristic data reading results.

[0219] In an optional embodiment, the business read request response module is used to respond to the business read request sent by the business read service; the number of business read requests sent by the business read service per second is less than a preset number threshold;

[0220] Accordingly, the device further comprises:

[0221] The reading result sending module is used to send the characteristic data reading result to the business reading service.

[0222] Figure 8 is a block diagram of a data reading device according to an exemplary embodiment. Figure 8 As shown, the data reading device comprises:

[0223] A second target write protocol data acquisition module 401 is used to, in response to a first remote procedure call request carrying characteristic data to be written, acquire second target write protocol data including the characteristic data to be written from at least two preset write protocol data; each of the preset write protocol data includes preset characteristic data; the preset characteristic data includes the characteristic data to be written;

[0224] The format processing module 403 is used to process the data format of the feature data to be written into data in a write protocol format based on the second target write protocol data; and determine the second target database corresponding to the second target write protocol data based on preset mapping information; the preset mapping information is used to characterize the mapping relationship between each of the preset write protocol data and the database written by the included preset feature data; the database written by the included preset feature data is determined based on the generation method and / or timeliness information of the included preset feature data;

[0225] The call request sending module 405 is used to send a second remote procedure call request to the second target database, wherein the second remote procedure call request carries data in the write protocol format, so that the second target database responds to the second remote procedure call request and writes the data in the write protocol format into the second target database.

[0226] In an optional embodiment, the format processing module is used to serialize the data format of the feature data to be written into binary data based on the second target write protocol data to obtain data in the write protocol format.

[0227] In an optional embodiment, the feature data to be written is streaming update feature data, the second target database is a database based on hybrid storage of memory and solid state drive, and the device further includes:

[0228] A first sending module of the streaming write central control is used to obtain the second target write protocol data from at least two preset write protocol data based on the first remote procedure call request sent by the streaming write central control in response to the business streaming data processing service;

[0229] A first processing module of the streaming write central control is used to process the data format of the feature data to be written into data in a write protocol format based on the streaming write central control through the second target write protocol data; and determine the database based on the hybrid storage of memory and solid state drive based on the preset mapping information;

[0230] The second sending module of the streaming write central control is used to send the second remote procedure call request to the database based on the hybrid storage of memory and solid-state hard disk based on the streaming write central control, so that the database based on the hybrid storage of memory and solid-state hard disk responds to the second remote procedure call request and writes the data in the write protocol format into the database based on the hybrid storage of memory and solid-state hard disk.

[0231] In an optional embodiment, the feature data to be written is real-time feature data, the second target database is a memory-based key-value database, and the device further includes:

[0232] A third sending module of the streaming write central control is used to obtain the second target write protocol data from at least two preset write protocol data based on the streaming write central control responding to the first remote procedure call request sent by the object storage service;

[0233] A second processing module of the streaming write central control is used to process the data format of the feature data to be written into data in a write protocol format based on the streaming write central control through the second target write protocol data; and determine the memory-based key-value database based on the preset mapping information;

[0234] The fourth sending module of the streaming write central control is used to send a second remote procedure call request to the memory-based key-value database based on the streaming write central control, so that the memory-based key-value database responds to the second remote procedure call request and writes the data in the write protocol format into the memory-based key-value database.

[0235] In an optional embodiment, the feature data to be written is batch update feature data, the second target database is a memory database based on a double buffer object, and the device further includes:

[0236] A component first sending module, configured to obtain the second target write protocol data from at least two preset write protocol data based on the first remote procedure call request sent by the distributed task platform component in response to the distributed file system service;

[0237] A component processing module, configured to process the data format of the feature data to be written into data in a write protocol format based on the distributed task platform component through the second target write protocol data; and determine a memory database based on a double cache object corresponding to the second target write protocol data based on the preset mapping information;

[0238] The component second sending module is used to send a second remote procedure call request to the memory database based on the double-cached object based on the distributed task platform component, so that the memory database based on the double-cached object responds to the second remote procedure call request and writes the data in the write protocol format into the database based on the hybrid storage of the memory and disk.

[0239] It should be noted that the device embodiment provided in the embodiments of the present application and the above-mentioned method embodiment are based on the same inventive concept.

[0240] An embodiment of the present application also provides an electronic device for reading data, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement a data reading method provided in any of the above embodiments.

[0241] An embodiment of the present application also provides an electronic device for data storage, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement a data storage method provided in any of the above embodiments.

[0242] An embodiment of the present application also provides a computer-readable storage medium, which can be set in a terminal to store at least one instruction or at least one program for implementing a method in a method embodiment, and the at least one instruction or at least one program is loaded and executed by a processor to implement a data reading method or a data storage method provided in the above method embodiment.

[0243] Optionally, in the embodiment of this specification, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the above storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0244] The memory of the embodiment of this specification can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, applications required for functions, etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0245] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the data reading method or data storage method provided by the above method embodiment.

[0246] The data reading method or data storage method provided in the embodiments of the present application can be executed in a terminal, a computer terminal, a server or a similar computing device. Taking running on a server as an example, Fig. 9 FIG. 1 is a hardware structure diagram of a server provided according to an exemplary embodiment. Fig. 9As shown, the server 500 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 510 (the central processing unit 510 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 530 for storing data, and one or more storage media 520 (such as one or more mass storage devices) for storing application programs 523 or data 522. Among them, the memory 530 and the storage medium 520 can be short-term storage or permanent storage. The program stored in the storage medium 520 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 510 can be configured to communicate with the storage medium 520 and execute a series of instruction operations in the storage medium 520 on the server 500. The server 500 may also include one or more power supplies 560, one or more wired or wireless network interfaces 550, one or more input and output interfaces 540, and / or one or more operating systems 521, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0247] The input / output interface 540 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 500. In one example, the input / output interface 540 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 540 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0248] It can be understood by those skilled in the art that Fig. 9 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig. 9 More or fewer components as shown, or with Fig. 9 Different configurations are shown.

[0249] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of this specification are described. Other embodiments are within the scope of the attached claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0250] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and server embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0251] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0252] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A data reading method, characterized in that: The method comprises: In response to a service read request carrying target read protocol data, at least two preset write protocol data are obtained; each of the preset write protocol data includes preset feature data; the target read protocol data includes at least one target feature data, and the at least one target feature data is obtained from the preset feature data included in each of the preset write protocol data; Determine, from the at least two preset write protocol data, first target write protocol data to which each target characteristic data belongs; Determine the database corresponding to the first target write protocol data based on the preset mapping information, and obtain the first target database corresponding to each of the target feature data; the preset mapping information is used to characterize the mapping relationship between each of the preset write protocol data and the database written by the included preset feature data; the database written by the included preset feature data is determined based on the generation method and / or timeliness information of the included preset feature data; Acquire a reading result corresponding to each target feature data from a first target database corresponding to each target feature data; According to the reading result corresponding to each of the target characteristic data, a characteristic data reading result corresponding to the service reading request is generated.

2. The method according to claim 1, characterized in that The step of determining, from the at least two preset write protocol data, the first target write protocol data to which each target characteristic data belongs includes: Determining preset characteristic data included in each of the preset write protocol data; Acquire preset characteristic data corresponding to each of the target characteristic data from the preset characteristic data included in each of the preset write protocol data; The preset write protocol data to which the preset characteristic data corresponding to each of the target characteristic data belongs is determined as the first target write protocol data.

3. The method according to claim 1 or 2, characterized in that: The step of determining a database corresponding to the first target write protocol data based on the preset mapping information to obtain a first target database corresponding to each of the target feature data includes: Acquire preset characteristic data and at least two databases included in the first target write protocol data; the at least two databases are determined based on the generation method and / or timeliness information of the preset characteristic data included in each of the preset write protocol data; Based on the preset mapping information, the database into which the preset characteristic data included in the first target write protocol data is written is obtained from the at least two databases to obtain a first target database corresponding to each of the target characteristic data.

4. The method according to claim 3, characterized in that The obtaining, from the first target database corresponding to each target feature data, a reading result corresponding to each target feature data, comprises: Determine the field position information of each target characteristic data in the first target write protocol data to which it belongs; Determine the offset address information of each target characteristic data according to the field position information; the offset address information is used to characterize the number of bytes between each target characteristic data and the starting data in the first target write protocol data to which it belongs; According to the offset address information of each target feature data, acquiring the binary data of each target feature data from the first target database corresponding to each target feature data; According to the binary data of each target characteristic data, a reading result corresponding to each target characteristic data is generated.

5. The method according to claim 4, characterized in that The target characteristic data is at least one of batch update characteristic data, streaming update characteristic data, and real-time characteristic data; the batch update characteristic data is data generated regularly by a preset platform; the streaming update characteristic data is data whose timeliness information is less than a preset timeliness threshold; the real-time characteristic data is data whose timeliness information is greater than or equal to the preset timeliness threshold; The first target database is at least one of a memory database based on dual cache objects, a database based on hybrid storage of memory and solid state disk, and a key-value database based on memory; In the case where the target feature data is batch update feature data, the first database corresponding to the target feature data is a memory database based on a double buffer object; In the case where the target characteristic data is streaming updated characteristic data, the first database corresponding to the target characteristic data is a database based on hybrid storage of memory and solid state drive; In the case where the target characteristic data is real-time characteristic data, the first target database corresponding to the target characteristic data is a memory-based key-value database.

6. The method according to claim 5, characterized in that The target feature data is the streaming update feature data, the first target database is a database based on a hybrid storage of a memory and a solid state drive, and according to the offset address information of each target feature data, the binary data of each target feature data is obtained from the first target database corresponding to each target feature data, including: According to the offset address information of each target feature data, searching for the binary data of each target feature data from the memory of the first target database corresponding to each target feature data; When the binary data of each target feature data is not found in the memory, the binary data of each target feature data is searched from the solid state hard disk of the first target database corresponding to each target feature data.

7. The method according to claim 1 or 6, characterized in that: The generating, according to the reading result corresponding to each of the target characteristic data, a characteristic data reading result corresponding to the service reading request includes: The reading results corresponding to each of the target characteristic data are spliced ​​to obtain the characteristic data reading results.

8. The method according to claim 1 or 6, characterized in that: The response to the service read request carrying the target read protocol data includes: In response to the business read request sent by the business read service; the number of business read requests sent by the business read service per second is less than a preset number threshold; After generating the characteristic data reading result corresponding to the service reading request, the method further includes: The characteristic data reading result is sent to the business reading service.

9. A data storage method, characterized in that: The method comprises: In response to a first remote procedure call request carrying characteristic data to be written, obtaining second target write protocol data including the characteristic data to be written from at least two preset write protocol data; each of the preset write protocol data includes preset characteristic data; the preset characteristic data includes the characteristic data to be written; Based on the second target write protocol data, the data format of the feature data to be written is processed into data in a write protocol format; and based on the preset mapping information, a second target database corresponding to the second target write protocol data is determined; the preset mapping information is used to characterize the mapping relationship between each of the preset write protocol data and the database into which the included preset feature data is written; the database into which the included preset feature data is written is determined based on the generation method and / or timeliness information of the included preset feature data; A second remote procedure call request is sent to the second target database, wherein the second remote procedure call request carries the data in the write protocol format, so that the second target database responds to the second remote procedure call request and writes the data in the write protocol format into the second target database.

10. The method according to claim 9, characterized in that The step of processing the data format of the to-be-written characteristic data into data in a write protocol format based on the second target write protocol data includes: Based on the second target write protocol data, the data format of the to-be-written characteristic data is serialized and processed into binary data to obtain data in the write protocol format.

11. The method according to claim 9 or 10, characterized in that: The feature data to be written is streaming update feature data, the second target database is a database based on hybrid storage of memory and solid state drive, and the method further includes: Based on the streaming write central control responding to the first remote procedure call request sent by the business streaming data processing service, acquiring the second target write protocol data from at least two preset write protocol data; Based on the streaming write central control, the data format of the feature data to be written is processed into data in a write protocol format through the second target write protocol data; and based on the preset mapping information, the database based on the hybrid storage of memory and solid state drive is determined; Based on the streaming write central control, the second remote procedure call request is sent to the database based on hybrid storage of memory and solid-state drive, so that the database based on hybrid storage of memory and solid-state drive responds to the second remote procedure call request and writes the data in the write protocol format into the database based on hybrid storage of memory and solid-state drive.

12. The method according to claim 9 or 10, characterized in that: The feature data to be written is real-time feature data, the second target database is a memory-based key-value database, and the method further includes: Based on the streaming write central control responding to the first remote procedure call request sent by the object storage service, obtaining the second target write protocol data from at least two preset write protocol data; Based on the streaming write central control, the data format of the feature data to be written is processed into data in a write protocol format through the second target write protocol data; and based on the preset mapping information, the memory-based key-value database is determined; Based on the streaming write central control, a second remote procedure call request is sent to the memory-based key-value database, so that the memory-based key-value database responds to the second remote procedure call request and writes the data in the write protocol format into the memory-based key-value database.

13. The method according to claim 9 or 10, characterized in that: The feature data to be written is batch update feature data, the second target database is a memory database based on a double buffer object, and the method further includes: Based on the distributed task platform component responding to the first remote procedure call request sent by the distributed file system service, acquiring the second target write protocol data from at least two preset write protocol data; Based on the distributed task platform component, the data format of the feature data to be written is processed into data in a write protocol format through the second target write protocol data; and based on the preset mapping information, a memory database based on a double cache object corresponding to the second target write protocol data is determined; Based on the distributed task platform component, a second remote procedure call request is sent to the memory database based on the double-cache object, so that the memory database based on the double-cache object responds to the second remote procedure call request and writes the data in the write protocol format to the database based on the hybrid storage of the memory and disk.

14. A data reading device, characterized in that: The device comprises: A business read request response module, configured to obtain at least two preset write protocol data in response to a business read request carrying target read protocol data; each of the preset write protocol data includes preset feature data; the target read protocol data includes at least one target feature data, and the at least one target feature data is obtained from the preset feature data included in each of the preset write protocol data; A first target write protocol data determination module, used to determine the first target write protocol data to which each of the target characteristic data belongs from the at least two preset write protocol data; A first target database determination module is used to determine the database corresponding to the first target write protocol data based on preset mapping information, and obtain the first target database corresponding to each of the target feature data; the preset mapping information is used to characterize the mapping relationship between each of the preset write protocol data and the database written by the included preset feature data; the database written by the included preset feature data is determined based on the generation method and / or timeliness information of the included preset feature data; A reading result acquisition module, used for acquiring a reading result corresponding to each target feature data from a first target database corresponding to each target feature data; The feature data reading result generating module is used to generate the feature data reading result corresponding to the service reading request according to the reading result corresponding to each target feature data.

15. A data storage device, characterized in that: The device comprises: A second target write protocol data acquisition module is used to respond to a first remote procedure call request carrying characteristic data to be written, and to acquire second target write protocol data including the characteristic data to be written from at least two preset write protocol data; each of the preset write protocol data includes preset characteristic data; and the preset characteristic data includes the characteristic data to be written; A format processing module, used for processing the data format of the feature data to be written into data in a write protocol format based on the second target write protocol data; and determining a second target database corresponding to the second target write protocol data based on preset mapping information; the preset mapping information is used to characterize the mapping relationship between each of the preset write protocol data and the database into which the included preset feature data is written; the database into which the included preset feature data is written is determined based on the generation method and / or timeliness information of the included preset feature data; A call request sending module is used to send a second remote procedure call request to the second target database, wherein the second remote procedure call request carries data in the write protocol format, so that the second target database responds to the second remote procedure call request and writes the data in the write protocol format into the second target database.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data reading method according to any one of claims 1 to 8 or the data storage method according to any one of claims 9 to 13 is implemented.

Images