Trilateral multiplexing data differentiation storage method
Through differentiated storage methods, the index configuration is dynamically adjusted according to the interface and multiplexing cycle, which solves the problem of resource waste and complex index management caused by data multiplexing cycle differences in ES, and improves storage resource utilization and system performance.
Patent Information
- Application Number
- CN202411401594.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2024-12-13
AI Technical Summary
Existing search servers (Elasticsearch, ES) cannot effectively deal with the differentiated demands for data reuse cycles in different interfaces and application scenarios, resulting in wasted storage resources and complicated index management, increasing the system's operation and maintenance costs and management burden.
A three-party multiplexed data differentiated storage method is adopted. By obtaining external data from different interfaces, the data is written to the matching storage index based on the preset multiplexed cycle rules, and the index configuration information is adjusted when the multiplexed cycle changes, so as to Avoid creating new indexes, improve storage resource utilization and simplify index management.
Differentiated storage of data for different interfaces is realized, which avoids waste of storage resources, simplifies index management, improves system maintainability and query efficiency, reduces operation and maintenance burden, and is suitable for high-concurrency scenarios of large-scale data.
Smart Images

Figure CN119127887A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a method for differential storage of three-party multiplexed data in the technical field of data processing. Background Art
[0002] In current data storage and management, many third-party data need to be reused and have strict timeliness requirements. For example, some data may need to be reused for 30 days, while other data may need to be reused for up to 180 days or even longer. The existing search server (Elasticsearch, ES) storage mechanism cannot effectively cope with these differentiated requirements. Specifically, different interfaces and application scenarios have different data reuse cycles, resulting in a unified storage and management method that can neither save resources nor ensure the efficiency of data access.
[0003] In related technologies, all reused data is stored in one ES index, but this often results in a waste of storage resources, especially for data with a short reuse period, which unnecessarily occupies long-term storage space. In addition, if the storage is segmented according to the interface or data source dimension, too many indexes may need to be created, which complicates index management and increases the system's operation and maintenance costs and management burden. Summary of the invention
[0004] In order to solve the technical problem that too many indexes need to be created when storing tripartite multiplexed data in the prior art, which complicates index management and increases the operation and maintenance cost and management burden of the system, the purpose of the present invention is to provide a method for differential storage of tripartite multiplexed data. The technical solution adopted is as follows: In a first aspect, an embodiment of the present invention provides a method for differential storage of three-party multiplexed data, the method being applied to a server, the method comprising: Get external data from different interfaces; Based on a preset reuse cycle rule, writing the external data into a matching storage index; In response to detecting that a multiplexing period of the external data changes, adjusting the configuration information of the storage index to obtain adjusted configuration information; Based on the adjusted configuration information, the external data is stored. In a second aspect, a three-party multiplexed data differentiated storage system is provided, the system comprising: Acquisition module, used to obtain external data from different interfaces; A writing module, used for writing the external data into a matching storage index based on a preset multiplexing cycle rule; An adjustment module, configured to adjust the configuration information of the storage index in response to detecting a change in the multiplexing period of the external data, so as to obtain adjusted configuration information; A storage module is used to store the external data based on the adjusted configuration information.
[0005] In a third aspect, a computer program product is provided, which includes: a computer program code, which, when executed on a computer, enables the computer to execute the method for differentiated storage of three-party multiplexed data in any one of the possible implementations described above.
[0006] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on a computer, the computer executes the method for differentiated storage of three-party multiplexed data in any one of the above possible implementations.
[0007] The present invention has the following beneficial effects: after acquiring external data of different interfaces, the external data is written into a matching storage index according to a preset reuse cycle rule, so that the external data of different interfaces can be differentially stored in the matching storage index. Afterwards, in response to detecting that the reuse cycle of the external data has changed, the configuration information of the storage index is adjusted to obtain the adjusted configuration information; and based on the adjusted configuration information, the external data is stored. In this way, by differentially storing the external data of different interfaces into matching storage indexes, and when the reuse cycle of the external data changes, it is sufficient to adjust the configuration information of the storage index without creating a new index, thereby improving the utilization rate of storage resources and simplifying index management. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0009] Figure 1 It is a schematic diagram of the implementation process of a method for differential storage of three-party multiplexed data provided by an embodiment of the present invention; Figure 2 It is another implementation flow diagram of a method for differential storage of three-party multiplexed data provided by an embodiment of the present invention; Figure 3 It is another implementation flow diagram of a method for differential storage of three-party multiplexed data provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of an implementation architecture of a method for differential storage of three-party multiplexed data provided by an embodiment of the present invention; Figure 5 It is a schematic diagram of the composition structure of a three-party multiplexed data differential storage system provided by an embodiment of the present invention; Figure 6 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0010] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a three-party multiplexed data differentiated storage method proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form as described.
[0011] Among them, in the description of the embodiments of the present invention, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present invention, "multiple" refers to two or more than two.
[0012] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.
[0013] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
[0014] In some systems of related technologies, third-party data is stored through a relational database management system (MySQL). The advantages of MySQL are its strong support for structured data and stable transaction processing capabilities, but its performance and scalability are insufficient when dealing with high-concurrency access and large-scale data storage. In addition, reused data has different reuse cycle requirements (such as 30 days or 180 days), and cannot be flexibly stored and managed through a simple table structure. It usually requires additional data management and query optimization at the application layer.
[0015] In recent years, ES has become one of the mainstream solutions for large-scale data storage and search. Many systems use ES to store reused data because of its powerful full-text search function and distributed architecture that can well support high-concurrency queries. However, when the reuse period of data is inconsistent (such as 30 days or 180 days of data), if all data is stored in the same index, short-term reused data will also occupy long-term storage resources, resulting in unnecessary waste of resources.
[0016] Another solution is to create multiple indexes according to the dimensions of interfaces or data sources. Although this can partially solve the problem of resource waste, it will also cause new problems: the sharp increase in the number of indexes complicates index management, and the efficiency of index maintenance, updating and query will decrease. Especially in multi-interface systems, the rapid growth of the number of indexes over time not only increases the difficulty of operation and maintenance, but may also lead to a decrease in system performance.
[0017] In this way, whether it is MySQL or ES, when data with different reuse periods are mixed and stored in the same index or table, data with a shorter reuse period still occupies storage resources, resulting in unnecessary waste of storage space. This waste of resources is particularly significant in large-scale data environments, resulting in increased storage costs. If indexes are created according to interface names or data sources, although resource waste can be avoided, new problems are also brought about. The number of indexes will increase rapidly, resulting in increased management complexity. As the number of indexes increases, the performance of ES may be affected in terms of query and update, and it will also bring additional operation and maintenance burdens when performing data maintenance and backup. As the number of indexes increases, it is necessary to search between multiple indexes when querying, which increases the complexity and time overhead of the query. Especially when the system needs to process a large number of requests in real time, the increase in the number of indexes will directly affect the response speed of the query and reduce the overall performance of the system.
[0018] Based on this, the embodiment of the present invention provides a specific solution for a method for storing three-way multiplexed data in a differentiated manner. Figure 1 , which shows a schematic diagram of the implementation process of a method for differential storage of three-party multiplexed data provided by an embodiment of the present invention. The method can be implemented by the following steps: 101, obtain external data from different interfaces.
[0019] Here, the external data of different interfaces can be the third-party data of different interfaces. For example, the data of multiple external partners. Different interfaces can be interfaces of different external partners, or interfaces with custom settings. The reuse period of the external data of these different interfaces can be the same or different. For example, some external data needs to be saved for 30 days, that is, the reuse period is 30 days, and some external data needs to be saved for 90 days, that is, the reuse period is 90 days.
[0020] 102. Based on a preset reuse cycle rule, write the external data into a matching storage index.
[0021] Here, the reuse cycle rule is preset, including: calculating the storage index to which the external data needs to be allocated according to the index name and the current timestamp.
[0022] In some possible implementations, the above step 102 can be performed by Figure 2 The steps shown achieve: 201. Classify the external data based on the reuse period of the external data to obtain classified data.
[0023] Here, for external data of different interfaces, the system first classifies the reuse period of the data. The reuse period of data of different interfaces varies. For example, some data only needs to be saved for 30 days, while other data may need to be saved for 90 days or longer. External data is classified according to different reuse periods to obtain classified data.
[0024] Based on the differences in these multiplexing cycles, different index configuration rules are created for data of each multiplexing cycle. Each interface can be partitioned and configured as follows: 1. The default data zone write permission is the "90-day partition" data zone. For storage space considerations, only one partition can be written; 2. The default data zone read permission is the "90-day partition" data zone. For query performance considerations, only two partitions can be selected; 3. Partitions with write permissions must be granted read permissions; 4. If a partition has read permission but write permission is turned off, the expiration time starts to be calculated, which is the current time + the future time after the reuse days. If the write permission is reopened, the expiration time is cleared.
[0025] 5. There is no partition setting for read and write permissions and no expiration time.
[0026] 202. Based on the preset reuse period rule, write the classified data into a matching storage index.
[0027] Here, a specific reuse period is configured for each storage index, so that the storage index matching the classified data can be determined through the preset reuse period rule, and the storage index is written to ensure that the data within the reuse period can be quickly read and retrieved. In this way, for external data of different interfaces, by classifying according to the reuse period, and then writing the classified data into the matching storage index, it is possible to achieve differentiated storage of external data.
[0028] In some possible implementations, the index names and current timestamps of multiple storage indexes are first obtained; then, based on the index names and the current timestamp of the classified data, the storage index that matches the classified data is determined from the multiple storage indexes. For example, the multiplexing period corresponding to the classified data is analyzed by combining the index name with the current timestamp of the classified data, so as to select the storage index with the same configured multiplexing period as the multiplexing period corresponding to the classified data, that is, the storage index that matches the classified data is obtained. In this way, although the external data of different interfaces are classified and stored according to different multiplexing periods, when the data is written, the embodiment of the present invention introduces differentiated data writing indexes. The writing operation of the external data will point to the configured index, and the external data will be assigned to the corresponding storage index according to the preset multiplexing period rule. In this scenario, the storage index to which the interface data needs to be assigned is calculated based on the index name + current timestamp. In this way, the user only needs to perform a data writing operation once, and the system will automatically handle the classified storage of the data, simplifying the process of data writing and management. In this way, according to the multiplexing period of different interfaces and data, the external data is classified and stored in the corresponding multiplexing period index. This differentiated storage strategy avoids the problem of storage resource waste caused by mixed storage of data with different reuse cycles.
[0029] 103 . In response to detecting that a multiplexing period of the external data changes, adjust the configuration information of the storage index to obtain adjusted configuration information.
[0030] Here, when the business requirements of different interfaces change, the reuse period of the external data of the corresponding interface changes. If it is detected that the reuse period of the external data changes, the configuration information of the storage index can be adjusted through the configuration page to obtain the adjusted configuration information.
[0031] In some possible implementations, in response to detecting a change in the reuse period of the external data, a configuration page is presented; thereafter, the configuration information of the storage index is adjusted on the configuration page to obtain the adjusted configuration information; wherein the adjusted configuration information includes: the adjusted storage period. In this way, the system supports dynamic adjustment of the reuse period of the index in response to changes in business requirements of different interfaces.
[0032] In some possible implementations, the storage period corresponding to the storage index is adjusted on the configuration page to obtain an adjusted storage period; and the adjusted configuration information is determined based on the adjusted storage period. For example, when the data reuse period of certain interfaces changes, the index configuration can be modified through the configuration page to adjust the storage period of the data without recreating a new index. This dynamic adjustment mechanism enables the system to flexibly respond to changes in business needs and further reduces the complexity of index management. In this way, the system supports dynamic adjustment of the index reuse period configuration according to business needs without recreating a new index, ensuring that the system can still respond flexibly when business needs change, and ensuring the long-term maintainability of the system.
[0033] 104. Store the external data based on the adjusted configuration information.
[0034] Here, the external data is stored according to the storage cycle in the adjusted configuration information. In this way, the external data of different interfaces are stored in a differentiated manner in the matching storage index. Afterwards, in response to detecting that the reuse cycle of the external data has changed, the configuration information of the storage index is adjusted to obtain the adjusted configuration information; and based on the adjusted configuration information, the external data is stored. In this way, by storing the external data of different interfaces in a differentiated manner in the matching storage index, the problems of storage resource waste and complex index management in the data storage scheme in the related art are solved, the utilization rate of storage resources is improved, and the index management is simplified.
[0035] In some embodiments, after writing external data to a matching storage index, if a data read request is received, it is possible to determine whether there is a call record for the data requested to be read. If there is a call record, the requested data is directly fed back. If there is no call record, the requested data is called from the corresponding storage index. Figure 3 As shown, this can be achieved by following the steps below: 301 : In response to a received data read request, determine a type and a target multiplexing period of the data read request.
[0036] Here, in response to the data read request, the type of the data read request is determined; wherein the type of the data read request includes: the requested data type, such as floating point type, character type or integer type, etc. Afterwards, based on the type of the data read request, the user basic information and the identification information of the corresponding interface are obtained. For example, if the type of the data read request is floating point type, the user basic information (such as user name, user identification, etc.) and the floating point identification information of the interface are obtained in floating point type; and based on the user basic information and the identification information, the target reuse period to which the corresponding interface belongs is determined. For example, the user basic information and the identification information are concatenated to obtain a character string. Through the identification information in the character string, the target reuse period configured for the corresponding interface can be obtained. In this way, by obtaining the user basic information and the identification information of the corresponding interface, the target reuse period to which the corresponding interface belongs can be quickly determined.
[0037] 302 : Read target external data based on the type of the data read request and the target multiplexing cycle.
[0038] Here, the target storage index is determined by the type of data read request and the target multiplexing cycle, and the target external data is read based on the target storage index. For example, by combining the type of data read request and the target multiplexing cycle, a storage index whose storage cycle is the target multiplexing cycle, i.e., the target storage index, can be obtained; then, the target external data is called from the target storage index. In this way, in response to a data read request, the requested target external data can be read quickly and accurately by analyzing the type of data read request and the target multiplexing cycle.
[0039] In some possible implementations, it is first determined whether there is a call record within the reuse period corresponding to the target storage index; if there is a call record within the reuse period corresponding to the target storage index, the target external data is returned based on the call record. In this way, by first determining whether there is already a call record in the target storage index, if there is a call record, the data already stored in the interface, that is, the target external data, is directly returned. If there is no call record, the target external data is directly read from the target storage index. In this way, when reading data, the system automatically selects the optimal storage index for reading according to the type of data reading request and the reuse period, thereby ensuring the improvement of query efficiency, especially in high-concurrency scenarios, and can effectively reduce the response time of the system.
[0040] In some embodiments, the multiplexing process of external data is as follows: Figure 4 As shown, combined Figure 4 The steps in are described below: 401: The calling layer obtains the basic user information transmitted from the upstream.
[0041] 402: The calling layer generates a cache string based on the user's basic information.
[0042] For example, the parameters passed by the user are naturally sorted, concatenated into a string, and finally converted into md5 in the format of k1=v1&k2=v2&k3=v3.
[0043] 403: The cache logic layer queries the identification information and write time of any interface.
[0044] For example, for an interface that queries the online status of a user's mobile phone, the identification information can be determined as the interface number + mobile phone number. After querying the identification information and write time of any interface in the cache logic layer, the Remote Dictionary Server (Redis) storage module obtains the cache record in the cache logic layer.
[0045] 404: The cache logic layer determines whether there is a call record that meets the conditions for any interface within the reuse cycle.
[0046] Here, if the cache logic layer determines that any interface has a call record that meets the conditions within the reuse period, go to step 405; determine whether a certain interface has a call record that meets the conditions within the reuse period, such as the above-mentioned mobile phone network status interface. If a mobile phone number has a call record within 30 days, then this record is directly returned. If the cache logic layer determines that any interface does not have a call record that meets the conditions within the reuse period, go to step 409.
[0047] 405. The cache logic layer generates a validity period according to the reuse period.
[0048] 406: The cache logic layer queries the ES index list configured by the interface within the validity period.
[0049] 407, the cache logic layer generates the index set to be queried based on the ES index list.
[0050] Here, the cache logic layer sends to the ES storage module.
[0051] 408. The ES storage module obtains the interface message from the index set.
[0052] 409, the calling layer reuses the external data corresponding to the interface by calling the interface.
[0053] In an embodiment of the present invention, by performing differentiated storage on data of different reuse periods, the situation of mixing data of short-term reuse periods with data of long-term reuse periods is avoided, thereby effectively saving storage space. This differentiated storage strategy can flexibly adjust the data storage method according to different business needs, greatly reducing the waste of storage resources. The method of creating indexes according to the interface or data source dimension in the related art will lead to too many indexes and complex management. By introducing a unified write index mechanism, the embodiment of the present invention allows users to write data into one index, and the system will automatically allocate the data to the corresponding storage index, thereby simplifying the creation and management of indexes. The difficulty of index maintenance is greatly reduced, the maintainability of the system is improved, and the index management is simplified. Through differentiated data reading strategies, the system automatically selects the corresponding index for retrieval for data of different reuse periods, thereby reducing the complexity of queries and improving query efficiency. Especially in high concurrency scenarios, it can quickly respond to data requests, improve data query efficiency, and significantly improve the overall performance and user experience of the system. The embodiment of the present invention supports dynamic adjustment of the reuse period configuration of the index, modifies the storage period of the data at any time according to the changes in different business needs, and ensures that the system can flexibly respond to different data reuse requirements. There is no need to frequently create or delete indexes, which reduces the system's operation and maintenance burden and improves the system's adaptability and flexibility. Since the creation and management of redundant indexes are reduced, the scalability of the system is effectively improved. The system can better cope with newly added reused data when the business expands, and through the automated index management mechanism, the stability and efficiency of the system in long-term operation are guaranteed. While achieving differentiated data storage and efficient management, it reduces resource waste, simplifies the index management process, and improves the system's query performance and expansion capabilities. It is suitable for high-concurrency scenarios of large-scale data and has broad application prospects and significant economic benefits.
[0054] The embodiment of the present invention provides a three-way multiplexed data differential storage system, such as Figure 5 As shown, the system 501 includes: Acquisition module 501, used to acquire external data from different interfaces; A writing module 502, configured to write the external data into a matching storage index based on a preset multiplexing cycle rule; An adjustment module 503, configured to adjust the configuration information of the storage index in response to detecting that a change has occurred in the multiplexing period of the external data, to obtain adjusted configuration information; The storage module 504 is used to store the external data based on the adjusted configuration information.
[0055] In some possible implementations, the writing module 502 is further used to classify the external data based on the reuse period of the external data to obtain classified data; and write the classified data into a matching storage index based on the preset reuse period rule.
[0056] In some possible implementations, the writing module 502 is further used to obtain index names and current timestamps of multiple storage indexes; and based on the index names and the current timestamp of the classified data, determine a storage index that matches the classified data from the multiple storage indexes.
[0057] In some possible implementations, the adjustment module 503 is further used to present a configuration page in response to detecting a change in the reuse period of the external data; and adjust the configuration information of the storage index on the configuration page to obtain the adjusted configuration information.
[0058] In some possible implementations, the adjustment module 503 is further configured to adjust the storage period corresponding to the storage index on the configuration page to obtain an adjusted storage period; and determine the adjusted configuration information based on the adjusted storage period.
[0059] In some possible implementations, the storage module 504 is further used to determine the type and target multiplexing cycle of the data read request in response to a received data read request; and read target external data based on the type and target multiplexing cycle of the data read request.
[0060] In some possible implementations, the storage module 504 is also used to determine the type of the data read request in response to the data read request; based on the type of the data read request, obtain user basic information and identification information of the corresponding interface; based on the user basic information and the identification information, determine the target multiplexing cycle to which the corresponding interface belongs.
[0061] In some possible implementations, the storage module 504 is further used to determine a target storage index based on the type of the data read request and the target multiplexing period; and read the target external data based on the target storage index.
[0062] In some possible implementations, the storage module 504 is also used to determine whether there is a call record that meets preset conditions within the reuse period corresponding to the target storage index; if there is a call record within the reuse period corresponding to the target storage index, the target external data is returned based on the call record.
[0063] It should be noted that: the system provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0064] Figure 6 is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention. Figure 6 As shown, the computer device 600 includes: a memory 601, a processor 602, and a computer program 603 stored in the memory 601 and running on the processor 602, wherein when the processor 602 executes the computer program 603, the computer device can execute any one of the three-party multiplexed data differentiated storage methods introduced above.
[0065] In addition, an embodiment of the present invention also protects a system, which may include a memory and a processor, wherein an executable program code is stored in the memory, and the processor is used to call and execute the executable program code to perform a three-party multiplexed data differentiated storage method provided by an embodiment of the present invention. This embodiment can divide the system into functional modules according to the above method example. For example, it can correspond to each functional module, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic, which is only a logical function division, and there may be other division methods in actual implementation. It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here.
[0066] It should be understood that the system provided in this embodiment is used to execute the above-mentioned three-party multiplexed data differentiated storage method, so the same effect as the above-mentioned implementation method can be achieved. In the case of an integrated unit, the system may include a processing module and a storage module. Among them, when the system is applied to a device, the processing module can be used to control and manage the actions of the device. The storage module can be used to support the device to execute mutual program codes, etc. Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logic blocks, modules and circuits described in combination with the disclosure of the present invention. The processor can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module can be a memory.
[0067] In addition, the system provided by the embodiment of the present invention may be specifically a chip, a component or a module, and the chip may include a connected processor and a memory; wherein the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute a three-party multiplexed data differential storage method provided in the above embodiment. This embodiment also provides a computer-readable storage medium, in which a computer program code is stored, and when the computer program code is run on a computer, the computer executes the above-mentioned related method steps to implement a three-party multiplexed data differential storage method provided in the above embodiment.
[0068] This embodiment also provides a computer program product, when the computer program product is run on a computer, the computer executes the above-mentioned related steps to implement a three-party multiplexed data differential storage method provided in the above embodiment. Among them, the system, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding method provided above, so the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here. Through the description of the above implementation mode, the technicians in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional modules as needed, that is, the internal structure of the system is divided into different functional modules to complete all or part of the functions described above. In the embodiments provided by the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiment described above is only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, system or unit, which may be electrical, mechanical or other forms.
[0069] It should be noted that the sequence of the above-mentioned embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or may be advantageous. The various embodiments in this specification are described in a progressive manner, and the same and similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments. The above content is only a specific implementation method of the present invention, but the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered within the protection scope of the present invention.
Claims
1. A method for differential storage of tripartite multiplexed data, characterized in that: The method comprises: Get external data from different interfaces; Based on a preset reuse cycle rule, writing the external data into a matching storage index; In response to detecting that a multiplexing period of the external data changes, adjusting the configuration information of the storage index to obtain adjusted configuration information; Based on the adjusted configuration information, the external data is stored.
2. A method for differential storage of three-party multiplexed data according to claim 1, characterized in that: The step of writing the external data into a matching storage index based on a preset multiplexing cycle rule includes: Based on the reuse period of the external data, the external data is classified to obtain classified data; Based on the preset reuse period rule, the classified data is written into a matching storage index.
3. A method for differential storage of three-party multiplexed data according to claim 2, characterized in that: Before writing the classified data into a matching storage index based on the preset reuse cycle rule, the method further includes: Obtaining index names of a plurality of storage indexes and a current timestamp of the classified data; Based on the index name and the current timestamp of the classified data, a storage index matching the classified data is determined from the multiple storage indexes.
4. A method for differential storage of three-party multiplexed data according to claim 1, characterized in that: In response to detecting that the multiplexing period of the external data has changed, adjusting the configuration information of the storage index to obtain the adjusted configuration information, includes: In response to detecting that a multiplexing period of the external data changes, presenting a configuration page; The configuration information of the storage index is adjusted on the configuration page to obtain the adjusted configuration information.
5. A method for differential storage of three-party multiplexed data according to claim 4, characterized in that: The step of adjusting the configuration information of the storage index on the configuration page to obtain the adjusted configuration information includes: Adjust the storage period corresponding to the storage index on the configuration page to obtain an adjusted storage period; The adjusted configuration information is determined based on the adjusted storage period.
6. A method for differential storage of three-party multiplexed data according to claim 1, characterized in that: After storing the external data based on the adjusted configuration information, the method further includes: In response to the received data read request, determining a type and a target multiplexing period of the data read request; Based on the type of the data read request and the target multiplexing cycle, target external data is read.
7. A method for differential storage of three-party multiplexed data according to claim 6, characterized in that: The step of determining, in response to the received data read request, a type and a target multiplexing period of the data read request comprises: In response to the data read request, determining a type of the data read request; Based on the type of the data reading request, obtain basic user information and identification information of the corresponding interface; Based on the user basic information and the identification information, a target multiplexing period to which the corresponding interface belongs is determined.
8. A method for differential storage of three-party multiplexed data according to claim 6, characterized in that: The step of reading target external data based on the type of the data read request and the target multiplexing cycle includes: Determining a target storage index based on the type of the data read request and the target multiplexing period; Based on the target storage index, the target external data is read.
9. A method for differential storage of three-party multiplexed data according to claim 8, characterized in that: The reading the external data based on the target storage index includes: Determine whether there is a call record that meets a preset condition within the reuse period corresponding to the target storage index; If there is a call record within the reuse period corresponding to the target storage index, the target external data is returned based on the call record.
10. A three-party multiplexed data differential storage system, characterized in that: The system comprises: Acquisition module, used to obtain external data from different interfaces; A writing module, used for writing the external data into a matching storage index based on a preset multiplexing cycle rule; An adjustment module, configured to adjust the configuration information of the storage index in response to detecting a change in the multiplexing period of the external data, so as to obtain adjusted configuration information; A storage module is used to store the external data based on the adjusted configuration information.
Citation Information
Patent Citations
Method an device for data storage
CN106919675A
Data query method and device and calculation equipment
CN108427684A
An implementation method for supporting data life cycle management of a multi-database engine
CN109815219A
Log management system and method based on ElasticSearch
CN117971606A
Index management method and device
CN117971832A
