System processing method and device, equipment and storage medium

Abstract

The invention discloses a system processing method and device, equipment and a storage medium, and the method comprises the steps: determining a simulation system corresponding to a cache system in a development process of the cache system in a content distribution network, and obtaining the configuration information of the simulation system; the configuration information is used for indicating cache parameters needing to be used by the simulation system, and the cache system is shared by multiple objects so as to be used for caching data generated by the multiple objects; obtaining a target data set through the simulation system, wherein the target data set comprises data generated by a plurality of objects; in the simulation system, cache parameters indicated by the configuration information are adopted to perform cache playback simulation on the target data set, and a cache simulation result corresponding to the target data set is obtained; visually outputting a cache simulation result corresponding to the target data set; according to the method, the function simulation of the cache system in the content distribution network can be realized based on the simulation system, so that the cache system can be developed based on the cache simulation result, and the developed cache system can achieve a better cache effect.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a system processing method, device, equipment and storage medium. Background Art

[0002] With the rapid development of Internet technology, applications with various functions have emerged; when the location of the user object is far away from the location of the data center of the application, when the user object uses the data provided by its data center through the application, during the data transmission process, due to the long distance, more long-distance traffic will be generated on the network link, making the data transmission cost high. Content Delivery Network (CDN) can cache the data of the application's data center through the cache system in the CDN server established in different locations, so that when the user object obtains the data provided by the corresponding application, it can obtain it from the CDN server close to the location of the user object, thereby reducing the traffic generated during the data transmission process, that is, reducing the data transmission cost; based on this, how to develop the cache system to achieve a better cache effect is a current research hotspot. Summary of the invention

[0003] The embodiments of the present application provide a system processing method, apparatus, device and storage medium, which can realize functional simulation of a cache system in a content distribution network based on a simulation system, and then develop the cache system based on the cache simulation results, so that the developed cache system can achieve better cache effect.

[0004] On the one hand, an embodiment of the present application provides a system processing method, including:

[0005] In the process of developing a cache system in a content distribution network, a simulation system corresponding to the cache system is determined, and configuration information of the simulation system is obtained; the configuration information is used to indicate cache parameters to be used by the simulation system, the cache parameters are determined in the process of developing the cache system, and the cache system is shared by multiple objects to cache data generated by the multiple objects;

[0006] Acquire a target data set through the simulation system, wherein the target data set includes data generated by the multiple objects;

[0007] In the simulation system, the cache parameters indicated by the configuration information are used to perform cache replay simulation on the target data set to obtain a cache simulation result corresponding to the target data set;

[0008] The cache simulation results corresponding to the target data set are output visually.

[0009] On the one hand, an embodiment of the present application provides a system processing device, including:

[0010] A processing unit, used to determine a simulation system corresponding to the cache system and obtain configuration information of the simulation system during the process of developing the cache system in the content distribution network; the configuration information is used to indicate cache parameters to be used by the simulation system, the cache parameters are determined during the process of developing the cache system, and the cache system is shared by multiple objects for caching data generated by the multiple objects;

[0011] The processing unit is further used to obtain a target data set through the simulation system, wherein the target data set includes data generated by the multiple objects;

[0012] The processing unit is further configured to perform cache replay simulation on the target data set in the simulation system using the cache parameters indicated by the configuration information to obtain a cache simulation result corresponding to the target data set;

[0013] The output unit is used to visually output the cache simulation result corresponding to the target data set.

[0014] On the one hand, an embodiment of the present application provides a system processing device, the system processing device comprising an input interface and an output interface, and further comprising:

[0015] a processor adapted to implement one or more instructions; and,

[0016] A computer storage medium storing one or more instructions, wherein the one or more instructions are suitable for being loaded by the processor and executing the above-mentioned system processing method.

[0017] On the one hand, an embodiment of the present application provides a computer storage medium, in which computer program instructions are stored. When the computer program instructions are executed by a processor, they are used to execute the above-mentioned system processing method.

[0018] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer storage medium; a processor of a system processing device reads the computer program from the computer storage medium, and the processor executes the computer program, so that the system processing device executes the above-mentioned system processing method.

[0019] In an embodiment of the present application, a simulation system corresponding to a cache system in a content distribution network can be determined, and configuration information of the simulation system can be obtained, wherein the configuration information is used to indicate cache parameters to be used by the simulation system, and the cache system is shared by multiple objects for caching data generated by multiple objects; after a target data set including data generated by multiple objects is obtained through the simulation system, the cache parameters indicated by the configuration information can be used in the simulation system to perform cache replay simulation on the target data set to obtain a cache simulation result corresponding to the target data set, and the cache simulation result corresponding to the target data set can be output visually; a functional simulation of a cache system in a content distribution network can be implemented based on the simulation system, and then the cache system can be developed based on the cache simulation result obtained by simulation, so that the developed cache system can achieve a better cache effect; and the cache parameters to be used by the simulation system can be configured based on the configuration information, so that the cache parameters to be simulated can be configured according to the development requirements of the cache system, and the cache simulation results of the same data under different cache parameters are output, which can provide a reference for the development of the cache system and increase the convenience of cache system development. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a structural schematic diagram of a processing system provided in an embodiment of the present application;

[0022] Figure 2 It is a structural schematic diagram of a simulation system provided in an embodiment of the present application;

[0023] Figure 3 is a structural diagram of another simulation system provided in an embodiment of the present application;

[0024] Figure 4 It is a flowchart of a system processing method provided in an embodiment of the present application;

[0025] Figure 5 It is a schematic diagram of an organizational structure for configuring different cache models and cache replacement algorithms provided in an embodiment of the present application;

[0026] Figure 6 is a structural diagram of another simulation system provided in an embodiment of the present application;

[0027] Figure 7is a structural diagram of another simulation system provided in an embodiment of the present application;

[0028] Figure 8 is a schematic diagram of another simulation system provided in an embodiment of the present application;

[0029] Fig. 9 It is a schematic diagram of a cache module performing cache replay simulation provided by an embodiment of the present application;

[0030] Fig.10 is a schematic diagram of data management by a data loader provided in an embodiment of the present application;

[0031] Fig.11 is a structural schematic diagram of a system processing device provided in an embodiment of the present application;

[0032] Fig.12 It is a structural diagram of a system processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] The embodiment of the present application provides a system processing solution, which can determine the simulation system corresponding to the cache system and obtain the configuration information of the simulation system during the process of developing the cache system in the content distribution network; wherein the configuration information is used to indicate the cache parameters to be used by the simulation system, and the cache parameters are determined during the process of developing the cache system, and the cache system is shared by multiple objects to cache the data generated by multiple objects; the target data set is obtained through the simulation system, and the target data set includes data generated by multiple objects; and then, in the simulation system, the cache parameters indicated by the configuration information can be used to perform cache replay simulation on the target data set to obtain the cache simulation results corresponding to the target data set, and the cache simulation results corresponding to the target data set can be visualized and output. Among them, the cache system in the content delivery network (Content Delivery Network, CDN) is a cache system deployed in the CDN server of the content delivery network, which can be shared by multiple objects to cache the data generated by multiple objects; the object here can refer to a content providing object, that is, an object that provides data content to the cache system of the CDN server, that is, an object that uses the cache system of the CDN server to cache data, for example, it can be a data center of various applications.

[0035] In one embodiment, the above-mentioned simulation system can be deployed in a system processing device, which can be a terminal device or a server; the terminal devices here may include but are not limited to: computers, smart phones, tablet computers, laptops, smart home appliances, vehicle terminals, smart wearable devices, etc.; the server here can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0036] Based on this, the above system processing solution can be executed by the system processing device. Further, optionally, the above system processing solution can also be executed in collaboration with a processing system including the system processing device. Figure 1 , is a structural diagram of a processing system provided in an embodiment of the present application, and the processing system may include at least a client 101 and a system processing device 102, wherein the client 101 may be a terminal device or an APP (Application) running in the terminal device, and the embodiment of the present application does not limit it. A simulation system is deployed in the system processing device 102; objects that need to use the simulation system for cache replay simulation (i.e., objects used by the simulation system, such as development objects of the cache system, test objects of the cache system, etc.) can initiate a content distribution request to the system processing device 102 through the client 101 during the process of developing the cache system in the content distribution network. A simulation request for a cache system in a network; the system processing device 102 responds to the simulation request, and in the process of developing a cache system in a content distribution network, determines the simulation system corresponding to the cache system and obtains configuration information of the simulation system; obtains a target data set through the simulation system, and the target data set includes data generated by multiple objects; in the simulation system, performs a cache replay simulation on the target data set using the cache parameters indicated by the configuration information to obtain a cache simulation result corresponding to the target data set; outputs the cache simulation result corresponding to the target data set to the client 101 in a visual manner; further, the user of the simulation system can develop a cache system based on the cache simulation result corresponding to the target data set. For ease of explanation, the subsequent embodiments of this application are explained by taking the above-mentioned system processing scheme executed by the system processing device as an example.

[0037] In one embodiment, the simulation system proposed in the present application provides a cache module (referred to as cache) for caching and replaying data, and the configuration information can be used to indicate the cache parameters to be used by the simulation system, and specifically can be used to indicate the cache parameters to be used by each cache module to be configured in the simulation system, that is, the configuration information can be used to configure the cache parameters to be used by the simulation system, and specifically can be used to configure the cache parameters to be used by each cache module to be configured in the simulation system, that is, to configure cache modules with different cache parameters in the simulation system. By configuring different cache parameters in the simulation system, that is, by configuring cache modules with different cache parameters in the simulation system, cache modules with different cache parameters can be used to perform cache replay simulation on the same data, and then cache simulation results of the same data under different cache modules can be obtained, that is, cache replay simulation can be performed on the same data under different cache parameters, and then cache simulation results of the same data under different cache parameters can be obtained. By visually outputting the cache simulation results of the same data under different cache parameters, the cache system can be developed based on the corresponding cache simulation results, for example, the cache parameters to be adopted by the cache system can be determined based on the cache simulation results of the same data under different cache parameters.

[0038] In one embodiment, since the cache system can be shared by multiple objects to cache data generated by multiple objects, when the cache system caches the data generated by multiple objects, different cache effects may occur based on different division methods of the multiple objects; for example, if the multiple objects are object 1, object 2, object 3, and object 4, if the multiple objects are divided into three object groups based on a division method, the first object group includes object 1, the second object group includes object 2 and object 3, and the third object group includes object 4, then under this division method, the cache system can respectively obtain the data generated by the first object group, the second object group, and the third object group, and perform the caching operations respectively. Cache processing; if the multiple objects are divided into two object groups based on another division method, the first object group includes object 1 and object 2, and the second object group includes object 3 and object 4, then under this division method, the cache system can respectively obtain the data generated by the first object group and the second object group, and perform cache processing on them respectively; although the cache system performs cache processing on the data generated by the same multiple objects under the two division methods, due to the different division methods, the cache system actually performs cache processing on the data generated by different object groups obtained by the division. Therefore, for the same multiple objects, there may be different cache effects due to different division methods.

[0039] Based on the above description, the present application proposes that the cache processing process of the cache system under different partitioning methods can be simulated by using a simulation system based on the configuration of the partitioning method of multiple object groups, and the corresponding cache simulation results can be obtained, so that the cache system can be developed based on the corresponding cache simulation results. For example, it can be determined that when the cache system caches the data generated by multiple objects, the data generated by the object group obtained under which partitioning method is cached. In this case, the configuration information can also include: M partitioning methods for multiple objects, M is an integer greater than 1; the M partitioning methods included in the configuration information can be used to configure the data acquisition method of the simulation system, that is, when the simulation system caches and replays the data generated by multiple objects, the data generated by the object group obtained under each partitioning method can be obtained respectively, and then the data generated by the corresponding object group is cached and replayed. Further, the simulation system proposed in the present application provides a data loader (referred to as dataLoader) for acquiring data, and the M partitioning methods included in the configuration information can be used to configure the data acquisition method of the simulation system, and can be specifically used to configure each data loader in the simulation system to obtain the data generated by which object group.

[0040] Further optionally, for any object group obtained by any division method, at least one data loader should be configured in the simulation system to obtain the data generated by the corresponding object group obtained under the corresponding division method. For example, if the multiple objects are object 1, object 2, object 3, and object 4, and the configuration information includes M division methods, M=3, if the multiple objects are divided into three object groups based on the first division method, the first object group includes object 1, the second object group includes object 2 and object 3, and the third object group includes object 4; if the multiple objects are divided into two object groups based on the second division method, the first object group includes object 1 and object 2, and the second object group includes object 3 and object 4; if the multiple objects are divided into two object groups based on the third division method, the first object group includes object 1, and the second object group includes object 2, object 3, and object 4, if the configuration information indicates: in the simulation system, multiple cache modules with different cache parameters are configured for any object group under any division method, and the same cache module is configured for each object group obtained by each division method. For example, in the simulation system, a cache module with cache parameter 1, a cache module with cache parameter 2, and a cache module with cache parameter 3 are configured for each object group. For example, if a data loader is configured in the simulation system for any object group obtained by any partitioning method, see Figure 2, is a structural diagram of a simulation system provided in an embodiment of the present application; the data generated by any object group under any division method is obtained by a data loader corresponding to the corresponding object group, and then cache replay simulation is performed by multiple cache modules corresponding to the corresponding object group. Exemplarily, if multiple data loaders are configured in the simulation system for any object group obtained by any division method, the number of configured data loaders is the same as the amount of data in the cache module configured for the corresponding object group, and the configured data loaders correspond one-to-one to the configured cache modules, see Figure 3 , which is a structural diagram of another simulation system provided in an embodiment of the present application; the data generated by any object group under any division method is acquired by multiple data loaders corresponding to the corresponding object group, and then cached and replayed by the cache modules corresponding to each data loader.

[0041] The collection and processing of relevant data in this application (such as configuration information of the simulation system, target data set, etc.) should be strictly in accordance with the requirements of laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0042] Based on the above system processing solution, the present application embodiment provides a system processing method. Figure 4 , which is a flow chart of a system processing method provided in an embodiment of the present application. Figure 4 The system processing method shown can be executed by a system processing device. Figure 4 The system processing method shown may include the following steps:

[0043] S401, in the process of developing a cache system in a content distribution network, determining a simulation system corresponding to the cache system, and obtaining configuration information of the simulation system.

[0044] The configuration information is used to indicate cache parameters to be used by the simulation system. The cache parameters are determined in the process of developing the cache system. The cache system is shared by multiple objects to cache data generated by multiple objects.

[0045] In one embodiment, the configuration information can be used to indicate the cache parameters to be used by the simulation system, that is, the configuration information can be used to configure the cache parameters to be used by the simulation system; the cache parameters include: a cache model and a cache replacement algorithm, that is, the configuration information can be used to configure the cache model and cache replacement algorithm to be used by the simulation system, and specifically can be used to configure the cache model and cache algorithm to be used by each cache module to be configured in the simulation system, that is, cache modules with different cache models and different cache replacement algorithms can be configured in the simulation system based on the configuration information for cache replay simulation of data. Among them, the cache model refers to the cache structure used to organize the cache, which can be used to describe the simulated cache system. The configurable cache model may include existing cache models, for example, it may include simple cache, hierarchical cache and partitioned cache; wherein, simple cache can indicate that a cache space uses the same cache replacement algorithm, hierarchical cache (also known as multi-level cache) can indicate that each level of cache can use a different cache replacement algorithm, and partitioned cache can indicate that the cache space is divided into multiple incoherent independent spaces based on the simple cache, and the hash value of the data to be cached is taken modulo to determine the space to which it belongs; further optionally, the configurable cache model can also include a cache model designed according to requirements, that is, by configuring the cache model designed according to requirements, the data can be cached and replayed in the corresponding cache model. Simulation. Among them, the cache replacement algorithm refers to an algorithm used to decide the data to be replaced from the data stored in the cache space, that is, when there is no remaining capacity in the cache space, if data still needs to be cached, then it is necessary to decide the data to be replaced from the data stored in the cache space, delete the decided data from the cache space, and cache the data to be cached in the cache space. This process can be implemented based on the cache replacement algorithm; further, the configurable cache replacement algorithm may include existing cache replacement algorithms, for example, it may include FIFO (First-In-First-Out), FIFO2 (First In First Out with Second Chance), LRU (Least Recently Used), LIRS (Low Inter-reference Recency Set), ARC (Adaptive Replacement Cache); further optionally, the configurable cache replacement algorithm may also include a cache replacement algorithm designed according to requirements, that is, by configuring the cache replacement algorithm designed according to requirements, cache replay simulation of data can be performed under the corresponding cache replacement algorithm.

[0046] Furthermore, the cache model and cache replacement algorithm support orthogonal configuration, that is, when configuring the cache parameters to be used by the simulation system, the configuration of a cache parameter should include both the configuration of the cache model and the configuration of the cache replacement algorithm. That is, when configuring the cache model and cache replacement algorithm of a cache module, the configuration of the cache model used by the cache module and the configuration of the cache replacement algorithm should be included. For example, you can choose a multi-level cache (cache model) composed of FIFO+LRU (cache replacement algorithm), you can choose a simple cache (cache model) of ARC (cache replacement algorithm), and so on. See Figure 5 , which is a schematic diagram of an organizational structure for configuring different cache models and cache replacement algorithms provided in an embodiment of the present application, wherein 501, 502 and 503 respectively indicate different configuration methods of the cache model and the cache replacement algorithm, wherein the cache model indicated by 502 is a cache model designed according to specific needs, and the dotted line represents a path that a cache access may pass through.

[0047] In one embodiment, the configuration information may also include: M division methods for multiple objects, where M is an integer greater than 1; the M division methods included in the configuration information can be used to configure the data acquisition method of the simulation system, that is, when the simulation system performs cache replay simulation on the data generated by multiple objects, it can respectively obtain the data generated by the object groups obtained under each division method, and then perform cache replay simulation on the data generated by the corresponding object groups. For example, if the multiple objects are object 1, object 2, object 3, and object 4, and the configuration information includes M division methods, M=3, if the first division method indicates that object 1 is divided together, object 2 and object 3 are divided together, and object 4 is divided together, then based on the first division method, the multiple objects are divided into three object groups, the first object group includes object 1, the second object group includes object 2 and object 3, and the third object group includes object 4; if the second division method indicates that object 1 and object 2 are divided together, and object 3 and object 4 are divided together, then based on the second division method, the multiple objects are divided into two object groups, the first object group includes object 1 and object 2, and the second object group includes object 3 and object 4; if the third division method indicates that object 1 is divided together, and object 2, object 3, and object 4 are divided together, then based on the third division method, the multiple objects are divided into two object groups, the first object group includes object 1, and the second object group includes object 2, object 3, and object 4. Taking the first division method as an example, under this division method, multiple cache modules corresponding to the object group can be configured for each object group obtained, and then the cache replay simulation of the data generated by different object groups can be realized based on different cache modules. For example, in the simulation system, Q_1 cache modules are configured for the first object group under this division method, and the cache parameters of the Q_1 cache modules are different from each other. Q_2 cache modules are configured for the second object group under this division method, and the cache parameters of the Q_2 cache modules are different from each other. Q_3 cache modules are configured for the third object group under this division method. , the cache parameters of the Q_3 cache modules are different from each other, wherein the number of cache modules configured for different object groups may be the same or different. Further, when the number of cache modules configured for different object groups is the same, the cache parameters used by the multiple cache modules configured for different object groups may be completely the same, may not be completely the same, or may be completely different. When the number of cache modules configured for different object groups is different, the cache parameters used by the multiple cache modules configured for different object groups may be partially the same, or may be completely different. This can be obtained according to the configuration information.Taking the first division method as an example, further, under this division method, at least one data loader corresponding to any object group obtained can be configured, and then the data generated by the object group can be obtained based on the data loader configured for the object group, and further based on the different cache modules configured for the object group, a cache replay simulation of the data generated by the object group can be implemented; exemplarily, a data loader corresponding to any object group obtained can be configured, and data loaders corresponding to multiple cache modules corresponding to the object group can also be configured for any object group obtained, that is, one data loader corresponds to one cache module, which can also be configured according to the configuration information.

[0048] In one embodiment, the configuration information of the simulation system may be determined in the process of developing the cache system, and the configuration information of the simulation system obtained by the system processing device may be manually set, and may be manually set according to specific needs, or may be set by the system processing device; in an exemplary embodiment, when the system processing device sets the configuration information of the simulation system, the information used to indicate cache parameters may be obtained, and the indication information of each configurable cache model and the indication information of each configurable cache replacement algorithm may be obtained, and then the information indicating the cache parameters in the configuration information may be obtained based on the orthogonal configuration of the indication information of each cache model and the indication information of each cache replacement algorithm; the system processing device sets the configuration information of the simulation system The M division methods included in the formula can be obtained based on the combination of multiple objects. For example, if the multiple objects include object 5, object 6 and object 7, the multiple objects are combined to obtain: {object 5}, {object 6}, {object 7}, {object 5, object 6}, {object 7}, {object 5, object 7}, {object 6}, {object 5}, {object 6, object 7}, and {object 5, object 6, object 7}. The set M division methods may include 6 division methods, each division method is used to indicate that the multiple objects are divided into the above-mentioned object groups. For example, one of the division methods is used to indicate that the multiple objects are divided into 3 object groups, and the 3 object groups are respectively {object 5}, {object 6} and {object 7}.

[0049] S402, obtaining a target data set through a simulation system.

[0050] Wherein, the target data set includes data generated by multiple objects; further, the target data set should include: data corresponding to each object group obtained by dividing multiple objects according to M division methods (that is, data generated by objects included in each object group); the system processing device can pull the data generated by each object in the corresponding object group from the target space through the data loader in the simulation system; wherein, the target space includes at least one of the following: local space and cache system; wherein, since the cache system will obtain the data generated by multiple objects in real time and cache the obtained data, the data generated by each object in the corresponding object group pulled from the cache system should be real-time data when the cache system performs cache processing, that is, data generated by multiple objects obtained in real time, and the data generated by each object in the corresponding object group pulled from the local space should be offline data stored in the local space, that is, the data generated by multiple objects obtained when the cache system performs cache processing can be stored in the local space, so that when cache replay simulation is required, the data generated by each object in the corresponding object group can be obtained from the local space.

[0051] In one embodiment, the system processing device obtains the target data set through the simulation system, which may include: traversing M partitioning methods and determining the mth partitioning method currently traversed, m∈[1,M]; dividing multiple objects according to the mth partitioning method to obtain N object groups under the mth partitioning method, where N is a positive integer; obtaining the data group of each object group in the N object groups through the simulation system to form a data subset corresponding to the mth partitioning method; wherein the data group of any object group includes the data generated by each object in the corresponding object group; continuing to traverse the M partitioning methods, and after all the M partitioning methods are traversed, obtaining the target data set, the target data set includes the data subset corresponding to each of the M partitioning methods. For example, if the multiple objects are object 1, object 2, object 3, and object 4, and the configuration information includes M division methods, M=3, if m=1, the first division method indicates that object 1 is divided together, object 2 and object 3 are divided together, and object 4 is divided together, then based on the first division method, the multiple objects are divided into three object groups (N=3), the first object group includes object 1, the second object group includes object 2 and object 3, and the third object group includes object 4; the data group of each of the three object groups can be obtained through the simulation system to constitute a data subset corresponding to the first division method, specifically including the data group of the first object group (including object 1) under the first division method, the data group of the second object group (including object 2 and object 3) under the first division method, and the data group of the third object group (including object 4) under the first division method, and the data group of any object group includes the data generated by each object in the corresponding object group.

[0052] In one embodiment, the simulation system may include multiple data loaders. Taking N object groups under the mth division method as an example, the system processing device obtains the data group of each of the N object groups through the simulation system, which may include: in the simulation system, N data loaders are assigned to the N object groups, and one object group is assigned one data loader; calling each of the N data loaders respectively to load the data group of the corresponding object group from the target space; wherein the target space includes at least one of the following: a local space and a cache system. Furthermore, the system processing device calls each data loader among the N data loaders respectively to load and obtain the data group of the corresponding object group from the target space, which may include: calling the nth data loader among the N data loaders to pull the data generated by each object in the corresponding object group from the target space, n∈[1,N]; dynamically creating a data filter associated with the nth data loader, and calling the dynamically created data filter to filter the data pulled by the nth data loader to obtain the data group of the object group corresponding to the nth data loader (that is, the data group of the nth object group under the mth division method); after obtaining the data group of the object group corresponding to the nth data loader, destroying the data filter associated with the nth data loader. Among them, when calling the dynamically created data filter to filter the data pulled by the nth data loader, the filtering can be performed according to the filtering rules. The filtering rules can be configured according to specific needs. For example, the filtering rules can indicate that a certain type of pulled data should be filtered out, and the filtering rules can also indicate that all pulled data should be retained, etc.; and, in the process of filtering the pulled data, if the pulled data cannot be directly used by the cache module, the pulled data needs to be parsed so that it can be used by the cache module.The data filter associated with the nth data loader is created dynamically, but optionally, a data filter associated with the nth data loader can also be created statically. The statically created data filter associated with the nth data loader will not be destroyed after filtering the data pulled by the nth data loader, but will be retained for filtering the newly pulled data by the nth data loader when the nth data loader pulls the newly generated data of each object in the corresponding object group from the target space again, so as to obtain the newly generated data group of the object group corresponding to the nth data loader. Further, the newly generated data group of the object group corresponding to the nth data loader can be cached and replayed for simulation until the simulation system is closed and the data filter is destroyed; further optionally, when creating a data filter associated with any data loader, it can be created dynamically or statically, and among the data filters associated with all data loaders in this simulation system, it can be created in a dynamic and static manner, that is, some data filters are created statically and some data filters are created dynamically. Based on this, in. Figure 2 Based on the structure of the simulation system shown, see Figure 6 , which is a structural diagram of another simulation system provided in an embodiment of the present application. In the simulation system, a data loader is configured for any object group under any division method, and one data loader corresponds to a data filter and corresponds to multiple cache modules configured for the object group.

[0053] In one embodiment, taking N data loaders allocated to N object groups under the m-th division mode as an example, the n-th data loader among the N data loaders (the n-th data loader corresponds to the n-th object group, that is, the data loader corresponding to the n-th object group under the m-th division mode) can sequentially pull the data generated by each object in the n-th object group from the target space, and pull a specified amount of data each time. The specified amount of data can be set according to specific needs, for example, it can be set to 512K, 1M, 2M, etc.; the data pulled by the n-th data loader among the N data loaders each time needs to be stored in the first buffer space corresponding to the n-th data loader, and the data filter associated with the n-th data loader (for example, a dynamically created data filter) obtains the data pulled by the n-th data loader from the first buffer space corresponding to the n-th data loader, and performs filtering processing, that is, the data pulled by the n-th data loader each time is sequentially obtained and filtered. Based on this, the system processing device can detect the remaining storage capacity of the first buffer space corresponding to the nth data loader; if it is detected that there is remaining storage capacity in the first buffer space, it triggers the execution of the step of calling the nth data loader among the N data loaders to pull the data generated by each object in the corresponding object group from the target space; if it is detected that there is no remaining storage capacity in the first buffer space, it waits for the storage capacity of the first buffer space to be released, and then triggers the execution of the step of calling the nth data loader among the N data loaders to pull the data generated by each object in the corresponding object group from the target space. Furthermore, the dynamically created data filter obtains the data pulled by the nth data loader from the first buffer space corresponding to the nth data loader and performs filtering processing; after obtaining the data group of the object group corresponding to the nth data loader, the system processing device can also store the data group of the object group corresponding to the nth data loader in the second buffer space corresponding to the nth data loader; the data obtained from the first buffer space by the dynamically created data filter is deleted from the first buffer space to release the storage capacity of the first buffer space; wherein, after completing the buffer replay simulation of any data group in the second buffer space corresponding to the nth data loader, the corresponding data group is deleted from the second buffer space.That is to say, when it is detected that the first buffer space corresponding to the nth data loader (i.e., the data loader corresponding to the nth object group under the mth division method) has remaining storage capacity, the nth data loader can sequentially pull the data generated by each object in the nth object group from the target space and store it in the first buffer space, pulling a specified amount of data each time (for the sake of convenience, the specified amount of data pulled each time is referred to as the original data group); if at this time the first buffer space includes three original data groups generated by the nth object group sequentially pulled by the nth data loader, namely original data group 1, original data group 2 and original data group 3, and the first buffer space does not have remaining storage capacity, then the first buffer space may be used to store the data generated by the nth object group. If there is remaining storage capacity, the data filter associated with the nth data loader can obtain the original data group 1 from the first buffer space, and perform data filtering on the original data group 1 to obtain the corresponding data group (that is, data group 1), and store data group 1 in the second buffer space. After storing data group 1 in the second buffer space, the original data group 1 in the first buffer space is deleted. At this time, it is detected that there is remaining storage capacity in the first buffer space, and the nth data loader can continue to pull the new original data group generated by the nth object group; and after completing the buffer replay simulation of data group 1 in the second buffer space, data group 1 is deleted from the second buffer space.

[0054] In one embodiment, in the above-mentioned simulation system, N data loaders are allocated to N object groups under the mth division mode, and one data loader is allocated to one object group; however, for any object group under the mth division mode (nth object group), multiple data loaders can also be allocated to the nth object group, and the number of allocated data loaders is equal to the number of cache modules corresponding to the nth object group, and they correspond one to one; based on this, after any data loader corresponding to the nth object group pulls the data generated by each object in the corresponding object group from the target space, a data filter associated with the data loader can also be created to filter the data pulled by the data loader to obtain the data group of the object group corresponding to the nth data loader (i.e., the nth object group); wherein, the creation method of the data filter associated with any data loader can be dynamic creation or static creation. Based on this, in Figure 3 Based on the structure of the simulation system shown, see Figure 7, is a structural diagram of another simulation system provided in an embodiment of the present application. In the simulation system, multiple data loaders are configured for any object group under any division method, the number of data loaders configured for any object group is equal to the number of cache modules configured for the any object group, and they correspond one to one, and one data loader corresponds to one data filter; when multiple data loaders are configured for any object group under any division method, the processing logic of each data loader and the corresponding data filter is similar to the processing logic when one data loader is configured for any object group under any division method, and will not be repeated here.

[0055] Based on the above, it can be known that in the simulation system, a data loading module corresponding to any object group is configured, whether one data loader is assigned to any object group or data loaders are assigned to any object group that are equal to the number of cache modules configured for any object group, a cache module corresponding to any object group corresponds to one data loading module.

[0056] S403, in the simulation system, using the cache parameters indicated by the configuration information to perform cache replay simulation on the target data set to obtain a cache simulation result corresponding to the target data set.

[0057] In one embodiment, the target data set includes M data subsets, different data subsets correspond to different division methods, and M is an integer greater than 1; the data subset corresponding to any division method includes a data group of at least one object group, and each object group involved in any data subset is obtained by dividing multiple objects according to the corresponding division method; wherein the simulation system includes multiple cache modules, and the configuration information is used to indicate the cache parameters to be used by each cache module. In a simulation system, a target data set is subjected to cache replay simulation using cache parameters indicated by configuration information to obtain a cache simulation result corresponding to the target data set, which may include: traversing M data subsets in the target data set to determine the mth data subset currently traversed, m∈[1,M]; determining, among multiple cache modules in the simulation system, cache modules corresponding to each data group in the mth data subset, one data group corresponds to at least two cache modules; calling each cache module corresponding to each data group in the mth data subset respectively, and performing cache replay simulation on the corresponding data group using the corresponding cache parameters indicated by the configuration information to obtain a cache simulation result of the mth data subset; the cache simulation result of the mth data subset includes: the cache simulation result of each data group in the mth data subset; the cache simulation result of any data group includes: the result obtained by performing cache replay simulation on each cache module corresponding to the corresponding data group; continuing to traverse the target data set until all M data subsets in the target data set are traversed, and obtaining a cache simulation result corresponding to the target data set; the cache simulation result corresponding to the target data set includes: the cache simulation result of each data subset in the target data set. That is to say, after obtaining the data group of any object group under the m-th division method, the data group of any object group can be subjected to cache replay simulation through multiple cache modules configured for the any object group, so as to obtain the cache simulation results of the data group of any object group obtained through each cache module corresponding to the any object group; further, the cache simulation results of the data group of each object group under the m-th division method obtained through each cache module corresponding to the corresponding object group constitute the cache simulation results under the m-th division method (that is, the cache simulation results of the m-th data subset); further, the cache simulation results under each division method in the M division methods constitute the cache simulation results of the target data set.

[0058] For example, if the multiple objects are object 1, object 2, object 3, and object 4, and the configuration information includes M division methods, M=3, if m=1, the multiple objects are divided into three object groups (N=3) based on the first division method, the first object group includes object 1, the second object group includes object 2 and object 3, and the third object group includes object 4, and the data groups of the three object groups constitute the first data subset; if in the simulation system, Q_1 cache modules are configured for the first object group under the first division method, and the cache parameters of the Q_1 cache modules are different, Q_2 cache modules are configured for the second object group under the first division method, and the cache parameters of the Q_2 cache modules are different, and Q_3 cache modules are configured for the third object group under the first division method, and the cache parameters of the Q_3 cache modules are different. Taking the data group of the first object group under the first division mode as an example, each cache module in the Q_1 cache modules corresponding to the data group (i.e., the Q_1 cache modules corresponding to the first object group) is called respectively, and the corresponding cache parameters indicated by the configuration information are used to perform cache replay simulation on the corresponding data group to obtain Q_1 cache simulation results of the data group. One cache module corresponds to one cache simulation result, that is, the cache simulation results obtained by the data group of the first object group through each cache module corresponding to the first object group are obtained; based on this, the Q_1 cache modules corresponding to the data group of the first object group under the first division mode are The cache simulation results corresponding to the data group of the second object group, and the Q_3 cache simulation results corresponding to the data group of the third object group constitute the cache simulation results under the first division method (i.e., the cache simulation results of the first data subset). Furthermore, the cache simulation results of the first data subset (i.e., the cache simulation results under the first division method), the cache simulation results of the second data subset (i.e., the cache simulation results under the second division method), and the cache simulation results of the third data subset (i.e., the cache simulation results under the third division method) constitute the cache simulation results of the target data set.

[0059] In one embodiment, taking the mth data subset obtained by the mth division method as an example, calling each cache module corresponding to each data group in the mth data subset respectively, and using the corresponding cache parameters indicated by the configuration information to perform cache replay simulation on the corresponding data group, it can include: performing data block processing on the nth data group in the mth data subset to obtain P data blocks; wherein n is a positive integer and is less than or equal to the number of data groups in the mth data subset, and P is a positive integer; calling each cache module corresponding to the nth data group respectively, and using the corresponding cache parameters indicated by the configuration information to perform cache replay simulation on each data block in the P data blocks. Among them, the block size used for data block processing on the nth data group in the mth data subset can be set according to specific needs, for example, it can be set to 32k, 64k, 128k, etc. When data block processing is performed on the nth data group in the mth data subset, the nth data group is preferentially divided into data blocks of the size indicated by the block size; further, the block size should be less than or equal to the space allocation unit size of the cache module, and the space allocation unit size of the cache module refers to: the size of a single cache unit in the cache space of the cache module. Furthermore, data blocking processing for the nth data group in the mth data subset may be performed by a data filter, that is, after the data filter performs filtering processing on the data generated by each object in any object group pulled by the data loader to obtain the data group of the object group, the data filter may also perform data blocking processing on the data group of the object group, so that the cache module performs cache replay simulation based on the data blocks obtained by the data blocking; wherein, the cache simulation result obtained by the nth data group via a corresponding cache module includes the P data blocks obtained by data blocking of the nth data group, and the cache simulation results obtained by cache replay simulation via the cache module respectively. Furthermore, a cache module may include a simulated cache process and a data statistics process in the process of performing cache replay simulation on a data block using corresponding cache parameters indicated by configuration information, wherein the simulated cache process includes simulating the cache processing process of the cache system on the data block based on the cache module, for example, including the following processing logic: whether it is necessary to cache the data block in the cache space of the cache module, and if it is necessary to cache the data block, then in which cache unit of which address in the cache space the data block needs to be cached, and the relevant processing logic may be collaboratively implemented based on the cache model and cache replacement algorithm configured by the cache module; further, the data statistics process is used to obtain, after the cache simulation process is completed, a cache simulation result obtained by performing cache replay simulation on the data block through the cache module.

[0060] In one embodiment, the cache simulation result may include an indicator value of a cache indicator for measuring the cache effect, wherein the cache indicator may include one or more cache indicators such as hit rate, access volume, and access rate; for the sake of convenience of explanation, the cache simulation result obtained by a data group of an object group through a corresponding cache module includes the cache simulation results obtained by performing cache replay simulation on each data block obtained by data segmentation of the data group of the object group through the cache module. Taking the cache simulation result obtained by performing cache replay simulation on a data block corresponding to an object group through a cache module as an example, the above-mentioned cache indicators are introduced. Among them, the index value of the access volume corresponding to a specific data block is: the number of data blocks (including the specific data block) that the cache module performs cache replay simulation on as of the end of the cache module performing cache replay simulation on the specific data block (i.e., the end of the cache simulation process); since the cache module is configured with a cache replacement algorithm, when there is no remaining capacity in the cache space corresponding to the cache module, if data still needs to be cached, then the data to be replaced can be decided from the data stored in the cache space based on the cache replacement algorithm, and the decided data can be deleted from the cache space, and the data to be cached can be cached in the cache space; in the process of the simulation system calling the cache module to perform cache replay simulation on the data block, if the data block to be cached detects that there is a data block identical to the data block in the cache space corresponding to the cache module during the cache replay simulation, then the process is recorded as a hit, based on this, the index value of the hit rate corresponding to a data block can be: the proportion of the number of hits of the cache module in the access volume as of the end of the cache module performing cache replay simulation on the specific data block (i.e., the end of the cache simulation process), and the higher the hit rate, the better the cache effect of the cache module.

[0061] In one embodiment, taking a cache module corresponding to an object group as an example, since the data loader corresponding to the cache module will pull the data generated by the object group in sequence, each time pulling a specified amount of data; for example, if the specified data amount is set to 1M, when the data loader pulls data for the first time, it can pull 1M of data from the starting point of the data generated by the object group, and the starting point of the data can be recorded as a data node. When the data loader pulls data for the second time, it can pull 1M of data from the data generated by the object group, 1M away from the starting point, and the place 1M away from the starting point can be recorded as a data node, and so on. The node in the data generated by the object group where the data loader starts pulling data each time can be considered as a data node. Based on this, the cache module can write a snapshot after the cache replay simulation of the data group filtered by the data pulled by the data loader each time is completed, wherein the snapshot written by the cache module after the cache replay simulation of the data group filtered by the data pulled by the data loader at a certain time is used to indicate the data consumption node of the cache replay simulation performed by the cache module this time, and the data consumption node is the data node corresponding to the data generated by the object group for the data pulled this time; by performing the cache replay simulation of the data group filtered by the data loader each time in the cache module after the cache replay simulation is completed. The post-write snapshot can make it possible that: after the cache module completes the cache replay simulation of the data group filtered by the data pulled by the data loader at any time, if the simulation system is shut down (for example, due to a simulation system crash, etc.), when the simulation system is reawakened, the data loader can, based on the written snapshot, pull a specified amount of data from the data generated by the object group, starting from the node indicated by the data consumption node in the written snapshot, and then continue to pull data from the corresponding data consumption node, so that the simulation system can continue to process the data generated by the object group from the corresponding data consumption node.

[0062] S404: Visually output the cache simulation results corresponding to the target data set.

[0063] In one embodiment, when the cache simulation results corresponding to the target data set are visually output, the visual output can be based on the cache simulation results corresponding to the data blocks; further, the visual output can be based on images, tables, etc. The embodiment of the present application does not limit the visual output method, and the visual output method can also be obtained through configuration. Further optionally, the effect of interval output of cache simulation results can be achieved by setting the output interval of the cache simulation results; taking the output of the cache simulation results corresponding to a cache module as an example, since the data loader corresponding to the cache module will sequentially obtain the data of a specified amount of data, and the data filter will perform filtering processing to obtain a data group, and the data group will be processed by data block processing to obtain multiple data blocks, so in the process of cache replay simulation of the cache module, each data block obtained by each data group block is obtained in turn, and each data block is cache replay simulation performed in turn; based on this, if the output interval is set to 0, then in this case, the system processing device will output the cache simulation results corresponding to the cache replay simulation performed by the cache module on each data block obtained by each data group block; If the output interval is set to the interval corresponding to a data group, and it is set to output only the cache simulation result of the last data block among the data blocks obtained for the data group blocks, then after the system processing device outputs the cache simulation result of the last data block obtained by the cache module for a data group block, it does not output the cache simulation result corresponding to the next data group, and only outputs the cache simulation result of the corresponding data block after obtaining the cache simulation result of the last data block obtained for the next data group block, and so on; optionally, among the cache simulation results of the data blocks obtained by the cache module, which data blocks will have their cache simulation results output can be set according to specific needs, such as the above-mentioned output of only the cache simulation result of the last data block among the data blocks obtained for the data group blocks. Further optionally, a post-processing module can be set in the simulation system to execute the relevant process of visually outputting the cache simulation results corresponding to the target data set.

[0064] In one embodiment, the system processing device outputs the cache simulation result corresponding to the target data set visually, so as to develop the cache system based on the cache simulation result corresponding to the target data set. Specifically, the target configuration information can be determined from the configuration information based on the cache simulation result corresponding to the target data set, and the cache system can be developed based on the target configuration information. The target configuration information may include: the target partitioning method determined from the M partitioning methods, and the cache parameters used by the target cache module are determined from the multiple cache modules configured corresponding to each object group under the target partitioning method as the target cache parameters. Optionally, the determination process of the target configuration information can be determined manually by analyzing the cache simulation result corresponding to the target data set, or it can be determined by the system processing device. An exemplary determination process of the target configuration information can be as follows: for any object group under any partitioning method, first, according to the cache simulation result obtained by each data block obtained by the data group block of the any object group through a cache module, the cache effect evaluation value of the data group of the any object group after the cache replay simulation is performed by the cache module can be determined, and then the cache effect evaluation value of the data group of the any object group after the cache replay simulation is performed by each cache module corresponding to the object group can be obtained. Then, the maximum cache effect evaluation value can be selected from the cache effect evaluation values ​​after cache replay simulation of each cache module corresponding to the object group. The maximum cache effect evaluation value can indicate the reference cache module with the best cache effect in each cache module corresponding to the object group, and then the maximum cache effect evaluation value corresponding to each object group under any division method can be obtained. Then, the maximum cache effect evaluation value corresponding to each object group under any division method can be averaged to obtain the average cache effect evaluation value under any division method, and the average cache effect evaluation value under any division method can be used to measure the optimal cache effect that can be achieved under any division method. By comparing the size of the average cache effect evaluation value under M division methods, the division method corresponding to the maximum average cache effect evaluation value can be selected as the target division method, and the reference cache module corresponding to each object group under the target division method is determined as the target cache module corresponding to the corresponding object group, and then the target cache parameters of each object group under the target division method can be obtained.Optionally, for any object group under any division method, when determining the cache effect evaluation value of the data group of any object group after cache replay simulation is performed on the data group of any object group according to the cache simulation results obtained by each data block obtained by the data group of any object group respectively through a cache module, a cache simulation result of a data block can be selected from the cache simulation results of each data block obtained by the data group of any object group (for example, a cache simulation result of a data block can be randomly selected, for example, the cache simulation result of the last data block can be selected), and the cache simulation result of the selected data block can be determined based on the cache simulation result of the selected data block. The cache effect evaluation value of the data group of the image group after the cache replay simulation is performed by the cache module; since the cache simulation result of the selected data block includes the index value of the cache index of the selected data block, one or more index values ​​of the cache index can be selected from the index values ​​of the cache index, and the corresponding cache effect evaluation value can be determined based on the index value of each selected cache index. Exemplarily, the index value of the hit rate can be selected to determine the corresponding cache effect evaluation value, and the determined cache effect evaluation value should be positively correlated with the index value of the hit rate. That is, the larger the index value of the hit rate, the better the cache effect is, and the larger the determined cache effect evaluation value is.

[0065] In one embodiment, since the block size used for data block processing of any data group can be set according to specific needs, taking the nth data group in the mth data subset as an example, the block size should be less than or equal to the space allocation unit size of the cache module, and the space allocation unit size of the cache module can also be configured according to specific needs; in a feasible implementation, the block size and the space allocation unit size can be set equal; based on this, the configuration information can also include the space allocation unit size of the cache module, and the space allocation unit size refers to: the size of a single cache unit in the cache space of each cache module ; The system processing device performs data block processing on the nth data group in the mth data subset to obtain P data blocks, which may include: obtaining the space allocation unit size from the configuration information, and performing data block processing on the nth data group in the mth data subset according to the space allocation unit size to obtain P data blocks; wherein the size of each data block in the P data blocks is less than or equal to the space allocation unit size; wherein, when performing data block processing on the nth data group in the mth data subset according to the space allocation unit size, the nth data group is preferentially divided into data blocks of a size indicated by the space allocation unit size (the same as the block size).

[0066] Furthermore, when the system processing device performs data block processing on the nth data group in the mth data subset according to the space allocation unit size, since the nth data group is preferentially divided into data blocks of the size indicated by the space allocation unit size (the same as the block size), it is also possible to obtain data blocks with a size smaller than the space allocation unit size. When the size of the data block is too small, if the corresponding data block is cached, the amount of data stored in the cache unit is small, which will lead to a waste of cache space storage resources. Based on this, the embodiment of the present application proposes that the size of the data block that needs to be simulated cached can be limited by configuring the disk granularity. When the size of the data block is smaller than the disk granularity, the corresponding data block does not need to be simulated cached. Then, the sizes of the P data blocks obtained by block- ing the nth data group should be greater than or equal to the disk granularity. In this case, the configuration information also includes a disk placement granularity, which is used to indicate the minimum size of a data block allowed to be cached; the system processing device performs data block processing on the nth data group in the mth data subset according to the space allocation unit size to obtain P data blocks, including: performing data block processing on the nth data group in the mth data subset according to the space allocation unit size to obtain multiple data blocks; obtaining the disk placement granularity from the configuration information, and filtering out P data blocks from the multiple data blocks according to the disk placement granularity and the size of each data block in the multiple data blocks; wherein the size of each data block in the P data blocks is greater than or equal to the disk placement granularity.

[0067] In one embodiment, when the cache module performs cache replay simulation on P data blocks obtained from the nth data group block, since the P data blocks are obtained from the same data group block, the cache module can preferentially cache the P data blocks in cache units with consecutive addresses; based on this, the configuration information can also include a cache allocation strategy, and the cache allocation strategy is used to indicate: from the cache space of the cache module, preferentially allocate cache units with consecutive addresses to store data blocks; based on this, the system processing device respectively calls each cache module corresponding to the nth data group, and uses the corresponding cache parameters indicated by the configuration information to perform cache replay simulation on each data block in the Pth data block, which can include: for any cache module corresponding to the nth data group, calling any cache module The block searches for cache units with continuous addresses from the corresponding cache space; if K cache units with continuous addresses are found, and K≥P, then call any cache module to use the corresponding cache parameters indicated by the configuration information to replay the P data block caches to P cache units in the K cache units, with one cache unit caching one data block; if K cache units with continuous addresses are found, and K<P, then call any cache module to use the corresponding cache parameters indicated by the configuration information to replay the K data block caches in the P data blocks to the K cache units; and randomly select PK cache units from the cache units other than the cache units with continuous addresses in the corresponding cache space, and replay the data block caches other than the K data blocks in the P data blocks to the randomly selected PK cache units. Further optionally, when the cache module performs a cache replay simulation on the P data blocks obtained by the nth data group block, the maximum number of cache units with continuous addresses required to cache the P data blocks (recorded as the number of continuous cache units K) can also be configured. * ), in this case, if K cache units with consecutive addresses are found, and K ≥ K * ≥P, then call any cache module to use the corresponding cache parameters indicated by the configuration information to cache and replay P data blocks to P cache units among the K cache units, one cache unit caches one data block, if K cache units with consecutive addresses are found, and K≥P>K * That is, even if more cache units with consecutive addresses than the number of cache units required for P data blocks can be found, due to the number of configured consecutive cache units (K * ) is limited, and only K in P data blocks can be * The data blocks are cached and replayed to K consecutive addresses. *cache units, and randomly cache the remaining data blocks in other cache units, for example, they can also be cached in unused cache units among the K cache units with consecutive addresses, or they can be cached in other cache units other than the K cache units with consecutive addresses. It is worth noting that the configuration information, such as the space allocation unit size, the disk granularity, and the number of consecutive cache units in the cache allocation strategy, will also have an impact on the cache effect when there are different configurations in the simulation system. Therefore, by configuring different allocation unit sizes in the simulation system, it is possible to determine the target allocation unit size required for the development of the cache system, by configuring different disk granularities in the simulation system, it is possible to determine the target disk granularity required for the development of the cache system, and by configuring different numbers of consecutive cache units in the simulation system, it is possible to determine the target number of consecutive cache units required for the development of the cache system.

[0068] In an embodiment of the present application, a simulation system corresponding to a cache system in a content distribution network can be determined, and configuration information of the simulation system can be obtained, wherein the configuration information is used to indicate cache parameters to be used by the simulation system, and the cache system is shared by multiple objects for caching data generated by multiple objects; after a target data set including data generated by multiple objects is obtained through the simulation system, the cache parameters indicated by the configuration information can be used in the simulation system to perform cache replay simulation on the target data set to obtain a cache simulation result corresponding to the target data set, and the cache simulation result corresponding to the target data set can be output visually; a functional simulation of a cache system in a content distribution network can be implemented based on the simulation system, and then the cache system can be developed based on the cache simulation result obtained by simulation, so that the developed cache system can achieve a better cache effect; and the cache parameters (including a cache model and a cache replacement algorithm) to be used by the simulation system can be configured based on the configuration information, so that the cache parameters to be simulated can be configured according to the development requirements of the cache system, and the cache simulation results of the same data under different cache parameters are output, which can provide a reference for the development of the cache system and increase the convenience of cache system development. For example, the cache parameters to be adopted by the cache system can be determined based on the cache simulation results of the same data under different cache parameters. Further, the present application also supports configuring the data acquisition method of the simulation system through M division methods for dividing multiple objects included in the configuration information, and then based on the configured multiple division methods, the simulation system can be used to simulate the cache processing process of the cache system under different division methods to obtain corresponding cache simulation results, so that the cache system can be developed based on the corresponding cache simulation results, which can provide a reference for the development of the cache system and increase the convenience of cache system development. For example, it can be determined that when the cache system caches the data generated by multiple objects, the data generated by the object group obtained under which division method is cached. Further, based on the configuration of cache parameters in the simulation system and the configuration of the division method of multiple objects, the cache simulation results of the same data under different cache parameters and different division methods can be output. The cache system is developed based on the output cache simulation results, which can provide a reference for the development of the cache system and increase the convenience of cache system development. For example, it can be determined that when the cache system caches the data generated by multiple objects, the data generated by the object group obtained under which division method is cached, and the cache parameters used for the data generated by the divided object group are cached. Furthermore, the present application also supports the configuration of space allocation unit size, disk granularity and cache allocation strategy during the simulation process, so as to provide a reference for the configuration of corresponding parameters when developing the cache system, thereby increasing the convenience of cache system development.Furthermore, the present application also supports cache replay simulation based on offline data and data actually used in the caching process of the CDN server, and there is no need to deploy a simulation system on the CDN server and execute the simulation process. Therefore, the simulation system is simple to deploy, easy to modify and debug, and will not affect the CDN server. Moreover, the cache simulation results output by the present application may include indicator values ​​of cache indicators indicating the cache effect, and support visual output in the form of images or tables, so that the developers of the cache system can understand the system performance and observe various cache indicators more intuitively and comprehensively.

[0069] Based on the relevant embodiments of the above-mentioned system processing method, this application will further introduce the processing process of the cache module and the processing process of the data loader. Taking any object group under any division method as an example, a data loader is configured for any object group in the simulation system, or a data loader with the same number of cache modules as that corresponding to any object group is configured for any object group in the simulation system. A cache module corresponding to any object group corresponds to a data loader, and different cache modules do not interfere with each other during cache replay simulation. Based on this, taking a cache module and a data loader in a simulation system as an example, see Figure 8 , is a schematic diagram of another simulation system provided in an embodiment of the present application, and the simulation system may include a data loader (dataLoader), a data filter (dataFilter), a cache module (cache) and a main program process (main); wherein, the data loader and the cache module in the simulation system are created by the main program process when the simulation system is started, and can be configured according to the configuration information when the simulation system is started, and can be specifically configured based on the configuration information and the provided data loader class (dataLoader class) and the cache module class (cache class).

[0070] Among them, the data loader can be used for related processes such as data pulling, data clearing and data maintenance, among which the data pulling process may include: when it is detected that there is remaining storage capacity in the first buffer space corresponding to the data loader (recorded as the original data buffer), the data generated by the corresponding object group of the data loader is pulled; the data clearing process can clear the data that has been used up, and may include: after filtering the data generated by the corresponding object group pulled once through the data filter associated with the data loader to obtain the corresponding data group, and storing the corresponding data group in the second buffer space (recorded as the available data buffer), the data generated by the corresponding object group pulled once (that is, the data obtained by the data filter from the first buffer space) is The method further comprises the steps of: deleting the data group from the first buffer space after the data group is segmented into two pieces, and deleting the corresponding data group from the second buffer space after completing the buffer replay simulation of any data group in the second buffer space; further, since the data filtering module will perform data segmentation processing on the data group to obtain data blocks, the second buffer space may store data blocks obtained by segmenting the data group, and deleting the data group from the second buffer space means deleting each data block obtained by segmenting the data group from the second buffer space; the data maintenance process may include the steps of transferring the data group in the second buffer space to the buffer module for cache replay simulation, and further includes the steps of transferring each data block obtained by segmenting the data group in the second buffer space to the buffer module in sequence for cache replay simulation.

[0071] Among them, the data filter associated with the data loader can be used for data filtering, data segmentation and other related processes, wherein the data filtering process may include: performing data filtering processing on the data generated by the object group pulled by the data loader each time to obtain the corresponding data group related process, the data segmentation process may include: performing data segmentation on the data group obtained by each filtration to obtain data blocks, and the data blocks obtained by segmenting the data group can be directly passed to the cache module for use (i.e., passed to the cache module for cache replay simulation), that is, the data filter can generate flow data (i.e., one data block at a time) that can be directly used by the cache module. Among them, the data filter associated with the data loader can be dynamically created, created each time data filtering and data segmentation are needed, and destroyed after use, without context; further, the data filter can apply different filtering and parsing methods, and after the data loader pulls a specified amount of data, the corresponding data filter is dynamically created and data filtering and data segmentation are performed, and the data filter will be destroyed after completion, so the filtering is stateless on a large time scale. The cache module can be used to perform cache replay simulation on the data (data block) obtained from the corresponding data loader (the second buffer space), which can include a simulated cache process and a data statistics process.

[0072] Among them, the main program process will be run when the simulation system is started. The data loader and cache module can be created according to the configuration information by running the main program process to realize the configuration of the data loader and cache module in the simulation system; the main program process can also be executed cyclically, including the cyclic execution of the following processing logic: (1) corresponding Figure 8 Step 1 in the above example clears the used data. This step may include: traversing each data loader to check whether there is any used data in the corresponding data loader. If so, calling the corresponding data loader to perform a data clearing process to clear the used data. (2) Figure 8 In step 2, data is pulled when there is remaining storage capacity in the first buffer space. The process may include: traversing each data loader, detecting whether there is remaining storage capacity in the first buffer space of the corresponding data loader, and if so, creating a thread for executing data pulling to pull data, that is, calling the data loader with remaining storage capacity in the first buffer space to execute processes related to data pulling. (3) Scheduling the execution of cache modules in the simulation system, that is, the main program process can schedule each cache module in the simulation system so that each cache module can be awakened only when the scheduling conditions are met. Taking a cache module as an example, after waking up the cache module, the data (data block) accumulated in the data loader from the last awakening to the current awakening that has not been cached and replayed by the cache module can be cached and replayed, and the cache module can be released after the cache replay simulation of the corresponding data is completed, and it can be awakened again when the scheduling conditions are met next time. Compared with keeping each cache module in the simulation system awake, this can save processing resources (of course, each cache module in the simulation system can also remain awake after being created after the simulation system is started, and when there is data that needs to be cached and replayed, the corresponding data can be cached and replayed, and when there is no data that needs to be cached and replayed, it waits for the arrival of the data to be processed). Based on this, the corresponding Figure 8 In step 3 of the above, the main program process can wake up the cache module that meets the scheduling condition according to the scheduling condition, wherein the scheduling condition can be set according to specific requirements, for example, the resources required for the operation of the cache module can be satisfied by the hardware; the process may include: by running the main program process, each cache module can be traversed to check whether each cache module can be scheduled for execution (i.e., whether the scheduling condition is met); if a cache module can be scheduled for execution, the cache module is woken up, and the awakened cache module can first read the last written snapshot and perform the following operations: (3.1) corresponding to Figure 8Step 4 in the above is to obtain the data (data block) that needs to be cached and replayed for this wakeup from the data loader corresponding to the awakened cache module, wherein the data that needs to be cached and replayed for this wakeup includes: the data in the interval from the end position of the cache module when it was last awakened to the data position available so far, which is checked from the data loader corresponding to the awakened cache module; the end position of the cache module when it was last awakened, that is, the position of the data processed by the cache module when it was last awakened, that is, the position indicated by the data consumption node in the last written snapshot; the data position available so far, that is, the position of the data node where the data pulled by the data loader is located when the cache module is awakened this time. For example, if the cache module ends its last awakening and is awakened this time, 5 data groups of the corresponding object group are obtained by pulling data through the data loader and filtering with the data filter, then the 5 data groups (that is, the individual data blocks obtained by the 5 data groups) are the data that need to be cached and replayed for this wakeup (that is, the data that needs to be used this wakeup). (3.2) Corresponding Figure 8 Step 5 in the above example performs a simulated cache process and a data statistics process for the data to be used in this wakeup. (3.3) Figure 8 In step 6, the data that has completed the cache replay processing is marked in the buffer to implement the relevant process of data clearing.

[0073] See also Fig. 9 , which is a schematic diagram of a cache module for cache replay simulation provided in an embodiment of the present application; firstly, the main program process (main) is run and the data maintenance-related process (corresponding to the data loader) is executed. Fig. 9 dataLoader.try_get_ready_file_end), so that the data loader can support the transfer of data to the cache module; after a cache module is awakened by the main program process, the main program process can detect whether the cache module has data to be used in this awakening. If it is detected that the cache module has data to be used in this awakening, a new thread is started to execute. This new thread is used to use the data to be used in this awakening (corresponding to Fig. 9 In the process of caching and replaying the data needed for this wakeup, the cache module will read the previous snapshot (corresponding to Fig. 9cache.readHistory), and perform cache replay simulation on the data required for this wakeup in turn, and write new snapshots until the data required for this wakeup is cached and replayed. In this process, the cache module can sequentially obtain each data group in the data required for this wakeup. For the data group obtained this time, the cache module can open each data block obtained by the data group obtained this time one by one. The cache module performs cache replay simulation on a data block in the data group obtained this time (including executing the simulated cache process and the data statistics process) to obtain the cache simulation result corresponding to the data block. After obtaining the cache simulation result corresponding to the data block, the data saving process will be called to store the cache simulation result of the data block (corresponding to Fig. 9 cache.save_statistic_data), optionally it can be stored in the redis database, further optionally, the data preservation conditions can be configured, and the data is stored only when the data preservation conditions are met, for example, it is set to indicate that only the cache simulation results of the last data block obtained from each data group block need to be output, then the cache simulation results of other data blocks obtained from the data group block do not need to be stored, wherein the cache module can implement the data statistics process by reading the function of the cached statistical information, and the cache module can also manage the cached data, for example, it can mark the data at the end of the cache replay simulation, and then facilitate the deletion from the corresponding buffer space after the corresponding data is cached and replayed by each cache module configured accordingly. Then the cache module performs a cache replay simulation on the next data block in the data group obtained this time. Before the cache module performs cache replay simulation on a data block in the data group obtained this time, it is necessary to construct a structure object (for example, a cur_chunk object) based on the data block, and after the structure object is verified (for example, including the validity verification), the cache module performs cache replay simulation on the structure object corresponding to the data block in the data group obtained this time, and the cache module constructs a corresponding structure object (corresponding to the next data block in the data group obtained this time) based on the next data block in the data group obtained this time. Fig. 9 After each data block in the acquired data group is cached and replayed, a snapshot is written (corresponding to Fig. 9 cache.writeHistory), and obtain the next data group in the data to be used in this wakeup, and pass it through the data loader (corresponding to Fig. 9 in dataLoader.inc_pre_process_file_end).

[0074] See also Fig.10, is a schematic diagram of a data loader provided in an embodiment of the present application for managing data. Taking the data loader corresponding to any object group under any division mode as an example, the storage path of the elements stored in the first buffer space (recorded as the original data buffer) of the data loader is recorded in the first file, and the elements stored in the first buffer space are data groups. The storage path of the elements stored in the second buffer space (recorded as the available data buffer) of the data loader is recorded in the second file, and the elements stored in the second buffer space are data blocks obtained by segmenting the data group. The main program process can detect whether there is remaining storage capacity in the first buffer space. If so, the data loader is called to execute the process related to data pulling (the data server in the figure can be a CDN server or a server storing offline data). The data of the specified data volume pulled by the data loader can be filtered through the data filtering module to obtain the corresponding data group, and the data group is segmented to obtain data blocks. After storing each data block obtained by segmentation in the second buffer space, the original data is deleted from the first buffer space (that is, the data corresponding to the specified data volume pulled is deleted from the first buffer space, and the corresponding storage path is deleted from the first file). The data loader can pass the data block in the second buffer space to the cache module. After the cache module completes the cache replay simulation of the data block, the data loader adds 1 to the counter corresponding to the corresponding position of (the data block) in the second buffer space (i.e., the available data buffer) to pass the next data block of the data block to the cache module, and after each cache module configured for the object group completes the cache replay simulation of the data block, the corresponding data block is deleted from the second buffer space, and the corresponding storage path is deleted from the second file; after the cache module completes the cache replay simulation of each data block of the data group, the data loader adds 1 to the counter corresponding to the corresponding position of (the data group) in the first buffer space (i.e., the original data buffer) to filter and segment the next data group of the data group through the data filtering module.

[0075] In an embodiment of the present application, a simulation system corresponding to a cache system in a content distribution network can be determined, and configuration information of the simulation system can be obtained, wherein the configuration information is used to indicate cache parameters to be used by the simulation system, and the cache system is shared by multiple objects for caching data generated by multiple objects; after a target data set including data generated by multiple objects is obtained through the simulation system, the cache parameters indicated by the configuration information can be used in the simulation system to perform cache replay simulation on the target data set to obtain a cache simulation result corresponding to the target data set, and the cache simulation result corresponding to the target data set can be output visually; a functional simulation of a cache system in a content distribution network can be implemented based on the simulation system, and then the cache system can be developed based on the cache simulation result obtained by simulation, so that the developed cache system can achieve a better cache effect; and the cache parameters to be used by the simulation system can be configured based on the configuration information, so that the cache parameters to be simulated can be configured according to the development requirements of the cache system, and the cache simulation results of the same data under different cache parameters are output, which can provide a reference for the development of the cache system and increase the convenience of cache system development. In addition, the present application also proposes to support the scheduling execution of the cache module in the simulation system, so that the cache module can be awakened only when the scheduling conditions are met. Compared with always keeping each cache module in the simulation system awake, processing resources can be saved.

[0076] Based on the above embodiments related to the system processing method, the present application embodiment provides a system processing device. Fig.11 , is a structural diagram of a system processing device provided in an embodiment of the present application, and the system processing device may include a processing unit 1101 and an output unit 1102. Fig.11 The system processing device shown can be used to perform the following operations:

[0077] The processing unit 1101 is used to determine a simulation system corresponding to the cache system and obtain configuration information of the simulation system during the process of developing the cache system in the content distribution network; the configuration information is used to indicate cache parameters to be used by the simulation system, the cache parameters are determined during the process of developing the cache system, and the cache system is shared by multiple objects to cache data generated by the multiple objects;

[0078] The processing unit 1101 is further configured to obtain a target data set through the simulation system, wherein the target data set includes data generated by the multiple objects;

[0079] The processing unit 1101 is further configured to perform cache replay simulation on the target data set in the simulation system using the cache parameters indicated by the configuration information to obtain a cache simulation result corresponding to the target data set;

[0080] The output unit 1102 is used to visually output the cache simulation result corresponding to the target data set.

[0081] In one embodiment, the configuration information further includes: M division methods for the multiple objects, where M is an integer greater than 1; when the processing unit 1101 obtains the target data set through the simulation system, the processing unit 1101 performs the following operations:

[0082] Traverse the M partitioning methods and determine the mth partitioning method currently traversed, m∈[1,M];

[0083] Dividing the multiple objects according to the m-th division method to obtain N object groups under the m-th division method, where N is a positive integer;

[0084] Acquire a data group of each of the N object groups through the simulation system to form a data subset corresponding to the m-th division method; wherein the data group of any object group includes data generated by each object in the corresponding object group;

[0085] Continue to traverse the M partitioning methods, and after all the M partitioning methods are traversed, obtain a target data set, wherein the target data set includes a data subset corresponding to each of the M partitioning methods.

[0086] In one embodiment, the simulation system includes a plurality of data loaders, and when the processing unit 1101 obtains the data group of each of the N object groups through the simulation system, it performs the following operations:

[0087] In the simulation system, N data loaders are allocated to the N object groups, and one data loader is allocated to one object group;

[0088] Each of the N data loaders is called respectively to load the data group of the corresponding object group from the target space; wherein the target space includes at least one of the following: a local space and the cache system.

[0089] In one embodiment, the processing unit 1101 respectively calls each of the N data loaders to load the data group of the corresponding object group from the target space and performs the following operations:

[0090] Calling the nth data loader among the N data loaders to pull data generated by each object in the corresponding object group from the target space, n∈[1,N];

[0091] Dynamically create a data filter associated with the nth data loader, and call the dynamically created data filter to filter the data pulled by the nth data loader to obtain a data group of the object group corresponding to the nth data loader;

[0092] After obtaining the data group of the object group corresponding to the nth data loader, destroying the data filter associated with the nth data loader.

[0093] In one embodiment, each time the data pulled by the nth data loader among the N data loaders needs to be stored in the first buffer space corresponding to the nth data loader; the processing unit 1101 is further used to:

[0094] Detecting the remaining storage capacity of the first buffer space corresponding to the nth data loader;

[0095] If it is detected that there is remaining storage capacity in the first buffer space, triggering the step of calling the nth data loader among the N data loaders to pull data generated by each object in the corresponding object group from the target space;

[0096] If it is detected that there is no remaining storage capacity in the first buffer space, wait until the storage capacity of the first buffer space is released, and then trigger the step of calling the nth data loader among the N data loaders to pull the data generated by each object in the corresponding object group from the target space.

[0097] In one embodiment, the dynamically created data filter obtains the data pulled by the nth data loader from the first buffer space corresponding to the nth data loader, and performs filtering processing;

[0098] After obtaining the data group of the object group corresponding to the nth data loader, the processing unit 1101 is further used to:

[0099] storing the data group of the object group corresponding to the nth data loader into the second buffer space corresponding to the nth data loader;

[0100] deleting the data acquired from the first buffer space by the dynamically created data filter from the first buffer space to release storage capacity of the first buffer space;

[0101] After completing the buffer replay simulation of any data group in the second buffer space corresponding to the nth data loader, the corresponding data group is deleted from the second buffer space.

[0102] In one embodiment, the cache parameters include: a cache model and a cache replacement algorithm; the cache model and the cache replacement algorithm support orthogonal configuration;

[0103] The cache model refers to a cache structure used to organize cache, and the cache replacement algorithm refers to an algorithm used to determine data to be replaced from data stored in the cache space.

[0104] In one embodiment, the target data set includes M data subsets, different data subsets correspond to different partitioning methods, and M is an integer greater than 1; the data subset corresponding to any partitioning method includes at least one data group of an object group, and each object group involved in any data subset is obtained by partitioning the multiple objects according to the corresponding partitioning method; wherein the simulation system includes multiple cache modules, and the configuration information is used to indicate the cache parameters to be used by each cache module;

[0105] The processing unit 1101 performs the following operations in the simulation system, when performing cache replay simulation on the target data set using the cache parameters indicated by the configuration information and obtaining the cache simulation result corresponding to the target data set:

[0106] Traverse the M data subsets in the target data set to determine the mth data subset currently traversed, m∈[1,M];

[0107] Determine, among a plurality of cache modules in the simulation system, a cache module corresponding to each data group in the mth data subset, wherein one data group corresponds to at least two cache modules;

[0108] Respectively call each cache module corresponding to each data group in the m-th data subset, use the corresponding cache parameters indicated by the configuration information to perform cache replay simulation on the corresponding data group, and obtain the cache simulation result of the m-th data subset; the cache simulation result of the m-th data subset includes: the cache simulation result of each data group in the m-th data subset; the cache simulation result of any data group includes: the result obtained by performing cache replay simulation on each cache module corresponding to the corresponding data group;

[0109] Continue to traverse the target data set until all M data subsets in the target data set are traversed, and obtain the cache simulation result corresponding to the target data set; the cache simulation result corresponding to the target data set includes: the cache simulation result of each data subset in the target data set.

[0110] In one embodiment, the processing unit 1101 respectively calls each cache module corresponding to each data group in the mth data subset, and performs the following operations when performing cache replay simulation on the corresponding data group using the corresponding cache parameters indicated by the configuration information:

[0111] Performing data block processing on the nth data group in the mth data subset to obtain P data blocks; wherein n is a positive integer and is less than or equal to the number of data groups in the mth data subset, and P is a positive integer;

[0112] Each cache module corresponding to the nth data group is called respectively, and the corresponding cache parameters indicated by the configuration information are used to perform cache replay simulation on each data block in the P data blocks.

[0113] In one embodiment, the configuration information further includes a space allocation unit size of the cache module, and the space allocation unit size refers to: the size of a single cache unit in the cache space of each cache module; when the processing unit 1101 performs data block processing on the nth data group in the mth data subset and obtains P data blocks, the processing unit 1101 performs the following operations:

[0114] Acquire the space allocation unit size from the configuration information, and perform data block processing on the nth data group in the mth data subset according to the space allocation unit size to obtain P data blocks;

[0115] Wherein, the size of each data block in the P data blocks is less than or equal to the size of the space allocation unit.

[0116] In one embodiment, the configuration information further includes a disk granularity, and the disk granularity is used to indicate the minimum size of a data block allowed to be cached; the processing unit 1101 performs data block processing on the nth data group in the mth data subset according to the space allocation unit size, and when P data blocks are obtained, performs the following operations:

[0117] Performing data block processing on the nth data group in the mth data subset according to the size of the space allocation unit to obtain a plurality of data blocks;

[0118] The disk-falling granularity is obtained from the configuration information, and P data blocks are filtered out from the multiple data blocks according to the disk-falling granularity and the size of each data block in the multiple data blocks; wherein the size of each data block in the P data blocks is greater than or equal to the disk-falling granularity.

[0119] In one embodiment, the configuration information further includes a cache allocation strategy, and the cache allocation strategy is used to indicate: from the cache space of the cache module, cache units with consecutive addresses are preferentially allocated to store data blocks;

[0120] The processing unit 1101 respectively calls each cache module corresponding to the nth data group, uses the corresponding cache parameters indicated by the configuration information, and performs the following operations when performing cache replay simulation on each data block in the Pth data block:

[0121] For any cache module corresponding to the nth data group, calling the any cache module to search for cache units with continuous addresses from the corresponding cache space;

[0122] If K cache units with consecutive addresses are found, and K≥P, then any cache module is called to use the corresponding cache parameters indicated by the configuration information to cache and replay the P data blocks to P cache units among the K cache units, with one cache unit caching one data block;

[0123] If K cache units with consecutive addresses are found and K<P, call any cache module to use the corresponding cache parameters indicated by the configuration information to replay the K data block caches among the P data blocks into the K cache units; and randomly select PK cache units from the cache units other than the cache units with consecutive addresses in the corresponding cache space, and replay the data block caches among the P data blocks other than the K data blocks into the randomly selected PK cache units.

[0124] According to one embodiment of the present application, Figure 4 The various steps involved in the system processing method shown can be Fig.11 The system processing device shown is executed by each unit. For example, Figure 4 Steps S401 to S403 shown in FIG. Fig.11 The processing unit 1101 in the system processing device shown is executed. Figure 4 Step S404 shown can be performed by Fig.11 The output unit 1102 in the system processing device shown is executed.

[0125] According to another embodiment of the present application, Fig.11The various units in the system processing device shown can be separately or completely combined into one or several other units to form, or some of the units can be further divided into multiple smaller units in function to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the functions of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. For example, the functions realized by the above-mentioned units can be realized by one processing unit. In other embodiments of the present application, the system processing device based on logical function division can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0126] According to another embodiment of the present application, the program can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 4 A computer program (including program code) for each step involved in the corresponding method shown in FIG. Fig.11 The system processing device shown in and the system processing method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer-readable storage medium, and loaded into the above-mentioned computing device through the computer-readable storage medium and run therein.

[0127] In an embodiment of the present application, a simulation system corresponding to a cache system in a content distribution network can be determined, and configuration information of the simulation system can be obtained, wherein the configuration information is used to indicate cache parameters to be used by the simulation system, and the cache system is shared by multiple objects for caching data generated by multiple objects; after a target data set including data generated by multiple objects is obtained through the simulation system, the cache parameters indicated by the configuration information can be used in the simulation system to perform cache replay simulation on the target data set to obtain a cache simulation result corresponding to the target data set, and the cache simulation result corresponding to the target data set can be output visually; a functional simulation of a cache system in a content distribution network can be implemented based on the simulation system, and then the cache system can be developed based on the cache simulation result obtained by simulation, so that the developed cache system can achieve a better cache effect; and the cache parameters to be used by the simulation system can be configured based on the configuration information, so that the cache parameters to be simulated can be configured according to the development requirements of the cache system, and the cache simulation results of the same data under different cache parameters are output, which can provide a reference for the development of the cache system and increase the convenience of cache system development.

[0128] Based on the above-mentioned system processing method and system processing device embodiments, the present application also provides a system processing device, which can be used to execute the above-mentioned system processing method. Fig.12 , is a structural diagram of a system processing device provided in an embodiment of the present application. Fig.12 The system processing device shown may include at least a processor 1201, an input interface 1202, an output interface 1203, and a computer storage medium 1204. The processor 1201, the input interface 1202, the output interface 1203, and the computer storage medium 1204 may be connected via a bus or other means.

[0129] The computer storage medium 1204 may be stored in the memory of the system processing device. The computer storage medium 1204 is used to store a computer program. The computer program includes program instructions. The processor 1201 is used to execute the program instructions stored in the computer storage medium 1204. The processor 1201 (or CPU (Central Processing Unit)) is the computing core and control core of the system processing device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the above-mentioned system processing method flow or corresponding functions.

[0130] The embodiment of the present application also provides a computer storage medium (Memory), which is a memory device in a system processing device for storing programs and data. It is understandable that the computer storage medium here can include both the built-in storage medium in the terminal and the extended storage medium supported by the terminal. The computer storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor 1201 are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer storage medium here can be a high-speed random access memory (RAM) memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer storage medium located away from the aforementioned processor.

[0131] In one embodiment, the processor 1201 may load and execute one or more instructions stored in a computer storage medium to implement the above-mentioned Figure 4 In the corresponding steps of the method in the embodiment of the system processing method, in a specific implementation, one or more instructions in the computer storage medium are loaded by the processor 1201 and execute the following steps:

[0132] In the process of developing a cache system in a content distribution network, a simulation system corresponding to the cache system is determined, and configuration information of the simulation system is obtained; the configuration information is used to indicate cache parameters to be used by the simulation system, the cache parameters are determined in the process of developing the cache system, and the cache system is shared by multiple objects to cache data generated by the multiple objects;

[0133] Acquire a target data set through the simulation system, wherein the target data set includes data generated by the multiple objects;

[0134] In the simulation system, the cache parameters indicated by the configuration information are used to perform cache replay simulation on the target data set to obtain a cache simulation result corresponding to the target data set;

[0135] The cache simulation results corresponding to the target data set are output visually.

[0136] In one embodiment, the configuration information further includes: M division methods for the multiple objects, where M is an integer greater than 1; when the processor 1201 obtains the target data set through the simulation system, the processor 1201 performs the following operations:

[0137] Traverse the M partitioning methods and determine the mth partitioning method currently traversed, m∈[1,M];

[0138] Dividing the multiple objects according to the m-th division method to obtain N object groups under the m-th division method, where N is a positive integer;

[0139] Acquire a data group of each of the N object groups through the simulation system to form a data subset corresponding to the m-th division method; wherein the data group of any object group includes data generated by each object in the corresponding object group;

[0140] Continue to traverse the M partitioning methods, and after all the M partitioning methods are traversed, obtain a target data set, wherein the target data set includes a data subset corresponding to each of the M partitioning methods.

[0141] In one embodiment, the simulation system includes a plurality of data loaders, and when the processor 1201 obtains the data group of each of the N object groups through the simulation system, the processor 1201 performs the following operations:

[0142] In the simulation system, N data loaders are allocated to the N object groups, and one data loader is allocated to one object group;

[0143] Each of the N data loaders is called respectively to load the data group of the corresponding object group from the target space; wherein the target space includes at least one of the following: a local space and the cache system.

[0144] In one embodiment, the processor 1201 calls each of the N data loaders respectively to load the data group of the corresponding object group from the target space and performs the following operations:

[0145] Calling the nth data loader among the N data loaders to pull data generated by each object in the corresponding object group from the target space, n∈[1,N];

[0146] Dynamically create a data filter associated with the nth data loader, and call the dynamically created data filter to filter the data pulled by the nth data loader to obtain a data group of the object group corresponding to the nth data loader;

[0147] After obtaining the data group of the object group corresponding to the nth data loader, destroying the data filter associated with the nth data loader.

[0148] In one embodiment, each time the data pulled by the nth data loader among the N data loaders needs to be stored in the first buffer space corresponding to the nth data loader; the processor 1201 is further used to:

[0149] Detecting the remaining storage capacity of the first buffer space corresponding to the nth data loader;

[0150] If it is detected that there is remaining storage capacity in the first buffer space, triggering the step of calling the nth data loader among the N data loaders to pull data generated by each object in the corresponding object group from the target space;

[0151] If it is detected that there is no remaining storage capacity in the first buffer space, wait until the storage capacity of the first buffer space is released, and then trigger the step of calling the nth data loader among the N data loaders to pull the data generated by each object in the corresponding object group from the target space.

[0152] In one embodiment, the dynamically created data filter obtains the data pulled by the nth data loader from the first buffer space corresponding to the nth data loader, and performs filtering processing;

[0153] After obtaining the data group of the object group corresponding to the nth data loader, the processor 1201 is further configured to:

[0154] storing the data group of the object group corresponding to the nth data loader into the second buffer space corresponding to the nth data loader;

[0155] deleting the data acquired from the first buffer space by the dynamically created data filter from the first buffer space to release storage capacity of the first buffer space;

[0156] After completing the buffer replay simulation of any data group in the second buffer space corresponding to the nth data loader, the corresponding data group is deleted from the second buffer space.

[0157] In one embodiment, the cache parameters include: a cache model and a cache replacement algorithm; the cache model and the cache replacement algorithm support orthogonal configuration;

[0158] The cache model refers to a cache structure used to organize cache, and the cache replacement algorithm refers to an algorithm used to determine data to be replaced from data stored in the cache space.

[0159] In one embodiment, the target data set includes M data subsets, different data subsets correspond to different partitioning methods, and M is an integer greater than 1; the data subset corresponding to any partitioning method includes at least one data group of an object group, and each object group involved in any data subset is obtained by partitioning the multiple objects according to the corresponding partitioning method; wherein the simulation system includes multiple cache modules, and the configuration information is used to indicate the cache parameters to be used by each cache module;

[0160] The processor 1201, in the simulation system, performs cache replay simulation on the target data set using the cache parameters indicated by the configuration information, and when obtaining a cache simulation result corresponding to the target data set, performs the following operations:

[0161] Traverse the M data subsets in the target data set to determine the mth data subset currently traversed, m∈[1,M];

[0162] Determine, among a plurality of cache modules in the simulation system, a cache module corresponding to each data group in the mth data subset, wherein one data group corresponds to at least two cache modules;

[0163] Respectively call each cache module corresponding to each data group in the m-th data subset, use the corresponding cache parameters indicated by the configuration information to perform cache replay simulation on the corresponding data group, and obtain the cache simulation result of the m-th data subset; the cache simulation result of the m-th data subset includes: the cache simulation result of each data group in the m-th data subset; the cache simulation result of any data group includes: the result obtained by performing cache replay simulation on each cache module corresponding to the corresponding data group;

[0164] Continue to traverse the target data set until all M data subsets in the target data set are traversed, and obtain the cache simulation result corresponding to the target data set; the cache simulation result corresponding to the target data set includes: the cache simulation result of each data subset in the target data set.

[0165] In one embodiment, the processor 1201 respectively calls each cache module corresponding to each data group in the mth data subset, and performs the following operations when performing cache replay simulation on the corresponding data group using the corresponding cache parameters indicated by the configuration information:

[0166] Performing data block processing on the nth data group in the mth data subset to obtain P data blocks; wherein n is a positive integer and is less than or equal to the number of data groups in the mth data subset, and P is a positive integer;

[0167] Each cache module corresponding to the nth data group is called respectively, and the corresponding cache parameters indicated by the configuration information are used to perform cache replay simulation on each data block in the P data blocks.

[0168] In one embodiment, the configuration information further includes a space allocation unit size of the cache module, where the space allocation unit size refers to the size of a single cache unit in the cache space of each cache module; when the processor 1201 performs data block processing on the nth data group in the mth data subset and obtains P data blocks, the processor 1201 performs the following operations:

[0169] Acquire the space allocation unit size from the configuration information, and perform data block processing on the nth data group in the mth data subset according to the space allocation unit size to obtain P data blocks;

[0170] Wherein, the size of each data block in the P data blocks is less than or equal to the size of the space allocation unit.

[0171] In one embodiment, the configuration information further includes a disk granularity, and the disk granularity is used to indicate the minimum size of a data block allowed to be cached; the processor 1201 performs data block processing on the nth data group in the mth data subset according to the space allocation unit size, and when P data blocks are obtained, performs the following operations:

[0172] Performing data block processing on the nth data group in the mth data subset according to the size of the space allocation unit to obtain a plurality of data blocks;

[0173] The disk-falling granularity is obtained from the configuration information, and P data blocks are filtered out from the multiple data blocks according to the disk-falling granularity and the size of each data block in the multiple data blocks; wherein the size of each data block in the P data blocks is greater than or equal to the disk-falling granularity.

[0174] In one embodiment, the configuration information further includes a cache allocation strategy, and the cache allocation strategy is used to indicate: from the cache space of the cache module, cache units with consecutive addresses are preferentially allocated to store data blocks;

[0175] The processor 1201 respectively calls each cache module corresponding to the nth data group, uses the corresponding cache parameters indicated by the configuration information, and performs the following operations when performing cache replay simulation on each data block in the Pth data block:

[0176] For any cache module corresponding to the nth data group, calling the any cache module to search for cache units with continuous addresses from the corresponding cache space;

[0177] If K cache units with consecutive addresses are found, and K≥P, then any cache module is called to use the corresponding cache parameters indicated by the configuration information to cache and replay the P data blocks to P cache units among the K cache units, with one cache unit caching one data block;

[0178] If K cache units with consecutive addresses are found and K<P, call any cache module to use the corresponding cache parameters indicated by the configuration information to replay the K data block caches among the P data blocks into the K cache units; and randomly select PK cache units from the cache units other than the cache units with consecutive addresses in the corresponding cache space, and replay the data block caches among the P data blocks other than the K data blocks into the randomly selected PK cache units.

[0179] The present application provides a computer program product, which includes a computer program stored in a computer storage medium; a processor of a system processing device reads the computer program from the computer storage medium, and the processor executes the computer program, so that the system processing device performs the above-mentioned Figure 4 The computer-readable storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0180] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A system processing method, characterized in that: include: In the process of developing a cache system in a content distribution network, determining a simulation system corresponding to the cache system and obtaining configuration information of the simulation system; The configuration information is used to indicate cache parameters to be used by the simulation system, the cache parameters are determined in the process of developing the cache system, and the cache system is shared by multiple objects to cache data generated by the multiple objects; Acquire a target data set through the simulation system, wherein the target data set includes data generated by the multiple objects; In the simulation system, the cache parameters indicated by the configuration information are used to perform cache replay simulation on the target data set to obtain a cache simulation result corresponding to the target data set; The cache simulation results corresponding to the target data set are output visually.

2. The method according to claim 1, characterized in that The configuration information further includes: M division methods for the multiple objects, where M is an integer greater than 1; and obtaining a target data set through the simulation system includes: Traverse the M partitioning methods and determine the mth partitioning method currently traversed, m∈[1,M]; Dividing the multiple objects according to the m-th division method to obtain N object groups under the m-th division method, where N is a positive integer; Acquire a data group of each of the N object groups through the simulation system to form a data subset corresponding to the m-th division method; wherein the data group of any object group includes data generated by each object in the corresponding object group; Continue to traverse the M partitioning methods, and after all the M partitioning methods are traversed, obtain a target data set, wherein the target data set includes a data subset corresponding to each of the M partitioning methods.

3. The method according to claim 2, characterized in that The simulation system includes a plurality of data loaders, and obtaining a data group of each of the N object groups through the simulation system includes: In the simulation system, N data loaders are allocated to the N object groups, and one data loader is allocated to one object group; Each of the N data loaders is called respectively to load the data group of the corresponding object group from the target space; wherein the target space includes at least one of the following: a local space and the cache system.

4. The method according to claim 3, characterized in that The calling of each of the N data loaders to load the data group of the corresponding object group from the target space comprises: Calling the nth data loader among the N data loaders to pull data generated by each object in the corresponding object group from the target space, n∈[1,N]; Dynamically create a data filter associated with the nth data loader, and call the dynamically created data filter to filter the data pulled by the nth data loader to obtain a data group of the object group corresponding to the nth data loader; After obtaining the data group of the object group corresponding to the nth data loader, destroying the data filter associated with the nth data loader.

5. The method according to claim 4, characterized in that Each time the data pulled by the nth data loader among the N data loaders needs to be stored in the first buffer space corresponding to the nth data loader; The method further comprises: Detecting the remaining storage capacity of the first buffer space corresponding to the nth data loader; If it is detected that there is remaining storage capacity in the first buffer space, triggering the step of calling the nth data loader among the N data loaders to pull data generated by each object in the corresponding object group from the target space; If it is detected that there is no remaining storage capacity in the first buffer space, wait until the storage capacity of the first buffer space is released, and then trigger the step of calling the nth data loader among the N data loaders to pull the data generated by each object in the corresponding object group from the target space.

6. The method according to claim 5, characterized in that The dynamically created data filter obtains the data pulled by the nth data loader from the first buffer space corresponding to the nth data loader, and performs filtering processing; After obtaining the data group of the object group corresponding to the nth data loader, the method further includes: storing the data group of the object group corresponding to the nth data loader into the second buffer space corresponding to the nth data loader; deleting the data acquired from the first buffer space by the dynamically created data filter from the first buffer space to release storage capacity of the first buffer space; After completing the buffer replay simulation of any data group in the second buffer space corresponding to the nth data loader, the corresponding data group is deleted from the second buffer space.

7. The method according to claim 1, characterized in that The cache parameters include: a cache model and a cache replacement algorithm; the cache model and the cache replacement algorithm support orthogonal configuration; The cache model refers to a cache structure used to organize cache, and the cache replacement algorithm refers to an algorithm used to determine data to be replaced from data stored in the cache space.

8. The method according to claim 1, characterized in that The target data set includes M data subsets, different data subsets correspond to different division methods, and M is an integer greater than 1; the data subset corresponding to any division method includes a data group of at least one object group, and each object group involved in any data subset is obtained by dividing the multiple objects according to the corresponding division method; wherein the simulation system includes multiple cache modules, and the configuration information is used to indicate the cache parameters to be used by each cache module; In the simulation system, using the cache parameters indicated by the configuration information to perform cache replay simulation on the target data set to obtain a cache simulation result corresponding to the target data set includes: Traverse the M data subsets in the target data set to determine the mth data subset currently traversed, m∈[1,M]; Determine, among a plurality of cache modules in the simulation system, a cache module corresponding to each data group in the mth data subset, wherein one data group corresponds to at least two cache modules; Respectively call each cache module corresponding to each data group in the m-th data subset, use the corresponding cache parameters indicated by the configuration information to perform cache replay simulation on the corresponding data group, and obtain the cache simulation result of the m-th data subset; the cache simulation result of the m-th data subset includes: the cache simulation result of each data group in the m-th data subset; the cache simulation result of any data group includes: the result obtained by performing cache replay simulation on each cache module corresponding to the corresponding data group; Continue to traverse the target data set until all M data subsets in the target data set are traversed, and obtain the cache simulation result corresponding to the target data set; the cache simulation result corresponding to the target data set includes: the cache simulation result of each data subset in the target data set.

9. The method according to claim 8, characterized in that The respectively calling each cache module corresponding to each data group in the mth data subset and performing cache replay simulation on the corresponding data group using the corresponding cache parameters indicated by the configuration information includes: Performing data block processing on the nth data group in the mth data subset to obtain P data blocks; wherein n is a positive integer and is less than or equal to the number of data groups in the mth data subset, and P is a positive integer; Each cache module corresponding to the nth data group is called respectively, and the corresponding cache parameters indicated by the configuration information are used to perform cache replay simulation on each data block in the P data blocks.

10. The method according to claim 9, characterized in that The configuration information also includes a space allocation unit size of the cache module, where the space allocation unit size refers to the size of a single cache unit in the cache space of each cache module; the data block processing of the nth data group in the mth data subset to obtain P data blocks includes: Acquire the space allocation unit size from the configuration information, and perform data block processing on the nth data group in the mth data subset according to the space allocation unit size to obtain P data blocks; Wherein, the size of each data block in the P data blocks is less than or equal to the size of the space allocation unit.

11. The method according to claim 10, characterized in that The configuration information also includes a disk placement granularity, where the disk placement granularity is used to indicate a minimum size of a data block allowed to be cached; the data block processing is performed on the nth data group in the mth data subset according to the space allocation unit size to obtain P data blocks, including: Performing data block processing on the nth data group in the mth data subset according to the size of the space allocation unit to obtain a plurality of data blocks; The disk-falling granularity is obtained from the configuration information, and P data blocks are filtered out from the multiple data blocks according to the disk-falling granularity and the size of each data block in the multiple data blocks; wherein the size of each data block in the P data blocks is greater than or equal to the disk-falling granularity.

12. The method according to claim 9, characterized in that The configuration information also includes a cache allocation strategy, wherein the cache allocation strategy is used to indicate: from the cache space of the cache module, cache units with consecutive addresses are preferentially allocated to store data blocks; The respectively calling each cache module corresponding to the nth data group and using the corresponding cache parameter indicated by the configuration information to perform cache replay simulation on each data block in the Pth data block includes: For any cache module corresponding to the nth data group, calling the any cache module to search for cache units with consecutive addresses from the corresponding cache space; If K cache units with consecutive addresses are found, and K≥P, then any cache module is called to use the corresponding cache parameters indicated by the configuration information to cache and replay the P data blocks to P cache units among the K cache units, with one cache unit caching one data block; If K cache units with consecutive addresses are found and K<P, call any cache module to use the corresponding cache parameters indicated by the configuration information to replay the K data block caches among the P data blocks into the K cache units; and randomly select PK cache units from the cache units other than the cache units with consecutive addresses in the corresponding cache space, and replay the data block caches among the P data blocks other than the K data blocks into the randomly selected PK cache units.

13. A system processing device, characterized in that: include: A processing unit, used to determine a simulation system corresponding to the cache system and obtain configuration information of the simulation system during the process of developing the cache system in the content distribution network; The configuration information is used to indicate cache parameters to be used by the simulation system, the cache parameters are determined in the process of developing the cache system, and the cache system is shared by multiple objects to cache data generated by the multiple objects; The processing unit is further used to obtain a target data set through the simulation system, wherein the target data set includes data generated by the multiple objects; The processing unit is further configured to perform cache replay simulation on the target data set in the simulation system using the cache parameters indicated by the configuration information to obtain a cache simulation result corresponding to the target data set; The output unit is used to visually output the cache simulation result corresponding to the target data set.

14. A system processing device, characterized in that: The system processing device includes an input interface and an output interface, and also includes: a processor adapted to implement one or more instructions; and, A computer storage medium storing one or more instructions, wherein the one or more instructions are suitable for being loaded by the processor and executing the system processing method according to any one of claims 1 to 12.

15. A computer storage medium, characterized in that: The computer storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, they are used to execute the system processing method according to any one of claims 1 to 12.

Images