Dictionary loading method, device, electronic equipment and storage medium

Abstract

The application discloses a dictionary loading method and device, electronic equipment and a storage medium, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: in response to a dictionary loading instruction, a shard set included in a data query cluster; for each shard in the shard set, obtaining corresponding shard running data, dictionary loading running data and request data to determine the running state of the shard; in response to an abnormal running state, updating the dictionary loading running data, updating the running state of the shard in combination with the shard running data and the request data, so that the updated running state is normal, and executing a dictionary loading program based on the updated dictionary loading running data; and in response to a normal running state, executing the dictionary loading program based on the dictionary loading running data. The embodiment can solve the problem that a full-amount dictionary loading process is implemented through concurrent threads, a large amount of processing resources and memory space need to be consumed to implement the loading of the dictionary, and the system performance is reduced.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and storage medium for loading a dictionary. Background Technology

[0002] In the e-commerce field, data analysis typically requires data querying to obtain data for analysis. ClickHouse (Click Stream Data Warehouse, a columnar database management system for online analytical processing) dictionaries are widely used for data querying. For example, e-commerce data analysis usually involves creating two dimension tables: one recording the relationship between SKUs (Item Units) and job departments, and the other recording the item information corresponding to each SKU. To improve system performance, the query processing of the relationship between SKUs and job departments is usually implemented using ClickHouse dictionaries. This involves loading the data table recording the relationship between SKUs and job departments into the memory of the data analysis cluster, generating a ClickHouse dictionary. During data querying, data on each shard in the data analysis cluster is correlated with the dictionary on that shard to retrieve the job department. Then, the query results from each shard are aggregated to obtain the final query result. Under this mechanism, each shard needs to store the full dictionary to ensure that the data on each shard can be queried based on the dictionary. Therefore, the full dictionary needs to be stored during dictionary loading to support data querying. Therefore, in existing technologies, when loading a dictionary, it is necessary to load the entire dictionary. Since the dictionary data is large, it requires a lot of processing resources and memory space to load the dictionary, which reduces system performance. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for loading a dictionary, which can solve the problem that loading a full dictionary through concurrent threads requires a large amount of processing resources and memory space, thus reducing system performance.

[0004] To achieve the above objectives, according to one aspect of the present invention, a dictionary loading method is provided.

[0005] A dictionary loading method according to an embodiment of the present invention includes:

[0006] In response to the dictionary loading instruction, obtain the set of shards included in the data query cluster;

[0007] For each shard in the shard set, obtain the corresponding shard running data, dictionary loading running data, and request data to determine the running status of the shard;

[0008] In response to the running status being normal, a dictionary loader is executed based on the dictionary loading running data; or, in response to the running status being abnormal, the dictionary loading running data is updated, and the running status of the shard is updated by combining the shard running data and the request data, so that the updated running status is normal, and the dictionary loader is executed based on the updated dictionary loading running data; in response to the running status being normal, a dictionary loader is executed based on the dictionary loading running data.

[0009] In one embodiment, the dictionary loading runtime data includes the number of concurrent threads for dictionary loading;

[0010] Executing the dictionary loader based on the dictionary loading runtime data includes:

[0011] The specified number of concurrent threads are used to read the corresponding dimension table in the shard;

[0012] Identify the first field set and the second field set in the dimension table, convert the data format of the data corresponding to the first field set to a preset format, and delete the data corresponding to the second field set to obtain an updated dimension table;

[0013] Generate a dictionary based on the updated dimension table.

[0014] In yet another embodiment, generating a dictionary based on the updated dimension table includes:

[0015] The number of parameter values ​​for the target field corresponding to the item identifier is determined based on the item identifier in the dimension table;

[0016] In response to the number of parameter values ​​exceeding a threshold, an array is generated based on the parameter values ​​of the target field corresponding to the item identifier, and a dictionary is generated in conjunction with the updated dimension table.

[0017] In yet another embodiment, the method further includes, prior to responding to the dictionary loading instruction:

[0018] In response to the dimension table import command, the number of shards in the data query cluster corresponding to the item identifier of the dimension table is obtained, and a preset calculation model is invoked to calculate the item identifier corresponding to each shard.

[0019] Data corresponding to each item identifier is obtained from an offline data source, based on a dimension table stored in the shard corresponding to each item identifier.

[0020] In yet another embodiment, before obtaining the corresponding slice runtime data, loading runtime data from the dictionary, and requesting data, the method further includes:

[0021] In response to the fact that a dictionary has been stored in the segment, the stored dictionary is deleted.

[0022] In yet another embodiment, the dictionary loading runtime data includes the number of concurrent threads for dictionary loading;

[0023] Update the dictionary loading and running data, including:

[0024] In response to the exception indicating a busy state, the number of concurrent threads is reduced to obtain an updated number of concurrent threads.

[0025] In yet another embodiment, after the dictionary loader is executed, the method further includes:

[0026] Calculate the first index value of the preset verification index based on the data in the dictionary, and calculate the second index value of the preset verification index based on the dimension table;

[0027] If the first indicator value and the second indicator value are the same, the dictionary verification is determined to be successful.

[0028] To achieve the above objectives, according to another aspect of the present invention, an apparatus for loading a dictionary is provided.

[0029] An embodiment of the present invention provides a dictionary loading apparatus comprising:

[0030] The retrieval unit is used to retrieve the set of shards included in the data query cluster in response to a dictionary loading instruction;

[0031] The processing unit is used to obtain the corresponding shard running data, dictionary loading running data, and request data for each shard in the shard set, so as to determine the running status of the shard;

[0032] The processing unit is further configured to, in response to the normal operating state, execute a dictionary loading program based on the dictionary loading operating data; or, in response to the abnormal operating state, update the dictionary loading operating data, update the operating state of the shard in combination with the shard operating data and the request data, so that the updated operating state is normal, and execute the dictionary loading program based on the updated dictionary loading operating data.

[0033] In one embodiment, the dictionary loading runtime data includes the number of concurrent threads for dictionary loading;

[0034] The processing unit is specifically used for:

[0035] The specified number of concurrent threads are used to read the corresponding dimension table in the shard;

[0036] Identify the first field set and the second field set in the dimension table, convert the data format of the data corresponding to the first field set to a preset format, and delete the data corresponding to the second field set to obtain an updated dimension table;

[0037] Generate a dictionary based on the updated dimension table.

[0038] In yet another embodiment, the processing unit is specifically used for:

[0039] The number of parameter values ​​for the target field corresponding to the item identifier is determined based on the item identifier in the dimension table;

[0040] In response to the number of parameter values ​​exceeding a threshold, an array is generated based on the parameter values ​​of the target field corresponding to the item identifier, and a dictionary is generated in conjunction with the updated dimension table.

[0041] In yet another embodiment, the apparatus further includes:

[0042] The calculation unit is used to respond to the dimension table import command, obtain the number of shards in the data query cluster corresponding to the item identifier of the dimension table, and call the preset calculation model to calculate the item identifier corresponding to each shard.

[0043] The storage unit is used to retrieve data corresponding to each item identifier from an offline data source, based on the dimension table stored in the shard corresponding to each item identifier.

[0044] In yet another embodiment, the processing unit is specifically used for:

[0045] In response to the fact that a dictionary has been stored in the segment, the stored dictionary is deleted.

[0046] In yet another embodiment, the dictionary loading runtime data includes the number of concurrent threads for dictionary loading;

[0047] The processing unit is specifically used for:

[0048] In response to the exception indicating a busy state, the number of concurrent threads is reduced to obtain an updated number of concurrent threads.

[0049] In another embodiment, the calculation unit is further configured to calculate a first index value of the preset verification index based on the data in the dictionary, and to calculate a second index value of the preset verification index based on the dimension table.

[0050] The device further includes:

[0051] A determining unit is configured to determine that the dictionary verification has passed in response to the first indicator value and the second indicator value being the same.

[0052] To achieve the above objectives, according to another aspect of the present invention, an electronic device is provided.

[0053] An electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the dictionary processing method provided in the embodiment of the present invention.

[0054] To achieve the above objectives, according to another aspect of the present invention, a computer-readable medium is provided.

[0055] An embodiment of the present invention provides a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the dictionary loading method provided in the embodiment of the present invention.

[0056] To achieve the above objectives, according to another aspect of the present invention, a computer program product is provided.

[0057] A computer program product according to an embodiment of the present invention includes a computer program that, when executed by a processor, implements the dictionary loading method provided in an embodiment of the present invention.

[0058] One embodiment of the above invention has the following advantages or beneficial effects: In this embodiment, after triggering the dictionary loading instruction, for each shard in the data query cluster, shard running data, dictionary loading running data, and request data can be obtained to determine the current running status of the shard. Based on the running status, it is then determined how to execute the dictionary loading program. That is, if the running status is normal, dictionary loading can be performed according to the dictionary loading running data; if the running status is abnormal, the dictionary loading running data can be adjusted and updated to obtain a normal running status, and dictionary loading can be performed according to the updated dictionary loading running parameters. Thus, in this embodiment, based on the shard's running status and request processing status, the dictionary loading running parameters can be adjusted in a timely manner so that the execution of the dictionary loading program does not excessively consume the shard's processing resources and memory, thereby ensuring the normal operation of the shard and avoiding degrading the shard's system performance.

[0059] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0060] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0061] Figure 1 This is a schematic diagram of a main process of a dictionary loading method according to an embodiment of the present invention;

[0062] Figure 2 This is a schematic diagram of another main process of the dictionary loading method according to an embodiment of the present invention;

[0063] Figure 3 This is a schematic diagram of another main process of the dictionary loading method according to an embodiment of the present invention;

[0064] Figure 4 This is a schematic diagram of the main units of a dictionary loading device according to an embodiment of the present invention;

[0065] Figure 5 This is yet another exemplary system architecture diagram to which embodiments of the present invention can be applied;

[0066] Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing embodiments of the present invention. Detailed Implementation

[0067] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0068] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with the relevant provisions of national laws and regulations.

[0069] This invention provides a dictionary loading system that can be used in scenarios where data queries are performed using the ClickHouse dictionary and the dictionary is loaded.

[0070] Specifically, this embodiment of the invention takes dictionary loading in a data analysis system in the e-commerce field as an example. Data analysis in e-commerce typically involves two data tables: one recording the relationship between SKUs and job departments, and the other recording the item information corresponding to each SKU. These two tables are linked by SKUs. Data queries are usually implemented through a cluster, which is equivalent to the dictionary loading system in this embodiment. The cluster includes one or more shards. To improve system performance, the query processing for the relationship between SKUs and job departments is usually implemented using a ClickHouse dictionary. This involves loading the dimension table recording the relationship between SKUs and job departments into memory to generate a ClickHouse dictionary. The disks in the shards can store detailed data tables that need to be retrieved through dictionary queries, such as the dimension table of item information corresponding to each SKU. The reason ClickHouse dictionaries can improve system performance is that they are stored in memory and computations are performed directly based on memory. Therefore, the size of the dictionary is strongly correlated with the memory size. Furthermore, dictionary queries operate in a local-to-local mode, meaning that the detailed data tables stored in each shard of the cluster need to be linked to the job information that can be retrieved from the dictionary for that shard. This ensures that data can be retrieved through dictionary lookups. To prevent data stored in shards from being unretrievalable through the dictionary, the full dictionary needs to be stored in each shard. However, since enterprises may have tens of thousands of purchasing and sales positions, organizational structures, brands, product categories, and other attributes, and these attributes change frequently, dimension tables need to record all job information over a long period and cover multiple business needs to adapt to various business requirements. Therefore, the data size in dimension tables becomes particularly large. In summary, it can be seen that when the dictionary size is large, the dictionary loading process consumes a significant amount of processing resources and memory space, placing a heavy burden on the shards in the cluster and reducing the system performance of each shard.

[0071] In this embodiment of the invention, the dictionary itself is optimized based on the characteristics of the dictionary, and the distribution method of the dictionary and the corresponding detailed table is improved, so that the dictionary can reduce its size while more completely covering various use cases.

[0072] Specifically, in this embodiment of the invention, the field format in the dictionary is optimized to reduce the required storage capacity while keeping the number of data entries in the dictionary unchanged. At the same time, considering that data queries for each shard are performed on the data stored in its shard, the data distribution rules of the detail table and the corresponding dimension table of the dictionary can be unified, so that the shard only stores the dictionary used to query the data stored in its detail table, thereby improving query efficiency and reducing the pressure on the cluster caused by dictionary loading.

[0073] It should be noted that the dictionary described in the embodiments of the present invention refers to the ClickHouse dictionary.

[0074] This invention provides a dictionary loading method, which can be executed by a data query cluster, such as... Figure 1 As shown, the method includes:

[0075] S101: In response to the dictionary load instruction, retrieve the set of shards included in the data query cluster.

[0076] In this process, the data query cluster can trigger a dictionary loading command during initialization or when a dictionary needs to be loaded. Since each shard in the cluster typically handles query operations independently, dictionary loading on each shard in a sharded cluster will not affect each other; that is, dictionary loading on each shard can be processed independently. In this step, after triggering the dictionary loading command, the shards included in the cluster can be obtained, allowing dictionary loading to be performed on each shard.

[0077] S102: For each shard in the sharded cluster, obtain the corresponding shard running data, dictionary loading running data, and request data to determine the running status of the shard.

[0078] S103: In response to a normal running status, execute the dictionary loader based on the dictionary loading running data; or, in response to an abnormal running status, update the dictionary loading running data, combine the shard running data and request data to update the shard running status, so that the updated running status is normal, and execute the dictionary loader based on the updated dictionary loading running data.

[0079] The steps involved in loading each shard dictionary are identical, so this step can be explained using one shard as an example.

[0080] Since the dictionary loading process is related to the memory and processing resources of the shards, this step can obtain shard runtime data, dictionary loading runtime data, and request data to determine whether performing dictionary loading in this way affects the operation of the shards. The shard runtime data can be data representing the current running status of the shards, such as shard CPU data, memory data, I / O data, TP99 (service monitoring, referring to the average response time of 100 requests), etc.; the dictionary loading runtime data can include the number of concurrent threads required for dictionary loading in the shard settings; and the request data can include the number of requests that the shard needs to process, such as the number of query requests.

[0081] It should be noted that the dictionary loading runtime data is pre-configured in the shards. In this step, the runtime status of the shards can be determined based on the pre-configured data. If the runtime status is abnormal, it can be updated. That is, in this step, the dictionary loading runtime data is updated in real time so that the shards maintain a normal runtime status when dictionary loading is performed based on the dictionary loading runtime data.

[0082] The sharding operation status can be normal or abnormal. In this step, if the sharding operation status is determined to be normal, it means that dictionary loading with the current dictionary loading operation data can guarantee the normal operation of the sharding, so the dictionary loading program can be executed based on the dictionary loading operation data. If the sharding operation status is determined to be abnormal, it means that dictionary loading with the current dictionary loading operation data cannot guarantee the normal operation of the sharding, so the dictionary loading operation data can be updated, and then the sharding operation status can be determined again by combining the sharding operation data and the request data, that is, the updated sharding operation status is obtained. This process is repeated until the updated operation status is normal, and the dictionary loading program is executed based on the updated dictionary loading operation data.

[0083] In this embodiment of the invention, a model for determining the running state can be preset and trained to derive the model used to determine the segment running state in this step. The model can be implemented using the SVM algorithm. Specifically, an initial model function is constructed as shown in Formula 1, where x i This represents the acquired data, y i x represents i The corresponding running status (usually -1 indicates normal running status, 1 indicates abnormal running status), N represents the number of data points acquired, and a represents the model parameters.

[0084]

[0085] In Formula 1, That is, with the penalty parameter set to 0.8, the training results of the initial model parameters can be obtained through model training.

[0086] in

[0087] ω * ·x+b * =0 (3)

[0088] f(x) = sign(ω) * ·x+b * (4)

[0089] After initializing the model parameters, we can further derive the function f(x) used to determine the segmentation running state in this step based on Equations 2, 3 (representing the separating hyperplane) and 4, i.e. Equation 4. In this step, the result of the segmentation running state can be obtained through the calculation of Equation 4.

[0090] In this embodiment of the invention, the dictionary loading runtime data includes the number of concurrent threads for dictionary loading. An abnormal shard running status typically indicates that the shard is in a busy state, requiring a reduction in the number of concurrent threads. Therefore, updating the dictionary loading runtime data in this step can be specifically executed as follows: in response to the abnormal busy state, reduce the number of concurrent threads to obtain the updated number of concurrent threads.

[0091] It should be noted that the number of concurrent threads for dictionary loading can be reduced based on preset rules, for example, by one less thread per update. When the number of concurrent threads for dictionary loading is reduced to one, if it is determined that the shard's running status is still busy, dictionary loading can be paused, such as by setting it to a waiting state, until the shard can maintain a normal running status for dictionary loading before resuming. Furthermore, during the execution of the dictionary loading program, the running status of the shard can be determined in real time. If the running status of a shard is determined to be abnormal, the dictionary loading running data can be updated again to ensure that the shard can run normally.

[0092] In this embodiment of the invention, shards are typically loaded with dictionaries based on stored dimension tables. Therefore, before executing the dictionary loading instruction, the corresponding dimension tables need to be stored in the shards. Specifically, before executing step S101, the following can also be performed: in response to the dimension table import instruction, obtain the number of shards in the cluster corresponding to the item identifier in the dimension table, and call the preset calculation model to calculate the item identifier corresponding to each shard; obtain the data corresponding to each item identifier from the offline data source, and use it based on the dimension table stored in the shard corresponding to each item identifier. The dimension table import instruction represents the import instruction for the dimension table corresponding to the dictionary. A dimension table can be pre-built in the shard, and then the response data can be imported into the dimension table to realize the dimension table import process. In this embodiment of the invention, the preset calculation model can be specifically: coalesce(cast(abs(hash(item identifier))as int)%number of shards in the cluster, 0), thus calculating the correspondence between each item identifier and the shard.

[0093] To ensure consistency in the distribution rules of the detailed tables and dictionaries stored in the shards, this embodiment of the invention treats the corresponding dimension tables of the detailed tables and dictionaries as a whole. Specifically, in this embodiment, the associated dimension of the detailed tables and the corresponding dimension tables of the dictionaries can be the item identifier. Therefore, the correspondence between the item identifier and each shard is calculated, meaning each shard stores the detailed table and dimension table corresponding to the item identifier. This allows loading the dictionary for that item identifier into the shard and retrieving all data corresponding to that item identifier from the stored detailed table. Thus, in this embodiment, each shard stores the detailed table and dictionary corresponding to the item identifier, without needing to store the entire dictionary, thereby reducing the size of the dictionary stored in the shard and consequently reducing the impact of dictionary loading on the shard's operational state.

[0094] It should be noted that, in this embodiment of the invention, to avoid abnormal import volume of dimension table data, anomaly detection can also be performed on the imported dimension table. Since the daily changes in the dimension table are relatively limited, the fluctuation of the dimension table data volume is relatively stable. To more quickly determine whether there are anomalies in the dimension table data volume, the k-sigma algorithm can be used. Specifically, since the date and data volume follow a normal distribution, outliers can be determined by whether the deviation from the average exceeds a threshold. Therefore, in this step, before importing data from the offline data source, the data volume of the corresponding data in the dimension table for each day within the past month can be obtained, and then the average data volume for each day can be calculated. Then, the data volume of the data in the offline data source is compared with the average data volume to determine whether the multiple of the data volume exceeding the average exceeds a threshold. If yes, it indicates that the data in the offline data source is abnormal, and the import can be paused; if no, it indicates that the data in the offline data source is normal. In this embodiment of the invention, the threshold can be set to 1.3.

[0095] In some embodiments, after executing the dictionary loading procedure, the obtained dictionary can also be validated. Specifically, this can be performed as follows: calculating a first value of a preset validation metric based on the data in the dictionary, and calculating a second value of the preset validation metric based on the dimension table; if the first and second metric values ​​are the same, the dictionary validation is determined to be successful. The preset metric can be set based on requirements, such as the amount of data.

[0096] It should be noted that, in this embodiment of the invention, the data in the dimension table can be checked for emptiness before loading, and if the shard includes multiple replicas, the amount of dictionary data in each replica can be compared to ensure that no dictionary is missing during the loading process.

[0097] It should be noted that before performing this step, you can also check whether the shard stores a dictionary. If it has been stored, it can be deleted directly, and then the latest dictionary can be loaded.

[0098] In this embodiment of the invention, the running parameters of dictionary loading can be adjusted in a timely manner based on the running status of the shards and the request processing status, so that the execution of the dictionary loading program does not consume too much of the shard processing resources and memory, thereby ensuring the normal operation of the shards and avoiding reducing the system performance of the shards.

[0099] This invention provides a dictionary loading method. Taking the number of concurrent threads for dictionary loading as an example, the steps of executing the dictionary loading program in step S102 are specifically explained, such as... Figure 2 As shown, the method includes:

[0100] S201: The number of concurrent threads is called to read the corresponding dimension table in the shard.

[0101] Once the number of concurrent threads for dictionary loading is determined, the specified number of threads can be used to execute the dictionary loading program. Since the dictionary is generated based on the corresponding dimension tables stored in the shards, the corresponding dimension tables in the shards can be read first.

[0102] S202: Identify the first and second field sets in the dimension table, convert the data format of the data corresponding to the first field set to a preset format, and delete the data corresponding to the second field set to obtain an updated dimension table.

[0103] In this embodiment of the invention, the field format in the dictionary is optimized to reduce the required storage capacity while maintaining the same number of data entries in the dictionary. Specifically, this involves converting the format of some dictionaries in the dimension table and deleting some useless fields (such as deleting either the department name or department ID fields). This embodiment of the invention can preset the set of fields to be converted and the set of fields to be deleted, thereby identifying the first and second field sets in the dimension table. The preset format can specifically be int format.

[0104] S203: Generate a dictionary based on the updated dimension table.

[0105] The above steps yield an updated dimension table, which can then be used to generate a dictionary.

[0106] In some embodiments, an item identifier may correspond to multiple different business scenarios. Therefore, a sub-table may be set in the dimension table to store the data of the business identifiers corresponding to the item identifier. In this embodiment of the invention, multiple business scenarios corresponding to the item identifier can be stored in an array. Therefore, in this embodiment of the invention, when there are multiple parameter values ​​in some dictionaries corresponding to the item identifier, a corresponding array can be generated. So this step can be specifically executed as follows: determine the number of parameter values ​​of the target field corresponding to the item identifier based on the item identifier in the dimension table; in response to the number of parameter values ​​being greater than a threshold, generate an array based on the parameter values ​​of the target field corresponding to the item identifier, and combine it with the updated dimension table to generate a dictionary. Here, the threshold can be set to 1, and the target field can be specifically a business scenario, sales model, etc. For example, the data of the sales model corresponding to the item identifier can be specifically: (GTMERP account, salesperson ERP account, procurement control officer ERP account, buyer ERP account, store position, category operation). When performing a query, after the shard uses the item identifier to match the corresponding array, it uses the has(arr, elem) method to determine whether the queried parameter value exists. This avoids the pressure of loading and storing multiple dictionaries and solves the problem of incomplete department information corresponding to the item identifier.

[0107] In this embodiment of the invention, the running parameters of dictionary loading can be adjusted in a timely manner based on the running status of the shards and the request processing status, so that the execution of the dictionary loading program does not consume too much of the shard processing resources and memory, thereby ensuring the normal operation of the shards and avoiding reducing the system performance of the shards.

[0108] This invention provides yet another dictionary loading method to specifically illustrate step S102, such as... Figure 3 As shown, the method includes:

[0109] S301: Obtain the corresponding shard running data, dictionary loading running data, and request data to determine the running status of the shard.

[0110] S302: In response to an abnormal running status, update the dictionary and load the running data.

[0111] In this step, after updating the dictionary loading and running data, it is also possible to determine whether the updated dictionary loading and running data is 0. If it is, the execution of subsequent steps can be paused; otherwise, the subsequent steps can be executed.

[0112] S303: Update the running status of the shards based on the updated dictionary, loaded running data, shard running data, and request data.

[0113] S304: In response to the updated running status being normal, execute the dictionary loader based on the updated dictionary loading running data.

[0114] S305: In response to the normal running status, execute the dictionary loader based on the dictionary loading running data.

[0115] It should be noted that the data processing principle in the embodiments of the present invention is the same as... Figure 1 The data processing principle is the same in the embodiments shown, and will not be repeated here.

[0116] To address the problems existing in the prior art, embodiments of the present invention provide a dictionary loading device 400, such as... Figure 4 As shown, the device 400 includes:

[0117] The acquisition unit 401 is used to acquire the set of shards included in the data query cluster in response to the dictionary loading instruction;

[0118] The processing unit 402 is used to obtain the corresponding shard running data, dictionary loading running data, and request data for each shard in the shard set, so as to determine the running status of the shard;

[0119] The processing unit 402 is further configured to, in response to the normal operating state, execute a dictionary loading program based on the dictionary loading operating data; or, in response to the abnormal operating state, update the dictionary loading operating data, and update the operating state of the shard by combining the shard operating data and the request data, so that the updated operating state is normal, and execute the dictionary loading program based on the updated dictionary loading operating data.

[0120] It should be understood that the manner in which embodiments of the present invention are implemented is different from the implementation method. Figure 1 The embodiments shown are the same and will not be described again here.

[0121] In one embodiment, the dictionary loading runtime data includes the number of concurrent threads for dictionary loading;

[0122] The processing unit 402 is specifically used for:

[0123] The specified number of concurrent threads are used to read the corresponding dimension table in the shard;

[0124] Identify the first field set and the second field set in the dimension table, convert the data format of the data corresponding to the first field set to a preset format, and delete the data corresponding to the second field set to obtain an updated dimension table;

[0125] Generate a dictionary based on the updated dimension table.

[0126] In yet another embodiment, the processing unit 402 is specifically used for:

[0127] The number of parameter values ​​for the target field corresponding to the item identifier is determined based on the item identifier in the dimension table;

[0128] In response to the number of parameter values ​​exceeding a threshold, an array is generated based on the parameter values ​​of the target field corresponding to the item identifier, and a dictionary is generated in conjunction with the updated dimension table.

[0129] In yet another embodiment, the device 400 further includes:

[0130] The calculation unit is used to respond to the dimension table import command, obtain the number of shards in the cluster corresponding to the item identifier of the dimension table, and call the preset calculation model to calculate the item identifier corresponding to each shard.

[0131] The storage unit is used to retrieve data corresponding to each item identifier from an offline data source, based on the dimension table stored in the shard corresponding to each item identifier.

[0132] In yet another embodiment, the processing unit 402 is specifically used for:

[0133] In response to the fact that a dictionary has been stored in the segment, the stored dictionary is deleted.

[0134] In yet another embodiment, the dictionary loading runtime data includes the number of concurrent threads for dictionary loading;

[0135] The processing unit 402 is specifically used for:

[0136] In response to the exception indicating a busy state, the number of concurrent threads is reduced to obtain an updated number of concurrent threads.

[0137] In another embodiment, the calculation unit is further configured to calculate a first index value of the preset verification index based on the data in the dictionary, and to calculate a second index value of the preset verification index based on the dimension table.

[0138] The device 400 further includes:

[0139] A determining unit is configured to determine that the dictionary verification has passed in response to the first indicator value and the second indicator value being the same.

[0140] It should be understood that the manner in which embodiments of the present invention are implemented is different from the implementation method. Figure 1 , 2 The same method is used in the embodiments shown in 1 and 3, and will not be described again here.

[0141] In this embodiment of the invention, the running parameters of dictionary loading can be adjusted in a timely manner based on the running status of the shards and the request processing status, so that the execution of the dictionary loading program does not consume too much of the shard processing resources and memory, thereby ensuring the normal operation of the shards and avoiding reducing the system performance of the shards.

[0142] According to embodiments of the present invention, an electronic device and a readable storage medium are also provided.

[0143] An electronic device according to an embodiment of the present invention includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the dictionary loading method provided in the embodiment of the present invention.

[0144] Figure 5 An exemplary system architecture 500 is shown where the dictionary loading method or apparatus of embodiments of the present invention can be applied.

[0145] like Figure 5As shown, system architecture 500 may include terminal devices 501, 502, and 503, a network 504, and a server 505. Network 504 serves as the medium for providing communication links between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0146] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Various client applications can be installed on terminal devices 501, 502, and 503.

[0147] Terminal devices 501, 502, and 503 can be, but are not limited to, smartphones, tablets, laptops, and desktop computers, etc.

[0148] Server 505 can be a server that provides various services. The server can analyze and process data such as received product information query requests, and feed back the processing results (such as product information - just an example) to the terminal device.

[0149] It should be noted that the dictionary loading method provided in this embodiment of the invention is generally executed by server 505, and correspondingly, the dictionary loading device is generally located in server 505.

[0150] It should be understood that Figure 5 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0151] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing embodiments of the present invention. Figure 6 The computer system shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0152] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0153] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0154] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this invention.

[0155] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a unit, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] The units described in the embodiments of the present invention can be implemented in software or hardware. The described units can also be located in a processor; for example, a processor can be described as including an acquisition unit and a processing unit. The names of these units do not necessarily limit the specific unit; for example, the acquisition unit can also be described as a "unit with slice acquisition function".

[0158] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to perform the dictionary loading method provided by the present invention.

[0159] In another aspect, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the dictionary loading method provided in the embodiments of the present invention.

[0160] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for loading a dictionary, characterized in that, include: In response to a dictionary load command, retrieve the set of shards included in the data query cluster; For each shard in the shard set, obtain the corresponding shard running data, dictionary loading running data, and request data to determine the running status of the shard; In response to the normal operating status, the dictionary loading program is executed based on the dictionary loading running data; Alternatively, in response to the abnormal running status, update the dictionary loading running data, combine the shard running data and the request data to update the running status of the shard, so that the updated running status is normal, and execute the dictionary loading program based on the updated dictionary loading running data; The dictionary loading runtime data includes the number of concurrent threads for dictionary loading; Executing the dictionary loader based on the dictionary loading runtime data includes: The specified number of concurrent threads are used to read the corresponding dimension table in the shard; Identify the first field set and the second field set in the dimension table, convert the data format of the data corresponding to the first field set to a preset format, and delete the data corresponding to the second field set to obtain an updated dimension table; Generate a dictionary based on the updated dimension table.

2. The method according to claim 1, characterized in that, Generate a dictionary based on the updated dimension table, including: The number of parameter values ​​for the target field corresponding to the item identifier is determined based on the item identifier in the dimension table; In response to the number of parameter values ​​exceeding a threshold, an array is generated based on the parameter values ​​of the target field corresponding to the item identifier, and a dictionary is generated in conjunction with the updated dimension table.

3. The method according to claim 1, characterized in that, Prior to responding to the dictionary loading instruction, the method further includes: In response to the dimension table import command, the number of shards in the data query cluster corresponding to the item identifier of the dimension table is obtained, and a preset calculation model is invoked to calculate the item identifier corresponding to each shard. Data corresponding to each item identifier is obtained from an offline data source, based on a dimension table stored in the shard corresponding to each item identifier.

4. The method according to claim 1, characterized in that, Before obtaining the corresponding slice runtime data, loading runtime data from the dictionary, and requesting data, the process also includes: In response to the fact that a dictionary has been stored in the segment, the stored dictionary is deleted.

5. The method according to claim 1, characterized in that, The dictionary loading runtime data includes the number of concurrent threads for dictionary loading; Update the dictionary loading and running data, including: In response to the exception indicating a busy state, the number of concurrent threads is reduced to obtain an updated number of concurrent threads.

6. The method according to claim 1, characterized in that, Following the execution of the dictionary loader, the following is also included: Calculate the first index value of the preset verification index based on the data in the dictionary, and calculate the second index value of the preset verification index based on the dimension table; If the first indicator value and the second indicator value are the same, the dictionary verification is determined to be successful.

7. A dictionary loading device, characterized in that, include: The retrieval unit is used to retrieve the set of shards included in the data query cluster in response to a dictionary loading instruction; The processing unit is used to obtain the corresponding shard running data, dictionary loading running data, and request data for each shard in the shard set, so as to determine the running status of the shard; The processing unit is further configured to execute a dictionary loading program based on the dictionary loading running data in response to the normal operating state. Alternatively, in response to the abnormal running status, update the dictionary loading running data, combine the shard running data and the request data to update the running status of the shard, so that the updated running status is normal, and execute the dictionary loading program based on the updated dictionary loading running data; The dictionary loading runtime data includes the number of concurrent threads for dictionary loading; The processing unit is specifically used for: The specified number of concurrent threads are used to read the corresponding dimension table in the shard; Identify the first field set and the second field set in the dimension table, convert the data format of the data corresponding to the first field set to a preset format, and delete the data corresponding to the second field set to obtain an updated dimension table; Generate a dictionary based on the updated dimension table.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images