Data processing method and device based on distributed architecture

Abstract

The invention provides a data processing method and device based on a distributed architecture, and the method comprises the steps: carrying out the text conversion of all to-be-processed data in a to-be-processed data set according to a preset conversion mode, and arranging the data texts according to a preset type arrangement order, dividing the text into n sub-to-be-processed data texts, and distributing the n sub-to-be-processed data texts to n server nodes in a one-to-one correspondence manner for data processing. The method has the beneficial effects that the data texts in the segmented sub-to-be-processed data texts are similar data texts, so that the corresponding server nodes only need to process similar data, and the data can be processed in batches without a conversion tool in the processing process, so that the processing efficiency of each server node can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a data processing method and device based on a distributed architecture. Background Art

[0002] Distributed architecture is a software architecture model that breaks down a large system into multiple independent subsystems or modules, distributes these subsystems on different server nodes, and communicates with each other through network protocols to form an overall system. At present, e-commerce platforms can significantly improve the overall performance of the system by breaking down and distributing tasks to different nodes for parallel processing according to the needs of business growth. In particular, when dealing with large-scale data and complex computing tasks, distributed architecture can be competent for data processing tasks.

[0003] In the prior art, the data sets to be processed on the e-commerce platform are often evenly distributed or randomly distributed, which will cause each server node to be unable to efficiently process the data sets to be processed, resulting in low processing efficiency. Summary of the invention

[0004] The main purpose of the present invention is to provide a data processing method and device based on a distributed architecture, aiming to solve the problem of low efficiency in data processing in the existing distributed architecture.

[0005] The present invention provides a data processing method based on a distributed architecture, comprising:

[0006] Get the data set to be processed;

[0007] Performing text conversion on each of the to-be-processed data in the to-be-processed data set according to a preset conversion method to obtain a data text of each of the to-be-processed data;

[0008] Obtaining the type of each of the data texts, arranging the data texts according to a preset type arrangement order, separating two adjacent data texts with a preset separator, and obtaining data texts to be processed;

[0009] Determining n server nodes in the distributed architecture for processing the data set to be processed;

[0010] Selecting n-1 of the spacers as target spacers according to the n server nodes;

[0011] Dividing the data text to be processed into n sub-data texts to be processed according to n-1 target separators;

[0012] The n sub-data texts to be processed are distributed one by one to n server nodes for data processing.

[0013] Furthermore, the step of selecting n-1 of the spacers as target spacers according to the n server nodes includes:

[0014] Randomly select n-1 of the spacers as initial spacers;

[0015] Dividing the data text to be processed into n initial sub-data texts to be processed according to the initial separator;

[0016] Marking n initial sub-data texts to be processed as the first text, the second text, ..., the nth text in sequence;

[0017] Vectorize the first text, the second text, ..., and the nth text respectively to obtain a first vector, a second vector, ..., and an nth vector;

[0018] Calculate the similarity between two adjacent vectors, map the obtained similarity to the interval (0,1), and find the linear distance from 1;

[0019] When all calculated linear distances are greater than a set threshold, the initial separator is recorded as a target separator.

[0020] Furthermore, after the step of calculating the similarity value between two adjacent vectors and mapping the obtained similarity value to the (0, 1) interval and finding the linear distance from 1, the method further includes:

[0021] If there is a calculated linear distance that is less than or equal to the set threshold, the initial separator is re-determined until the calculated linear distance is greater than the set threshold;

[0022] The initial separators corresponding to the calculated linear distances that are all greater than the set threshold are recorded as target separators.

[0023] Furthermore, the step of distributing the n sub-data texts to be processed to n server nodes one by one for data processing includes:

[0024] Get the processed historical data texts from n server nodes;

[0025] Calculate the similarity between each sub-data text to be processed and each historical data text;

[0026] According to the similarity, each server node is assigned a corresponding sub-data text to be processed for processing.

[0027] Furthermore, after the step of distributing the n sub-data texts to be processed to n server nodes one by one for data processing, the method further includes:

[0028] The server node obtains a corresponding subset of data to be processed according to the obtained sub-data to be processed text;

[0029] The data to be processed in the subset of data to be processed is processed.

[0030] Furthermore, the step of performing text conversion on each of the to-be-processed data in the to-be-processed data set according to a preset conversion method to obtain the data text of each to-be-processed data includes:

[0031] Identify the type of data structure to be processed;

[0032] Determine a traversal method according to the type of the data structure to be processed;

[0033] The data structure to be processed is traversed according to the traversal method, and the corresponding data text is generated according to the traversed data type and value.

[0034] Furthermore, the step of selecting n-1 of the spacers as target spacers according to the n server nodes includes:

[0035] Randomly select n-1 of the spacers as initial spacers;

[0036] Dividing the data text to be processed into n initial sub-data texts to be processed according to the initial separator;

[0037] Marking n initial sub-data texts to be processed as the first text, the second text, ..., the nth text in sequence;

[0038] Calculating the text lengths of the first text, the second text, ..., the nth text;

[0039] Subtract the minimum text length from the maximum text length to get the text length range;

[0040] Determine whether the text length range is less than a preset range;

[0041] If it is smaller than the preset range, the initial separator is used as the target separator.

[0042] The present invention also provides a data processing device with a distributed architecture, comprising:

[0043] An acquisition module is used to obtain the data set to be processed;

[0044] A conversion module, used to perform text conversion on each of the to-be-processed data in the to-be-processed data set according to a preset conversion method to obtain a data text of each to-be-processed data;

[0045] An arrangement module, used for obtaining the type of each of the data texts, arranging the data texts according to a preset type arrangement order, separating two adjacent data texts with a preset separator, and obtaining data texts to be processed;

[0046] A determination module, used to determine n server nodes in the distributed architecture for processing the data set to be processed;

[0047] A selection module, used for selecting n-1 of the spacers as target spacers according to the n server nodes;

[0048] A division module, used for dividing the data text to be processed into n sub-data texts to be processed according to n-1 target separators;

[0049] The distribution module is used to distribute the n sub-data texts to be processed one by one to n server nodes for data processing.

[0050] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0051] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0052] The beneficial effects of the present invention are as follows: by performing text conversion on each data to be processed in the data set to be processed according to a preset conversion method, arranging the data text according to a preset type arrangement order, and dividing it into n sub-data texts to be processed, the data texts in the divided sub-data texts to be processed can be made to be similar data texts, so that the corresponding server nodes only need to process similar data. During the processing, no conversion tools are required, and the data can be processed in batches, thereby significantly improving the processing efficiency of each server node. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flowchart of a data processing method based on a distributed architecture according to an embodiment of the present invention;

[0054] Figure 2 is a schematic block diagram of the structure of a data processing device based on a distributed architecture according to an embodiment of the present invention;

[0055] Figure 3 A schematic block diagram of the structure of a computer device according to an embodiment of the present application.

[0056] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly. The connection can be a direct connection or an indirect connection.

[0059] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone.

[0060] In addition, in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0061] Reference Figure 1 The present invention proposes a data processing method based on a distributed architecture, comprising:

[0062] S1: Get the data set to be processed;

[0063] S2: performing text conversion on each data to be processed in the data set to be processed according to a preset conversion method to obtain a data text of each data to be processed;

[0064] S3: Obtain the type of each of the data texts, arrange the data texts according to a preset type arrangement order, and separate two adjacent data texts with a preset separator to obtain data texts to be processed;

[0065] S4: Determine n server nodes in the distributed architecture for processing the data set to be processed;

[0066] S5: selecting n-1 of the spacers as target spacers according to the n server nodes;

[0067] S6: dividing the data text to be processed into n sub-data texts to be processed according to n-1 target separators;

[0068] S7: Distribute the n sub-data texts to be processed one by one to n server nodes for data processing.

[0069] As described in the above step S1, a data set to be processed is obtained, wherein the data set to be processed includes a plurality of data to be processed, and the data set to be processed may be obtained from an e-commerce platform via the Internet or may be obtained by manual uploading.

[0070] As described in step S2 above, text conversion is performed on each of the to-be-processed data in the to-be-processed data set according to a preset conversion method to obtain data text of each to-be-processed data. The conversion method is not limited and needs to be determined according to the data structure type of the to-be-processed data. In some embodiments, different conversion rules can be set for data of different structural types, such as data conversion of date and time, data conversion of tables, etc., and the data is subjected to encoding conversion, field splicing and other operations to form a data text that meets the expected format and requirements.

[0071] As described in the above step S3, the type of each data text is obtained, and the data text is arranged in a preset type arrangement order, and two adjacent data texts are separated by a preset separator to obtain the data text to be processed. That is, the type of each data text is obtained, and the data text is arranged in a preset type arrangement order. The type of the data text here is the type of data, not the type of data structure. It is classified according to the specific attributes or rules of the data, such as vertical splitting according to the business, such as classification by commodity application, member center, etc., and can also be classified according to categories such as commodity information, sales records, user behavior, etc. The specific classification rules can be pre-set, and the arrangement order can also be pre-set, so that after the type of the data text is obtained, it is arranged to obtain the data text to be processed. The data text is classified and arranged, so that in the subsequent process, the data allocated to each server node is similar, which can improve the processing efficiency of each server.

[0072] As described in the above step S4, n server nodes for processing the data set to be processed in the distributed architecture are determined. The determination method can be to calculate the idle rate of each server node, and use the n server nodes with an idle rate greater than a preset value as the n server nodes for processing the data set to be processed. Of course, if the amount of data to be processed in the data set to be processed is not large, n server nodes can also be selected from the server nodes with an idle rate greater than the preset value for processing. The specific selected number can be set in a correspondence table between the number of data to be processed and the number of servers, and the number of servers can be determined according to the amount of data to be processed.

[0073] As described in the above steps S5-S6, n-1 of the separators are selected as target separators according to the n server nodes, and the data text to be processed is divided into n sub-data texts to be processed according to the n-1 target separators. That is, n server nodes need to select n-1 separators as target separators to correspond to n sub-data texts to be processed.

[0074] As described in step S7, the n sub-data texts to be processed are distributed one by one to n server nodes for data processing. Since the data texts are arranged according to the preset type arrangement order and divided into n sub-data texts to be processed, the data texts in the divided sub-data texts to be processed are all similar data texts, so that the corresponding server nodes only need to process similar data. During the processing, no conversion tools are required, and the data can be processed in batches, thereby significantly improving the processing efficiency of each server node.

[0075] In one embodiment, the step S5 of selecting n-1 of the spacers as target spacers according to the n server nodes includes:

[0076] S501: Randomly select n-1 separators as initial separators;

[0077] S502: dividing the data text to be processed into n initial sub-data texts to be processed according to the initial separator;

[0078] S503: Mark the n initial sub-data texts to be processed as the first text, the second text, ..., the nth text in sequence;

[0079] S504: vectorizing the first text, the second text, ..., and the nth text respectively to obtain a first vector, a second vector, ..., and an nth vector;

[0080] S505: Calculate the similarity value between two adjacent vectors, and map the obtained similarity value to the (0,1) interval to find the linear distance from 1;

[0081] S506: When all calculated linear distances are greater than a set threshold, the initial separator is recorded as a target separator.

[0082] As described in the above steps S501-S506, the determination of the target separator is achieved, that is, n-1 separators are randomly selected in advance as initial separators, and the data text to be processed is divided into n initial sub-data texts to be processed according to the initial separators; the n initial sub-data texts to be processed are marked as the first text, the second text, ..., the nth text in sequence; the first text, the second text, ..., the nth text are vectorized respectively to obtain the first vector, the second vector, ..., the nth vector; the similarity value between two adjacent vectors is calculated, and the obtained similarity value is mapped to the (0,1) interval to find the linear distance from 1. The similarity value can be calculated by any similarity calculation method, such as Euclidean distance, Manhattan distance, cosine similarity, etc. Specifically, it can be calculated by the formula The similarity value is calculated, wherein, is the similarity value, represents the first vector, represents the second vector, represents the i-th dimension of the first vector, represents the i-th dimension of the second vector; through the formula Calculate the mapping value of the nonlinear mapping to the (0,1) interval; where, Represents a mapping value; the first vector and the second vector are two adjacent vectors in the first vector, the second vector, ..., the nth vector, respectively. When all calculated linear distances are greater than a set threshold, the initial separator is recorded as a target separator, otherwise, an initial separator needs to be set until all calculated linear distances are greater than a set threshold, and then the initial separator can be set as a target separator. It should be noted that only when all linear distances are greater than a set threshold can it be said that there is no correlation between two adjacent vectors. At this time, it can be regarded as further classification of the data, so that the corresponding server nodes can process data of the same category, further improving processing efficiency.

[0083] In one embodiment, after the step S505 of calculating the similarity value between two adjacent vectors and mapping the obtained similarity value to the interval (0, 1) to obtain the linear distance from 1, the method further includes:

[0084] S5061: If there is a calculated linear distance that is less than or equal to the set threshold, re-determine the initial separator until the calculated linear distances are all greater than the set threshold;

[0085] S5062: Record the initial separator corresponding to the calculated linear distance being greater than the set threshold as the target separator.

[0086] As described in the above steps S5061-S5062, if there is a calculated linear distance that is less than or equal to the set threshold, the initial separator is re-determined until the calculated linear distances are all greater than the set threshold, and the initial separator corresponding to the calculated linear distances that are all greater than the set threshold is recorded as the target separator. It should be noted that the method of re-determining the initial separator is not limited, and the initial separator between two adjacent vectors whose linear distance is less than or equal to the set threshold can be re-determined without re-determining other initial separators, so as to save the time of re-determination and improve efficiency.

[0087] In one embodiment, the step S7 of distributing the n sub-data texts to be processed to n server nodes for data processing one by one includes:

[0088] S701: Obtain the processed historical data texts in n server nodes;

[0089] S702: Calculate the similarity between each sub-data text to be processed and each historical data text;

[0090] S703: Allocate corresponding sub-data text to be processed to each server node according to the similarity for processing.

[0091] As described in the above steps S701-S703, the processed historical data texts in n server nodes are obtained; the similarity between each sub-data text to be processed and each historical data text is calculated; and the corresponding sub-data text to be processed is allocated to each server node for processing according to the similarity. In order to make the sub-data text to be processed more matched with the server node, the historical data text processed in the server node can be obtained, and the similarity between each sub-data text to be processed and each historical data text can be calculated. The similarity can be calculated by converting it into a vector, and then calculating its Euclidean distance or cosine similarity to obtain the similarity value. According to the size of the similarity, each server node allocates the corresponding sub-data text to be processed for processing. The specific allocation method is to first allocate the one with the greatest similarity, and then select the one with the greatest similarity from the remaining data for allocation until all the sub-data texts to be processed are allocated. In some embodiments, assuming that the first server node has the greatest similarity with the first sub-data text to be processed, the first sub-data text to be processed is preferentially allocated to the first server node at this time. If the second server node has the second greatest similarity with the first sub-data text to be processed, but since the first sub-data text to be processed has been allocated, this data should also be deleted and not considered. At this time, all data corresponding to all first server nodes and the first sub-data text to be processed should be eliminated, and then the one with the greatest similarity is selected from the remaining data for matching until all the sub-data texts to be processed are allocated.

[0092] In one embodiment, after the step S7 of distributing the n sub-data texts to be processed one by one to the n server nodes for data processing, the method further includes:

[0093] S801: The server node obtains a corresponding subset of data to be processed according to the obtained sub-data text to be processed;

[0094] S802: Processing the data to be processed in the subset of data to be processed.

[0095] As described in the above steps S801-S802, the server node obtains the corresponding subset of data to be processed based on the obtained sub-data to be processed text, and processes the data to be processed in the sub-data to be processed subset. After each server node obtains the sub-data to be processed text, the data contained in the sub-data to be processed text may not be comprehensive. Therefore, the corresponding subset of data to be processed can be obtained for processing. In some embodiments, if the data contained in the sub-data to be processed text can meet the processing requirements, it is not necessary to obtain the original data to be processed.

[0096] In one embodiment, the step S2 of performing text conversion on each of the to-be-processed data in the to-be-processed data set according to a preset conversion method to obtain the data text of each to-be-processed data includes:

[0097] S201: Identify the type of data structure to be processed;

[0098] S202: Determine a traversal method according to the type of the data structure to be processed;

[0099] S203: Traverse the data structure to be processed according to the traversal method, and generate corresponding data text according to the traversed data type and value.

[0100] As described in the above steps S201-S203, the generation of data text is realized. Identify the type of data structure to be processed; determine the traversal method according to the type of data structure to be processed; traverse the data structure to be processed according to the traversal method, and generate the corresponding data text according to the traversed data type and value. It should be noted that different types of data structures to be processed have different processing methods. Specifically, the processing methods corresponding to different data structures to be processed can be pre-set, and then traversal can be performed to generate the corresponding data text.

[0101] In one embodiment, the step S5 of selecting n-1 of the spacers as target spacers according to the n server nodes includes:

[0102] S511: Randomly select n-1 of the separators as initial separators;

[0103] S512: dividing the data text to be processed into n initial sub-data texts to be processed according to the initial separator;

[0104] S513: marking the n initial sub-data texts to be processed as the first text, the second text, ..., the nth text in sequence;

[0105] S514: Calculate the text lengths of the first text, the second text, ..., and the nth text;

[0106] S515: Subtract the minimum value of the text length from the maximum value of the text length to obtain the text length range;

[0107] S516: Determine whether the text length range is less than a preset range;

[0108] S517: If it is less than the preset range, the initial separator is used as the target separator.

[0109] In one embodiment, in order to make the amount of tasks processed by each server node equal, the length of the initial sub-data text to be processed can be monitored, that is, the text length of the first text, the second text, ..., and the nth text is calculated; the text length range is obtained by subtracting the minimum text length from the maximum text length. If the text length range is less than the preset range, the initial separator is used as the target separator. If the text length range is not less than the preset range, the initial separator is re-determined until the text length range is less than the preset range.

[0110] Reference Figure 2 The present invention also provides a data processing device with a distributed architecture, comprising:

[0111] An acquisition module 10 is used to acquire a data set to be processed;

[0112] The conversion module 20 is used to perform text conversion on each of the to-be-processed data in the to-be-processed data set according to a preset conversion method to obtain a data text of each to-be-processed data;

[0113] The arrangement module 30 is used to obtain the type of each of the data texts, arrange the data texts according to a preset type arrangement order, and separate two adjacent data texts with a preset separator to obtain the data text to be processed;

[0114] A determination module 40, configured to determine n server nodes in the distributed architecture for processing the data set to be processed;

[0115] A selection module 50, configured to select n-1 of the spacers as target spacers according to the n server nodes;

[0116] A division module 60, configured to divide the data text to be processed into n sub-data texts to be processed according to n-1 target separators;

[0117] The distribution module 70 is used to distribute the n sub-data texts to be processed to the n server nodes one by one for data processing.

[0118] In one embodiment, the selection module 50 includes:

[0119] A selection submodule, used for randomly selecting n-1 of the spacers as initial spacers;

[0120] A division submodule, used for dividing the data text to be processed into n initial sub-data texts to be processed according to the initial separator;

[0121] A first marking submodule, used for marking n initial sub-data texts to be processed as a first text, a second text, ..., an nth text in sequence;

[0122] A vectorization submodule, used for vectorizing the first text, the second text, ..., and the nth text respectively to obtain a first vector, a second vector, ..., and an nth vector;

[0123] The calculation submodule is used to calculate the similarity value between two adjacent vectors, and map the obtained similarity value to the (0,1) interval to find the linear distance from 1;

[0124] The second marking submodule is used to mark the initial separator as a target separator when all calculated linear distances are greater than a set threshold.

[0125] In one embodiment, the meter selection module 50 further includes:

[0126] A redetermining submodule, for redetermining the initial separator if the calculated linear distance is less than or equal to the set threshold, until the calculated linear distances are all greater than the set threshold;

[0127] The third marking submodule is used to mark the initial separators corresponding to the calculated linear distances being greater than the set threshold as target separators.

[0128] In one embodiment, the distribution module 70 includes:

[0129] The historical data text acquisition submodule is used to obtain the processed historical data texts in n server nodes;

[0130] A similarity calculation submodule is used to calculate the similarity between each sub-data text to be processed and each historical data text;

[0131] The processing submodule is used to allocate corresponding sub-data text to be processed to each server node for processing according to the similarity.

[0132] In one embodiment, the data processing device of the distributed architecture further includes:

[0133] A module for acquiring a subset of data to be processed, used for the server node to acquire a corresponding subset of data to be processed according to the acquired sub-data text to be processed;

[0134] The to-be-processed data subset processing module is used to process the to-be-processed data in the to-be-processed data subset.

[0135] In one embodiment, the conversion module 20 includes:

[0136] An identification submodule, used to identify the type of data structure to be processed;

[0137] A traversal mode determination submodule, used to determine a traversal mode according to the type of the data structure to be processed;

[0138] The traversal submodule is used to traverse the data structure to be processed according to the traversal method, and generate corresponding data text according to the traversed data type and value.

[0139] In one embodiment, the selection module 50 includes:

[0140] An initial separator selection submodule is used to randomly select n-1 separators as initial separators;

[0141] A to-be-processed data text division submodule, used for dividing the to-be-processed data text into n initial sub-to-be-processed data texts according to the initial separator;

[0142] An initial sub-data to be processed text marking submodule, used for marking n initial sub-data to be processed texts as a first text, a second text, ..., an nth text in sequence;

[0143] A text length calculation submodule, used to calculate the text lengths of the first text, the second text, ..., the nth text;

[0144] The text length range calculation submodule is used to obtain the text length range by subtracting the minimum text length from the maximum text length;

[0145] A text length range judgment submodule is used to judge whether the text length range is less than a preset range;

[0146] As a submodule, it is used to use the initial separator as the target separator if it is smaller than the preset range.

[0147] The beneficial effects of the present invention are as follows: by performing text conversion on each data to be processed in the data set to be processed according to a preset conversion method, arranging the data texts according to a preset type arrangement order, and dividing them into n sub-data texts to be processed, the data texts in the divided sub-data texts to be processed can be made to be similar data texts, so that the corresponding server nodes only need to process similar data. During the processing, no conversion tools are required, and the data can be processed in batches, thereby significantly improving the processing efficiency of each server node.

[0148] Reference Figure 3 In an embodiment of the present application, a computer device is also provided. The computer device may be a server node, and its internal structure may be as follows: Figure 3As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store various data to be processed, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the data processing method based on the distributed architecture described in any of the above embodiments can be implemented.

[0149] Those skilled in the art will understand that Figure 3 The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0150] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the data processing method based on a distributed architecture described in any of the above embodiments can be implemented.

[0151] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0152] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, device, article or method including the element.

[0153] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0154] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0155] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the claims of the present invention.

Claims

1. A data processing method based on a distributed architecture, characterized in that: include: Get the data set to be processed; Performing text conversion on each of the to-be-processed data in the to-be-processed data set according to a preset conversion method to obtain a data text of each of the to-be-processed data; Obtaining the type of each of the data texts, arranging the data texts according to a preset type arrangement order, separating two adjacent data texts with a preset separator, and obtaining data texts to be processed; Determining n server nodes in the distributed architecture for processing the data set to be processed; Selecting n-1 of the spacers as target spacers according to the n server nodes; Dividing the data text to be processed into n sub-data texts to be processed according to n-1 target separators; The n sub-data texts to be processed are distributed one by one to n server nodes for data processing.

2. The data processing method based on distributed architecture according to claim 1, characterized in that: The step of selecting n-1 of the spacers as target spacers according to the n server nodes comprises: Randomly select n-1 of the spacers as initial spacers; Dividing the data text to be processed into n initial sub-data texts to be processed according to the initial separator; Marking n initial sub-data texts to be processed as the first text, the second text, ..., the nth text in sequence; Vectorize the first text, the second text, ..., and the nth text respectively to obtain a first vector, a second vector, ..., and an nth vector; Calculate the similarity between two adjacent vectors, map the obtained similarity to the interval (0,1), and find the linear distance from 1; When all calculated linear distances are greater than a set threshold, the initial separator is recorded as a target separator.

3. The data processing method based on distributed architecture according to claim 2, characterized in that: After the step of calculating the similarity value between two adjacent vectors and mapping the obtained similarity value to the interval (0, 1) to find the linear distance from 1, the method further includes: If there is a calculated linear distance that is less than or equal to the set threshold, the initial separator is re-determined until the calculated linear distance is greater than the set threshold; The initial separators corresponding to the calculated linear distances that are all greater than the set threshold are recorded as target separators.

4. The data processing method based on a distributed architecture according to claim 1, characterized in that: The step of distributing the n sub-data texts to be processed one by one to n server nodes for data processing includes: Get the processed historical data texts from n server nodes; Calculate the similarity between each sub-data text to be processed and each historical data text; According to the similarity, each server node is assigned a corresponding sub-data text to be processed for processing.

5. The data processing method based on distributed architecture according to claim 1, characterized in that: After the step of distributing the n sub-data texts to be processed one by one to n server nodes for data processing, the method further includes: The server node obtains a corresponding subset of data to be processed according to the obtained sub-data to be processed text; The data to be processed in the subset of data to be processed is processed.

6. The data processing method based on distributed architecture according to claim 1, characterized in that: The step of performing text conversion on each of the to-be-processed data in the to-be-processed data set according to a preset conversion method to obtain the data text of each to-be-processed data includes: Identify the type of data structure to be processed; Determine a traversal method according to the type of the data structure to be processed; The data structure to be processed is traversed according to the traversal method, and the corresponding data text is generated according to the traversed data type and value.

7. The data processing method based on distributed architecture according to claim 1, characterized in that: The step of selecting n-1 of the spacers as target spacers according to the n server nodes comprises: Randomly select n-1 of the spacers as initial spacers; Dividing the data text to be processed into n initial sub-data texts to be processed according to the initial separator; Marking n initial sub-data texts to be processed as the first text, the second text, ..., the nth text in sequence; Calculating the text lengths of the first text, the second text, ..., the nth text; Subtract the minimum text length from the maximum text length to get the text length range; Determine whether the text length range is less than a preset range; If it is smaller than the preset range, the initial separator is used as the target separator.

8. A data processing device with a distributed architecture, characterized in that: include: An acquisition module is used to obtain the data set to be processed; A conversion module, used to perform text conversion on each of the to-be-processed data in the to-be-processed data set according to a preset conversion method to obtain a data text of each to-be-processed data; An arrangement module, used for obtaining the type of each of the data texts, arranging the data texts according to a preset type arrangement order, separating two adjacent data texts with a preset separator, and obtaining data texts to be processed; A determination module, used to determine n server nodes in the distributed architecture for processing the data set to be processed; A selection module, used for selecting n-1 of the spacers as target spacers according to the n server nodes; A division module, used for dividing the data text to be processed into n sub-data texts to be processed according to n-1 target separators; The distribution module is used to distribute the n sub-data texts to be processed one by one to n server nodes for data processing.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images