Multi-platform merchant data intelligent preprocessing method, device and equipment and storage medium

Abstract

The invention relates to the field of data preprocessing, and discloses a multi-platform merchant data intelligent preprocessing method and device, equipment and a storage medium, and the method comprises the steps: obtaining a target initial operation data set from a target e-commerce platform server set based on target merchant information; performing header configuration and key data extraction on the target initial operation data set based on a first preprocessing strategy to obtain a first preprocessed data set; and performing data cleaning on the first preprocessed data set based on a second preprocessing strategy to obtain a second preprocessed data set, and sending the second preprocessed data set to the target storage space for storage. According to the embodiment of the invention, a plurality of target initial operation data subsets with a plurality of data types and formats can be obtained from a plurality of different e-commerce platform servers, and the target initial operation data subsets are converted into preprocessed data with uniform data structures and formats after multi-dimensional preprocessing; the differential data of each e-commerce platform server can be smoothed, and the preprocessed data can be conveniently and efficiently applied.

Description

Technical Field

[0001] The present invention relates to the technical field of data preprocessing, and in particular to a method, device, equipment and storage medium for intelligent preprocessing of merchant data on multiple platforms. Background Art

[0002] At present, there are various ways to sell goods, and live streaming sales on live streaming platforms is one of them. For example, when the same merchant sells goods on different live streaming platforms, historical operating data corresponding to the merchant will be generated.

[0003] Different live streaming platforms have their own customized settings for the full process data of product sales. For example, the main process of product sales on each live streaming platform is ordering, shipping, receiving, and settlement. However, combined with the data rules of each live streaming platform itself, the sub-node links, sub-nodes, and branch processes under each main node are different. As a result, the data structures and fields of order data, settlement data, logistics data, and data to be settled for the same merchant on each live streaming platform are different, which causes great difficulties for subsequent data aggregation analysis and comprehensive calculations. Summary of the invention

[0004] The embodiments of the present invention provide a multi-platform merchant data intelligent preprocessing method, device, equipment and storage medium, aiming to solve the problem in the prior art that historical business data generated by live-streaming merchants selling goods on different live-streaming platforms have different data structures, fields, etc., which greatly difficulties the subsequent data aggregation analysis and comprehensive calculation.

[0005] In a first aspect, an embodiment of the present invention provides a multi-platform merchant data intelligent preprocessing method, which includes:

[0006] In response to the data preprocessing instruction, obtaining target merchant information and a target e-commerce platform server set corresponding to the data preprocessing instruction; wherein the target e-commerce platform server set includes a plurality of e-commerce platform servers, and each e-commerce platform server stores a subset of historical business data corresponding to the target merchant information;

[0007] Acquiring a target initial business data set from the target e-commerce platform server set based on the target merchant information; wherein the target initial business data set includes a plurality of target initial business data subsets having a plurality of data types and formats, and each target initial business data subset is obtained by screening from one of a plurality of historical business data subsets corresponding to the target merchant information;

[0008] Based on a preset first preprocessing strategy, header configuration and key data extraction are performed on the target initial business data set to obtain a first preprocessing data set;

[0009] Performing data cleaning on the first preprocessed data set based on a preset second preprocessing strategy to obtain a second preprocessed data set; and

[0010] A target storage space corresponding to the second preprocessed data set is acquired, and the second preprocessed data set is sent to the target storage space for storage.

[0011] In a second aspect, an embodiment of the present invention further provides a multi-platform merchant data intelligent pre-processing device, which includes:

[0012] A target merchant information acquisition unit, configured to, in response to a data preprocessing instruction, acquire target merchant information and a target e-commerce platform server set corresponding to the data preprocessing instruction; wherein the target e-commerce platform server set includes a plurality of e-commerce platform servers, and each e-commerce platform server stores a subset of historical business data corresponding to the target merchant information;

[0013] a target data set acquisition unit, configured to acquire a target initial business data set from the target e-commerce platform server set based on the target merchant information; wherein the target initial business data set includes a plurality of target initial business data subsets having a plurality of data types and formats, and each target initial business data subset is obtained by screening from one of a plurality of historical business data subsets corresponding to the target merchant information;

[0014] A first preprocessing unit, configured to perform table header configuration and key data extraction on the target initial business data set based on a preset first preprocessing strategy to obtain a first preprocessing data set;

[0015] A second preprocessing unit, configured to perform data cleaning on the first preprocessing data set based on a preset second preprocessing strategy to obtain a second preprocessing data set; and

[0016] The preprocessed data storage unit is used to obtain a target storage space corresponding to the second preprocessed data set, and send the second preprocessed data set to the target storage space for storage.

[0017] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method described in the first aspect above can be implemented.

[0019] Embodiments of the present invention provide a multi-platform merchant data intelligent preprocessing method, device, equipment and storage medium, the method comprising: in response to a data preprocessing instruction, obtaining target merchant information and a target e-commerce platform server set corresponding to the data preprocessing instruction; wherein the target e-commerce platform server set includes multiple e-commerce platform servers, and each e-commerce platform server stores a historical business data subset corresponding to the target merchant information; based on the target merchant information, obtaining a target initial business data set from the target e-commerce platform server set; wherein the target initial business data set includes multiple target initial business data subsets with multiple data types and formats, and each target initial business data subset is screened from one of the multiple historical business data subsets corresponding to the target merchant information; based on a preset first preprocessing strategy, the target initial business data set is header configured and key data is extracted to obtain a first preprocessing data set; based on a preset second preprocessing strategy, the first preprocessing data set is cleaned to obtain a second preprocessing data set; obtaining a target storage space corresponding to the second preprocessing data set, and sending the second preprocessing data set to the target storage space for storage. In the embodiment of the present invention, multiple target initial business data subsets with multiple data types and formats can be obtained from multiple different e-commerce platform servers to form a target initial business data set, and after multi-dimensional preprocessing, they are converted into preprocessed data with a unified data structure and format, thereby achieving smooth processing of differentiated data from each e-commerce platform server and facilitating the subsequent efficient application of preprocessed data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0021] Figure 1 A schematic diagram of an application scenario of the multi-platform merchant data intelligent pre-processing method provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a process flow of a multi-platform merchant data intelligent pre-processing method provided by an embodiment of the present invention;

[0023] Figure 3A schematic diagram of a sub-process of a multi-platform merchant data intelligent pre-processing method provided by an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of another sub-process of the multi-platform merchant data intelligent pre-processing method provided by an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of another sub-process of the multi-platform merchant data intelligent pre-processing method provided by an embodiment of the present invention;

[0026] Figure 6 A schematic block diagram of a multi-platform merchant data intelligent pre-processing device provided by an embodiment of the present invention;

[0027] Figure 7 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0030] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0031] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0032] Please also see Figure 1 and Figure 2 , Figure 1 Schematic diagram of a scenario of a multi-platform merchant data intelligent preprocessing method according to an embodiment of the present invention. Figure 21 is a flow chart of a multi-platform merchant data intelligent preprocessing method provided by an embodiment of the present invention. The method is applied to a server 10, which can be connected to an e-commerce platform server set 20 for communication. Figure 2 As shown, the method includes the following steps S110-S150.

[0033] S110. In response to a data preprocessing instruction, obtain target merchant information and a target e-commerce platform server set corresponding to the data preprocessing instruction.

[0034] The target e-commerce platform server set includes a plurality of e-commerce platform servers, and each e-commerce platform server stores a subset of historical business data corresponding to the target merchant information.

[0035] In this embodiment, the technical solution is described with the server as the execution subject. The server can be regarded as a data middle station or a data processing server integrated with data preprocessing function, and the server can communicate with multiple e-commerce platform servers included in the target e-commerce platform server set. Moreover, the server can obtain business data related to the merchant from each e-commerce platform server based on the merchant's authorized login data for further data analysis of the merchant (such as risk level determination, etc.).

[0036] For example, the server operation and maintenance personnel can first obtain a merchant list in the data preprocessing system deployed by the server (the merchant list includes several merchant names, and the merchant corresponding to each merchant name has merchant live broadcast type business data stored in at least one e-commerce platform server in the target e-commerce platform server set); then select at least one merchant from the merchant list and obtain the corresponding merchant information (such as the merchant information of the merchant includes at least the merchant name, merchant unique code, merchant contact representative, etc.) to form the target merchant information; finally, according to the merchant name or merchant unique code included in the target merchant information, determine the target e-commerce platform server set related to each target merchant in the target merchant information in the server. More specifically, the server operation and maintenance personnel select a merchant A1 in the merchant list, and then automatically obtain the target merchant information corresponding to merchant A1, and finally determine the e-commerce platform server corresponding to the live broadcast platform where merchant A1 has specifically opened a live broadcast sales business to form the target e-commerce platform server set. It can be seen that based on the above method, the target merchant information and the target e-commerce platform server set can be quickly determined.

[0037] S120. Acquire a target initial business data set from the target e-commerce platform server set based on the target merchant information.

[0038] The target initial business data set includes multiple target initial business data subsets with multiple data types and formats, and each target initial business data subset is screened from one of the multiple historical business data subsets corresponding to the target merchant information.

[0039] In this embodiment, still referring to the above example, if the target merchant information includes at least merchant A1, and it is determined that merchant A1 has opened live sales business on live broadcast platform B1, live broadcast platform B2 and live broadcast platform B3, when merchant A1 carries out live sales business on live broadcast platform B1-live broadcast platform B3 respectively, the corresponding historical business data will be stored in the e-commerce platform servers corresponding to live broadcast platform B1-live broadcast platform B3 respectively, that is, the e-commerce platform servers corresponding to the above-mentioned live broadcast platforms respectively store the historical business data subset corresponding to merchant A1. The server can obtain the historical business data subset corresponding to merchant A1 from the e-commerce platform servers corresponding to the above-mentioned live broadcast platforms based on the authorized login data provided by merchant A1, thereby forming the target initial business data set corresponding to merchant A1. In order to reduce the amount of data acquired for merchant A1, the instruction trigger time t1 corresponding to the data preprocessing instruction can be used as the end point of the target time interval, and m days (such as m = 30, 60, 90, etc., and not limited to the above values, can be customized according to actual needs) can be calculated forward to determine the starting point of the target time interval, that is, [t1-m, t1) is used as the target time interval. The subsets of historical business data of merchant A1 acquired by the server in the above manner are all composed of historical business data in the above target time interval.

[0040] Take the example that the same merchant selects one historical business data from the business data subset corresponding to each e-commerce platform server. For example, the merchant is still merchant A1 in the above example, and obtains historical business data C1 from the historical business data subset in the e-commerce platform server corresponding to live broadcast platform B1, obtains historical business data C2 from the historical business data subset in the e-commerce platform server corresponding to live broadcast platform B2, and obtains historical business data C3 from the historical business data subset in the e-commerce platform server corresponding to live broadcast platform B3. If the historical business data C1, historical business data C2, and historical business data C3 all include at least some of the order data, settlement data, logistics data, and data to be settled, and the order data, settlement data, logistics data, and data to be settled have different data structures and field names (and / or field types) in different e-commerce platform servers, then the historical business data C1, historical business data C2, and historical business data C3 are obtained by the server. In order to facilitate the subsequent data analysis, it is necessary to pre-process the historical business data C1, historical business data C2, and historical business data C3 to unify the data structure and field name (and / or field type). Of course, during the specific implementation, the historical operating data may include not only the data listed above, but also store rating data corresponding to the merchant, store violation record data, store reward and punishment status data, store violation points data, store payment recharge data, etc., and the above data of the merchant can be used to conduct risk control assessment on the merchant.

[0041] For example, the historical operating data C1 corresponding to merchant A1 includes logistics data C11 and settlement data C12. The logistics data C11 includes fields such as order number, merchant actual payment amount (yuan), after-sales status, express number, order transaction time, and delivery time; the settlement data C12 includes fields such as merchant order number | occurrence time, income amount, expenditure amount, account type, etc.

[0042] The historical operating data C2 corresponding to merchant A1 includes logistics data C21 and settlement order data C22. The logistics data C21 includes fields such as order number, order creation time, order status, actual payment, express order number, return and refund; the settlement order data C22 includes fields such as order number, order creation time, order status, actual payment, express order number, after-sales status, etc.

[0043] The historical operating data C3 corresponding to merchant A1 includes logistics data C31 and settlement order data C32. The logistics data C31 includes fields such as main order number, sub-order number, order submission time, order status, delivery time, order payable amount, after-sales status, etc.; the settlement order data C32 includes fields such as settlement time, order number, sub-order number, settlement amount, settlement order type, order time, total revenue, etc.

[0044] Once the detailed data and fields respectively included in the historical operating data C1-C3 of merchant A1 are known, they can be further preprocessed in various ways so that the preprocessed data C1' that ultimately corresponds to the historical operating data C1, the preprocessed data C2' that ultimately corresponds to the historical operating data C2, and the preprocessed data C3' that ultimately corresponds to the historical operating data C3 have unified data structures and field names.

[0045] S130. Performing header configuration and key data extraction on the target initial business data set based on a preset first preprocessing strategy to obtain a first preprocessing data set.

[0046] In this embodiment, the target initial business data set may be firstly configured with a header and key data may be extracted based on a first preprocessing strategy preset in the server to obtain a first preprocessing data set, thereby obtaining data that is convenient for subsequent unified data analysis.

[0047] In one embodiment, as an embodiment in which the server processes the target initial business data set without performing file data splitting, the first preprocessing strategy includes a header configuration sub-strategy and a core field screening sub-strategy, such as Figure 3 As shown, step S130 includes:

[0048] S1301, performing effective table screening and header uniform conversion on the target initial business data set based on the unified header configuration file in the header configuration sub-strategy to obtain a first parsed data set;

[0049] S1302: Perform core field screening on each data table in the first parsed data set based on the core field header set in the core field screening sub-strategy to obtain a second parsed data set as the first preprocessed data set.

[0050] In this embodiment, the target initial business data set includes multiple data tables (each data table is in one of the xlsx format or the csv format). For example, still referring to the above example, data table 1 is the order data table generated by merchant A1 on the live broadcast platform B1, data table 2 is the settlement data table generated by merchant A1 on the live broadcast platform B1, data table 3 is the merchant live broadcast time statistics table generated by merchant A1 on the live broadcast platform B1, data table 4 is the order data table generated by merchant A1 on the live broadcast platform B2, data table 5 is the settlement data table generated by merchant A1 on the live broadcast platform B2, data table 6 is the merchant live broadcast time statistics table generated by merchant A1 on the live broadcast platform B2, data table 7 is the order data table generated by merchant A1 on the live broadcast platform B3, data table 8 is the settlement data table generated by merchant A1 on the live broadcast platform B3, and data table 9 is the merchant live broadcast time statistics table generated by merchant A1 on the live broadcast platform B3. At this time, the target initial business data set can be first screened for effective tables based on the unified header configuration file. For example, a unified header for the order data table and a unified header for the settlement data table are set in the unified header configuration file. Then, the respective headers in the above-mentioned data table 3, data table 6 and data table 9 are not related to the unified header for the order data table and the unified header for the settlement data table. Therefore, data table 3, data table 6 and data table 9 can be deleted from the target initial business data set to obtain the first initial parsed data set. Afterwards, the field names of each data table in the first initial parsed data set are converted according to the corresponding unified header in the unified header configuration file. For example, data table 1, data table 4 and data table 7 are converted based on the unified header of the order data table (obtaining data table 1A corresponding to data table 1, data table 4A corresponding to data table 4 and data table 7A corresponding to data table 7), and data table 2, data table 5 and data table 8 are converted based on the unified header of the settlement data table (obtaining data table 2A corresponding to data table 2, data table 5A corresponding to data table 5 and data table 8A corresponding to data table 8). After completing the above-mentioned unified header conversion processing, the first parsed data set can be obtained.

[0051] Among them, in the above determination that data table 1, data table 4 and data table 7 are based on the unified header of the order data table to convert the names of the fields in the header, it is first determined that the vector similarity (such as cosine similarity) between the semantic vectors corresponding to the headers of data table 1, data table 4 and data table 7 and the semantic vector corresponding to the unified header of the order data table exceeds the first preset similarity threshold. Similarly, in determining that data table 2, data table 5 and data table 8 are based on the unified header of the settlement data table to convert the names of the fields in the header, it is first determined that the vector similarity (such as cosine similarity) between the semantic vectors corresponding to the headers of data table 2, data table 5 and data table 8 and the semantic vector corresponding to the unified header of the settlement data table exceeds the second preset similarity threshold.

[0052] Afterwards, the core fields of the order data table corresponding to the unified header of the order data table are obtained in the core field header set, and based on the core fields of the order data table, core fields are screened for each data table corresponding to the unified header of the order data table in the first parsed data set (such as the above-mentioned data table 1A, data table 4A and data table 7A). For example, data table 1A includes 11 field names, and the core fields of the order data table include 7 field names. Then, of the 11 field names of data table 1A, 7 field names that are exactly the same as the core fields of the order data table are retained, and the remaining 4 field names and corresponding field values ​​are deleted or screened as non-core fields. Similarly, the core fields of the settlement data table corresponding to the unified header of the settlement data table in the core field header set are obtained, and the core fields of each data table corresponding to the unified header of the settlement data table in the first parsed data set (such as the above-mentioned data table 2A, data table 5A and data table 8A) are screened based on the core fields of the settlement data table. For example, if data table 2A includes 8 field names, and the core fields of the settlement data table include 6 field names, then the 8 field names of data table 2A retain 6 field names that are exactly the same as the core fields of the settlement data table, and the remaining 2 field names and corresponding field values ​​are deleted or screened as non-core fields. After completing the header configuration and key data extraction of the target initial operating data set, the first preprocessed data set after the first round of preprocessing can be obtained.

[0053] In one embodiment, the first preprocessing strategy further includes a non-core field screening sub-strategy, and after step S1302, further includes:

[0054] Based on the non-core field header set in the non-core field screening sub-strategy, core fields are screened for each data table in the first parsed data set to obtain a third parsed data set, and a parsing warning log corresponding to the third parsed data set is generated.

[0055] In this embodiment, when the first pre-processing strategy also sets a non-core field screening sub-strategy, the non-core field screening sub-strategy can be flexibly effective, for example, it can be effective after step S1302, or it can also be that the above-mentioned non-core field screening step is performed after all the steps of this application are executed. Moreover, in order to intuitively prompt whether the first parsing data set corresponding to the target initial business data set has been parsed in the server, as long as the non-core field screening instruction is received in the server and the non-core field screening sub-strategy is executed accordingly to perform core field screening on each data table in the first parsing data set, it is necessary to generate a parsing warning log corresponding to the parsing process in the server to prompt the user the occurrence time of the above-mentioned parsing process and the parsing object targeted (that is, the first parsing data set corresponding to the target initial business data set).

[0056] In one embodiment, as an embodiment in which the server splits the target initial business data set into file data and performs parallel processing, the first preprocessing strategy includes a data splitting sub-strategy, a header configuration sub-strategy, and a core field screening sub-strategy, such as Figure 4 As shown, step S130 includes:

[0057] S1311, split and group the target initial business data set based on the data splitting sub-strategy to obtain multiple target split data subsets, and send each target split data subset to a task queue;

[0058] S1312, each task queue performs effective table screening and header uniform conversion on the received target split data subset based on the unified header configuration file in the header configuration sub-strategy to obtain a first parsed data subset;

[0059] S1313, each task queue performs core field screening on each data table in the first parsed data subset based on the core field header set in the core field screening sub-strategy for the obtained first parsed data subset, to obtain a second parsed data subset;

[0060] S1314. Obtain a second parsed data subset in each task queue to form the first preprocessed data set.

[0061] In this embodiment, as an embodiment of the server splitting the file data of the target initial business data set and performing parallel processing, the server can be regarded as a server cluster and includes multiple sub-servers. A task queue is set in each sub-server, thereby realizing parallel processing of multiple target split data subsets obtained by splitting the target initial business data set, thereby improving the overall processing efficiency.

[0062] Still taking the target initial business data set including multiple data tables as an example, and continuing to refer to the above example, the target initial business data set includes data tables 1-9, at this time, the target initial business data set can be first split and grouped based on the data splitting sub-strategy to obtain multiple target split data subsets, and each target split data subset is sent to the task queues in each sub-server included in the server. Preferably, the data splitting sub-strategy is to determine the total number of data split groups according to the total number of all task queues in the sub-servers included in the server. For example, the server includes 2 sub-servers, and one of the sub-servers is deployed with 4 task queues and the other sub-server is deployed with 5 task queues. Then the data splitting sub-strategy can split the target initial business data set into 9 target split data subsets, and send the above 9 target split data subsets to the only task queue of the sub-server for processing, thereby realizing parallel processing of data. Of course, when specifically implemented, the data splitting sub-strategy is not limited to determining the total number of data split groups according to the total number of all task queues in the sub-servers included in the server, and can also be combined with the total data volume of the target initial business data set and the data table attributes to perform grouping.

[0063] Afterwards, each task queue performs header configuration and key data extraction on the target split data subset. The specific process can also refer to the above-mentioned embodiment of processing the target initial business data set without file data splitting, that is, each task queue first performs effective table screening and header uniform conversion on each data table for the received target split data subset based on the unified header configuration file in the header configuration sub-strategy, to obtain the first parsed data subset; then each task queue continues to perform core field screening on each data table for the processed first parsed data subset based on the core field header set in the core field screening sub-strategy, to obtain the second parsed data subset; finally, the second parsed data subsets obtained in each task queue are summarized to obtain the first preprocessed data set.

[0064] Among them, in order to ensure that the target split data subsets obtained by splitting the same target initial business data set can be accurately summarized after a series of data processing, when a target initial business data set is split and grouped based on the data splitting sub-strategy, it is necessary to add the same data set unique identification ID value to each obtained target split data subset. When determining the data set unique identification ID value of the target initial business data set, the merchant unique code in the target merchant information can be directly used as the data set unique identification ID value of the target initial business data set.

[0065] In one embodiment, step S1311 includes:

[0066] Obtaining the total number of task queues in the server and the total data volume of the target initial business data set;

[0067] Based on the data splitting sub-strategy, the total number of task queues and the total data volume, the data tables in the target initial operating data set are evenly grouped according to the total number of task queues and the total data volume to obtain a target split data subset whose total number is the total number of task queues; wherein the target split data subset includes several data tables in the target initial operating data set.

[0068] In this embodiment, when grouping is performed in combination with the total number of task queues in the server, the total data volume of the target initial business data set, and the total number of data tables, the division can be performed in accordance with the principle of grouping as close to the average as possible. For example, when the total data volume of the target initial business data set is N1 (the unit is MB, GB or TB, and N1 is a positive number), the total number of task queues in the server is N2 (N2 is a positive integer), and the total number of data tables of the target initial business data set is N3 (N3 is a positive integer), the result of N1 / N2 can be rounded up to obtain a first calculation result, which can represent the maximum data volume of a target differential data subset, and then the data size of each data table in the target initial business data set is obtained, and the N3 data tables are flexibly grouped and combined in the form of non-splitting tables (that is, the same data table will not be split and divided into different target split data subsets, but each complete data table will be divided into a unique target split data subset), to obtain multiple target differential data subsets. Among them, each target differential data subset includes several complete data tables, and the total amount of data corresponding to several data tables in each target differential data subset is less than the first calculation result, and the sum of the total amount of data corresponding to each target differential data subset is equal to the total data amount N1.

[0069] In one embodiment, before step S130, the method further includes:

[0070] Obtain a field name mapping subfile and a field value mapping subfile, and form a header configuration substratum in the first preprocessing strategy; wherein the field name mapping subfile is used to uniformly convert the header field names of the data table, and the field value mapping subfile is used to map and convert the field values ​​corresponding to the field names in the data table that have undergone uniform conversion.

[0071] The header configuration sub-strategy in the first pre-processing strategy is stored in an intermediate table created in the server.

[0072] In this embodiment, when performing unified conversion processing on the headers of each data table, in addition to converting each field name into a field name corresponding to the unified header configuration file (specifically, the field name mapping subfile therein), it is also necessary to convert the field values ​​corresponding to each field name in the data table into corresponding field values ​​according to the field value mapping subfile in the unified header configuration file. For example, the field name of the order status in data table 1 is converted into the order status based on the unified header configuration file (it can be regarded as the field name of the order status in data table 1 that meets the requirements of the field name mapping subfile in the unified header configuration file, and the original field name can be retained). If the field value corresponding to the order status in data table 1 is "completed", the field value corresponding to the order status after conversion based on the field value mapping subfile in the unified header configuration file is "shipped".

[0073] Data formatting can also be implemented in the field value mapping subfile. For example, in the example above, after the field value mapping subfile in the unified header configuration file is converted and the field value corresponding to the order status is "shipped", the data type unification in the data formatting is implemented. The field value mapping subfile can format the date and time in the original data table (such as the above data table 1), unify the date and time fields into a standard format (such as YYYY-MM-DD, HH:MM:SS); and can also convert all text values ​​in the original data table into a unified text encoding (such as UTF-8 encoding, which is a text encoding standard).

[0074] The header configuration sub-strategy in the first preprocessing strategy is stored in the intermediate table of the server. The intermediate table can be regarded as an intermediate anti-corrosion layer design. If the actual needs of the user change, only the field name mapping sub-file and field value mapping sub-file in the intermediate table need to be modified, and there is no need to prompt the e-commerce platform server to adjust the specific field name and field value range of the field, thus realizing the decoupling of data cleaning and processing and basic business logic. Of course, if the first preprocessing strategy also includes a core field screening sub-strategy, the core field screening sub-strategy can also be stored in the above intermediate table.

[0075] S140: Perform data cleaning on the first preprocessed data set based on a preset second preprocessing strategy to obtain a second preprocessed data set.

[0076] In this embodiment, the first preprocessed data set may be cleaned based on the second preprocessing strategy preset in the server to obtain a second preprocessed data set, thereby obtaining data that is convenient for subsequent standardized unified storage and data analysis.

[0077] In one embodiment, the second preprocessing strategy includes a data deduplication sub-strategy, a missing value filling sub-strategy and an outlier processing sub-strategy, such as Figure 5 As shown, step S140 includes:

[0078] S141, performing data deduplication on the first pre-processed data set based on the data deduplication sub-strategy to obtain a first processed data set;

[0079] S142, completing missing values ​​of the first processed data set based on the missing value filling sub-strategy to obtain a second processed data set;

[0080] S143: Delete outliers from the second processed data set based on the outlier processing sub-strategy to obtain the second pre-processed data set.

[0081] In this embodiment, the data deduplication sub-strategy in the second preprocessing strategy is used to select and save duplicate data in the first preprocessing data set, the missing value filling sub-strategy is used to fill in missing values ​​in some missing values ​​in the first processing data set based on a preset missing value filling method (such as mean value filling, etc.), and the abnormal value processing sub-strategy is used to identify abnormal values ​​in the second processing data set and delete abnormal data to which the abnormal values ​​belong (for example, based on the 3σ principle, it can be determined whether the values ​​of each field of a certain data in the data table are abnormal values. When there is an abnormal value in the field value of the data, the data can be deleted as abnormal data). After completing the data cleaning of the first preprocessing data set, the second preprocessing data set that can be used to store in the target area can be obtained.

[0082] S150: Acquire a target storage space corresponding to the second preprocessed data set, and send the second preprocessed data set to the target storage space for storage.

[0083] In this embodiment, after completing the multiple data preprocessing of the target initial business data set corresponding to the target merchant information, the second preprocessed data set can be obtained, and then the second preprocessed data set can be sent to the target storage space corresponding to the second preprocessed data set in the server for storage. If a risk control model is also deployed in the server, the second preprocessed data set can be input into the corresponding risk control model to obtain the risk level corresponding to the target merchant information.

[0084] It can be seen that the implementation example of the method can obtain multiple target initial business data subsets with multiple data types and formats from multiple different e-commerce platform servers to form a target initial business data set, and convert them into preprocessed data with a unified data structure and format after multi-dimensional preprocessing, thereby achieving smooth processing of differentiated data from each e-commerce platform server and facilitating the subsequent efficient application of preprocessed data.

[0085] Figure 6 is a schematic block diagram of a multi-platform merchant data intelligent pre-processing device provided by an embodiment of the present invention. Figure 6 As shown, corresponding to the above multi-platform merchant data intelligent preprocessing method, the present invention also provides a multi-platform merchant data intelligent preprocessing device 100. The multi-platform merchant data intelligent preprocessing device includes: a target merchant information acquisition unit 110, a target data set acquisition unit 120, a first preprocessing unit 130, a second preprocessing unit 140 and a preprocessed data storage unit 150.

[0086] The target transaction data set acquisition unit 110 is used to respond to the data preprocessing instruction and acquire the target merchant information and the target e-commerce platform server set corresponding to the data preprocessing instruction.

[0087] The target e-commerce platform server set includes a plurality of e-commerce platform servers, and each e-commerce platform server stores a subset of historical business data corresponding to the target merchant information.

[0088] In this embodiment, the technical solution is described with the server as the execution subject. The server can be regarded as a data middle station or a data processing server integrated with data preprocessing function, and the server can communicate with multiple e-commerce platform servers included in the target e-commerce platform server set. Moreover, the server can obtain business data related to the merchant from each e-commerce platform server based on the merchant's authorized login data for further data analysis of the merchant (such as risk level determination, etc.).

[0089] For example, the server operation and maintenance personnel can first obtain a merchant list in the data preprocessing system deployed by the server (the merchant list includes several merchant names, and the merchant corresponding to each merchant name has merchant live broadcast type business data stored in at least one e-commerce platform server in the target e-commerce platform server set); then select at least one merchant from the merchant list and obtain the corresponding merchant information (such as the merchant information of the merchant includes at least the merchant name, merchant unique code, merchant contact representative, etc.) to form the target merchant information; finally, according to the merchant name or merchant unique code included in the target merchant information, determine the target e-commerce platform server set related to each target merchant in the target merchant information in the server. More specifically, the server operation and maintenance personnel select a merchant A1 in the merchant list, and then automatically obtain the target merchant information corresponding to merchant A1, and finally determine the e-commerce platform server corresponding to the live broadcast platform where merchant A1 has specifically opened a live broadcast sales business to form the target e-commerce platform server set. It can be seen that based on the above method, the target merchant information and the target e-commerce platform server set can be quickly determined.

[0090] The target data set acquisition unit 120 is used to acquire a target initial business data set from the target e-commerce platform server set based on the target merchant information.

[0091] The target initial business data set includes multiple target initial business data subsets with multiple data types and formats, and each target initial business data subset is screened from one of the multiple historical business data subsets corresponding to the target merchant information.

[0092] In this embodiment, still referring to the above example, if the target merchant information includes at least merchant A1, and it is determined that merchant A1 has opened live sales business on live broadcast platform B1, live broadcast platform B2 and live broadcast platform B3, when merchant A1 carries out live sales business on live broadcast platform B1-live broadcast platform B3 respectively, the corresponding historical business data will be stored in the e-commerce platform servers corresponding to live broadcast platform B1-live broadcast platform B3 respectively, that is, the e-commerce platform servers corresponding to the above-mentioned live broadcast platforms respectively store the historical business data subset corresponding to merchant A1. The server can obtain the historical business data subset corresponding to merchant A1 from the e-commerce platform servers corresponding to the above-mentioned live broadcast platforms based on the authorized login data provided by merchant A1, thereby forming the target initial business data set corresponding to merchant A1. In order to reduce the amount of data acquired for merchant A1, the instruction trigger time t1 corresponding to the data preprocessing instruction can be used as the end point of the target time interval, and m days (such as m = 30, 60, 90, etc., and not limited to the above values, can be customized according to actual needs) can be calculated forward to determine the starting point of the target time interval, that is, [t1-m, t1) is used as the target time interval. The subsets of historical business data of merchant A1 acquired by the server in the above manner are all composed of historical business data in the above target time interval.

[0093] Take the example that the same merchant selects one historical business data from the business data subset corresponding to each e-commerce platform server. For example, the merchant is still merchant A1 in the above example, and obtains historical business data C1 from the historical business data subset in the e-commerce platform server corresponding to live broadcast platform B1, obtains historical business data C2 from the historical business data subset in the e-commerce platform server corresponding to live broadcast platform B2, and obtains historical business data C3 from the historical business data subset in the e-commerce platform server corresponding to live broadcast platform B3. If the historical business data C1, historical business data C2, and historical business data C3 all include at least some of the order data, settlement data, logistics data, and data to be settled, and the order data, settlement data, logistics data, and data to be settled have different data structures and field names (and / or field types) in different e-commerce platform servers, then the historical business data C1, historical business data C2, and historical business data C3 are obtained by the server. In order to facilitate the subsequent data analysis, it is necessary to pre-process the historical business data C1, historical business data C2, and historical business data C3 to unify the data structure and field name (and / or field type). Of course, during the specific implementation, the historical operating data may include not only the data listed above, but also store rating data corresponding to the merchant, store violation record data, store reward and punishment status data, store violation points data, store payment recharge data, etc., and the above data of the merchant can be used to conduct risk control assessment on the merchant.

[0094] For example, the historical operating data C1 corresponding to merchant A1 includes logistics data C11 and settlement data C12. The logistics data C11 includes fields such as order number, merchant actual payment amount (yuan), after-sales status, express number, order transaction time, and delivery time; the settlement data C12 includes fields such as merchant order number | occurrence time, income amount, expenditure amount, account type, etc.

[0095] The historical operating data C2 corresponding to merchant A1 includes logistics data C21 and settlement order data C22. The logistics data C21 includes fields such as order number, order creation time, order status, actual payment, express order number, return and refund; the settlement order data C22 includes fields such as order number, order creation time, order status, actual payment, express order number, after-sales status, etc.

[0096] The historical operating data C3 corresponding to merchant A1 includes logistics data C31 and settlement order data C32. The logistics data C31 includes fields such as main order number, sub-order number, order submission time, order status, delivery time, order payable amount, after-sales status, etc.; the settlement order data C32 includes fields such as settlement time, order number, sub-order number, settlement amount, settlement order type, order time, total revenue, etc.

[0097] Once the detailed data and fields respectively included in the historical operating data C1-C3 of merchant A1 are known, they can be further preprocessed in various ways so that the preprocessed data C1' that ultimately corresponds to the historical operating data C1, the preprocessed data C2' that ultimately corresponds to the historical operating data C2, and the preprocessed data C3' that ultimately corresponds to the historical operating data C3 have unified data structures and field names.

[0098] The first preprocessing unit 130 is used to perform header configuration and key data extraction on the target initial business data set based on a preset first preprocessing strategy to obtain a first preprocessed data set.

[0099] In this embodiment, the target initial business data set may be firstly configured with a header and key data may be extracted based on a first preprocessing strategy preset in the server to obtain a first preprocessing data set, thereby obtaining data that is convenient for subsequent unified data analysis.

[0100] In one embodiment, as an embodiment in which the server processes the target initial business data set without performing file data splitting, the first preprocessing strategy includes a header configuration sub-strategy and a core field screening sub-strategy, and the first preprocessing unit 130 is specifically used to:

[0101] Based on the unified header configuration file in the header configuration sub-strategy, effective table screening and header unified conversion are performed on the target initial business data set to obtain a first parsed data set;

[0102] Based on the core field header set in the core field screening sub-strategy, core fields of each data table in the first parsed data set are screened to obtain a second parsed data set as the first preprocessed data set.

[0103] In this embodiment, the target initial business data set includes multiple data tables (each data table is in one of the xlsx format or the csv format). For example, still referring to the above example, data table 1 is the order data table generated by merchant A1 on the live broadcast platform B1, data table 2 is the settlement data table generated by merchant A1 on the live broadcast platform B1, data table 3 is the merchant live broadcast time statistics table generated by merchant A1 on the live broadcast platform B1, data table 4 is the order data table generated by merchant A1 on the live broadcast platform B2, data table 5 is the settlement data table generated by merchant A1 on the live broadcast platform B2, data table 6 is the merchant live broadcast time statistics table generated by merchant A1 on the live broadcast platform B2, data table 7 is the order data table generated by merchant A1 on the live broadcast platform B3, data table 8 is the settlement data table generated by merchant A1 on the live broadcast platform B3, and data table 9 is the merchant live broadcast time statistics table generated by merchant A1 on the live broadcast platform B3. At this time, the target initial business data set can be first screened for effective tables based on the unified header configuration file. For example, a unified header for the order data table and a unified header for the settlement data table are set in the unified header configuration file. Then, the respective headers in the above-mentioned data table 3, data table 6 and data table 9 are not related to the unified header for the order data table and the unified header for the settlement data table. Therefore, data table 3, data table 6 and data table 9 can be deleted from the target initial business data set to obtain the first initial parsed data set. Afterwards, the field names of each data table in the first initial parsed data set are converted according to the corresponding unified header in the unified header configuration file. For example, data table 1, data table 4 and data table 7 are converted based on the unified header of the order data table (obtaining data table 1A corresponding to data table 1, data table 4A corresponding to data table 4 and data table 7A corresponding to data table 7), and data table 2, data table 5 and data table 8 are converted based on the unified header of the settlement data table (obtaining data table 2A corresponding to data table 2, data table 5A corresponding to data table 5 and data table 8A corresponding to data table 8). After completing the above-mentioned unified header conversion processing, the first parsed data set can be obtained.

[0104] Among them, in the above determination that data table 1, data table 4 and data table 7 are based on the unified header of the order data table to convert the names of the fields in the header, it is first determined that the vector similarity (such as cosine similarity) between the semantic vectors corresponding to the headers of data table 1, data table 4 and data table 7 and the semantic vector corresponding to the unified header of the order data table exceeds the first preset similarity threshold. Similarly, in determining that data table 2, data table 5 and data table 8 are based on the unified header of the settlement data table to convert the names of the fields in the header, it is first determined that the vector similarity (such as cosine similarity) between the semantic vectors corresponding to the headers of data table 2, data table 5 and data table 8 and the semantic vector corresponding to the unified header of the settlement data table exceeds the second preset similarity threshold.

[0105] Afterwards, the core fields of the order data table corresponding to the unified header of the order data table are obtained in the core field header set, and based on the core fields of the order data table, core fields are screened for each data table corresponding to the unified header of the order data table in the first parsed data set (such as the above-mentioned data table 1A, data table 4A and data table 7A). For example, data table 1A includes 11 field names, and the core fields of the order data table include 7 field names. Then, of the 11 field names of data table 1A, 7 field names that are exactly the same as the core fields of the order data table are retained, and the remaining 4 field names and corresponding field values ​​are deleted or screened as non-core fields. Similarly, the core fields of the settlement data table corresponding to the unified header of the settlement data table in the core field header set are obtained, and the core fields of each data table corresponding to the unified header of the settlement data table in the first parsed data set (such as the above-mentioned data table 2A, data table 5A and data table 8A) are screened based on the core fields of the settlement data table. For example, if data table 2A includes 8 field names, and the core fields of the settlement data table include 6 field names, then the 8 field names of data table 2A retain 6 field names that are exactly the same as the core fields of the settlement data table, and the remaining 2 field names and corresponding field values ​​are deleted or screened as non-core fields. After completing the header configuration and key data extraction of the target initial operating data set, the first preprocessed data set after the first round of preprocessing can be obtained.

[0106] In one embodiment, the first preprocessing strategy further includes a non-core field screening sub-strategy, and the multi-platform merchant data intelligent preprocessing device 100 further includes:

[0107] The non-core field screening unit is used to perform core field screening on each data table in the first parsed data set based on the non-core field header set in the non-core field screening sub-strategy to obtain a third parsed data set, and generate a parsing warning log corresponding to the third parsed data set.

[0108] In this embodiment, when the first pre-processing strategy also sets a non-core field screening sub-strategy, the non-core field screening sub-strategy can be flexibly effective, for example, it can be effective after step S1302, or it can also be that the above-mentioned non-core field screening step is performed after all the steps of this application are executed. Moreover, in order to intuitively prompt whether the first parsing data set corresponding to the target initial business data set has been parsed in the server, as long as the non-core field screening instruction is received in the server and the non-core field screening sub-strategy is executed accordingly to perform core field screening on each data table in the first parsing data set, it is necessary to generate a parsing warning log corresponding to the parsing process in the server to prompt the user the occurrence time of the above-mentioned parsing process and the parsing object targeted (that is, the first parsing data set corresponding to the target initial business data set).

[0109] In one embodiment, as an embodiment in which the server splits the target initial business data set into file data and performs parallel processing, the first preprocessing strategy includes a data splitting sub-strategy, a header configuration sub-strategy, and a core field screening sub-strategy, and the first preprocessing unit 130 is further specifically used to:

[0110] Based on the data splitting sub-strategy, the target initial operating data set is split and grouped to obtain a plurality of target split data subsets, and each target split data subset is sent to a task queue;

[0111] Each task queue performs effective table screening and header uniform conversion on the received target split data subset based on the unified header configuration file in the header configuration sub-strategy to obtain a first parsed data subset;

[0112] Each task queue performs core field screening on each data table in the first parsed data subset based on the core field header set in the core field screening sub-strategy for the obtained first parsed data subset to obtain a second parsed data subset;

[0113] A second parsed data subset in each task queue is obtained to form the first pre-processed data set.

[0114] In this embodiment, as an embodiment of the server splitting the file data of the target initial business data set and performing parallel processing, the server can be regarded as a server cluster and includes multiple sub-servers. A task queue is set in each sub-server, thereby realizing parallel processing of multiple target split data subsets obtained by splitting the target initial business data set, thereby improving the overall processing efficiency.

[0115] Still taking the target initial business data set including multiple data tables as an example, and continuing to refer to the above example, the target initial business data set includes data tables 1-9, at this time, the target initial business data set can be first split and grouped based on the data splitting sub-strategy to obtain multiple target split data subsets, and each target split data subset is sent to the task queues in each sub-server included in the server. Preferably, the data splitting sub-strategy is to determine the total number of data split groups according to the total number of all task queues in the sub-servers included in the server. For example, the server includes 2 sub-servers, and one of the sub-servers is deployed with 4 task queues and the other sub-server is deployed with 5 task queues. Then the data splitting sub-strategy can split the target initial business data set into 9 target split data subsets, and send the above 9 target split data subsets to the only task queue of the sub-server for processing, thereby realizing parallel processing of data. Of course, when specifically implemented, the data splitting sub-strategy is not limited to determining the total number of data split groups according to the total number of all task queues in the sub-servers included in the server, and can also be combined with the total data volume of the target initial business data set and the data table attributes to perform grouping.

[0116] Afterwards, each task queue performs header configuration and key data extraction on the target split data subset. The specific process can also refer to the above-mentioned embodiment of processing the target initial business data set without file data splitting, that is, each task queue first performs effective table screening and header uniform conversion on each data table for the received target split data subset based on the unified header configuration file in the header configuration sub-strategy, to obtain the first parsed data subset; then each task queue continues to perform core field screening on each data table for the processed first parsed data subset based on the core field header set in the core field screening sub-strategy, to obtain the second parsed data subset; finally, the second parsed data subsets obtained in each task queue are summarized to obtain the first preprocessed data set.

[0117] Among them, in order to ensure that the target split data subsets obtained by splitting the same target initial business data set can be accurately summarized after a series of data processing, when a target initial business data set is split and grouped based on the data splitting sub-strategy, it is necessary to add the same data set unique identification ID value to each obtained target split data subset. When determining the data set unique identification ID value of the target initial business data set, the merchant unique code in the target merchant information can be directly used as the data set unique identification ID value of the target initial business data set.

[0118] In one embodiment, the step of performing data splitting and grouping on the target initial operating data set based on the data splitting sub-strategy to obtain a plurality of target split data subsets, and sending each target split data subset to a task queue includes:

[0119] Obtaining the total number of task queues in the server and the total data volume of the target initial business data set;

[0120] Based on the data splitting sub-strategy, the total number of task queues and the total data volume, the data tables in the target initial operating data set are evenly grouped according to the total number of task queues and the total data volume to obtain a target split data subset whose total number is the total number of task queues; wherein the target split data subset includes several data tables in the target initial operating data set.

[0121] In this embodiment, when grouping is performed in combination with the total number of task queues in the server, the total data volume of the target initial business data set, and the total number of data tables, the division can be performed in accordance with the principle of grouping as close to the average as possible. For example, when the total data volume of the target initial business data set is N1 (the unit is MB, GB or TB, and N1 is a positive number), the total number of task queues in the server is N2 (N2 is a positive integer), and the total number of data tables of the target initial business data set is N3 (N3 is a positive integer), the result of N1 / N2 can be rounded up to obtain a first calculation result, which can represent the maximum data volume of a target differential data subset, and then the data size of each data table in the target initial business data set is obtained, and the N3 data tables are flexibly grouped and combined in the form of non-splitting tables (that is, the same data table will not be split and divided into different target split data subsets, but each complete data table will be divided into a unique target split data subset), to obtain multiple target differential data subsets. Among them, each target differential data subset includes several complete data tables, and the total amount of data corresponding to several data tables in each target differential data subset is less than the first calculation result, and the sum of the total amount of data corresponding to each target differential data subset is equal to the total data amount N1.

[0122] In one embodiment, the multi-platform merchant data intelligent pre-processing device 100 further includes:

[0123] The first preprocessing strategy preconfiguration unit is used to obtain a field name mapping subfile and a field value mapping subfile, and constitute a header configuration substratum in the first preprocessing strategy; wherein the field name mapping subfile is used to uniformly convert the header field names of the data table, and the field value mapping subfile is used to map and convert the field values ​​corresponding to the field names in the data table that have undergone uniform conversion processing.

[0124] The header configuration sub-strategy in the first pre-processing strategy is stored in an intermediate table created in the server.

[0125] In this embodiment, when performing unified conversion processing on the headers of each data table, in addition to converting each field name into a field name corresponding to the unified header configuration file (specifically, the field name mapping subfile therein), it is also necessary to convert the field values ​​corresponding to each field name in the data table into corresponding field values ​​according to the field value mapping subfile in the unified header configuration file. For example, the field name of the order status in data table 1 is converted into the order status based on the unified header configuration file (it can be regarded as the field name of the order status in data table 1 that meets the requirements of the field name mapping subfile in the unified header configuration file, and the original field name can be retained). If the field value corresponding to the order status in data table 1 is "completed", the field value corresponding to the order status after conversion based on the field value mapping subfile in the unified header configuration file is "shipped".

[0126] Data formatting can also be implemented in the field value mapping subfile. For example, in the example above, after the field value mapping subfile in the unified header configuration file is converted and the field value corresponding to the order status is "shipped", the data type unification in the data formatting is implemented. The field value mapping subfile can format the date and time in the original data table (such as the above data table 1), unify the date and time fields into a standard format (such as YYYY-MM-DD, HH:MM:SS); and can also convert all text values ​​in the original data table into a unified text encoding (such as UTF-8 encoding, which is a text encoding standard).

[0127] The header configuration sub-strategy in the first preprocessing strategy is stored in the intermediate table of the server. The intermediate table can be regarded as an intermediate anti-corrosion layer design. If the actual needs of the user change, only the field name mapping sub-file and field value mapping sub-file in the intermediate table need to be modified, and there is no need to prompt the e-commerce platform server to adjust the specific field name and field value range of the field, thus realizing the decoupling of data cleaning and processing and basic business logic. Of course, if the first preprocessing strategy also includes a core field screening sub-strategy, the core field screening sub-strategy can also be stored in the above intermediate table.

[0128] The second preprocessing unit 140 is configured to perform data cleaning on the first preprocessing data set based on a preset second preprocessing strategy to obtain a second preprocessing data set.

[0129] In this embodiment, the first preprocessed data set may be cleaned based on the second preprocessing strategy preset in the server to obtain a second preprocessed data set, thereby obtaining data that is convenient for subsequent standardized unified storage and data analysis.

[0130] In one embodiment, the second preprocessing strategy includes a data deduplication sub-strategy, a missing value filling sub-strategy, and an outlier processing sub-strategy, and the second preprocessing unit 140 is specifically used to:

[0131] Deduplication of the first pre-processed data set is performed based on the data deduplication sub-strategy to obtain a first processed data set;

[0132] Filling missing values ​​in the first processed data set based on the missing value filling sub-strategy to obtain a second processed data set;

[0133] Outliers are deleted from the second processed data set based on the outlier processing sub-strategy to obtain the second pre-processed data set.

[0134] In this embodiment, the data deduplication sub-strategy in the second preprocessing strategy is used to select and save duplicate data in the first preprocessing data set, the missing value filling sub-strategy is used to fill in missing values ​​in some missing values ​​in the first processing data set based on a preset missing value filling method (such as mean value filling, etc.), and the abnormal value processing sub-strategy is used to identify abnormal values ​​in the second processing data set and delete abnormal data to which the abnormal values ​​belong (for example, based on the 3σ principle, it can be determined whether the values ​​of each field of a certain data in the data table are abnormal values. When there is an abnormal value in the field value of the data, the data can be deleted as abnormal data). After completing the data cleaning of the first preprocessing data set, the second preprocessing data set that can be used to store in the target area can be obtained.

[0135] The preprocessed data storage unit 150 is used to obtain a target storage space corresponding to the second preprocessed data set, and send the second preprocessed data set to the target storage space for storage.

[0136] In this embodiment, after completing the multiple data preprocessing of the target initial business data set corresponding to the target merchant information, the second preprocessed data set can be obtained, and then the second preprocessed data set can be sent to the target storage space corresponding to the second preprocessed data set in the server for storage. If a risk control model is also deployed in the server, the second preprocessed data set can be input into the corresponding risk control model to obtain the risk level corresponding to the target merchant information.

[0137] It can be seen that the implementation example of the device can obtain multiple target initial business data subsets with multiple data types and formats from multiple different e-commerce platform servers to form a target initial business data set, and convert them into preprocessed data with a unified data structure and format after multi-dimensional preprocessing, thereby achieving smooth processing of differentiated data from each e-commerce platform server and facilitating the subsequent efficient application of preprocessed data.

[0138] The multi-platform merchant data intelligent pre-processing device can be implemented in the form of a computer program. The computer program can be used in Figure 7 Runs on the computer device shown.

[0139] See also Figure 7 , Figure 7 The schematic block diagram of a computer device provided by an embodiment of the present invention is shown in FIG. The computer device integrates any multi-platform merchant data intelligent pre-processing device provided by an embodiment of the present invention.

[0140] See also Figure 7 The computer device 400 includes a processor 402 , a memory and a network interface 405 connected via a system bus 401 , wherein the memory may include a storage medium 403 and an internal memory 404 .

[0141] The storage medium 403 can store an operating system 4031 and a computer program 4032. The computer program 4032 includes program instructions, and when the program instructions are executed, the processor 402 can execute the above-mentioned multi-platform merchant data intelligent pre-processing method.

[0142] The processor 402 is used to provide computing and control capabilities to support the operation of the entire computer device.

[0143] The internal memory 404 provides an environment for the operation of the computer program 4032 in the storage medium 403. When the computer program 4032 is executed by the processor 402, the processor 402 can execute the above-mentioned multi-platform merchant data intelligent pre-processing method.

[0144] The network interface 405 is used to communicate with other devices over the network. Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0145] The processor 402 is used to run the computer program 4032 stored in the memory to implement the multi-platform merchant data intelligent pre-processing method as described above.

[0146] It should be understood that in the embodiment of the present invention, the processor 402 may be a central processing unit (CPU), and the processor 402 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0147] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0148] Therefore, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the multi-platform merchant data intelligent pre-processing method as described above.

[0149] The computer-readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0150] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0151] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0152] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0154] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multi-platform merchant data intelligent preprocessing method, characterized in that: include: In response to the data preprocessing instruction, obtaining target merchant information and a target e-commerce platform server set corresponding to the data preprocessing instruction; wherein the target e-commerce platform server set includes a plurality of e-commerce platform servers, and each e-commerce platform server stores a subset of historical business data corresponding to the target merchant information; Acquiring a target initial business data set from the target e-commerce platform server set based on the target merchant information; wherein the target initial business data set includes a plurality of target initial business data subsets having a plurality of data types and formats, and each target initial business data subset is obtained by screening from one of a plurality of historical business data subsets corresponding to the target merchant information; Based on a preset first preprocessing strategy, header configuration and key data extraction are performed on the target initial business data set to obtain a first preprocessing data set; Performing data cleaning on the first preprocessed data set based on a preset second preprocessing strategy to obtain a second preprocessed data set; and A target storage space corresponding to the second preprocessed data set is acquired, and the second preprocessed data set is sent to the target storage space for storage.

2. The method according to claim 1, characterized in that The first preprocessing strategy includes a header configuration sub-strategy and a core field screening sub-strategy; The step of performing header configuration and key data extraction on the target initial business data set based on the preset first preprocessing strategy to obtain the first preprocessing data set includes: Based on the unified header configuration file in the header configuration sub-strategy, effective table screening and header unified conversion are performed on the target initial business data set to obtain a first parsed data set; Based on the core field header set in the core field screening sub-strategy, core fields of each data table in the first parsed data set are screened to obtain a second parsed data set as the first preprocessed data set.

3. The method according to claim 2, characterized in that The first preprocessing strategy further includes a non-core field screening sub-strategy; after the step of screening the core fields of each data table in the first parsed data set based on the core field header set in the core field screening sub-strategy to obtain a second parsed data set as the first preprocessed data set, the method further includes: Based on the non-core field header set in the non-core field screening sub-strategy, core fields are screened for each data table in the first parsed data set to obtain a third parsed data set, and a parsing warning log corresponding to the third parsed data set is generated.

4. The method according to claim 1, characterized in that: The first preprocessing strategy includes a data splitting sub-strategy, a header configuration sub-strategy, and a core field screening sub-strategy; The step of performing header configuration and key data extraction on the target initial business data set based on the preset first preprocessing strategy to obtain the first preprocessing data set includes: Based on the data splitting sub-strategy, the target initial operating data set is split and grouped to obtain a plurality of target split data subsets, and each target split data subset is sent to a task queue; Each task queue performs effective table screening and header uniform conversion on the received target split data subset based on the unified header configuration file in the header configuration sub-strategy to obtain a first parsed data subset; Each task queue performs core field screening on each data table in the first parsed data subset based on the core field header set in the core field screening sub-strategy for the obtained first parsed data subset to obtain a second parsed data subset; A second parsed data subset in each task queue is obtained to form the first pre-processed data set.

5. The method according to claim 4, characterized in that The step of performing data splitting and grouping on the target initial operating data set based on the data splitting sub-strategy to obtain a plurality of target split data subsets, and sending each target split data subset to a task queue includes: Obtaining the total number of task queues in the server and the total data volume of the target initial business data set; Based on the data splitting sub-strategy, the total number of task queues and the total data volume, the data tables in the target initial operating data set are evenly grouped according to the total number of task queues and the total data volume to obtain a target split data subset whose total number is the total number of task queues; wherein the target split data subset includes several data tables in the target initial operating data set.

6. The method according to any one of claims 2 or 4, characterized in that: Before the step of performing header configuration and key data extraction on the target initial business data set based on the preset first preprocessing strategy to obtain the first preprocessing data set, the method further includes: Obtain a field name mapping subfile and a field value mapping subfile, and form a header configuration substratum in the first preprocessing strategy; wherein the field name mapping subfile is used to uniformly convert the header field names of the data table, and the field value mapping subfile is used to map and convert the field values ​​corresponding to the field names in the data table that have undergone uniform conversion. The header configuration sub-strategy in the first pre-processing strategy is stored in an intermediate table created in the server.

7. The method according to claim 1, characterized in that The second preprocessing strategy includes a data deduplication sub-strategy, a missing value filling sub-strategy and an outlier processing sub-strategy; The step of performing data cleaning on the first preprocessed data set based on a preset second preprocessing strategy to obtain a second preprocessed data set includes: Deduplication of the first pre-processed data set is performed based on the data deduplication sub-strategy to obtain a first processed data set; Perform missing value completion on the first processed data set based on the missing value filling sub-strategy to obtain a second processed data set; Outliers are deleted from the second processed data set based on the outlier processing sub-strategy to obtain the second pre-processed data set.

8. A multi-platform merchant data intelligent pre-processing device, characterized in that: include: A target merchant information acquisition unit, configured to, in response to a data preprocessing instruction, acquire target merchant information and a target e-commerce platform server set corresponding to the data preprocessing instruction; wherein the target e-commerce platform server set includes a plurality of e-commerce platform servers, and each e-commerce platform server stores a subset of historical business data corresponding to the target merchant information; a target data set acquisition unit, configured to acquire a target initial business data set from the target e-commerce platform server set based on the target merchant information; wherein the target initial business data set includes a plurality of target initial business data subsets having a plurality of data types and formats, and each target initial business data subset is obtained by screening from one of a plurality of historical business data subsets corresponding to the target merchant information; A first preprocessing unit, configured to perform table header configuration and key data extraction on the target initial business data set based on a preset first preprocessing strategy to obtain a first preprocessing data set; A second preprocessing unit, configured to perform data cleaning on the first preprocessing data set based on a preset second preprocessing strategy to obtain a second preprocessing data set; and The preprocessed data storage unit is used to obtain a target storage space corresponding to the second preprocessed data set, and send the second preprocessed data set to the target storage space for storage.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multi-platform merchant data intelligent pre-processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the multi-platform merchant data intelligent pre-processing method according to any one of claims 1 to 7 can be implemented.

Images