Method and computing device for obtaining big data of multiple e-commerce platforms

Abstract

The application provides a method for obtaining multi-e-commerce big data and a computing device. The method comprises the following steps: simultaneously starting a plurality of first threads with a number of N; using the plurality of first threads to request first type data of different online stores from an e-commerce platform respectively; storing the first type data in a distributed file system; and storing the first type data in the file system in a distributed database system. The capability of the N first threads for requesting data from the e-commerce platform is greater than the request limit of the e-commerce platform for the first type data, and the capability of N-2 first threads for requesting data from the e-commerce platform is less than the request limit of the e-commerce platform for the first type data. According to the scheme of the application, the interface capability provided by the e-commerce platform can be maximized.

Description

Technical Field

[0001] This application relates to the fields of machine learning and commercial big data technology, and more specifically, to a method and computing device for acquiring big data from multiple e-commerce platforms. Background Technology

[0002] With the development of network computing technology, e-commerce activities have generated a large amount of commercial big data. For example, compared with traditional industries, e-commerce generates massive amounts of raw e-commerce data within its ecosystem value chain. The acquisition, processing, or effective utilization of this data can assist business operations or support business decision-making.

[0003] Some e-commerce platforms have already opened their data interfaces. However, how to efficiently obtain data from these platforms is a noteworthy issue, especially for data analysis service systems that need to acquire data from numerous e-commerce stores.

[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] This application aims to provide a method and computing device for acquiring big data from multiple e-commerce platforms, which can maximize the use of the interface capabilities provided by the e-commerce platforms.

[0006] The user features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0007] According to one aspect of this application, a method for obtaining big data from multiple e-commerce platforms is provided, comprising: simultaneously launching multiple first threads of number N; using the multiple first threads to request first type data from different online stores on the e-commerce platform; storing the first type data in a distributed file system; and storing the first type data in the file system in a distributed database system, wherein the ability of N first threads to request data from the e-commerce platform is greater than the e-commerce platform's request limit for first type data, and the ability of N-2 first threads to request data from the e-commerce platform is less than the e-commerce platform's request limit for first type data.

[0008] According to another aspect of this application, a computing device is provided, comprising: a processor; a memory storing a computer program; and implementing the aforementioned method when the processor executes the computer program.

[0009] According to another aspect of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the aforementioned method.

[0010] According to some implementations, by controlling the number of threads, the API capabilities provided by the e-commerce platform can be maximized.

[0011] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0012] The above and other objectives, features and advantages of this application will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0013] Figure 1 This diagram illustrates one application scenario of the technical solution of this application.

[0014] Figure 2 A flowchart illustrating a method for obtaining big data from multiple e-commerce platforms according to an example embodiment of this application is shown.

[0015] Figure 3 The flowchart illustrates the operation of using multiple first threads to obtain online store IDs and request first-type data from different online stores from the e-commerce platform in a thread-safe manner, according to an embodiment.

[0016] Figure 4 The diagram illustrates a process flow according to another embodiment, which utilizes multiple first threads to obtain online store IDs and request first-type data from different online stores from an e-commerce platform in a thread-safe manner.

[0017] Figure 5 The flowchart illustrates a method for obtaining big data from multiple e-commerce platforms using a priority queue, according to an example embodiment.

[0018] Figure 6 The flowchart illustrates a method for requesting corresponding e-commerce data from an e-commerce platform based on an example embodiment, prioritizing the online store ID in the non-execution priority queue.

[0019] Figure 7 The flowchart illustrates a method for requesting corresponding e-commerce data from an e-commerce platform based on a store ID in a non-standard queue, according to an example embodiment.

[0020] Figure 8 The flowchart illustrates a method for obtaining big data from multiple e-commerce platforms using multithreading and priority queues, according to an example embodiment.

[0021] Figure 9 A schematic diagram of a system for acquiring big data from multiple e-commerce platforms according to another embodiment of this application is shown.

[0022] Figure 10 A block diagram of a computing device according to an example embodiment of this application is shown. Detailed Implementation

[0023] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0024] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0025] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0026] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0027] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0028] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments.

[0029] E-commerce operations generate a large amount of raw e-commerce data in storage, and some e-commerce platforms have opened data interfaces for e-commerce companies to access this data. However, e-commerce platforms typically impose request limits on the same user's interface access, such as a limit on the number of requests per second. Therefore, it is necessary to design an efficient data request method to acquire big data from multiple e-commerce platforms as quickly as possible while obtaining data from numerous online stores.

[0030] Therefore, this application proposes a method for obtaining big data from multiple e-commerce platforms. Through multi-threaded control and other methods, e-commerce big data can be obtained efficiently from e-commerce platforms.

[0031] The technical solution of this application will be described in detail below with reference to the embodiments.

[0032] Figure 1 This diagram illustrates one application scenario of the technical solution of this application.

[0033] See Figure 1 In e-commerce systems, data generated along the e-commerce value chain is stored in the e-commerce platform's database. To access this stored data, the data can be authorized for use by the data processing system according to the embodiments of this application through the e-commerce platform's API interface. Furthermore, the system can also connect with other major participants in the e-commerce business (including third-party payment providers, logistics providers, and warehousing service providers) via interfaces to obtain raw e-commerce data from multiple dimensions (sales, inventory, traffic, policy violations, logistics, settlement, etc.) in real time, and can use distributed technologies for storage and computation.

[0034] According to embodiments of this application, the data processing system, upon receiving authorization, retrieves the original e-commerce data of the corresponding e-commerce platform from the e-commerce platform and stores it in a storage system associated with the data processing system. According to some embodiments, this storage system may be a distributed storage system.

[0035] The data processing system according to the embodiments of this application processes the data, for example, through standardization, to obtain data that can be used subsequently; then, it can combine industry experience, business models, financial models and / or statistical models, etc., and obtain the risk rating results and risk profiles of e-commerce enterprises through machine learning and other methods, and provide them to financial institutions such as banks as a reliable basis for financing, etc.

[0036] Figure 2 A flowchart illustrating a method for obtaining big data from multiple e-commerce platforms according to an example embodiment of this application is shown.

[0037] See Figure 2 In S201, multiple first threads of number N are started simultaneously.

[0038] According to the example implementation, the capacity of N first threads to request data from the e-commerce platform exceeds the e-commerce platform's request limit for the first type of data, and the capacity of N-2 first threads to request data from the e-commerce platform is less than the e-commerce platform's request limit for the first type of data. Thus, by controlling the number of threads, the interface capabilities provided by the e-commerce platform can be maximized. Furthermore, by controlling the number of threads, computational and network resources are minimized, and excessive thread idleness is avoided.

[0039] In S203, multiple first threads are used to request first-class data from different online stores on the e-commerce platform.

[0040] After obtaining authorization from the online store, you can access the e-commerce platform's data interfaces. For example, after confirming the customer's authorization, the e-commerce platform will return the online store's token. The token is stored in the backend, and all interfaces need to include the token when making requests.

[0041] E-commerce platform interfaces may include order data interfaces, sales data interfaces, report data interfaces, document data interfaces, etc. Different e-commerce platforms may provide different interfaces. The first type of data can be data obtained from one of these interfaces, and this application is not limited in this regard. For ease of understanding, the term "first type of interface" is used generically here, intended to indicate that the technical solution of this application can obtain data from any similar interface, and of course, data from these interfaces can also be obtained simultaneously through different threads (e.g., the second, ..., nth threads).

[0042] According to some embodiments, the ability of N-1 first threads to request data from the e-commerce platform is less than the e-commerce platform's request limit for the first type of data.

[0043] According to some embodiments, the request responses received by multiple first threads can also be monitored, thereby dynamically adjusting the number of first threads and thus making more precise dynamic adjustments to the number of threads.

[0044] According to some embodiments, when the performance of a single server is insufficient, multiple first threads can run on multiple different servers.

[0045] In S205, the first type of data is stored in a distributed file system.

[0046] According to the example embodiment, the acquired first type of data is stored in a distributed file system to improve data acquisition efficiency and increase data security.

[0047] For example, a distributed file system may include a FastDFS system, but this application is not limited to this.

[0048] In S207, the first type of data in the file system is stored in a distributed database system.

[0049] According to the example embodiment, the first type of data in the file system can be further stored in a distributed database system. The file system stores a large number of small text files. To facilitate statistical processing and utilization, these large numbers of small text files are parsed, and the data is stored in the distributed database system.

[0050] For example, a distributed database system may include the TiDB system, but this application is not limited to this.

[0051] According to the embodiments described above, by employing multiple threads whose request capabilities appropriately exceed the limits of the e-commerce platform's interface, the interface capabilities provided by the e-commerce platform can be maximized, while minimizing the waste of computing and network resources.

[0052] Figure 3 The flowchart illustrates the operation of using multiple first threads to obtain online store IDs and request first-type data from different online stores from the e-commerce platform in a thread-safe manner, according to an embodiment.

[0053] See Figure 3 In S301, each first thread acquires the first distributed lock.

[0054] According to some embodiments, the open-source distributed lock Redisson can be used, but this application is not limited thereto.

[0055] In S303, the first thread that acquires the first distributed lock obtains the online store ID and releases the first distributed lock.

[0056] According to some implementations, the store ID (identifier) ​​of the online store for which data needs to be obtained is stored in the unexecuted queue.

[0057] The first thread retrieves the store ID from the unexecuted queue, and then adds the store ID to the executing queue. In this way, since the first thread has already acquired the first distributed lock, the problem of the store ID being retrieved repeatedly will not occur.

[0058] According to some implementations, the list of authorized online stores can be queried periodically to obtain a set of online store IDs. After deduplicating the obtained set of online store IDs, they are added to the unexecuted queue. For example, deduplication can be achieved by removing online store IDs from both the unexecuted queue and the executing queue from the obtained set of online store IDs.

[0059] In S305, the first thread that obtains the online store ID requests the corresponding first type of data from the e-commerce platform.

[0060] After obtaining the online store ID, the first thread can obtain a token based on the online store ID, as well as parameters such as the time range of the data to be obtained, and request the first type of data of the corresponding online store from the e-commerce platform, such as single data interface, sales data interface, report data interface or document data interface, etc.

[0061] According to the example implementation, after the first thread completes the request for the first type of data corresponding to the online store ID from the e-commerce platform, it can remove the online store ID from the execution queue, obtain another online store ID from the unexecuted queue, and then repeat the above operation.

[0062] Figure 4 The diagram illustrates a process flow according to another embodiment, which utilizes multiple first threads to obtain online store IDs and request first-type data from different online stores from an e-commerce platform in a thread-safe manner.

[0063] See Figure 4 In S401, multiple first threads retrieve the online store ID from the never-executed queue.

[0064] According to some embodiments, the store ID (identifier) ​​of the online store whose data needs to be retrieved is placed in an unexecuted queue. The unexecuted queue is a queue based on a single-threaded memory storage system, such as Redis. However, this application is not limited to this; other thread-safe queues can also be used to store the store ID to ensure that the store ID is not retrieved repeatedly.

[0065] According to some implementations, the list of authorized online stores can be queried periodically to obtain a set of online store IDs. After deduplicating the obtained set of online store IDs, they are added to the unexecuted queue. For example, deduplication can be achieved by removing online store IDs from both the unexecuted queue and the executing queue from the obtained set of online store IDs.

[0066] In S403, the first thread that obtains the online store ID puts the obtained online store ID into the execution queue.

[0067] In this way, the online store's data scraping process can be visualized and monitored in real time through this execution queue.

[0068] In S405, the first thread that obtains the online store ID requests the corresponding first type of data from the e-commerce platform.

[0069] After obtaining the online store ID, the first thread can obtain a token based on the online store ID, as well as parameters such as the time range of the data to be obtained, and request the first type of data of the corresponding online store from the e-commerce platform, such as single data interface, sales data interface, report data interface or document data interface, etc.

[0070] According to the example implementation, after the first thread completes the request for the first type of data corresponding to the online store ID from the e-commerce platform, it can remove the online store ID from the execution queue, obtain another online store ID from the unexecuted queue, and then repeat the above operation.

[0071] Figure 5 The flowchart illustrates a method for obtaining big data from multiple e-commerce platforms using a priority queue, according to an example embodiment.

[0072] See Figure 5 In step S501, a queue for storing online store IDs is set up. According to the example embodiment, the queue includes an unexecuted regular queue, an executing regular queue, an unexecuted priority queue, and an executing priority queue.

[0073] By setting up different queues, a foundation can be laid for prioritizing the acquisition of data from certain online stores. Furthermore, by configuring queues during execution, it's possible to visualize and monitor the online stores during the data scraping process in real time, and further ensure priority acquisition of data from newly added online stores, for example.

[0074] In S503, the online store IDs in the queue are updated periodically.

[0075] According to some embodiments, the list of authorized online stores can be queried periodically, for example, from a database, to obtain a set of online store IDs. Then, online store IDs in the queue are excluded from the set. The retained IDs of newly added online stores are then added to the unexecuted priority queue, and the retained IDs of non-new online stores are added to the unexecuted regular queue.

[0076] According to some embodiments, the acquired set of online store IDs may include a first set and a second set. The first set contains the online store IDs of newly added online stores, and the second set contains the online store IDs of non-new online stores. Thus, when removing online store IDs from the queue of the online store ID set, online store IDs that have not been executed in the priority queue and those that are being executed in the priority queue are removed from the first set, and online store IDs that have not been executed in the regular queue and those that are being executed in the regular queue are removed from the second set.

[0077] In S505, requests for corresponding e-commerce data from the e-commerce platform are prioritized based on the online store IDs in the unexecuted priority queue.

[0078] When a new online store is authorized to access a data interface, since the newly authorized online store has a large amount of historical data, a method is needed to prioritize obtaining the data of the newly added online store.

[0079] According to the example implementation, by processing the online store IDs in the non-execution priority queue, the request for the corresponding e-commerce data from the e-commerce platform is completed.

[0080] In S507, when there is no data in either the non-execution priority queue or the execution priority queue, the corresponding e-commerce data is requested from the e-commerce platform based on the online store ID in the non-execution regular queue.

[0081] According to the example implementation, data acquisition from online stores in the priority queue is completed before data acquisition from online stores in the regular queue begins. For example, data acquisition from existing online stores is only performed after data acquisition from newly added online stores on the e-commerce platform is complete. This not only ensures the optimal acquisition of specific data but also achieves efficient data acquisition in a simple manner, minimizing switching or waiting time during the data acquisition process.

[0082] According to some embodiments, the acquired data can be stored in a distributed file system to improve data acquisition efficiency and increase data security. For example, the distributed file system may include the FastDFS system, but this application is not limited thereto.

[0083] According to some embodiments, the acquired data from the file system can be further stored in a distributed database system. The file system stores a large number of small text files. To facilitate statistical processing and utilization, these numerous small text files are parsed, and the data is stored in the distributed database system. For example, the distributed database system may include the TiDB system, but this application is not limited thereto.

[0084] According to some embodiments, S505 and S507 can be executed in multiple threads. The non-executed regular queue and the non-executed priority queue are queues based on a single-threaded memory storage system, which may include a Redis system. However, this application is not limited to this; other thread-safe queues can also be used to store the online store ID, or other thread-safe methods can be used to obtain the online store ID to ensure that the problem of duplicate acquisition of the online store ID does not occur.

[0085] Figure 6 The flowchart illustrates a method for requesting corresponding e-commerce data from an e-commerce platform based on an example embodiment, prioritizing the online store ID in the non-execution priority queue.

[0086] See Figure 6 In S601, the priority queue for retrieving the online store ID was never executed.

[0087] According to the example implementation, each time a store ID is obtained to request corresponding data from the e-commerce platform, the store ID is first retrieved from the unexecuted priority queue. This allows for the preferential retrieval of data from certain stores, such as retrieving data from newly added stores first.

[0088] In S603, the obtained online store ID is placed in the execution priority queue.

[0089] According to an embodiment, by placing the acquired online store ID into a priority queue during execution, the online stores during the data crawling process can be visualized and monitored in real time. Furthermore, it can further ensure, for example, the priority acquisition of data from newly added online stores, as described below.

[0090] In S605, after requesting the corresponding e-commerce data from the e-commerce platform based on the obtained online store ID, the online store ID is removed from the execution priority queue.

[0091] After requesting the data corresponding to the online store ID from the e-commerce platform, the online store ID can be removed from the execution priority queue, and then the above operation can be repeated.

[0092] Figure 7 The flowchart illustrates a method for requesting corresponding e-commerce data from an e-commerce platform based on a store ID in a non-standard queue, according to an example embodiment.

[0093] See Figure 7 In S701, the priority queue for retrieving the online store ID was never executed.

[0094] According to the example implementation, each time a store ID is obtained to request corresponding data from the e-commerce platform, the store ID is first retrieved from the unexecuted priority queue. This ensures that when a new store ID is added to the unexecuted priority queue, it can be processed first.

[0095] If no store ID is obtained from the priority queue that has never been executed, but the priority queue in the execution phase is not empty, the process returns to S701 after a delay. This ensures that data from stores in the priority queue is retrieved from the e-commerce platform before data from stores in the regular queue is retrieved.

[0096] If the online store ID is not obtained from the never-executed priority queue and the executing priority queue is empty, proceed to S703.

[0097] In S703, the regular queue for retrieving the online store ID was never executed.

[0098] Since there are no more online stores that need to be prioritized at this point, the processing of online stores in the regular queue, such as existing online stores, begins.

[0099] In S705, the obtained online store ID is placed into the regular queue during execution.

[0100] As before, by placing the obtained online store IDs into the regular queue during execution, the online stores in the data crawling process can be visualized and monitored in real time.

[0101] In S707, after requesting the corresponding e-commerce data from the e-commerce platform based on the obtained online store ID, the online store ID is removed from the regular queue during execution.

[0102] After requesting the data corresponding to the online store ID from the e-commerce platform, the online store ID can be removed from the regular queue during execution, and then the above operation can be repeated.

[0103] Figure 8 The flowchart illustrates a method for obtaining big data from multiple e-commerce platforms using multithreading and priority queues, according to an example embodiment.

[0104] See Figure 8 In step S801, a queue for storing online store IDs is set up. According to the example embodiment, the queue includes a non-executed regular queue, an executing regular queue, a non-executed priority queue, and an executing priority queue.

[0105] By setting up different queues, a foundation can be laid for prioritizing the acquisition of data from certain online stores. Furthermore, by configuring queues during execution, it's possible to visualize and monitor the online stores during the data scraping process in real time, and further ensure priority acquisition of data from newly added online stores, for example.

[0106] In S803, multiple first threads of number N are started simultaneously.

[0107] According to the example implementation, data acquisition efficiency is further improved by using a multi-threaded approach.

[0108] According to some embodiments, the ability of N first threads to request data from the e-commerce platform exceeds the e-commerce platform's request limit for the first type of data. Furthermore, according to some embodiments, it can be determined that the ability of N-2 or N-1 first threads to request data from the e-commerce platform is less than the e-commerce platform's request limit for the first type of data.

[0109] In this way, by controlling the number of threads, the API capabilities provided by the e-commerce platform can be maximized. Furthermore, controlling the number of threads minimizes the waste of computing and network resources.

[0110] In S805, the online store IDs in the queue are updated periodically.

[0111] According to some embodiments, the list of authorized online stores can be queried periodically, for example, from a database, to obtain a set of online store IDs. Then, online store IDs in the queue are excluded from the set. The retained IDs of newly added online stores are then added to the unexecuted priority queue, and the retained IDs of non-new online stores are added to the unexecuted regular queue.

[0112] According to some embodiments, the acquired set of online store IDs may include a first set and a second set. The first set contains the online store IDs of newly added online stores, and the second set contains the online store IDs of non-new online stores. Thus, when removing online store IDs from the queue of the online store ID set, online store IDs that have not been executed in the priority queue and those that are being executed in the priority queue are removed from the first set, and online store IDs that have not been executed in the regular queue and those that are being executed in the regular queue are removed from the second set.

[0113] In S807, multiple first threads are used to request the first type of data from different online stores from the e-commerce platform based on the online store IDs in the unexecuted priority queue.

[0114] When a new online store is authorized to access a data interface, since the newly authorized online store has a large amount of historical data, a method is needed to prioritize obtaining the data of the newly added online store.

[0115] According to the example embodiment, by processing the online store IDs in the non-execution priority queue, the request for the corresponding first type of data from the e-commerce platform is completed. By using a multi-threaded approach to acquire the first type of data, data acquisition efficiency can be improved.

[0116] Furthermore, according to some embodiments, by controlling the number of threads, the interface capabilities provided by the e-commerce platform can be maximized, while minimizing the waste of computing and network resources.

[0117] E-commerce platform interfaces may include order data interfaces, sales data interfaces, report data interfaces, document data interfaces, etc. Different e-commerce platforms may provide different interfaces. The first type of data can be data obtained from one of these interfaces, and this application is not limited in this regard. For ease of understanding, the term "first type of interface" is used generically here, intended to indicate that the technical solution of this application can obtain data from any similar interface, and of course, data from these interfaces can also be obtained simultaneously through different threads (e.g., the second, ..., nth threads).

[0118] In S809, when there is no data in either the non-execution priority queue or the execution priority queue, multiple first threads are used to request the first type of data from different online stores from the e-commerce platform based on the online store ID in the non-execution regular queue.

[0119] According to the example implementation, data acquisition from online stores in the priority queue is completed before data acquisition from online stores in the regular queue begins. For example, data acquisition from existing online stores is only performed after data acquisition from newly added online stores on the e-commerce platform is complete. This not only ensures the optimal acquisition of specific data but also achieves efficient data acquisition in a simple manner, minimizing switching or waiting time during the data acquisition process.

[0120] As mentioned before, using a multi-threaded approach to acquire the first type of data can improve data acquisition efficiency.

[0121] according to Figure 8 The method of the illustrated embodiment may be employed in other respects in a manner similar to that of the other embodiments described above, and will not be repeated here.

[0122] Figure 9 A schematic diagram of a system for acquiring big data from multiple e-commerce platforms according to another embodiment of this application is shown.

[0123] See Figure 9 According to the example embodiment, multiple threads for retrieving online store data are executed on at least two servers, A and B.

[0124] like Figure 9 As shown, the data to be acquired includes four categories: order data, sales data, report data, and document data. Three threads are created on each server (A or B) for each category to retrieve data from the online store on the e-commerce platform. As mentioned earlier, the number of threads can be adjusted based on server and network resource availability, as well as the e-commerce platform's restrictions on API requests.

[0125] See Figure 9 For each thread, the first step is to obtain the online store ID, and then the corresponding online store data is obtained from the e-commerce platform.

[0126] According to the embodiment, each thread first retrieves a store ID from the unexecuted priority queue and places the retrieved store ID into the executing priority queue. Then, based on the retrieved store ID, it requests the corresponding e-commerce data from the e-commerce platform and removes the store ID from the executing priority queue. Next, it retrieves another store ID from the unexecuted priority queue and continues the process of retrieving the corresponding data.

[0127] If no store ID is retrieved from the priority queue that has not yet been executed, but the priority queue is not empty during execution, the process will return after a delay and continue retrieving the store ID. This ensures that data from stores in the priority queue from the e-commerce platform is retrieved first, before data from stores in the regular queue from the e-commerce platform is retrieved.

[0128] If no store ID is found in the unexecuted priority queue and the executing priority queue is empty, it indicates that there are no more stores requiring priority processing. Therefore, processing begins on the stores in the regular queue, retrieving store IDs from the unexecuted regular queue. The retrieved store ID is placed into the executing regular queue, and the corresponding e-commerce data is requested from the e-commerce platform based on the retrieved store ID. Then, the store ID is removed from the executing regular queue, and another store ID is retrieved from the unexecuted priority queue, continuing the aforementioned data retrieval process.

[0129] Figure 10A block diagram of a computing device according to an example embodiment of this application is shown.

[0130] like Figure 10 As shown, the computing device 30 includes a processor 12 and a memory 14. The computing device 30 may also include a bus 22, a network interface 16, and an I / O interface 18. The processor 12, memory 14, network interface 16, and I / O interface 18 can communicate with each other via the bus 22.

[0131] The processor 12 may include one or more general-purpose CPUs (Central Processing Units), microprocessors, or application-specific integrated circuits, etc., for executing relevant program instructions.

[0132] Memory 14 may include a machine system readable medium in the form of volatile memory, such as random access memory (RAM), read-only memory (ROM), and / or cache memory. Memory 14 is used to store one or more programs containing instructions, as well as data. Processor 12 may read the instructions stored in memory 14 to perform the methods described above according to embodiments of this application.

[0133] The computing device 30 can also communicate with one or more networks via the network interface 16. The network interface 16 can be a wired network interface, a wireless network interface, or a virtual network interface.

[0134] The computing device 30 can also communicate with one or more external devices (such as audio input devices, audio output devices, cameras, keyboards, mice, monitors, various sensors, etc.) via the input / output (I / O) interface 18.

[0135] Bus 22 may include an address bus, a data bus, a control bus, etc. Bus 22 provides a path for exchanging information between components.

[0136] It should be noted that, in specific implementations, the computing device 30 may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the device described above may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0137] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), network storage devices, cloud storage devices, or any type of medium or device suitable for storing instructions and / or data.

[0138] This application also provides a computer program product including a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments.

[0139] Those skilled in the art will clearly understand that the technical solutions of this application can be implemented using software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware capable of independently performing or cooperating with other components to perform a specific function, where the hardware may be, for example, a field-programmable gate array (FPGA), integrated circuit, etc.

[0140] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0141] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0142] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0143] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0144] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0145] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0146] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0147] The embodiments of this application have been described and explained in detail above. It should be clearly understood that this application describes how specific examples are formed and used, but this application is not limited to any details of these examples. Rather, based on the teachings of the disclosure of this application, these principles can be applied to many other embodiments.

[0148] Through the description of the example embodiments, those skilled in the art will readily understand that the risk rating prediction method according to the embodiments of this application has at least one or more of the following advantages.

[0149] According to some implementations, by controlling the number of threads, the API capabilities provided by the e-commerce platform can be maximized.

[0150] According to some implementations, by controlling the number of threads, computing and network resources are minimized.

[0151] According to some implementations, by using a thread-safe queue to store the online store ID, it can be guaranteed that the online store ID will not be obtained repeatedly.

[0152] According to some embodiments, setting up an execution queue can enable visualization and real-time monitoring of the online store's data scraping process.

[0153] According to some embodiments, by setting a priority queue, it is possible to prioritize the acquisition of data, such as newly added online stores, in a simple way.

[0154] According to some embodiments, by setting an execution priority queue, priority access to data such as newly added online stores can be further ensured.

[0155] According to some embodiments, each time a store ID is obtained to request corresponding data from the e-commerce platform, the store ID is first obtained from the unexecuted priority queue. This ensures that if a new store ID is added to the unexecuted priority queue, it can be processed first.

[0156] According to some embodiments, when the priority queue is not empty during execution, the online store ID is not obtained from the regular queue. This ensures that the data of online stores in the priority queue is obtained from the e-commerce platform first, and then the data of online stores in the regular queue is obtained from the e-commerce platform.

[0157] According to some implementations, by setting different queues and combining them with multi-threaded data acquisition, it is possible to prioritize the acquisition of data from certain online stores while simultaneously improving data acquisition efficiency.

[0158] The foregoing can be better understood in accordance with the following terms:

[0159] 1. A method for acquiring big data from multiple e-commerce platforms, characterized by comprising:

[0160] Simultaneously start multiple first threads, with a quantity of N;

[0161] The multiple first threads are used to request the first type of data from different online stores on the e-commerce platform;

[0162] The first type of data is stored in a distributed file system;

[0163] The first type of data in the file system is stored in a distributed database system.

[0164] Among them, the ability of N first threads to request data from the e-commerce platform is greater than the e-commerce platform's request limit for the first type of data, and the ability of N-2 first threads to request data from the e-commerce platform is less than the e-commerce platform's request limit for the first type of data.

[0165] 2. The method as described in Clause 1, characterized in that the ability of N-1 first threads to request data from the e-commerce platform is less than the e-commerce platform's request limit for the first type of data.

[0166] 3. The method as described in Clause 1, characterized in that it further comprises:

[0167] Monitor the request responses received by the multiple first threads to dynamically adjust the number of first threads.

[0168] 4. The method as described in Clause 1, wherein the plurality of first threads originate from a plurality of servers.

[0169] 5. The method as described in Clause 1, characterized in that, utilizing the plurality of first threads to request first-type data from different online stores on the e-commerce platform, includes:

[0170] Each thread acquires the first distributed lock.

[0171] The first thread that acquires the first distributed lock obtains the online store ID and releases the first distributed lock;

[0172] The first thread that obtains the online store ID requests the corresponding first type of data from the e-commerce platform.

[0173] 6. The method described in Clause 5, characterized in that, the first thread that obtains the first distributed lock acquires the online store ID, including:

[0174] The first thread never retrieves the online store ID from the queue;

[0175] The first thread puts the online store ID into the execution queue.

[0176] 7. The method as described in Clause 1, characterized in that, utilizing the plurality of first threads to request first-type data from different online stores on the e-commerce platform, includes:

[0177] The multiple first threads each obtain the online store ID from the unexecuted queue, which is a queue based on a single-threaded memory storage system.

[0178] The first thread that obtains the online store ID will put the obtained online store ID into the execution queue;

[0179] The first thread that obtains the online store ID requests the corresponding first type of data from the e-commerce platform.

[0180] 8. The method as described in Clause 7, wherein the single-threaded memory storage system includes the Redis system.

[0181] 9. The method as described in clause 6 or 7, characterized in that it further comprises:

[0182] Regularly query the list of authorized online stores and obtain a set of online store IDs;

[0183] After deduplicating the obtained set of online store IDs, they are added to the unexecuted queue.

[0184] 10. The method as described in clause 6 or 7, characterized in that it further comprises:

[0185] After the first thread that obtained the online store ID completes the request for the corresponding first type of data from the e-commerce platform, it removes the online store ID from the execution queue and obtains another online store ID from the unexecuted queue.

[0186] 11. The method as described in Clause 1, wherein the distributed file system includes the FastDFS system.

[0187] 12. The method as described in Clause 1, wherein the distributed database system includes the TiDB system.

[0188] 13. The method as described in Clause 1, characterized in that it further comprises:

[0189] Simultaneously start multiple second threads, with a quantity of M;

[0190] The multiple second threads are used to request the second type of data from different online stores on the e-commerce platform.

[0191] Among them, the ability of M second threads to request data from the e-commerce platform is greater than the e-commerce platform's request limit for the second type of data, and the ability of M-2 first threads to request data from the e-commerce platform is less than the e-commerce platform's request limit for the second type of data.

[0192] 14. A computing device, characterized in that it comprises:

[0193] processor;

[0194] A memory, on which computer programs are stored;

[0195] When the processor executes the computer program, it implements the method as described in any one of clauses 1-13.

[0196] Exemplary embodiments of this application have been specifically shown and described above. It should be understood that this application is not limited to the detailed structures, arrangements, or implementation methods described herein; rather, this application is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended provisions.

Claims

1. A method for acquiring big data from multiple e-commerce platforms, characterized in that, include: Simultaneously start multiple first threads, with a quantity of N; The multiple first threads are used to request the first type of data from different online stores on the e-commerce platform; The first type of data is stored in a distributed file system; The first type of data in the file system is stored in a distributed database system. Among them, the ability of N first threads to request data from the e-commerce platform is greater than the e-commerce platform's request limit for the first type of data, and the ability of N-2 first threads to request data from the e-commerce platform is less than the e-commerce platform's request limit for the first type of data.

2. The method as described in claim 1, characterized in that, The ability of N-1 first threads to request data from the e-commerce platform is less than the e-commerce platform's request limit for the first type of data.

3. The method as described in claim 1, characterized in that, Also includes: Monitor the request responses received by the multiple first threads to dynamically adjust the number of first threads.

4. The method as described in claim 1, characterized in that, The multiple first threads originate from multiple servers.

5. The method as described in claim 1, characterized in that, The multiple first threads are used to request first-type data from different online stores on the e-commerce platform, including: Each thread acquires the first distributed lock. The first thread that acquires the first distributed lock obtains the online store ID and releases the first distributed lock; The first thread that obtains the online store ID requests the corresponding first type of data from the e-commerce platform.

6. The method as described in claim 5, characterized in that, The first thread that acquires the first distributed lock obtains the online store ID, including: The first thread never retrieves the online store ID from the queue; The first thread puts the online store ID into the execution queue.

7. The method as described in claim 1, characterized in that, The multiple first threads are used to request first-type data from different online stores on the e-commerce platform, including: The multiple first threads each obtain the online store ID from the unexecuted queue, which is a queue based on a single-threaded memory storage system. The first thread that obtains the online store ID will put the obtained online store ID into the execution queue; The first thread that obtains the online store ID requests the corresponding first type of data from the e-commerce platform.

8. The method as described in claim 7, characterized in that, The single-threaded memory storage system includes the Redis system.

9. The method as described in claim 6 or 7, characterized in that, Also includes: Regularly query the list of authorized online stores and obtain a set of online store IDs; After deduplicating the obtained set of online store IDs, they are added to the unexecuted queue.

10. The method as described in claim 6 or 7, characterized in that, Also includes: After the first thread that obtained the online store ID completes the request for the corresponding first type of data from the e-commerce platform, it removes the online store ID from the execution queue and obtains another online store ID from the unexecuted queue.

11. The method as described in claim 1, characterized in that, The distributed file system includes the FastDFS system.

12. The method as described in claim 1, characterized in that, The distributed database system includes the TiDB system.

13. The method as described in claim 1, characterized in that, Also includes: Simultaneously start multiple second threads, with a quantity of M; The multiple second threads are used to request the second type of data from different online stores on the e-commerce platform. Among them, the ability of M second threads to request data from the e-commerce platform is greater than the e-commerce platform's request limit for the second type of data, and the ability of M-2 first threads to request data from the e-commerce platform is less than the e-commerce platform's request limit for the second type of data.

14. A computing device, characterized in that, include: processor; A memory, on which computer programs are stored; When the processor executes the computer program, it implements the method as described in any one of claims 1-13.

Images