Modular social media data collection method and device

Abstract

In an embodiment of the present invention, a modular social media data collection method, device and electronic device are provided, which belongs to the field of data processing technology. The method includes: setting an engine layer, a collection layer and a data layer in a data collection system; receiving request information input by an external system through an Engines module for data collection scheduling, and verifying the legitimacy of the request information according to a preset security protocol; when the request information meets the legitimacy requirements, parsing the request information using the Engines module to obtain a parsing result; executing corresponding data collection tasks through an APIs module or a crawling module, and passing the collected data to a Pipelines module for data parsing and formatting to obtain standard data; passing the standard data to the Engines module, and using the Engines module to send the standard data to an external system. The present invention improves the ease of use and flexibility of social media data collection.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a modular social media data collection method, device and electronic equipment. Background Art

[0002] Social media data collection refers to the process of collecting public or user-authorized data from various social media platforms (such as Weibo). This data can include text content, pictures, videos, user comments, likes, forwarding, and other interactive information. The purposes of data collection vary and may be used in areas such as data analysis, user monitoring, and advertising optimization.

[0003] The existing technology has the following defects in the process of social media data collection:

[0004] 1. The system structure is chaotic and maintenance is difficult

[0005] The system lacks modular design. As the number of functions increases, the code base becomes increasingly complex and difficult to understand and maintain. Different parts are closely connected, and a small change may trigger a chain reaction, affecting the stability of the entire system.

[0006] 2. Poor scalability and inability to quickly access new platforms

[0007] Support for specific social media platforms may be hard-coded, which means that every time a new platform needs to be supported, a lot of code must be written from scratch. The APIs of different platforms vary greatly, and without an abstraction layer to handle these differences, integrating a new platform each time will be very time-consuming and error-prone.

[0008] 3. No standards, high code duplication

[0009] Different developers may use different programming styles and habits, resulting in inconsistent code styles and increasing the difficulty of reading and understanding the code. Due to the lack of good design patterns and reuse mechanisms, many similar functions will be implemented multiple times, causing the code base to be bloated.

[0010] 4. Unable to uniformly manage various collections

[0011] Each collection task may be run in isolation, without a centralized control panel or dispatch center for coordination. Collection parameters and settings are scattered in different places, making it easy to miss or make mistakes when changing the configuration.

[0012] The above problems become technical problems that need to be solved. Summary of the invention

[0013] In view of this, embodiments of the present invention provide a modular social media data collection method, device, and electronic device, which at least partially solve the problems existing in the prior art.

[0014] In a first aspect, an embodiment of the present invention provides a modular social media data collection method, comprising:

[0015] An engine layer, a collection layer and a data layer are set in the data collection system. The engine layer includes an engine module and business items. The business items include multiple different types of data sets. The collection layer includes an APIs module responsible for encapsulating API calls and a crawling module responsible for data collection. The data layer includes a Pipelines module for data format conversion.

[0016] Receive request information input from the external system through the Engines module for data acquisition scheduling, and verify the legitimacy of the request information according to the preset security protocol;

[0017] When the request information meets the legality requirements, the Engines module is used to parse the request information to obtain a parsing result. If the parsing result is an object ID, the Engines module calls the APIs module to obtain API details. If the parsing result is a link, the Engines module calls the crawling module to collect the link.

[0018] Execute the corresponding data collection tasks through the APIs module or the crawling module, and pass the collected data to the Pipelines module for data parsing and formatting to obtain standard data;

[0019] The standard data is passed to the Engines module, and the Engines module is used to send the standard data to an external system.

[0020] According to a specific implementation of an embodiment of the present invention, the setting of the engine layer, the collection layer and the data layer in the data collection system includes:

[0021] The APIs module encapsulates the collection and calling logic of the open APIs of multiple social media platforms, processes the paging mechanism in the API response, and sets the rate limiting strategy in the APIs module. Each API request consumes a token, and the token is generated at a fixed rate R.

[0022] A multi-threaded asynchronous data crawling mechanism is set in the Crawlers module, and a finite state machine is used to manage the state of data crawling. At the same time, the risk of IP blocking is avoided through proxy pool and user proxy rotation;

[0023] In the Engines module, a task queue is defined and a task scheduling algorithm is set to achieve dynamic adjustment of priority and resource allocation.

[0024] According to a specific implementation of an embodiment of the present invention, the receiving of request information input by an external system through the Engines module for data acquisition scheduling and verifying the legitimacy of the request information according to a preset security protocol includes:

[0025] Set one or more listening ports in the Engines module to receive HTTP / HTTPS requests from external systems;

[0026] After receiving the request, check whether the Content-Type header format carried in the POST method in the request is correct;

[0027] If the format is incorrect, a 400 Bad Request response is returned and the request is rejected.

[0028] According to a specific implementation of an embodiment of the present invention, the receiving of request information input by an external system through the Engines module for data acquisition scheduling and verifying the legitimacy of the request information according to a preset security protocol also includes:

[0029] Extract the API key and OAuth access token from the request header according to the preset security protocol;

[0030] For API key-based authentication, check that the provided key exists in the whitelist database and confirms that it has not expired or been revoked;

[0031] For OAuth 0 authentication, use the OAuth server to verify the validity of the access token, including checking the token's scope, validity period, and associated user;

[0032] If authentication fails, a 401 Unauthorized response is returned and the attempt is logged;

[0033] After the authorization verification is passed, further check whether the requested operation is within the authorization scope and whether the requested resource path matches the permissions assigned to the user;

[0034] If the request exceeds the authorization scope, a 403 Forbidden response is returned and the abnormal behavior is recorded;

[0035] Limit the number of requests within a preset time based on IP address and user ID, and limit the transmission speed of request data through a speed limit algorithm;

[0036] When the rate limit is exceeded, a 429 Too Many Requests response is returned with a prompt for the retry time.

[0037] Detailed logging of all successful authentication and authorization processes, including timestamp, source IP, user ID, and request type;

[0038] For interactions that need to maintain state, a session management mechanism is introduced, and a session token is issued after each successful authentication.

[0039] According to a specific implementation of an embodiment of the present invention, when the request information meets the legality requirement, parsing the request information using the Engines module includes:

[0040] Load the configuration file of the Engines module, initialize the data structure and set the default parameters;

[0041] Extract all relevant header information from HTTP / HTTPS requests;

[0042] Determine whether there is JSON data in the parsed request body based on the Content-Type header. If so, use the standard library to convert the JSON string into an object or dictionary structure in a programming language.

[0043] According to a specific implementation of an embodiment of the present invention, the corresponding data collection task is performed by the APIs module or the crawling module, and the collected data is passed to the Pipelines module for data parsing and formatting to obtain standard data, including:

[0044] According to the parsed request information, specific collection parameters are set, wherein the collection parameters include: target API endpoint, identity authentication information, request parameters, and data behavior rules;

[0045] Constructing an HTTP request, wherein the HTTP request includes header information and query parameters, and sending the HTTP request to a target API endpoint;

[0046] Start a crawl task using a preconfigured data crawler instance and store the crawled content in a temporary cache.

[0047] According to a specific implementation of an embodiment of the present invention, the APIs module or the crawling module performs the corresponding data collection task, and passes the collected data to the Pipelines module for data parsing and formatting to obtain standard data, and further includes:

[0048] After the collection is completed, the collected raw data is cleaned to remove redundant information, blank fields and obviously erroneous data entries. For data captured from web pages, HTML tags are removed to keep only plain text;

[0049] Encapsulate the cleaned data into a format suitable for processing by the Pipelines module, and pass the cleaned data to the Pipelines module through the queue system;

[0050] After receiving the data, the Pipelines module parses the data according to predefined rules and pattern matching algorithms to extract useful information. For structured data, it directly maps it to the database table structure or preset data model. For unstructured data, it applies natural language processing to extract target information.

[0051] The parsed data is organized according to a predetermined standard format to form standard data.

[0052] According to a specific implementation of an embodiment of the present invention, the passing of the standard data to the Engines module, and using the Engines module to send the standard data to an external system includes:

[0053] Construct a response object containing standard data according to the expected format of the external system, and add metadata to the response object, including: timestamp, request ID and status code;

[0054] Securely pass the constructed response object to the Engines module through a predefined internal communication protocol;

[0055] After receiving the data, the Engines module parses the response object and extracts the standard data to be sent and its related metadata;

[0056] Set HTTP request header, content type, and authentication information according to the API documentation of the external system or the pre-agreed interface specifications;

[0057] Construct a complete HTTP request, including URL, GET / POST, request body and header information, and send the HTTP request to the specified endpoint of the external system through the network.

[0058] In a second aspect, an embodiment of the present invention provides a modular social media data collection device, comprising:

[0059] Setting module, setting engine layer, collection layer and data layer in the data collection system, the engine layer includes engine module and business sub-items, the business sub-items include multiple different types of data sets, the collection layer includes APIs module responsible for encapsulating API calls and crawling module responsible for data collection, the data layer includes Pipelines module for data format conversion;

[0060] The receiving module receives the request information input by the external system through the Engines module for data acquisition scheduling, and verifies the legitimacy of the request information according to the preset security protocol;

[0061] A parsing module, when the request information meets the legality requirements, uses the Engines module to parse the request information to obtain a parsing result. If the parsing result is an object ID, the Engines module calls the APIs module to obtain API details. If the parsing result is a link, the Engines module calls the crawling module to collect the link.

[0062] The execution module performs the corresponding data collection tasks through the APIs module or the crawling module, and passes the collected data to the Pipelines module for data parsing and formatting to obtain standard data;

[0063] The sending module transfers the standard data to the Engines module, and uses the Engines module to send the standard data to an external system.

[0064] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0065] at least one processor; and,

[0066] a memory communicatively connected to the at least one processor; wherein,

[0067] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the modular social media data collection method in any of the aforementioned first aspects or any of the implementations of the first aspect.

[0068] In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the modular social media data collection method in the aforementioned first aspect or any implementation of the first aspect.

[0069] In a fifth aspect, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the modular social media data collection method in the aforementioned first aspect or any implementation of the first aspect.

[0070] The modular social media data collection scheme in the embodiment of the present invention includes: setting an engine layer, a collection layer and a data layer in a data collection system, the engine layer includes an engine module and a business item, the business item includes multiple different types of data sets, the collection layer includes an APIs module responsible for encapsulating API calls and a crawling module responsible for data collection, and the data layer includes a Pipelines module for data format conversion; receiving request information input by an external system through an Engines module for data collection scheduling, and verifying the legitimacy of the request information according to a preset security protocol; when the request information meets the legitimacy requirements, the Engines module is used to parse the request information to obtain a parsing result, if the parsing result is an object ID, the Engines module calls the APIs module to obtain API details, if the parsing result is a link, the Engines module calls the crawling module to collect links; executing corresponding data collection tasks through the APIs module or the crawling module, and passing the collected data to the Pipelines module for data parsing and formatting to obtain standard data; passing the standard data to the Engines module, and using the Engines module to send the standard data to the external system. This scheme has the following beneficial effects:

[0071] 1. Improve system maintainability

[0072] By dividing the system into the engine layer, the collection layer, and the data layer, each layer focuses on specific functions, making the code structure clearer and reducing coupling. This reduces the difficulty of maintenance and improves the readability and maintainability of the code.

[0073] Due to the loose coupling between modules, changes or optimizations in one module will not have a significant impact on other modules, thus simplifying the process of troubleshooting and functional upgrades.

[0074] 2. Enhance the scalability and flexibility of the system

[0075] The introduction of APIs module and crawling module can flexibly support different types of social media platforms. For new platforms, only corresponding adapters need to be developed or configured without large-scale modification of existing codes.

[0076] The plug-in architecture makes it easy to add new data sources or adjust existing collection strategies, ensuring that the system can quickly respond to market demands and technological changes.

[0077] 3. Reduce code duplication and improve code quality

[0078] By defining standardized data processing processes (such as the Pipelines module), we avoid repeatedly implementing similar functions in different places and reduce redundant code.

[0079] The internal functions of each module are closely related, but they maintain low coupling with external modules, which ensures the consistency and stability of the code and promotes code reuse.

[0080] 4. Achieve centralized management and efficient scheduling

[0081] The Engines module serves as the core coordinator, responsible for receiving requests, assigning tasks and ultimately returning results, thus achieving centralized control over all collection activities.

[0082] 5. Optimize resource utilization and improve efficiency

[0083] It supports concurrent processing of multiple tasks, optimizes resource allocation through queue management and load balancing algorithms, and improves the overall efficiency of the system.

[0084] Automatically adjust collection parameters such as crawling depth and frequency according to actual workload to maximize resource utilization and data collection accuracy.

[0085] The data collection scheme of the present invention complies with relevant regulations, and is specifically described as follows:

[0086] 1. Not violating the law

[0087] The data collection scheme of the present invention strictly complies with the current laws and regulations of the country from design concept to actual operation. In all aspects of data collection, including but not limited to determining the collection target, selecting the collection method, implementing the collection behavior, and storing and transmitting the data, legality and compliance are the primary principles.

[0088] For example, before collecting data, for data involving personal information, the solution strictly follows the provisions of relevant laws and regulations such as the Cybersecurity Law of the People's Republic of China, the Data Security Law of the People's Republic of China, and the Personal Information Protection Law of the People's Republic of China to obtain the explicit authorization and consent of the data subject. Through clear and easy-to-understand notification methods, the data subject is fully informed of key information such as the purpose, method, scope, storage period, and usage of data collection, ensuring that the data subject makes the authorization decision voluntarily with full knowledge.

[0089] During the data collection process, we strictly abide by the legal restrictions on data collection methods and avoid illegal means such as illegal intrusion, theft, fraud, etc. For data collected from public channels, we also strictly follow the relevant laws and regulations on the use of public data to ensure that the data source is legal and the collection behavior is compliant.

[0090] 2. Do not violate social ethics

[0091] The data collection scheme of the present invention adheres to the principles of social morality such as fairness, equity, integrity, respect and protection of the rights and interests of others. Data collection activities focus on protecting personal privacy and personal dignity, and do not conduct any data collection activities that may infringe on the privacy of others, cause adverse social impacts, or damage public order and morality.

[0092] The collection plan is designed to fully consider the differences in culture, beliefs and values ​​of different groups, so as to avoid discrimination, prejudice or hurting the feelings of specific groups due to data collection activities. At the same time, in the process of using and sharing data, we follow ethical standards and do not use the collected data in any way to maliciously attack, defame others or undermine social harmony and stability.

[0093] 3. No harm to public interest

[0094] The data collection scheme of the present invention takes promoting social progress, advancing scientific and technological development and protecting public welfare as its starting point and end point, and does not contain any factors that hinder the public interest.

[0095] The purpose of data collection is to achieve specific scientific research, technological innovation or social service goals, and to create value for society, such as improving the quality of public services, enhancing the level of social governance, promoting the development of medical and health care, and promoting environmental protection. The collected data will be used for legitimate and beneficial purposes and will not be used for any undesirable purpose such as endangering national security, undermining social stability, damaging public safety or affecting public health.

[0096] During the data collection process, strict security measures are taken to ensure the security and integrity of the data, prevent data leakage, abuse or malicious use, and avoid potential threats to the public interest. For example, the collected sensitive data is encrypted for storage and transmission, and a complete data access control mechanism is established to limit data access rights, so that only authorized personnel can use the data within the prescribed scope.

[0097] 4. No illegal acquisition or use of genetic resources

[0098] The data collection scheme of the present invention does not involve the acquisition and utilization of genetic resources. Therefore, there is no violation of the provisions of laws and administrative regulations in acquiring or utilizing genetic resources and relying on such genetic resources to complete inventions and creations.

[0099] In summary, the data collection scheme of the present invention strictly complies with relevant regulations in all aspects, does not violate any law, social morality or public interest, and does not involve any illegal acquisition or use of genetic resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0101] Figure 1 A schematic diagram of a modular social media data collection method flow chart provided by an embodiment of the present invention;

[0102] Figure 2 A schematic flow chart of another modular social media data collection method provided by an embodiment of the present invention;

[0103] Figure 3 A schematic diagram of the structure of a modular social media data collection device provided by an embodiment of the present invention;

[0104] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0105] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0106] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0107] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0108] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0109] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0110] The embodiment of the present disclosure provides a modular social media data collection method. The modular social media data collection method provided in this embodiment can be executed by a computing device, which can be implemented as software, or as a combination of software and hardware, and the computing device can be integrated in a server, terminal device, etc.

[0111] See also Figure 1 and Figure 2 The present disclosure provides a modular social media data collection method, including:

[0112] S101, setting an engine layer, a collection layer and a data layer in a data collection system, wherein the engine layer includes an engine module and business sub-items, wherein the business sub-items include multiple different types of data sets, the collection layer includes an APIs module responsible for encapsulating API calls and a crawling module responsible for data collection, and the data layer includes a Pipelines module for data format conversion.

[0113] The architecture design of setting up the engine layer, collection layer and data layer in the data collection system can effectively organize and optimize the collection and processing process of social media data. The following is a detailed description of the three-layer architecture and its components:

[0114] 1. Engine layer

[0115] As the central nervous system of the system, it is responsible for receiving external requests, parsing request information, scheduling tasks, and finally returning the results to the external system. It verifies the legitimacy of the received requests to ensure that only authorized requests can enter the subsequent processing flow.

[0116] The engine module receives requests from external systems, parses the request information, and determines the type of operation to be performed (such as obtaining user details, crawling web page content, etc.). Based on the parsed results, the corresponding module (APIs or Crawlers) is called to perform specific data collection tasks. The collected data is collected and organized, and the appropriate response format is constructed, and the results are then sent back to the external system.

[0117] The business segment includes multiple different types of data sets, each corresponding to a specific type of social media platform or data source.

[0118] 2. Collection layer

[0119] There are different ways to obtain raw data from social media platforms, which can be structured API data or unstructured web page content.

[0120] The APIs module uses the official API interface to programmatically obtain structured data such as user information, post details, etc., and handles OAuth tokens or other forms of authentication to ensure legal access to API resources.

[0121] The crawling module uses web crawler technology to extract unstructured or semi-structured data from web pages, such as text, image links, comments, etc. It follows preset behavioral rules, such as crawling depth and frequency limits, to avoid excessive load on the target website.

[0122] 3. The data layer cleans, parses and formats the collected raw data to generate standard data sets for further analysis or display.

[0123] The Pipelines module is used to remove redundant information, blank fields, and obviously erroneous data entries to ensure data integrity and accuracy. Pattern matching algorithms or natural language processing techniques are applied to extract valuable information from unstructured data. The parsed data is converted into a unified standard format, such as JSON, CSV, etc., for easy storage, transmission, and analysis.

[0124] This layered architecture not only improves the maintainability and scalability of the system, but also enhances its flexibility and reliability. The clear division of labor between the layers makes development and maintenance more efficient, while also providing a solid foundation for future functional expansion and technology upgrades. In this way, the data collection system can more effectively respond to different types of social media platforms and diverse business needs, ensuring the security, accuracy and efficiency of the data collection process.

[0125] S102, receiving request information input from an external system through the Engines module for data collection scheduling, and verifying the legitimacy of the request information according to a preset security protocol.

[0126] Receiving request information input from the external system through the Engines module for data acquisition scheduling and verifying the legitimacy of the request information according to the preset security protocol is a key step to ensure system security and data integrity. The following is a detailed implementation flow that describes how to complete this process effectively:

[0127] 1. Receive Request

[0128] The Engines module sets one or more listening ports to receive HTTP / HTTPS requests from external systems. When a request is received, all relevant header information (such as API keys, OAuth tokens, etc.) and request body content are immediately captured.

[0129] 2. Preliminary Verification

[0130] Check whether the basic format of the request is correct, for example, make sure the request is a POST method and carries the correct ContentType header (such as application / json). If the format is incorrect, return a 400BadRequest response and refuse to process the request.

[0131] 3. Identity verification

[0132] Extract authentication information (such as API key, OAuth access token, etc.) from the request header. For API key-based authentication, check whether the provided key exists in the whitelist database and confirm that it has not expired or been revoked. If the key is invalid or does not exist, return a 401Unauthorized response and log the attempt.

[0133] Use the OAuth server to verify the validity of the access token, including checking the scope, validity period, and associated user or application permissions of the token. If the token is invalid or has insufficient permissions, a 401Unauthorized response is returned and abnormal behavior is logged.

[0134] 4. Authorization Check

[0135] After verification, further check whether the requested operation is within the authorized scope. For example, check whether the requested resource path matches the permissions assigned to the user. Ensure that the user or application initiating the request has the permissions required to perform the specific operation. If the request exceeds the authorized scope, a 403Forbidden response is returned and the abnormal behavior is recorded.

[0136] 5. Rate Limiting

[0137] To prevent abuse of the API interface, use the LeakyBucket or TokenBucket algorithm to manage rate limits. Limit the number of requests within a certain period of time based on IP address, user ID or other identifiers. When the rate limit is exceeded, a 429TooManyRequests response is returned with a prompt for the retry time.

[0138] 6. Session Management and State Tracking

[0139] For interactions that need to maintain state, a session management mechanism can be introduced, such as using JWT (JSON Web Token) or other forms of session tokens. Session tokens can be issued after each successful authentication and carried in subsequent requests to reduce the need for repeated authentication. Keep detailed logs of all successful authentication and authorization processes, including timestamps, source IP, user ID, request type, and other information. Monitor system performance indicators, such as response time and error rates, to identify potential security threats or performance bottlenecks.

[0140] Through the above steps, the Engines module can effectively receive and verify the request information of the external system, ensuring that only legitimate and authorized requests can enter the subsequent processing stage.

[0141] S103, when the request information meets the legality requirements, the Engines module is used to parse the request information to obtain a parsing result. If the parsing result is an object ID, the Engines module calls the APIs module to obtain API details. If the parsing result is a link, the Engines module calls the crawling module to collect the link.

[0142] After ensuring that the request information meets the legality requirements, the next task of the Engines module is to parse the request information and call the corresponding module (APIs or Crawlers) based on the parsing results to perform specific data collection tasks. The following are the detailed implementation steps:

[0143] 1. Receive and verify the request

[0144] Prior to this, the legitimacy of the request has been ensured through steps such as identity authentication, authorization check, and rate limiting. Now, the Engines module is ready to parse the legitimate request information.

[0145] 2. Parsing request information

[0146] Extract all necessary parameters from the request, including but not limited to the operation type, target ID, link, time range, paging parameters, and filter conditions. Based on the provided parameters, clarify the specific operation the user wants to perform, such as obtaining detailed information about a specific object or crawling web page content.

[0147] You can parse the metadata in the HTTP request header and the specific content in the request body. If the request body is in JSON or other encoding format, use an appropriate library or framework to convert it into an object or dictionary structure in the programming language. For natural language input or other unstructured text content, apply natural language processing (NLP) technology to understand the user's intent and convert it into instructions that the system can recognize.

[0148] 3. Call the corresponding module according to the analysis results

[0149] If the parsed result is an object ID, the APIs module is called to construct a request for a specific social media platform API based on the parsed object ID. Necessary authentication information (such as API key, OAuth token) is added to ensure the security of the API call. The constructed request is sent to the target API endpoint to obtain detailed data related to the object ID. The status code and data format returned by the API are checked to ensure data integrity and correctness, and the results are passed to the Pipelines module for further processing.

[0150] If the parsed result is a link, the crawling module is called to start one or more distributed crawler instances according to the pre-configured crawler behavior rules (such as crawling depth and frequency limit). Following the set behavior rules, the web page content, including text, pictures, comments, etc., is crawled from the specified link. HTML tags and other redundant information are removed, only useful content fragments are retained, and the crawled data is stored in a temporary cache.

[0151] The cleaned data is encapsulated into a format suitable for processing by the Pipelines module and safely passed to the module for subsequent data parsing and formatting.

[0152] Through the above steps, the Engines module can effectively parse the received request information and call the corresponding APIs or crawling modules according to the parsing results to perform specific data collection tasks. This design not only improves the flexibility and adaptability of the system, but also ensures the security and accuracy of the data collection process.

[0153] S104, executing corresponding data collection tasks through the APIs module or the crawling module, and passing the collected data to the Pipelines module for data parsing and formatting to obtain standard data.

[0154] The key link in the social media data collection system is to perform the corresponding data collection tasks through the APIs module or the crawling module, and pass the collected data to the Pipelines module for data parsing and formatting, and finally obtain the standard data. The following are the detailed implementation steps:

[0155] 1. Choose the right collection method

[0156] If the request information indicates an object ID (such as a user ID or a post ID), select the APIs module. This is usually suitable for obtaining structured data, such as the official API interface provided by social media platforms. If the request information is a link, select the crawling module. This method is used to obtain unstructured or semi-structured data, such as web page content.

[0157] 2. Configure collection parameters

[0158] Set API request parameters, including: target API endpoint, authentication information (such as API key, OAuth token), request parameters (such as time range, paging information, filter conditions, etc.).

[0159] 3. Perform Data Acquisition

[0160] Case 1: API call

[0161] Based on the parsed object ID, construct a request for a specific social media platform API, including the necessary authentication information. Send the constructed request to the target API endpoint, making sure to use HTTPS to ensure the security of data transmission. Check the status code and data format returned by the API to ensure data integrity and correctness, and store the results in a temporary cache.

[0162] Case 2: Web crawling

[0163] Start one or more distributed crawler instances according to the preconfigured crawler behavior rules. Follow the set behavior rules to crawl web page content from the specified link, including text, pictures, comments, etc. Remove HTML tags and other redundant information, keep only useful content fragments, and store the crawled data in a temporary cache.

[0164] 4. Pass to Pipelines module

[0165] Encapsulate the collected raw data (whether from API or crawler) into a format suitable for Pipelines module processing, such as JSON object or CSV file. Pass the data to Pipelines module securely through queue system or other mechanisms to ensure the stability and reliability of data flow.

[0166] 5. Data analysis and conversion

[0167] After receiving the data, the Pipelines module parses the data according to predefined rules and pattern matching algorithms to extract useful information. For structured data, it is directly mapped to the database table structure or preset data model. For unstructured data, natural language processing (NLP), image recognition and other technologies are applied to extract valuable information.

[0168] 6. Data Formatting

[0169] Organize the parsed data in a predetermined standard format to ensure consistency. Define clear data fields and types to facilitate subsequent analysis and presentation.

[0170] Through these steps, the system can efficiently and accurately collect data from social media platforms and convert it into a standardized format to provide users with valuable information and services. Each step is carefully designed to ensure the integrity and accuracy of the data while maintaining the flexibility and scalability of the system.

[0171] S105, passing the standard data to the Engines module, and using the Engines module to send the standard data to an external system.

[0172] The process of passing standard data to the Engines module and using the Engines module to send this data to external systems requires ensuring data security, accuracy, and efficient transmission. The following are the detailed implementation steps:

[0173] 1. Prepare standardized data

[0174] Ensure that the standardized data has been properly organized in the predetermined format and that all fields meet the expected type and length requirements. If the data volume is large, the data can be compressed to reduce transmission time and bandwidth consumption. For sensitive data, use appropriate encryption algorithms (such as AES, RSA) to encrypt the data to ensure that it will not be stolen or tampered with during transmission.

[0175] 2. Build a response object

[0176] Construct a response object containing standard data according to the expected format of the external system (such as JSON, XML). Add necessary metadata to the response, such as timestamp, request ID, status code, etc., so that the external system can understand the content and context of the response.

[0177] 3. Pass to Engines module

[0178] The constructed response object is securely passed to the Engines module through pre-defined internal communication protocols (such as message queues, shared memory, database tables, etc.).

[0179] 4.Engines module is ready to send

[0180] After receiving the data, the Engines module parses the response object and extracts the standard data to be sent and its related metadata. According to the API documentation of the external system or the pre-agreed interface specifications, set the HTTP request header, content type, authentication information and other parameters. Determine the transmission protocol to be used (such as HTTP / HTTPS) and ensure that all security measures (such as TLS encryption) are enabled.

[0181] 5. Execute the send operation

[0182] Construct a complete HTTP request, including the URL, method (GET / POST), request body, and all necessary header information. Send an HTTP request to the specified endpoint of the external system over the network to ensure that the data successfully reaches the target location. For external systems that support asynchronous operations, set callback functions or polling mechanisms to track task progress and obtain final results.

[0183] Through the above steps, the Engines module can effectively send standard data to the external system to ensure the security and reliability of data transmission. Each link has been carefully designed to cope with various possible situations, thereby ensuring the stable operation of the system.

[0184] According to a specific implementation of an embodiment of the present invention, the setting of the engine layer, the collection layer and the data layer in the data collection system includes:

[0185] The APIs module encapsulates the collection and calling logic of the open APIs of multiple social media platforms, processes the paging mechanism in the API response, and sets the rate limiting strategy in the APIs module. Each API request consumes a token, and the token is generated at a fixed rate R.

[0186] A multi-threaded asynchronous data crawling mechanism is set in the Crawlers module, and a finite state machine is used to manage the state of data crawling. At the same time, the risk of IP blocking is avoided through proxy pool and user proxy rotation;

[0187] In the Engines module, a task queue is defined and a task scheduling algorithm is set to achieve dynamic adjustment of priority and resource allocation.

[0188] According to a specific implementation of an embodiment of the present invention, the receiving of request information input by an external system through the Engines module for data acquisition scheduling and verifying the legitimacy of the request information according to a preset security protocol includes:

[0189] Set one or more listening ports in the Engines module to receive HTTP / HTTPS requests from external systems;

[0190] After receiving the request, check whether the Content-Type header format carried in the POST method in the request is correct;

[0191] If the format is incorrect, a 400 Bad Request response is returned and the request is rejected.

[0192] According to a specific implementation of an embodiment of the present invention, the receiving of request information input by an external system through the Engines module for data acquisition scheduling and verifying the legitimacy of the request information according to a preset security protocol also includes:

[0193] Extract the API key and OAuth access token from the request header according to the preset security protocol;

[0194] For API key-based authentication, check that the provided key exists in the whitelist database and confirms that it has not expired or been revoked;

[0195] For OAuth 0 authentication, use the OAuth server to verify the validity of the access token, including checking the token's scope, validity period, and associated user;

[0196] If authentication fails, a 401 Unauthorized response is returned and the attempt is logged;

[0197] After the authorization verification is passed, further check whether the requested operation is within the authorization scope and whether the requested resource path matches the permissions assigned to the user;

[0198] If the request exceeds the authorization scope, a 403 Forbidden response is returned and the abnormal behavior is recorded;

[0199] Limit the number of requests within a preset time based on IP address and user ID, and limit the transmission speed of request data through a speed limit algorithm;

[0200] When the rate limit is exceeded, a 429 Too Many Requests response is returned with a prompt for the retry time.

[0201] Detailed logging of all successful authentication and authorization processes, including timestamp, source IP, user ID, and request type;

[0202] For interactions that need to maintain state, a session management mechanism is introduced, and a session token is issued after each successful authentication.

[0203] As an optional manner, before the Engines module sends the standard data to the external system, the Engines module further processes the multiple social media included in the standard data, including the following steps S201-S205:

[0204] S201, setting a BaseClient abstract class and a BaseResource abstract class, wherein the BaseClient abstract class is used to encapsulate the operation interface of a general client, and the BaseResource abstract class is used to define a resource operation interface that is independent of a social media platform.

[0205] The BaseClient abstract class is used to encapsulate the operation interface of the general client. In different social media platforms or other scenarios where clients need to interact, there are often some common operations, such as connecting to the server, sending basic requests, receiving responses, etc. By defining the BaseClient abstract class, these common operations are abstracted to form an interface. As long as the subsequent specific client class inherits this abstract class, it needs to implement the methods related to these common operations in accordance with the regulations, so as to ensure that different clients have unified specifications and calling methods at the basic functional level.

[0206] The BaseResource abstract class is used to define resource operation interfaces that are independent of social media platforms. Resource operations refer to operations such as obtaining, modifying, and deleting various data resources (such as user information, dynamic content, etc.). Although the details and implementation methods of specific resources on different social media platforms are different, from a more abstract level, there are some common operation logics, such as reading resources, updating resources, etc. This abstract class abstracts these common operations to form a unified interface definition, which makes it easier to derive specific resource classes for different platforms to implement these interfaces and add detailed logic related to each platform.

[0207] S202, based on the BaseResource abstract class, derive derived resource classes applied to social media platforms, each derived resource class contains data processing logic required for interacting with the corresponding social media platform, and the data processing logic calculates the first performance indicator Pf of each derived resource class based on the call frequency Fd, response time Tr and parallel processing capability Pc.

[0208] Based on the BaseResource abstract class, specific resource classes are derived for different social media platforms. For example, if there are Weibo platforms and Douyin platforms, WeiboResource classes and DouyinResource classes can be derived respectively. Each derived class contains the data processing logic required to interact with the corresponding social media platform, which means that it knows how to obtain, process and operate data resources according to the rules and interfaces of the corresponding platform. For example, the WeiboResource class knows the API call method and data format of the Weibo platform, so that it can accurately perform operations such as obtaining the dynamics of Weibo users and publishing new Weibo, while the DouyinResource class is familiar with the operation logic related to the Douyin platform.

[0209] The execution of these data processing logics will affect the performance of the resource class, so it needs to be measured based on some key indicators, namely the call frequency Fd (the number of times the resource is operated per unit time, reflecting the frequency of resource use), response time Tr (the time spent from issuing an operation request to receiving a platform response, reflecting the timeliness of the interaction) and parallel processing capability Pc (how many related operations can be processed simultaneously, which is related to the concurrent efficiency of resource processing).

[0210] Through a specific calculation method, combined with the above parameters of call frequency Fd, response time Tr and parallel processing capability Pc, the first performance index Pf of each derived resource class is obtained. This index can quantitatively reflect the comprehensive performance of the resource class when performing resource operations on the corresponding platform.

[0211] As a specific implementation method, the following multiple performance indicators can be calculated:

[0212] Calculate the call frequency Fd:

[0213]

[0214] k is the number of observed events, λ1 is the average occurrence rate;

[0215] Calculate the response time Tr:

[0216]

[0217] L is the average number of requests in the system, λ2 is the request arrival rate;

[0218] Calculate the parallel processing capacity Pc:

[0219]

[0220] p is the parallelizable part of the program, and s is the speedup ratio.

[0221] The calculation formula of the first performance index Pf is further set as:

[0222]

[0223] Np is the total number of events observed in a specific time period, Fdi represents the frequency of the ith call, and α1, β1, γ1, δ1 are weight coefficients.

[0224] S203, create a unified client class Client inherited from the BaseClient abstract class, the unified client class includes n social media platform resource objects, dynamically instantiate the n social media platform resource objects included in the unified client class using a factory pattern, and calculate a second performance indicator Ps of the social media platform resource objects based on the data transmission rate Ts, memory usage Mu, network delay Ds, and resource allocation efficiency Ef of the social media platform resource objects.

[0225] Creating a unified client class Client that inherits from the BaseClient abstract class means that it has the capabilities specified by the general client operation interfaces defined in BaseClient. At the same time, this class contains n social media platform resource objects, which means that it can integrate the resource operation capabilities related to multiple different social media platforms in one client class, playing a role in unified management and unified external service provision. For example, if an application wants to interact with multiple platforms such as Weibo, Douyin, and WeChat at the same time to obtain user information and publish content, it can be achieved through this unified client class.

[0226] The factory pattern is used to dynamically instantiate n social media platform resource objects. The factory pattern can flexibly create corresponding social media platform resource object instances according to different conditions (such as configuration file specifications, user selection, etc.), thereby improving the scalability and maintainability of the code.

[0227] For each social media platform resource object, based on their data transmission rate Ts (the speed at which data is transmitted between the platform and the client, which affects the efficiency of interaction), memory usage Mu (the proportion of memory occupied by resource objects, which is related to system resource consumption), network delay Ds (the delay time caused by network communication, which affects the timeliness of operations), and resource allocation efficiency Ef (whether resource allocation is reasonable and efficient, reflecting the degree of optimization of resource utilization), the second performance indicator Ps is obtained through a specific calculation method. This indicator is also a quantitative measure of the performance of resource objects, which helps to evaluate their performance and applicability in the entire client from another perspective.

[0228] Specifically, the data transmission rate Ts can be calculated:

[0229]

[0230] B is the bandwidth, S / N is the signal-to-noise ratio;

[0231] Calculate the memory usage Mu:

[0232] Mu

[0233] U is the used memory and T is the total memory;

[0234] Calculate network delay Ds:

[0235] Ds=Dp+Dq

[0236] Dp is the propagation delay and Dq is the queue delay.

[0237] Calculate resource allocation efficiency Ef:

[0238]

[0239] Cc is the amount of work done and Tz is the total time.

[0240] Furthermore, the calculation formula of the second performance indicator Ps is set as:

[0241]

[0242] w1, w2, w3, w4 are weight coefficients, Bmax and Emax are the maximum data transmission rate and maximum resource allocation efficiency respectively, η1, η2 are power parameters, γ2 and β2 are parameters in the exponential decay function, which are used to adjust the impact of Mu and Ds on performance.

[0243] S204, using the task complexity Tf, resource availability Rz and mapping function fy, generates a signature random number Qs for each derived resource class, digitally signs n social media platform resource objects based on the random number Qs, and generates a data call interface J={J1, J2,…Jn} of the n social media platform resource objects based on the digital signature.

[0244] Based on the task complexity Tf (the complexity of the task to be performed, for example, the complexity of the task of obtaining a large amount of user data may be higher than that of the task of obtaining a single user's information), resource availability Rz (the degree to which the corresponding resource can be normally used at present, which is affected by factors such as platform status and permissions) and mapping function fy (a custom conversion function that maps and converts relevant input parameters according to business logic to obtain specific values, which may be used for adjustment, encryption, etc.), a signature random number Qs for each derived resource class is generated through a specific algorithm or calculation logic. This random number has a certain degree of randomness and uniqueness (generating a specific result based on the given input parameters), and can be used for subsequent security and identification-related operations such as digital signatures, and plays a role in distinguishing different resource classes and ensuring data integrity and security.

[0245] Based on the previously generated signature random number Qs, digital signatures are performed for n social media platform resource objects. Digital signatures can ensure the legitimacy, integrity, and non-repudiation of the data source. For example, the recipient can verify the signature to confirm whether the data comes from a legitimate resource object and whether it has been tampered with during transmission. Then, based on these digital signatures, data call interfaces J={J1, J2, … Jn} for n social media platform resource objects are generated. Each interface corresponds to a resource object. These interfaces define how the client calls the specific operations of the corresponding resource object. From the outside, it provides a unified and standardized call entry, hiding the complexity of the internal implementation of resource objects on different platforms.

[0246] S205, when a call request Qr of a social media platform resource is received, based on the first performance indicator Pf, the second performance indicator Ps and the data call interface J, m resource interfaces J'={J'1, J'2, ... J'm} corresponding to the call request Qr are generated, m≤n, to realize the same-name call on different social media platform resources, so that users can interact with different social media platforms in the same way.

[0247] When the client receives a call request Qr for social media platform resources, such as a user clicking in the application to obtain user information on a certain platform or to post a dynamic message, the client needs to decide which resource interfaces to use to process the request based on the existing resource conditions and performance.

[0248] Based on the previously calculated first performance index Pf (reflecting resource data processing performance), the second performance index Ps (reflecting resource object's own resource-related performance) and the generated data call interface J (defining the call entry of the resource object), through certain screening, matching and other logic, the m resource interfaces J'={J'1, J'2, ... J'm} (m≤n, meaning that the number of resource interfaces that meet the conditions will not exceed the total number of resource objects) corresponding to the call request Qr are generated. In this way, the same-name call on resources on different social media platforms can be realized, that is, for users, whether operating the Weibo platform or the Douyin platform, as long as the request is for the same function (such as obtaining user information), the call can be initiated in the same way (through a unified client entry), and the client will automatically select the appropriate platform resource interface to process the request based on performance, resources, etc., ultimately achieving the purpose of users interacting with different social media platforms in the same way, improving user experience and facilitating the integrated management of multi-platform resources by applications.

[0249] According to a specific implementation of an embodiment of the present invention, when the request information meets the legality requirement, parsing the request information using the Engines module includes:

[0250] Load the configuration file of the Engines module, initialize the data structure and set the default parameters;

[0251] Extract all relevant header information from HTTP / HTTPS requests;

[0252] Determine whether there is JSON data in the parsed request body based on the Content-Type header. If so, use the standard library to convert the JSON string into an object or dictionary structure in a programming language.

[0253] According to a specific implementation of an embodiment of the present invention, the corresponding data collection task is performed by the APIs module or the crawling module, and the collected data is passed to the Pipelines module for data parsing and formatting to obtain standard data, including:

[0254] According to the parsed request information, specific collection parameters are set, wherein the collection parameters include: target API endpoint, identity authentication information, request parameters, and data behavior rules;

[0255] Constructing an HTTP request, wherein the HTTP request includes header information and query parameters, and sending the HTTP request to a target API endpoint;

[0256] Start a crawl task using a preconfigured data crawler instance and store the crawled content in a temporary cache.

[0257] According to a specific implementation of an embodiment of the present invention, the APIs module or the crawling module performs the corresponding data collection task, and passes the collected data to the Pipelines module for data parsing and formatting to obtain standard data, and further includes:

[0258] After the collection is completed, the collected raw data is cleaned to remove redundant information, blank fields and obviously erroneous data entries. For data captured from web pages, HTML tags are removed to keep only plain text;

[0259] Encapsulate the cleaned data into a format suitable for processing by the Pipelines module, and pass the cleaned data to the Pipelines module through the queue system;

[0260] After receiving the data, the Pipelines module parses the data according to predefined rules and pattern matching algorithms to extract useful information. For structured data, it directly maps it to the database table structure or preset data model. For unstructured data, it applies natural language processing to extract target information.

[0261] The parsed data is organized according to a predetermined standard format to form standard data.

[0262] According to a specific implementation of an embodiment of the present invention, the passing of the standard data to the Engines module, and using the Engines module to send the standard data to an external system includes:

[0263] Construct a response object containing standard data according to the expected format of the external system, and add metadata to the response object, including: timestamp, request ID and status code;

[0264] Securely pass the constructed response object to the Engines module through a predefined internal communication protocol;

[0265] After receiving the data, the Engines module parses the response object and extracts the standard data to be sent and its related metadata;

[0266] Set HTTP request header, content type, and authentication information according to the API documentation of the external system or the pre-agreed interface specifications;

[0267] Construct a complete HTTP request, including URL, GET / POST, request body and header information, and send the HTTP request to the specified endpoint of the external system through the network.

[0268] Corresponding to the above method embodiment, see Figure 3The embodiment of the present invention further discloses a modular social media data collection device 30, comprising:

[0269] Setting module 301, setting an engine layer, a collection layer and a data layer in the data collection system, wherein the engine layer includes an engine module and business sub-items, wherein the business sub-items include multiple different types of data sets, the collection layer includes an APIs module responsible for encapsulating API calls and a crawling module responsible for data collection, and the data layer includes a Pipelines module for data format conversion;

[0270] The receiving module 302 receives the request information input by the external system through the Engines module for data collection and scheduling, and verifies the legitimacy of the request information according to a preset security protocol;

[0271] Parsing module 303, when the request information meets the legality requirement, uses the Engines module to parse the request information to obtain a parsing result. If the parsing result is an object ID, the Engines module calls the APIs module to obtain API details. If the parsing result is a link, the Engines module calls the crawling module to collect the link.

[0272] The execution module 304 executes the corresponding data collection task through the APIs module or the crawling module, and passes the collected data to the Pipelines module for data parsing and formatting to obtain standard data;

[0273] The sending module 305 passes the standard data to the Engines module, and uses the Engines module to send the standard data to an external system.

[0274] See also Figure 4 The embodiment of the present invention further provides an electronic device 60, the electronic device comprising:

[0275] at least one processor; and,

[0276] a memory communicatively connected to the at least one processor; wherein,

[0277] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the modular social media data collection method in the aforementioned method embodiment.

[0278] An embodiment of the present invention further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the aforementioned method embodiment.

[0279] An embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the modular social media data collection method in the aforementioned method embodiment.

[0280] Reference below Figure 4 , which shows a schematic diagram of the structure of an electronic device 60 suitable for implementing the embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0281] like Figure 4 As shown, the electronic device 60 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 60 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0282] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, a gage, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 60 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 60 is shown with various devices, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0283] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0284] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A modular social media data collection method, characterized in that: include: An engine layer, a collection layer and a data layer are set in the data collection system. The engine layer includes an engine module and business items. The business items include multiple different types of data sets. The collection layer includes an APIs module responsible for encapsulating API calls and a crawling module responsible for data collection. The data layer includes a Pipelines module for data format conversion. Receive request information input from the external system through the Engines module for data acquisition scheduling, and verify the legitimacy of the request information according to the preset security protocol; When the request information meets the legality requirements, the Engines module is used to parse the request information to obtain a parsing result. If the parsing result is an object ID, the Engines module calls the APIs module to obtain API details. If the parsing result is a link, the Engines module calls the crawling module to collect the link. Execute the corresponding data collection tasks through the APIs module or the crawling module, and pass the collected data to the Pipelines module for data parsing and formatting to obtain standard data; Passing the standard data to the Engines module, and using the Engines module to send the standard data to an external system; in The engine layer, the collection layer and the data layer are set in the data collection system, including: The APIs module encapsulates the collection and calling logic of the open APIs of multiple social media platforms, processes the paging mechanism in the API response, and sets the rate limiting strategy in the APIs module. Each API request consumes a token, and the token is generated at a fixed rate R. A multi-threaded asynchronous data crawling mechanism is set in the crawling module, and a finite state machine is used to manage the state of data crawling. At the same time, the risk of IP blocking is avoided through proxy pool and user proxy rotation; In the Engines module, a task queue is defined and a task scheduling algorithm is set to achieve dynamic adjustment of priority and resource allocation.

2. The method according to claim 1, characterized in that The receiving of request information input from an external system through the Engines module for data acquisition scheduling and verifying the legitimacy of the request information according to a preset security protocol includes: Set one or more listening ports in the Engines module to receive HTTP / HTTPS requests from external systems; After receiving the request, check whether the Content-Type header format carried in the POST method in the request is correct; If the format is incorrect, a 400 Bad Request response is returned and the request is rejected.

3. The method according to claim 2, characterized in that The receiving of request information input by the external system through the Engines module for data acquisition scheduling and verifying the legitimacy of the request information according to a preset security protocol also includes: Extract the API key and OAuth access token from the request header according to the preset security protocol; For API key-based authentication, check that the provided key exists in the whitelist database and confirms that it has not expired or been revoked; For OAuth 2.0 authentication, use the OAuth server to verify the validity of the access token, including checking the token's scope, validity period, and associated user; If authentication fails, a 401 Unauthorized response is returned and the attempt is logged; After the authorization verification is passed, further check whether the requested operation is within the authorization scope and whether the requested resource path matches the permissions assigned to the user; If the request exceeds the authorization scope, a 403 Forbidden response is returned and the abnormal behavior is recorded; Limit the number of requests within a preset time based on IP address and user ID, and limit the transmission speed of request data through a speed limit algorithm; When the rate limit is exceeded, a 429 Too Many Requests response is returned with a prompt for the retry time. Detailed logging of all successful authentication and authorization processes, including timestamp, source IP, user ID, and request type; For interactions that need to maintain state, a session management mechanism is introduced, and a session token is issued after each successful authentication.

4. The method according to claim 3, characterized in that When the request information meets the legality requirement, the request information is parsed using the Engines module, including: Load the configuration file of the Engines module, initialize the data structure and set the default parameters; Extract all relevant header information from HTTP / HTTPS requests; Determine whether there is JSON data in the parsed request body based on the Content-Type header. If so, use the standard library to convert the JSON string into an object or dictionary structure in a programming language.

5. The method according to claim 4, characterized in that The corresponding data collection task is performed through the APIs module or the crawling module, and the collected data is passed to the Pipelines module for data parsing and formatting to obtain standard data, including: According to the parsed request information, specific collection parameters are set, wherein the collection parameters include: target API endpoint, identity authentication information, request parameters, and data behavior rules; Constructing an HTTP request, wherein the HTTP request includes header information and query parameters, and sending the HTTP request to a target API endpoint; Start a crawl task using a preconfigured data crawler instance and store the crawled content in a temporary cache.

6. The method according to claim 5, characterized in that The APIs module or the crawling module performs the corresponding data collection task, and passes the collected data to the Pipelines module for data parsing and formatting to obtain standard data, and also includes: After the collection is completed, the collected raw data is cleaned to remove redundant information, blank fields and obviously erroneous data entries. For data captured from web pages, HTML tags are removed to keep only plain text; Encapsulate the cleaned data into a format suitable for processing by the Pipelines module, and pass the cleaned data to the Pipelines module through the queue system; After receiving the data, the Pipelines module parses the data according to predefined rules and pattern matching algorithms to extract useful information. For structured data, it directly maps it to the database table structure or preset data model. For unstructured data, it applies natural language processing to extract target information. The parsed data is organized according to a predetermined standard format to form standard data.

7. The method according to claim 6, characterized in that The step of passing the standard data to the Engines module and sending the standard data to an external system by using the Engines module includes: Construct a response object containing standard data according to the expected format of the external system, and add metadata to the response object, including: timestamp, request ID and status code; Securely pass the constructed response object to the Engines module through a predefined internal communication protocol; After receiving the data, the Engines module parses the response object and extracts the standard data to be sent and its related metadata; Set HTTP request header, content type, and authentication information according to the API documentation of the external system or the pre-agreed interface specifications; Construct a complete HTTP request, including URL, GET / POST, request body and header information, and send the HTTP request to the specified endpoint of the external system through the network.

8. A modular social media data collection device, characterized in that: include: Setting module, setting engine layer, collection layer and data layer in the data collection system, the engine layer includes engine module and business sub-items, the business sub-items include multiple different types of data sets, the collection layer includes APIs module responsible for encapsulating API calls and crawling module responsible for data collection, the data layer includes Pipelines module for data format conversion; The receiving module receives the request information input by the external system through the Engines module for data acquisition scheduling, and verifies the legitimacy of the request information according to the preset security protocol; A parsing module, when the request information meets the legality requirements, uses the Engines module to parse the request information to obtain a parsing result. If the parsing result is an object ID, the Engines module calls the APIs module to obtain API details. If the parsing result is a link, the Engines module calls the crawling module to collect the link. The execution module performs the corresponding data collection tasks through the APIs module or the crawling module, and passes the collected data to the Pipelines module for data parsing and formatting to obtain standard data; A sending module, which transmits the standard data to the Engines module, and uses the Engines module to send the standard data to an external system; in The engine layer, the collection layer and the data layer are set in the data collection system, including: The APIs module encapsulates the collection and calling logic of the open APIs of multiple social media platforms, processes the paging mechanism in the API response, and sets the rate limiting strategy in the APIs module. Each API request consumes a token, and the token is generated at a fixed rate R. A multi-threaded asynchronous data crawling mechanism is set in the crawling module, and a finite state machine is used to manage the state of data crawling. At the same time, the risk of IP blocking is avoided through proxy pool and user proxy rotation; In the Engines module, a task queue is defined and a task scheduling algorithm is set to achieve dynamic adjustment of priority and resource allocation.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the modular social media data collection method described in any one of claims 1 to 7.

Images