Social media data acquisition method and device, electronic equipment and storage medium

Abstract

The social media data acquisition method provided by the invention relates to the technical field of data acquisition, and comprises the following steps: establishing and maintaining MTPro long connection, serializing and controlling API (Application Program Interface) calling by utilizing an asynchronous task scheduling mechanism, and calling a corresponding TelegramAPI interface based on a group type. And aiming at the speed limit problem, a self-adaptive exponential backoff mechanism is constructed, and an API speed limit strategy is effectively avoided in combination with a dynamic backoff interval and speed limit state feedback. Meanwhile, an independent heartbeat mechanism is set to maintain the connection activity, and the request success rate and the system stability are improved through a speed limiting queue and a dynamic retry strategy. Experiments show that the method ensures high concurrency control, increases the request success rate by 65%, reduces the delay by 47%, significantly increases the resource utilization rate, and has good practical value and popularization prospects.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology, and in particular to a method, apparatus, electronic device, and storage medium for acquiring social media data. Background Technology

[0002] With the widespread use of social media platforms, information in Telegram public groups and channels has significant data collection value. Existing Telegram data monitoring technologies are mainly based on HTTP polling or simple API calls, employing a polling mechanism with fixed time intervals. This lacks the ability to intelligently respond to API limitations and is inefficient when handling large-scale group monitoring.

[0003] How to dynamically adjust the request frequency to avoid rate limiting under Telegram's strict API restrictions, while improving data collection efficiency, reducing latency, optimizing resource utilization through efficient concurrency control and connection management, and ensuring the stability of long-term connections is a technical problem worth solving. Summary of the Invention

[0004] This invention aims to at least partially address one of the technical problems in related technologies. Therefore, one object of this invention is to provide a method, apparatus, electronic device, and storage medium for collecting social media data, thereby improving the security of the target path.

[0005] One aspect of the present invention provides a method for collecting social media data, comprising the following steps: Establish and maintain a long-lived MTProto connection with the Telegram platform, and output a session channel for subsequent data interaction; On the aforementioned session channel, an asynchronous task scheduling mechanism is used to serialize the API calls of each Telegram account, ensuring that only one API request is sent to the group processing unit at any given time. The group processing unit calls the corresponding Telegram API according to the link type of the group to be joined. If the group is a public group, it calls JoinChannelRequest; if the group is a private group, it calls ImportChatInviteRequest. The API return information is then output to the error parsing unit to obtain the API return information. If the API returns FLOOD_WAIT or HTTP429, an adaptive exponential backoff mechanism is initiated. The backoff status information is established and maintained through the adaptive exponential backoff mechanism. The backoff status information includes at least the number of consecutive rate limits, the time of the most recent rate limit, and the current backoff interval. The current backoff interval is then fed back to the asynchronous task scheduling mechanism. The asynchronous task scheduling mechanism is used to adjust the API request sending rhythm according to the current backoff interval, and groups that cannot be joined temporarily due to rate limiting are written into the rate limiting queue. The time for lifting the limit for each group is calculated and recorded. When the limit is lifted, the rate limit queue automatically triggers a retry, resends the group to be retried to the group processing unit, and inserts a dynamic random delay based on the current number of consecutive rate limits. An independent heartbeat mechanism is initiated in parallel to maintain the activity of the session channel. The initial heartbeat interval is 300 seconds. If a heartbeat rate limit error is detected, heartbeat backoff management is initiated. The maximum heartbeat backoff interval is 1800 seconds, and the heartbeat status is fed back to the adaptive exponential backoff mechanism.

[0006] Preferably, the rate-limiting queue is organized and retrieved using a multi-level dictionary structure nested by account and group URL.

[0007] Preferably, the dynamic random delay satisfies delay=min(3.0+consecutive_floods×1.0,10.0).

[0008] Preferably, the asynchronous task scheduling mechanism is implemented based on Python's asyncio library and generates independent coroutine tasks for each account.

[0009] Preferably, the error parsing unit includes extracting the rate-limiting waiting time using regular expressions and classifying different error types using a tree-structured decision process.

[0010] Preferably, the step of establishing and maintaining backoff state information through an adaptive exponential backoff mechanism includes: When the first speed limit event occurs, the backoff interval is initialized to 300 seconds; when a FLOOD_WAIT error is detected, a new backoff interval is calculated according to backoff_interval=min(300×2^consecutive_floods,86400); When an HTTP 429 error is detected, the number of consecutive rate-limiting attempts will be doubled, and the maximum backoff interval will be 86,400 seconds. If the time since the last speed limit exceeds 2 hours, the current retreat interval will be automatically halved but not less than 300 seconds. When an unlimited-speed API call is successfully completed, the current backoff interval is reduced by 10%.

[0011] Preferably, the dynamic random delay time is perturbed by a random amount at the millisecond level before being inserted into the asynchronous task scheduling mechanism, and the overall delay does not exceed 12 seconds.

[0012] The present invention also provides a system for collecting social media data, the system being used to perform a method for collecting social media data, comprising: The connection module is used to establish and maintain a long MTProto connection with the Telegram platform and output a session channel for subsequent data interaction. The control module is used to serialize the API calls of each Telegram account on the session channel through an asynchronous task scheduling mechanism, ensuring that only one API request is sent to the group processing unit at any given time. The parsing module is used by the group processing unit to call the corresponding Telegram API according to the link type of the group to be joined. If the group is a public group, JoinChannelRequest is called, and if the group is a private group, ImportChatInviteRequest is called. The API return information is then output to the error parsing unit to obtain the API return information. The backoff module is used to initiate an adaptive exponential backoff mechanism if the API returns FLOOD_WAIT or HTTP429. The adaptive exponential backoff mechanism establishes and maintains backoff status information, which includes at least the number of consecutive rate limits, the time of the most recent rate limit, and the current backoff interval. The current backoff interval is then fed back to the asynchronous task scheduling mechanism. The evaluation module is used to adjust the API request sending rhythm based on the current backoff interval using an asynchronous task scheduling mechanism, and write groups that cannot be joined temporarily due to rate limiting into the rate limiting queue, and calculate and record the time when the limit is lifted for each group; The processing module is used to automatically trigger a retry when the limit is lifted, resend the group to be retried to the group processing unit, and insert a dynamic random delay based on the current number of consecutive rate limits. The feedback module is used to initiate an independent heartbeat mechanism in parallel to maintain the activity of the session channel. The initial heartbeat interval is 300 seconds. If a heartbeat rate limit error is detected, the system enters heartbeat backoff management with an upper limit of 1800 seconds. The heartbeat status is then fed back to the adaptive exponential backoff mechanism.

[0013] The present invention also provides an electronic device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the method for collecting social media data.

[0014] The present invention also provides a storage medium storing a computer program adapted for loading by a processor to perform the steps in the method for collecting social media data. Beneficial effects

[0015] This invention significantly improves data acquisition efficiency and reduces latency through an adaptive exponential backoff mechanism and MTProto long-connection optimization. Specifically, it utilizes a strategy of dynamically adjusting the backoff interval, initially setting the rate limit to 300 seconds, and exponentially increasing it to a maximum of 86,400 seconds. Combined with a continuous rate-limiting count tracking and automatic recovery mechanism, the backoff interval is halved if the rate limit exceeds 2 hours, effectively circumventing Telegram's API rate-limiting mechanism. Real-world testing shows that this mechanism increases the success rate of valid requests by 65% ​​and reduces average response latency by 47%. Simultaneously, it establishes long connections based on the MTProto protocol and supplements them with a 300-second heartbeat maintenance, with a backoff limit of 1800 seconds, reducing connection reconstruction overhead. Compared to traditional solutions, this reduces API call counts by 40% and improves connection stability by over 50%, fundamentally solving the problems of high latency and frequent rate limits caused by fixed polling.

[0016] Regarding system resource utilization and stability, this invention achieves efficient concurrent control through asynchronous task scheduling and rate-limiting queue management. Employing Pythonasyncio's coroutine task mechanism serializes API calls for each account, with a concurrent semaphore limit of 1. In I / O-intensive scenarios, this increases the number of groups a single node can handle by 3-5 times, while CPU utilization only increases by 20-30%. The rate-limiting queue uses a multi-level dictionary structure organized by account and group URL, accurately calculating release times and triggering retries. Combined with dynamic random delays satisfying delay=min(3.0+consecutive_floods×1.0,10.0), long-term monitoring efficiency remains around 80% even with large-scale rate limiting. Furthermore, a multi-layered error handling strategy uses regular expressions to parse rate-limiting times with 99.9% accuracy. Combined with a state persistence mechanism, it achieves 100% state recovery after system crashes, comprehensively improving data acquisition stability and resource utilization efficiency in complex network environments. Attached Figure Description

[0017] Figure 1 This is a flowchart of a social media data collection method according to the present invention; Figure 2 This is a system diagram of social media data collection according to the present invention; Figure 3 The curves showing the request success rate over time for different monitoring technologies in the embodiments of this invention are shown. Figure 4 This is a comparison curve of resource utilization rates for different monitoring technologies in the embodiments of the present invention; Figure 5 This is a comparison of the average response delay under different speed-limiting conditions in the embodiments of the present invention; Figure 6 This is a schematic diagram of an electronic device structure provided in one embodiment of the present invention; Figure 7 This is a schematic diagram of a computer storage medium structure provided in one embodiment of the present invention. Detailed Implementation

[0018] To better understand the invention, various aspects of the invention will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely illustrative of exemplary embodiments of the invention and are not intended to limit the scope of the invention in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0019] In the accompanying drawings, the size, dimensions, and shapes of the elements have been slightly adjusted for ease of illustration. The drawings are for illustrative purposes only and are not strictly to scale. As used herein, the terms “approximately,” “about,” and similar terms are used to indicate approximation rather than degree, and are intended to illustrate inherent deviations in measured or calculated values ​​that will be recognized by one of ordinary skill in the art. Furthermore, the order in which the steps are described in this invention does not necessarily indicate the order in which these steps occur in actual operation, unless otherwise expressly defined or deduced from the context.

[0020] It should also be understood that expressions such as "comprising," "including," "having," "containing," and / or "comprising" are open-ended rather than closed-ended expressions in this specification, indicating the presence of the stated features, elements, and / or components, but not excluding the presence of one or more other features, elements, components, and / or combinations thereof. Furthermore, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features, not just individual elements in the list. Additionally, when describing embodiments of the invention, the word "may" is used to mean "one or more embodiments of the invention." And the term "exemplary" is intended to refer to examples or illustrations.

[0021] Unless otherwise specified, all terms used herein (including engineering and technical terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that, unless expressly stated herein, terms defined in common dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not in an idealized or overly formalized sense.

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other. The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] Step 1: Establish and maintain a long MTProto connection with the Telegram platform to output a session channel for subsequent data interaction.

[0024] Specifically, this step establishes a persistent connection using Telegram's MTProto protocol. This protocol is a Telegram-customized encrypted transmission protocol that offers lower latency and higher reliability compared to traditional HTTP polling. In practice, Telegram's underlying API interfaces are used to complete client authentication, key negotiation, and session initialization, forming a persistent session channel. Maintaining a persistent connection is crucial to avoiding frequent connection establishment and termination—existing technologies like HTTP polling require a re-handshake for each request, leading to latency overhead and wasted API quotas. Persistent connections can reuse underlying network resources, reducing system resource consumption in large-scale group monitoring scenarios. This is because traditional solutions consume TCP connection resources with each request, while persistent connections only require a single initialization to support continuous data interaction.

[0025] By leveraging long-lived connections, the effective payload ratio of API calls is significantly increased, directly reducing data acquisition latency and effectively addressing the shortcomings of existing technologies in "improper connection management." This provides a stable transmission foundation for operations such as asynchronous task scheduling and heartbeat maintenance. Without this step, all subsequent session-based operations (such as group joining and message retrieval) would be impossible due to frequent connection interruptions, leading to insufficient overall system availability. Furthermore, this step, in conjunction with the heartbeat mechanism in step S7, forms a dual stabilization strategy of "long-lived connections + heartbeat keep-alive," a synergistic effect that cannot be achieved by simply using short connections in existing technologies.

[0026] For example, the Telegram client (TelegramClient class) is initialized using the Telegram library, passing in the api_id and api_hash parameters to establish a session. The client.start() method is called to complete the MTProto protocol handshake, generate an encrypted session key, and establish a long-lived connection. An independent heartbeat task, keep_alive(), is started, periodically calling client.get_me() (default interval 300 seconds) to maintain connection activity and prevent session expiration due to prolonged inactivity. Step 2: On the session channel, the API calls of each Telegram account are serialized through an asynchronous task scheduling mechanism to ensure that only one API request is sent to the group processing unit at any given time. This step implements the asynchronous task scheduling mechanism using the Python asyncio framework, generating an independent coroutine task for each Telegram account, and strictly limiting the concurrency of a single account to 1 using a semaphore. The core logic of the serialization control is that after an API request is sent, the task enters a suspended state until a response is received or the timeout period is reached, at which point the semaphore is released to allow the next request to be sent. This design addresses the risk control mechanism for the Telegram API—this mechanism has strict limitations on high-concurrency requests, specifically: when the number of concurrent requests for the same account exceeds 1, the probability of triggering the FLOOD_WAIT rate limiting mechanism increases significantly. For example, an independent coroutine task monitor_groups_task() is created for each Telegram account, and an account-level semaphore is created using asyncio.Semaphore(1) to ensure that the concurrency of a single account is 1.

[0027] Step 3: The group processing unit calls the corresponding Telegram API according to the link type of the group to be joined. If the group is a public group, JoinChannelRequest is called; if the group is a private group, ImportChatInviteRequest is called. The API return information is then output to the error parsing unit to obtain the API return information. The error parsing unit uses regular expressions to extract the rate limit waiting time and uses a tree-structured decision process to classify and process different error types.

[0028] Specifically, the group type is identified by link characteristics, such as the presence of "joinchat" or a "+" symbol: public groups can be joined directly via JoinChannelRequest, while private groups require the ImportChatInviteRequest method to parse the invitation link. The error parsing module uses regular expressions r'wait of (\d+) second' and r'FLOOD_WAIT_(\d+)' to accurately extract the FLOOD_WAIT wait time, achieving high accuracy and avoiding retry timing deviations caused by parsing errors in existing technologies, such as misjudging 1800 seconds as 180 seconds. A tree-structured decision-making process handles errors according to their severity: for example, ChannelPrivateError (group private) is directly marked as requiring an invitation link; FLOOD_WAIT errors enter a backoff process; and network errors are retried three times before being queued.

[0029] By intelligently identifying group link types and calling corresponding APIs, combined with regular expressions to accurately extract error information and a tree-structured decision-making process for classification, this system overcomes the shortcomings of existing technologies such as "coarse link type processing" (failing to distinguish between public / private groups when calling appropriate APIs) and "inaccurate error parsing" (inability to accurately extract rate-limiting times and classify error types). Specifically, the system distinguishes between public / private groups based on link characteristics, such as the presence of "joinchat" or the "+" symbol, and calls the appropriate APIs accordingly. JoinChannelRequest or ImportChatInviteRequestAPI; Regular expression extraction is performed using the following method: pattern1=r'waitof(\d+)second'; pattern2=r'FLOOD_WAIT_(\d+)'; Regular expressions are used to accurately extract the FLOOD_WAIT wait time, and a tree-structured decision-making process is used to classify and handle error types such as FLOOD_WAIT and ChannelPrivateError.

[0030] For example, the group processing unit first determines whether the link is an invitation hash or a public URL based on its form: if '+' or joinchat is detected, it parses the invite_hash and calls ImportChatInviteRequest(invite_hash); otherwise, it directly uses JoinChannelRequest(channel_url) to join the public channel. The API return value is immediately sent to the error parsing unit, which uses the double regular expression patterns r'waitof(\\d+)second' and r'FLOOD_WAIT_(\\d+)' to accurately extract the waiting seconds, and maps the error to nine categories such as rate limiting / permission / invalid link based on a tree-structured decision table, which then drives state transitions and backoff actions.

[0031] Step 4: If the API returns FLOOD_WAIT or HTTP429, then the adaptive exponential backoff mechanism is initiated. The backoff status information is established and maintained through the adaptive exponential backoff mechanism. The backoff status information includes at least the number of consecutive rate limits, the time of the most recent rate limit, and the current backoff interval. The current backoff interval is then fed back to the asynchronous task scheduling mechanism. It should be noted that the method uses a unified conversion between UTC and Shanghai time zone for all time data to ensure consistency of time across modules. The backoff interval is initialized to 300 seconds upon the first rate-limiting event. When a FLOOD_WAIT error is detected, a new backoff interval is calculated using `backoff_interval = min(300 × 2^consecutive_floods, 86400)`. When an HTTP 429 error is detected, the number of consecutive rate-limiting events is doubled, and the maximum backoff interval is 86400 seconds. When the time since the last rate-limiting event exceeds 2 hours, the current backoff interval is automatically halved but not reduced to less than 300 seconds. When an unlimited rate-limiting API call is successfully completed, the current backoff interval is reduced. Here, `backoff_interval` represents the current backoff interval, `consecutive_floods` represents the number of consecutive FLOOD_WAIT errors, `min()` is the minimum value function, and 300 is the initial backoff value in seconds, meaning a 300-second wait for the first backoff. This function formula indicates that, initially, when a rate-limiting event (such as FLOOD_WAIT) is detected for the first time, the backoff time is set to 300 seconds. If the same type of error is detected again in succession, the backoff time will increase exponentially, doubling with each additional error (× 2^n); the maximum backoff interval is 86,400 seconds (24 hours), meaning the system will not be delayed indefinitely, ensuring the controllability of service recovery.

[0032] Specifically: Initial value setting: The backoff interval is set to 300 seconds during the initial speed limit. This value has been verified through extensive testing—a value below 300 seconds easily triggers Telegram's basic protection layer, while a value above 300 seconds increases data acquisition latency by 40%. Exponential growth model: For FLOOD_WAIT errors, the backoff interval is calculated using the following method. backoff_interval=min(300×2^consecutive_floods,86400); The "2" is used as an exponential growth factor, determined through testing based on the quasi-exponential growth characteristics of the Telegram API rate limiting mechanism, achieving a 95% fit with the actual rate limiting curve. For HTTP 429 errors, the backoff strength is enhanced by doubling the number of consecutive rate limits (consecutive_floods += 2), extending the maximum backoff interval to 86,400 seconds to cover extreme rate limiting scenarios. An automatic recovery mechanism is implemented: if more than 2 hours have passed since the last rate limiting, the backoff interval is halved but not less than 300 seconds; after each successful completion of an unlimited rate API call, the backoff interval is reduced, achieving a dynamic balance between "limiting" and "recovery."

[0033] For example, if the parsing result is FLOOD_WAIT or HTTP429, the backoff manager will accumulate consequential_floods in the account-level backoff_status structure and calculate the next round of check intervals using the formula backoff=min(300×2^consecutive_floods,86400). For HTTP429 errors, the count is doubled and the upper limit is allowed to be extended to 86400 seconds. The system also implements "automatic recovery" logic: if more than 2 hours have passed since the last rate limit, the backoff interval will be halved but not less than 300 seconds; if there are no rate limits in a whole round of operations and the user successfully joins the group, the interval will be further reduced while maintaining the safety threshold.

[0034] By constructing an adaptive exponential backoff mechanism and an asynchronous scheduling collaborative architecture, this approach completely solves the shortcomings of existing technologies, such as the inability to dynamically adjust request frequency due to "fixed-interval polling" and the long-term invalid waiting caused by "single backoff strategies." Dynamic adjustment significantly improves the request success rate compared to traditional solutions; for example, it maintains a high success rate even under a limit of >15 requests per hour. Simultaneously, by directly using the backoff interval as the asynchronous task suspension time, it dynamically adapts to the asynchronous scheduling mechanism in step S2, allowing I / O waiting resources during the backoff period to be reused, reducing resource consumption compared to independent backoff algorithms. Furthermore, the combination of an 86,400-second backoff cap and automatic recovery shortens the average backoff time and reduces data latency, resolving the data lag problem caused by the fixed 24-hour backoff in existing technologies.

[0035] Step 5: Use the asynchronous task scheduling mechanism to adjust the API request sending rhythm according to the current backoff interval, and write the groups that cannot be joined temporarily due to rate limiting into the rate limiting queue, calculate and record the time when the limit is lifted for each group; The rate-limiting queue is organized and retrieved using a multi-level dictionary structure nested by account and group URL. The asynchronous task scheduling mechanism is implemented based on Python's asyncio library and generates an independent coroutine task for each account.

[0036] Specifically: The rate-limiting queue data structure adopts a multi-level nested dictionary structure, organized by account and group URL, supporting time-complexity queries and updates. When the API returns FLOOD_WAIT, the system calculates the release time based on the wait time in the error, such as the current time + 1800 seconds, and stores it with dual indexes by account and URL. The asynchronous scheduling mechanism dynamically adjusts the task execution frequency based on the backoff interval: for example, when the backoff interval is 600 seconds, the task is executed once every 600 seconds, during which the CPU can handle other account tasks. This design solves the problem of "resource blocking caused by synchronous processing" in existing technologies—traditional solutions occupy thread resources while waiting, while asynchronous scheduling allows one CPU core to manage the backoff wait of 200 accounts simultaneously, increasing CPU utilization from 20% to 60%. In addition, accurate release time calculation avoids premature or late retries: existing technologies retry at fixed intervals, such as 10 minutes, which may trigger a second rate limit before the limit is lifted, while this mechanism significantly improves the accuracy of retry timing, reducing the number of rate limits from 47 times in 24 hours to 5 times, as shown in Table 1.

[0037] Step 6: When the limit is lifted, the rate-limiting queue automatically triggers a retry, resends the group to be retried to the group processing unit, and inserts a dynamic random delay based on the current number of consecutive rate-limiting attempts; the dynamic random delay satisfies the following conditions: delay=min(3.0+consecutive_floods×1.0,10.0); Here, `delay` represents a dynamic random delay, `consecutive_floods` represents the number of consecutively detected abnormal high-frequency operations, such as network flood attacks or API call abuse, and `min()` represents taking the minimum value. Before being inserted into the asynchronous task scheduling mechanism, the delay is perturbed by a millisecond-level random quantity, and the overall delay does not exceed 12 seconds.

[0038] Specifically, the dynamic random delay mechanism uses a base value of 3 seconds plus a linear increase of 1 second for each consecutive number of requests, superimposed with a random perturbation of ±500ms, to create an "irregular delay" similar to human operation. For example, after three consecutive rate limits, the delay is 3+3=6 seconds, which fluctuates between 5.5-6.5 seconds after the random perturbation. The upper limit of 12 seconds is based on Telegram's risk control threshold, because exceeding 12 seconds significantly reduces data real-time performance, while being below 12 seconds maintains efficiency while avoiding detection. This design simulates human operation patterns, such as random pauses during manual page turning, significantly increasing the probability that Telegram's risk control system will identify the request as "normal user behavior" compared to existing technologies.

[0039] Step 7: Initiate an independent heartbeat mechanism in parallel to maintain the activity of the session channel. The initial heartbeat interval is 300 seconds. If a heartbeat rate limiting error is detected, heartbeat backoff management is initiated. The maximum heartbeat backoff interval is 1800 seconds. The heartbeat status is fed back to the adaptive exponential backoff mechanism. The heartbeat mechanism maintains an independent heartbeat backoff status and records the number of consecutive errors, the time of the last error, and the time of the next retry.

[0040] Specifically: The heartbeat mechanism maintains connection activity by sending lightweight API requests, such as retrieving self-information. An initial interval of 300 seconds represents the optimal balance between Telegram session activity and API call frequency—tests show that intervals less than 300 seconds increase API consumption, while intervals greater than 300 seconds increase connection drop rates. When the heartbeat detects a rate-limiting error, an independent backoff is initiated, which is gentler than the main backoff and avoids connection failure due to excessively long heartbeat backoff times. Heartbeat status, such as the last success time and the number of errors, is fed back to the backoff mechanism in step S4 in real time. If the heartbeat fails three times consecutively, the main backoff interval automatically increases, improving system robustness.

[0041] By constructing an independent heartbeat backoff mechanism in conjunction with the MTProto long-connection collaborative architecture, the deficiency of "insufficient connection activity" in existing technologies is addressed: an independent heartbeat mechanism with an initial interval of 300 seconds is used to maintain session channel activity, and a dedicated backoff is initiated when a heartbeat rate limiting error is detected. API call quotas are more economical due to reduced overhead from reconnection. This mechanism avoids connection drops caused by 24-hour main operation backoff in existing technologies by using a heartbeat-specific backoff strategy independent of the main operation backoff, resulting in improved connection stability in actual tests. Simultaneously, in conjunction with the MTProto long connection in step S1 (which reduces connection reconstruction overhead and the heartbeat mechanism maintains connection activity), data interaction latency is reduced from average, achieving a low-latency effect that existing short-connection, heartbeat-free solutions cannot achieve.

[0042] Performance Testing and Effect Verification: To verify the technical effectiveness of the above steps, comparative tests were conducted on 50 / 100 / 300 groups for 24 hours, 7 days, and 5 days respectively. The results are shown in Tables 1-3. Table 1 compares the performance of different monitoring technologies under moderate speed-limited conditions (monitoring 50 groups, 24-hour test); Table 2 compares the test results for monitoring scenarios with different group sizes (7-day continuous test); and Table 3 compares the system performance under different speed-limited conditions (monitoring 100 groups, 5-day test). These tables objectively evaluate the technical advantages of the present invention. We conducted systematic comparative tests for different group sizes and speed-limited conditions. The following table shows a comparison of the adaptive exponential backoff scheme of the present invention with existing technical solutions in terms of key performance indicators.

[0043] Table 1: Performance comparison of different monitoring technologies under moderate speed limit environment (monitoring 50 groups, 24-hour test)

[0044] Table 2: Comparison of test results for monitoring scenarios of different group sizes (7-day continuous test)

[0045] Table 3: Comparison of system performance under different speed limit intensities (monitoring 100 groups, 5-day test)

[0046] The comparison data in Tables 1, 2, and 3, along with the performance curves, are presented above. Figure 3 , Figure 4 and Figure 5 The significant advantages of this invention under various conditions are clearly visible: 1. High efficiency and low latency: In all test scenarios, the success rate of requests using this invention is on average 123.4% higher than that of traditional solutions, and the average response latency is reduced by 65.9%. Especially in high-rate-limited environments, it can still maintain a success rate of 82.5%, while the traditional solution is only 18.4%.

[0047] 2. Significantly reduced resource consumption: The CPU utilization rate of this invention is reduced by an average of 46.5%, and memory consumption is reduced by 42.2%, enabling the system to monitor more groups under the same hardware conditions. In large-scale testing (300 groups), the resource consumption of this system is only 47.2% of that of traditional solutions.

[0048] 3. Significantly Reduced Rate Limit Triggers: Thanks to the intelligent backoff strategy, the number of rate limit triggers in this invention was only 10.6% of that of traditional solutions during a 24-hour test, significantly improving account security. This means that during long-term operation, this system can maintain stable data collection efficiency without performance degradation due to frequent rate limit triggers.

[0049] 4. Strong adaptability: As can be seen from the performance curve, the performance of traditional solutions drops sharply as the speed limit intensity increases, while the present invention can still maintain a relatively stable performance curve, demonstrating excellent environmental adaptability.

[0050] 5. Significant advantages in scalability: As the number of monitoring groups increases, the performance gap between this invention and traditional solutions gradually widens. In a large-scale test with 300 groups, the success rate of this invention (87.6%) is 4.11 times that of the traditional solution (21.3%), demonstrating its excellent scalability.

[0051] The test results above fully demonstrate that the innovative technical solutions of this invention, such as the adaptive exponential backoff algorithm, rate-limiting queue intelligent management, and asynchronous concurrent architecture, are not only theoretically sound but also achieve significant performance improvements in practical applications, providing an efficient, stable, and resource-friendly solution for Telegram group data collection.

[0052] Figure 6 This is a schematic diagram of an electronic device structure provided in one embodiment of the present invention. Figure 6 As shown, according to another aspect of the present invention, an electronic device 500 is also provided. The electronic device 500 may include one or more processors and one or more memories. The memories store computer-readable code that, when executed by the one or more processors, can perform the social media data collection method described above.

[0053] The method or system according to embodiments of the present invention can also be used by means of Figure 6 The architecture of the electronic device shown is used to implement this. For example... Figure 6 As shown, the electronic device 500 may include a bus 501, one or more CPUs 502, a read-only memory (ROM) 503, a random access memory (RAM) 504, a communication port 505 connected to a network, an input / output component 506, a hard disk 507, etc. The storage device in the electronic device 500, such as the ROM 503 or the hard disk 507, may store the social media data acquisition method provided by this invention. The social media data acquisition method may, for example, include: establishing and maintaining a long-lived MTProto connection with the Telegram platform, and outputting a session channel for subsequent data interaction; On the aforementioned session channel, an asynchronous task scheduling mechanism serializes API calls for each Telegram account, ensuring that only one API request is sent to the group processing unit at any given time. The group processing unit then calls the corresponding Telegram API based on the link type of the group to be joined. If the group is public, it calls JoinChannelRequest; if the group is private, it calls ImportChatInviteRequest. The API return information is then output to the error parsing unit. If the API return information is FLOOD_WAIT or HTTP429, an adaptive exponential backoff mechanism is activated. This mechanism establishes and maintains backoff status information. The backoff status information includes at least the number of consecutive rate-limiting attempts, the time of the most recent rate-limiting attempt, and the current backoff interval. The current backoff interval is fed back to the asynchronous task scheduling mechanism. The asynchronous task scheduling mechanism adjusts the API request sending rhythm based on the current backoff interval, and adds groups that cannot be joined due to rate limiting to a rate-limiting queue. The restriction lifting time for each group is calculated and recorded. When the restriction lifting time arrives, the rate-limiting queue automatically triggers a retry, resending the groups to be retried to the group processing unit, and inserting a dynamic random delay based on the current number of consecutive rate-limiting attempts. An independent heartbeat mechanism is initiated in parallel to maintain the activity of the session channel. The initial heartbeat interval is 300 seconds. If a heartbeat rate-limiting error is detected, heartbeat backoff management is initiated, with a maximum heartbeat backoff interval of 1800 seconds. The heartbeat status is fed back to the adaptive exponential backoff mechanism. Furthermore, the electronic device 500 may also include a user interface 508. Of course, Figure 6 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 6 One or more components in the illustrated electronic device.

[0054] Figure 7 This is a schematic diagram of a computer storage medium structure provided in one embodiment of the present invention. Figure 7 The diagram shows a computer storage medium 600 according to an embodiment of the present invention. The computer storage medium 600 stores computer-readable instructions. When executed by a processor, the computer-readable instructions can perform a method for collecting social media data according to an embodiment of the present invention, as described with reference to the above figures. The storage medium 600 includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0055] Furthermore, according to embodiments of the present invention, the process described in the above-mentioned flowchart can be implemented as a computer software program. For example, the present invention provides a non-transitory machine storage medium storing machine-readable instructions that can be executed by a processor to perform instructions corresponding to the method steps provided by the present invention, such as: establishing and maintaining an MTProto long connection with the Telegram platform, and outputting a session channel for subsequent data interaction; on the session channel, serializing API calls for each Telegram account through an asynchronous task scheduling mechanism to ensure that only one API request is sent to the group processing unit at any given time; the group processing unit calls the corresponding Telegram API according to the link type of the group to be joined, wherein if the group is a public group, JoinChannelRequest is called, and if the group is a private group, ImportChatInviteRequest is called, and the API return information is output to the error parsing unit to obtain the API return information; if the API If the returned information is FLOOD_WAIT or HTTP429, an adaptive exponential backoff mechanism is initiated. This mechanism establishes and maintains backoff status information, including at least the number of consecutive rate-limiting attempts, the most recent rate-limiting time, and the current backoff interval. This current backoff interval is then fed back to the asynchronous task scheduling mechanism. The asynchronous task scheduling mechanism adjusts the API request sending rhythm based on the current backoff interval, and groups temporarily unable to join due to rate limiting are added to a rate-limiting queue. The restriction release time for each group is calculated and recorded. When the restriction release time arrives, the rate-limiting queue automatically triggers a retry, resending the groups to be retried to the group processing unit, and inserting a dynamic random delay based on the current number of consecutive rate-limiting attempts. An independent heartbeat mechanism is initiated in parallel to maintain the activity of the session channel. The initial heartbeat interval is 300 seconds. If a heartbeat rate-limiting error is detected, heartbeat backoff management is initiated, with a maximum heartbeat backoff interval of 1800 seconds. The heartbeat status is then fed back to the adaptive exponential backoff mechanism. When this computer program is executed by the central processing unit (CPU), the functions defined in the method of this invention are performed.

[0056] The methods, apparatus, and devices of the present invention may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the method is for illustrative purposes only, and the steps of the method of the present invention are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present invention may also be implemented as a program recorded on a recording medium, the program comprising machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers recording media storing programs for performing the method according to the present invention.

[0057] In addition, the parts of the technical solutions provided in the embodiments of the present invention that are consistent with the implementation principles of the corresponding technical solutions in the prior art have not been described in detail, so as to avoid excessive elaboration.

[0058] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for collecting social media data, characterized in that, Includes the following steps: Establish and maintain a long-lived MTProto connection with the Telegram platform, and output a session channel for subsequent data interaction; On the aforementioned session channel, an asynchronous task scheduling mechanism is used to serialize the API calls of each Telegram account, ensuring that only one API request is sent to the group processing unit at any given time. The group processing unit calls the corresponding Telegram API according to the link type of the group to be joined. If the group is a public group, it calls JoinChannelRequest; if the group is a private group, it calls ImportChatInviteRequest. The API return information is then output to the error parsing unit to obtain the API return information. If the API returns FLOOD_WAIT or HTTP429, an adaptive exponential backoff mechanism is initiated. The backoff status information is established and maintained through the adaptive exponential backoff mechanism. The backoff status information includes at least the number of consecutive rate limits, the time of the most recent rate limit, and the current backoff interval. The current backoff interval is then fed back to the asynchronous task scheduling mechanism. The asynchronous task scheduling mechanism is used to adjust the API request sending rhythm according to the current backoff interval, and groups that cannot be joined temporarily due to rate limiting are written into the rate limiting queue. The time for lifting the limit for each group is calculated and recorded. When the limit is lifted, the rate limit queue automatically triggers a retry, resends the group to be retried to the group processing unit, and inserts a dynamic random delay based on the current number of consecutive rate limits. An independent heartbeat mechanism is initiated in parallel to maintain the activity of the session channel. The initial heartbeat interval is 300 seconds. If a heartbeat rate limit error is detected, heartbeat backoff management is initiated. The maximum heartbeat backoff interval is 1800 seconds, and the heartbeat status is fed back to the adaptive exponential backoff mechanism.

2. The method according to claim 1, characterized in that, The rate-limiting queue is organized and retrieved using a multi-level dictionary structure nested by account and group URL.

3. The method according to claim 1, characterized in that, The dynamic random delay satisfies delay=min(3.0+consecutive_floods×1.0,10.0), where consecutive_floods represents the number of consecutively detected abnormal high-frequency operations; min( , ) represents the minimum value function; and delay represents the dynamic random delay.

4. The method according to claim 1, characterized in that, The asynchronous task scheduling mechanism is implemented based on Python's asyncio library and generates independent coroutine tasks for each account.

5. The method according to claim 1, characterized in that, The error parsing unit This includes using regular expressions to extract rate-limited waiting times and using a tree-structured decision process to classify and handle different error types.

6. The method according to claim 1, characterized in that, The establishment and maintenance of backoff state information through the adaptive exponential backoff mechanism includes: When the first speed limit event occurs, the backoff interval is initialized to 300 seconds; when a FLOOD_WAIT error is detected, a new backoff interval is calculated according to backoff_interval=min(300×2^consecutive_floods,86400); backoff_interval represents the current backoff interval time that should be applied, and consecutive_floods represents the number of consecutive FLOOD_WAIT errors; When an HTTP 429 error is detected, the number of consecutive rate-limiting attempts will be doubled, and the maximum backoff interval will be 86,400 seconds. If the time since the last speed limit exceeds 2 hours, the current retreat interval will be automatically halved but not less than 300 seconds. When an unlimited-speed API call is successfully completed, the current backoff interval is reduced by 10%.

7. The method according to claim 1, characterized in that, The dynamic random delay time is perturbed by a random amount at the millisecond level before being inserted into the asynchronous task scheduling mechanism, and the overall delay does not exceed 12 seconds.

8. A system for collecting social media data, said system being used to perform any one of the methods of claims 1 to 8, characterized in that, include: The connection module is used to establish and maintain a long MTProto connection with the Telegram platform and output a session channel for subsequent data interaction. The control module is used to serialize the API calls of each Telegram account on the session channel through an asynchronous task scheduling mechanism, ensuring that only one API request is sent to the group processing unit at any given time. The parsing module is used by the group processing unit to call the corresponding Telegram API according to the link type of the group to be joined. If the group is a public group, JoinChannelRequest is called, and if the group is a private group, ImportChatInviteRequest is called. The API return information is then output to the error parsing unit to obtain the API return information. The backoff module is used to initiate an adaptive exponential backoff mechanism if the API returns FLOOD_WAIT or HTTP429. The adaptive exponential backoff mechanism establishes and maintains backoff status information, which includes at least the number of consecutive rate limits, the time of the most recent rate limit, and the current backoff interval. The current backoff interval is then fed back to the asynchronous task scheduling mechanism. The evaluation module is used to adjust the API request sending rhythm based on the current backoff interval using an asynchronous task scheduling mechanism, and write groups that cannot be joined temporarily due to rate limiting into the rate limiting queue, and calculate and record the time when the limit is lifted for each group; The processing module is used to automatically trigger a retry when the limit is lifted, resend the group to be retried to the group processing unit, and insert a dynamic random delay based on the current number of consecutive rate limits. The feedback module is used to initiate an independent heartbeat mechanism in parallel to maintain the activity of the session channel. The initial heartbeat interval is 300 seconds. If a heartbeat rate limit error is detected, the system enters heartbeat backoff management with an upper limit of 1800 seconds. The heartbeat status is then fed back to the adaptive exponential backoff mechanism.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the method for collecting social media data as described in any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores a computer program adapted for loading by a processor to perform the steps of the method for collecting social media data as described in any one of claims 1-7.

Images