Data processing method, platform and electronic device based on big data

Abstract

The application relates to the technical field of communication data processing, in particular to a data processing method and platform based on big data and electronic equipment, which comprises the following steps: collecting real-time state data of a plurality of user equipment in a target communication cell in a non-periodic triggering mode, and structurally packaging the real-time state data to form an initial data stream; performing dynamic quality evaluation and abnormality repair on the initial data stream to generate a purified data sequence, and synchronously extracting statistical features of the purified data sequence; constructing a statistical feature trend curve based on the statistical features of the purified data sequence, performing segmented fluctuation analysis, and setting a to-be-verified label based on the fluctuation analysis result; in response to the to-be-verified label, performing space-time clustering analysis on original multi-domain feature vectors in a time period corresponding to the to-be-verified label, identifying a high-load interference area, and generating a resource reallocation scheme. The application effectively improves the efficiency of communication data processing.

Description

Technical Field

[0001] This invention relates to the field of communication data processing technology, and in particular to a data processing method, platform and electronic device based on big data. Background Technology

[0002] In current communication networks, traditional data processing methods often employ fixed-period collection of user equipment status data. This approach has significant shortcomings when facing ultra-dense 5G network scenarios. Fixed-period collection cannot be dynamically adjusted according to the actual network load, resulting in unnecessary resource waste under low load conditions and potential omission of critical data during sudden high load surges. Consequently, data collection is inefficient and lacks specificity, making it difficult to meet the real-time requirements of complex network environments.

[0003] Furthermore, existing technologies also have shortcomings in data quality assurance and resource scheduling. Anomaly detection methods are limited, and repair mechanisms lack intelligence, easily affecting the accuracy of subsequent analysis. Simultaneously, resource allocation strategies often lack comprehensive consideration of spatiotemporal interference characteristics, failing to accurately identify high-load interference areas and implement effective resource reallocation. This makes it difficult to effectively alleviate network congestion problems, thus limiting overall system performance and user experience. Summary of the Invention

[0004] The purpose of this invention is to provide a data processing method, platform, and electronic device based on big data to solve at least one of the problems existing in the prior art.

[0005] To achieve the above objectives, according to one aspect of this application, the present invention provides a data processing method based on big data, comprising:

[0006] Real-time status data of multiple user equipment in the target communication cell are collected in an aperiodic triggering manner, and the real-time status data is encapsulated in a structured manner to form an initial data stream;

[0007] Dynamic quality assessment and anomaly repair are performed on the initial data stream to generate a cleaned data sequence, and the statistical characteristics of the cleaned data sequence are extracted simultaneously.

[0008] A statistical characteristic trend curve is constructed based on the statistical characteristics of the purified data sequence, and a segmented fluctuation analysis is performed. A label to be verified is set based on the fluctuation analysis results.

[0009] In response to the label to be verified, spatiotemporal clustering analysis is performed on the original multi-domain feature vectors within the time period corresponding to the label to be verified, high-load interference regions are identified, and resource reallocation schemes are generated.

[0010] Optionally, the sliding window is a variable-length adaptive window, and its window length L is determined by the following formula:

[0011] L=max{Lmin,floor(Nact / Mt×200)};

[0012] Where Lmin is the predefined minimum window length, Nact is the number of user devices currently active, Mt is the total number of registered users in the target communication cell, and floor() is the floor function;

[0013] The stepping strategy is set as follows: the step size of each sliding window movement is equal to one-third of the current window length, and the step size is rounded up.

[0014] Optionally, when calculating multi-dimensional local outlier factors and identifying anomalous data record units within each sliding window, a three-dimensional local outlier index is set: the spatial proximity outlier factor, temporal consistency outlier factor, and conversational semantic outlier factor of the i-th data record unit within the sliding window;

[0015] If the spatial proximity outlier factor of the i-th data record unit within the sliding window is greater than the first outlier threshold and the temporal consistency outlier factor is greater than the second outlier threshold, or the session semantic outlier factor is greater than the third outlier threshold, then the current data record unit is determined to be an abnormal data record unit.

[0016] Optionally, if neither of the two data recording units before and after the abnormal data recording unit is an abnormal data recording unit, then the Lagrange interpolation method is used to calculate the intermediate interpolation vector based on the multi-domain feature vectors of the two data recording units before and after to replace the multi-domain feature vector of the abnormal data recording unit.

[0017] If a continuous sequence of abnormal data record units exists, all data record units in the continuous sequence of abnormal data record units are discarded and recorded as data holes. Based on the multi-domain feature vectors of the last normal data record unit before and the first normal data record unit after the data hole, linear decay weighted interpolation is performed to repair the multi-domain feature vectors of the data record units within the data hole.

[0018] The data record units created after the repair are used as the purified data sequence.

[0019] Optionally, when extracting statistical features from the purified data sequence, the statistical features include: mean offset and variance convergence.

[0020] Optionally, the mean offset Δμ and variance convergence ρ corresponding to the continuous sliding windows are arranged in time order to construct the mean offset trend curve Cμ(t) and variance convergence trend curve Cρ(t), respectively, where t is the sliding window number;

[0021] Perform the following piecewise processing on Cμ(t) and Cρ(t):

[0022] Cμ(t) and Cρ(t) are divided into several curve segments with a fixed time length T, and the sample variance σμ within each curve segment is calculated. 2 and σρ 2 ;

[0023] Detect the peaks in each curve segment that exceed the mean ± 2 standard deviations of that curve segment, and count the proportion of abnormal peaks to the total number of peaks in that segment. Record the results as rμ and rρ.

[0024] If any of the following conditions are met for each curve segment, then the time domain corresponding to that curve segment is determined to be an abnormal fluctuation period:

[0025] Condition 1: σμ 2 Greater than Vμmax or σρ 2 Greater than Vρmax;

[0026] Condition 2: rμ is greater than Rmax or rρ is greater than Rmax;

[0027] Where Vμmax and Vρmax are the offset variance threshold and convergence variance threshold, respectively, and Rμmax is the abnormal peak ratio threshold.

[0028] A verification label is set for the set of data recording units corresponding to all sliding windows during the abnormal fluctuation period, and no processing is performed on the remaining sliding windows.

[0029] Optionally, the spatiotemporal clustering analysis includes:

[0030] Geographic location estimates and multi-domain feature vectors for each data record unit within the corresponding time period of the label to be verified are extracted, and the DBSCAN density clustering algorithm is used to divide the data record units into several clusters, where:

[0031] If the average HARQ failure rate within a cluster is greater than 5% and the average signal-to-noise ratio within a cluster is less than 10dB, the cluster is identified as a high-load interference cluster.

[0032] When generating a resource reallocation scheme, all user equipment in the high-load interference cluster is added to the queue of users to be transferred, and arranged in descending order according to the emergency transfer index of the user equipment.

[0033] Obtain the PRB occupancy bitmap at the current time from the base station scheduler, extract all consecutive idle PRB segments, and record the starting index, length, center frequency, and bandwidth of each consecutive idle PRB segment;

[0034] For the top M user devices in the user transfer queue, sorted in descending order, select and allocate consecutive idle PRB segments that meet the needs of each user device based on the starting index, length, center frequency, and bandwidth of the consecutive idle PRB segments.

[0035] Optionally, the method further includes: performing resource reallocation and implementing closed-loop verification and rollback mechanisms;

[0036] The system sends resource reconfiguration commands to user equipment through radio resource control command signals and resource reallocation schemes, and starts an 800ms verification window after the handover is completed. The average HARQ failure rate F of the whole cell and the average signal-to-noise ratio change ΔSNR of the frequency-hopped user are collected during the verification period.

[0037] The closed-loop verification process includes:

[0038] Set the verification pass and rollback conditions in the verification window;

[0039] When the verification conditions are met, the resource reallocation is deemed successful and the reallocation state is exited.

[0040] If the rollback conditions are met, and resource reallocation fails, a rollback is triggered:

[0041] The system sends a command to the user equipment that has been issued a resource reconfiguration instruction to return the original occupied PRB segment, determines that the resource reconfiguration has failed, and adjusts the emergency transfer index calculation process for the user equipment.

[0042] According to another aspect of this application, a big data-based data processing platform is provided, comprising:

[0043] The initial data stream generation unit is used to collect real-time status data of multiple user equipment in the target communication cell in an aperiodic triggering manner, and to encapsulate the real-time status data in a structured manner to form an initial data stream.

[0044] The statistical feature extraction unit is used to perform dynamic quality assessment and anomaly repair on the initial data stream, generate a cleaned data sequence, and simultaneously extract the statistical features of the cleaned data sequence.

[0045] The label setting unit is used to construct a statistical feature trend curve based on the statistical features of the cleaned data sequence, perform segmented fluctuation analysis, and set the label to be verified based on the fluctuation analysis results.

[0046] The scheme generation unit is used to perform spatiotemporal clustering analysis on the original multi-domain feature vectors of the label to be verified within the corresponding time period, identify high-load interference areas, and generate resource reallocation schemes in response to the label to be verified.

[0047] The rollback update unit is used to perform resource reallocation and execute closed-loop verification and rollback mechanisms.

[0048] According to another aspect of this application, an electronic device is provided, comprising:

[0049] One or more processors;

[0050] Storage device for storing one or more programs;

[0051] When the one or more programs are executed by the one or more processors, the one or more processors implement the big data-based data processing method.

[0052] Compared with the prior art, the beneficial effects of the present invention are as follows: by using a non-periodic triggered data acquisition mechanism, combined with dynamic quality assessment and anomaly repair technology, the resource waste and data loss problems caused by the traditional fixed-period acquisition method are effectively solved. The acquisition frequency can be dynamically adjusted according to the network load, capturing key data in a timely manner under high load and reducing system overhead under low load. At the same time, through multi-dimensional local outlier detection and intelligent interpolation repair technology, the integrity and accuracy of the data are ensured, providing a high-quality data foundation for subsequent analysis.

[0053] During the data analysis and processing phase, the solution employs a combination of trend curve construction and segmented fluctuation analysis. It accurately identifies abnormal fluctuation periods using multi-dimensional indicators such as mean offset and variance convergence, and precisely locates high-load interference areas using spatiotemporal clustering analysis. Based on the emergency transfer index of user equipment and PRB occupancy bitmap, it intelligently generates the optimal resource reallocation scheme, realizing refined resource scheduling based on actual network conditions. This effectively alleviates interference problems in high-load areas and improves the overall network performance.

[0054] A robust closed-loop verification and rollback mechanism is established. Reconfiguration commands are issued via radio resource control commands, and a verification window is activated after the handover to monitor key indicators. Clear verification pass and rollback conditions are set to ensure the effectiveness of resource reallocation and network stability. When reallocation fails, the system can promptly roll back to the original state and dynamically adjust the emergency transfer index calculation weights, avoiding performance degradation caused by erroneous reallocation and improving system reliability and robustness. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating the data processing method based on big data in this embodiment.

[0057] Figure 2 This is a schematic diagram of the structure of the big data-based data processing platform provided in this embodiment.

[0058] Figure 3 This is a schematic diagram of the structure of the electronic device provided in this embodiment. Detailed Implementation

[0059] To more clearly illustrate the present invention, the following description, in conjunction with preferred embodiments and accompanying drawings, further explains the invention. Similar components in the drawings are indicated by the same reference numerals. Those skilled in the art should understand that the specific description below is illustrative rather than restrictive and should not be construed as limiting the scope of protection of the present invention.

[0060] It should be noted that although the terms first, second, third, etc., may be used in the embodiments of this application for description, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, without departing from the scope of the embodiments of this application, first can also be referred to as second, and similarly, second can also be referred to as first.

[0061] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0062] Specifically, the big data-based data processing method, platform, and electronic device described in this application are applied to intelligent load balancing scenarios in communication networks; specifically, they are applied to an ultra-dense networking scenario in a fifth-generation mobile communication system, in which there are more than 200 simultaneously active user devices within the coverage area of ​​a single macro cell, and the mobility characteristics of the user devices exhibit a "group-style sudden movement" pattern; the method aims to suppress the congestion and collapse of communication links through a multi-dimensional data-driven dynamic resource reallocation mechanism.

[0063] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely for explaining the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0064] To apply the above-mentioned application scenarios, this application provides a data processing method based on big data, the flowchart of which can be found in the document. Figure 1 As shown, it includes:

[0065] Step S1: Collect real-time status data of multiple user devices in the target communication cell in an aperiodic triggering manner, and encapsulate the real-time status data in a structured manner according to device type and service priority to form an initial data stream.

[0066] The real-time status data includes data sources uploaded by each user device.

[0067] Specifically, the non-periodic triggering method refers to a system that does not rely on a fixed time interval. Instead, data collection is initiated when either the physical resource block occupancy rate within the cell or the average number of hybrid automatic repeat request attempts within 100ms exceeds its corresponding trigger threshold. When neither indicator exceeds its respective threshold, the system does not initiate data collection and remains silent. In particular, in this embodiment, the trigger threshold for the physical resource block occupancy rate within the cell is set to 78%, and the trigger threshold for the average number of hybrid automatic repeat request attempts is set to 2.3 times.

[0068] The process of structuring and encapsulating the real-time status data further includes:

[0069] Data sources from the same user equipment within a transmission time interval are packaged into a data record unit according to a predefined data frame format. Each data record unit contains a timestamp field, a user equipment temporary identifier field, and a multi-domain feature vector.

[0070] A data block is formed by combining 32 consecutively generated data record units, and a verification header is attached to the beginning of the data block. The verification header stores the hash verification value of all data record units in the data block, as well as the number of data record units in the data block.

[0071] The data block is used as the smallest transmission unit of the initial data stream and pushed to the downstream data processing queue.

[0072] The multi-domain feature vector field includes four-dimensional sub-vectors, specifically [signal-to-noise ratio, channel quality indicator, number of bytes occupied in the buffer, and HARQ failure rate]. The transmission time interval is the minimum time unit required for a complete data block transmission between the base station and the user equipment. It is a fixed time tick during data transmission, specifically 1ms in this application.

[0073] Specifically, real-time status data of user equipment within the target communication cell is collected through a non-periodic triggering method and then structured and encapsulated to form an initial data stream. This non-periodic triggering mechanism can dynamically adjust the data collection frequency according to network load, avoiding resource waste or data loss problems caused by fixed-periodic collection. It captures critical data promptly under high load and reduces system overhead under low load, effectively improving the targeting and efficiency of data collection and providing a high-quality raw data foundation for subsequent data processing.

[0074] Please continue reading. Figure 1 As shown, the data processing method based on big data also includes:

[0075] Step S2: Perform dynamic quality assessment and anomaly repair on the initial data stream to generate a cleaned data sequence, and simultaneously extract the statistical features of the cleaned data sequence; when performing dynamic quality assessment and anomaly repair on the initial data stream, the data blocks need to be decoded to obtain each data record unit.

[0076] This includes performing dynamic quality assessment and anomaly repair on the initial data stream, including:

[0077] Step S21: Determine the window length and stepping strategy of the sliding window; the length of the sliding window refers to the number of data record units it contains.

[0078] Step S22: Calculate the multi-dimensional local outlier factor within each sliding window and identify anomalous data record units.

[0079] Specifically, the sliding window is a variable-length adaptive window, and its window length L is determined by the following formula:

[0080] L=max{Lmin,floor(Nact / Mt×200)};

[0081] Where Lmin is the predefined minimum window length, Nact is the number of user devices currently active, Mt is the total number of registered users in the target communication cell, and floor() is the floor function;

[0082] The stepping strategy is set as follows: the step size of each sliding window movement is equal to one-third of the current window length, and the step size is rounded up.

[0083] Preferably, the predefined minimum window length in this application is 20 data record units, and 200 represents the data record units generated by 200 transmission time intervals.

[0084] Specifically, when calculating the multi-dimensional local outlier factor and identifying anomalous data record units within each sliding window, a three-dimensional local outlier index is set:

[0085] Calculate the Mahalanobis distance between the multi-domain feature vector of the i-th data record unit in the sliding window and the multi-domain feature vectors of the other data record units in the sliding window, and sort the Mahalanobis distances in ascending order, then take the arithmetic mean of the first K distances as the spatial neighbor outlier factor of the i-th data record unit in the sliding window.

[0086] The multi-domain feature vector of the i-th data recording unit is subjected to difference operation with the multi-domain feature vector of the (i-1)-th data recording unit and the multi-domain feature vector of the (i+1)-th data recording unit respectively to obtain the Euclidean distance between the two difference vectors. The harmonic average of the two Euclidean distances is taken as the temporal consistency outlier factor of the i-th data recording unit within the sliding window.

[0087] Retrieve the multi-domain feature vectors of all data record units of the same user equipment in the past 500 milliseconds, and calculate their average value as the historical mean vector. Then, calculate the magnitude of the relative deviation ratio vector between the i-th data record unit and the historical mean vector, and use the result as the session semantic outlier factor of the i-th data record unit within the sliding window.

[0088] If the spatial proximity outlier factor of the i-th data record unit within the sliding window is greater than the first outlier threshold and the temporal consistency outlier factor is greater than the second outlier threshold, or the session semantic outlier factor is greater than the third outlier threshold, then the current data record unit is determined to be an abnormal data record unit.

[0089] Preferably, the first outlier threshold, the second outlier threshold, and the third outlier threshold are determined by pre-collecting historical data for the target communication cell under its typical interference environment and calculating the 90th percentile of the above three outlier factors. In this embodiment, after offline calibration, the first outlier threshold is set to 1.8, the second outlier threshold is set to 2.1, and the third outlier threshold is set to 0.65. The typical interference environment is set to a neighboring cell interference intensity range of -85dBm to -75dBm; at the same time, the value of K is set to min(5, floor(L / 2)).

[0090] Specifically, the anomaly repair process described in this application can be implemented through the following steps:

[0091] If neither of the two data recording units before and after the abnormal data recording unit is an abnormal data recording unit, then the Lagrange interpolation method is used to calculate the intermediate interpolation vector based on the multi-domain feature vectors of the two data recording units before and after the abnormal data recording unit to replace the multi-domain feature vector of the abnormal data recording unit.

[0092] If a continuous sequence of anomalous data record units exists, all data record units in that sequence are discarded and designated as "data holes". Based on the multi-domain feature vectors x(t0) and x(t1) of the last normal data record unit before and the first normal data record unit after the data hole, linear decay weighted interpolation is performed, calculated using the following formula:

[0093] x(i)'=(1-(i-t0) / (t1-t0))×x(t0)+(i-t0) / (t1-t0)×x(t1);

[0094] Where x(i)' is the multi-domain feature vector of the i-th data record unit to be repaired in the sliding window within the data hole, and t0 and t1 are the numerical positions of the last normal data record unit before the data hole and the first normal data record unit after the data hole within the sliding window, respectively.

[0095] The data record units created after the repair are used as the purified data sequence.

[0096] Specifically, when extracting the statistical features of the purified data sequence, the statistical features include: mean offset and variance convergence.

[0097] The mean offset is defined as the difference between the arithmetic mean of the signal-to-noise ratios of all data record units in the cleaned data sequence of the current sliding window and the arithmetic mean of the signal-to-noise ratios of the cleaned data sequence of the previous sliding window, and then divided by the absolute value of the arithmetic mean of the signal-to-noise ratios of the cleaned data sequence of the previous sliding window.

[0098] The variance convergence is defined as the square root of the ratio of the variance of the signal-to-noise ratio of all data record units in the cleaned data sequence of the current sliding window to the variance of the signal-to-noise ratio of the cleaned data sequence of the previous sliding window.

[0099] Specifically, dynamic quality assessment and anomaly repair are performed on the initial data stream to generate a cleaned data sequence and extract statistical features. A variable-length adaptive sliding window and multi-dimensional local outlier detection methods are employed to accurately identify anomalous data records. Repair techniques such as Lagrange interpolation and linear decay weighted interpolation are used to effectively restore data integrity. This method not only improves data quality but also preserves the time-series characteristics of the data, providing a reliable data foundation for subsequent analysis.

[0100] Please continue reading. Figure 1 As shown, the data processing method based on big data also includes:

[0101] S3 constructs a statistical characteristic trend curve based on the statistical characteristics of the purified data sequence, performs segmented fluctuation analysis, and sets a label to be verified based on the fluctuation analysis results.

[0102] Specifically, the mean offset Δμ and variance convergence ρ corresponding to the continuous sliding window are arranged in chronological order to construct the mean offset trend curve Cμ(t) and variance convergence trend curve Cρ(t), respectively, where t is the sliding window number;

[0103] Perform the following piecewise processing on Cμ(t) and Cρ(t):

[0104] Cμ(t) and Cρ(t) are divided into several curve segments with a fixed time length T, and the sample variance σμ within each curve segment is calculated. 2 and σρ 2 ;

[0105] Detect the peaks in each curve segment that exceed the mean ± 2 standard deviations of that curve segment, and count the proportion of abnormal peaks to the total number of peaks in that segment. Record the results as rμ and rρ.

[0106] If any of the following conditions are met for each curve segment, then the time domain corresponding to that curve segment is determined to be an abnormal fluctuation period:

[0107] Condition 1: σμ 2 Greater than Vμmax or σρ 2 Greater than Vρmax;

[0108] Condition 2: rμ is greater than Rmax or rρ is greater than Rmax;

[0109] Where Vμmax and Vρmax are the offset variance threshold and convergence variance threshold, respectively, and Rμmax is the abnormal peak ratio threshold.

[0110] A verification label is set for the set of data recording units corresponding to all sliding windows during the abnormal fluctuation period, and no processing is performed on the remaining sliding windows.

[0111] Specifically, Vμmax, Vρmax, and Rmax mentioned in this application are determined in advance through offline analysis of the target communication cell. In this application, they are all set by taking the 90th percentile of historical data. In this embodiment, Vμmax, Vρmax, and Rmax are 0.15, 0.22, and 0.25, respectively.

[0112] Specifically, a trend curve is constructed based on the statistical characteristics of the purified data sequence, and segmented fluctuation analysis is performed, with labels to be verified. This step accurately identifies abnormal fluctuation periods by analyzing the trend changes of mean offset and variance convergence, combined with indicators such as sample variance and the proportion of abnormal peaks, effectively distinguishing between normal and abnormal fluctuations, avoiding misjudgments and omissions, and improving the targeting and accuracy of subsequent resource reallocation.

[0113] Please continue reading. Figure 1 As shown, the data processing method based on big data also includes:

[0114] Step S4: In response to the label to be verified, perform spatiotemporal clustering analysis on the original multi-domain feature vector within the time period corresponding to the label to be verified, identify high-load interference regions, and generate a resource reallocation scheme.

[0115] The spatiotemporal clustering analysis includes:

[0116] Geographic location estimates and multi-domain feature vectors for each data record unit within the corresponding time period of the label to be verified are extracted, and the DBSCAN density clustering algorithm is used to divide the data record units into several clusters, where:

[0117] If the average HARQ failure rate within a cluster is greater than 5% and the average signal-to-noise ratio within a cluster is less than 10dB, the cluster is identified as a high-load interference cluster.

[0118] Record the centroid location and the range of physical resource blocks covered by each cluster.

[0119] Specifically, the geographical location estimate of the data recording unit extracted in this application can be approximated by the angle of arrival or the sector identifier of the target communication cell. The implementation process is a technical means well known to those skilled in the art and will not be described in detail in this application.

[0120] The generated resource reallocation scheme includes:

[0121] All user equipment (UEs) within the high-load interference cluster are added to the queue of users to be transferred, and arranged in descending order of their emergency transfer index. The emergency transfer index of each UE is calculated using the following formula:

[0122] E(j,k) = 0.5 × HARQ failure rate normalized value (j,k) + 0.3 × buffer occupancy normalized value (j,k) + 0.2 × (1 − dwell time normalized value (j,k));

[0123] Obtain the PRB occupancy bitmap at the current time from the base station scheduler, extract all consecutive idle PRB segments, and record the starting index, length, center frequency, and bandwidth of each consecutive idle PRB segment;

[0124] For the top M user devices in the user transfer queue, sorted in descending order, select the consecutive idle PRB segments that meet the bandwidth requirements and have the lowest frequency adjustment cost for allocation.

[0125] Where k is the cluster number obtained by clustering, j is the user equipment number, and the dwell time normalized value (j,k) is a dimensionless value normalized to the dwell time (j,k). The dwell time (j,k) is: when the k-th cluster is a high-load interference cluster, the continuous dwell time of the j-th user equipment corresponding to the data recording unit in the cluster within the target communication cell. Similarly, the HARQ failure rate normalized value (j,k) and the buffer occupancy normalized value (j,k) are both dimensionless values ​​normalized to the HARQ failure rate (j,k) and the number of bytes occupied in the buffer (j,k). The continuous idle PRB segment is an idle PRB segment with a continuous length of greater than or equal to 5 PRBs. 0.5, 0.3, and 0.2 are the weights of the HARQ failure rate normalized value (j,k), the buffer occupancy normalized value (j,k), and the dwell time normalized value (j,k), respectively.

[0126] Specifically, when allocating consecutive idle PRB segments that meet bandwidth requirements and have the lowest frequency adjustment cost in this application, the following are included:

[0127] Obtain the central PRB index fc and required bandwidth br of its current user equipment;

[0128] From all available PRB segments, a candidate segment set S that meets the bandwidth constraint is selected: if S is empty, the user equipment is skipped; if S is not empty, the cost index C(s) of each available PRB segment in the candidate segment set S is calculated, s∈S, and C(s) is set as a1×|fc-fc(s)| / NP+a2×(1-SNR(s) / SNRmax), where fc(s) is the starting index of the s-th available PRB segment in the candidate segment set S, NP is the total number of PRBs in the target communication cell, SNR(s) is the historical average signal-to-noise ratio of the s-th available PRB segment in the candidate segment set S, SNRmax is the maximum signal-to-noise ratio that can be achieved in the target communication cell, and a1 and a2 are weighting coefficients and a1+a2=1;

[0129] Select the free PRB segment corresponding to min{C(s)} as the allocated free PRB segment for the user equipment;

[0130] Update the PRB occupancy bitmap in the base station scheduler.

[0131] M represents the number of users to be transferred. It is determined as follows: estimate the minimum number of PRBs required by each user device in the queue of user devices arranged in descending order, and start accumulating the required PRBs from the head of the queue of user devices arranged in descending order until the accumulated required PRBs first exceed the total number of PRBs. The number of users accumulated at this time is M. If the sum of the demands of all users in the queue is less than the total number of PRBs, then M is the actual length of the queue. The total number of PRBs is the sum of the number of PRBs in all consecutive idle PRB segments.

[0132] This embodiment provides a simple normalization method. The normalized value of HARQ failure rate is set as min(HARQ failure rate (j,k) / maximum tolerance failure rate,1), with the maximum tolerance failure rate set to 10%. The normalized value of buffer usage is set as min(number of bytes occupied in the buffer (j,k) / maximum device buffer capacity,1), where the maximum device buffer capacity is reported by the user equipment. If the user equipment does not report its maximum device buffer capacity, the 90th percentile of the maximum device buffer capacity reported by all user equipment in the target communication cell is taken as the unified benchmark. The normalized value of dwell time is set as min(dwell time (j,k) / reference dwell time,1), with the reference dwell time set to 1800 seconds.

[0133] Specifically, in response to the tags to be verified, spatiotemporal clustering analysis is performed to identify high-load interference areas and generate resource reallocation schemes. Utilizing the DBSCAN density clustering algorithm, combined with geographic location estimation and multi-domain feature vectors, high-load interference clusters are accurately identified. By calculating the emergency transfer index of user equipment and prioritizing them, and combining this with the PRB occupancy bitmap, the optimal idle resource segment is selected for allocation. This achieves intelligent resource scheduling based on actual network conditions, alleviating interference problems in high-load areas.

[0134] Please continue reading. Figure 1 As shown, the data processing method based on big data also includes:

[0135] Step S5: Perform resource reallocation and execute closed-loop verification and rollback mechanism.

[0136] Specifically, a resource reconfiguration command is sent to the user equipment through the radio resource control command signal and the resource reallocation scheme to switch the user equipment to the continuous idle PRB segment allocated in step S4. After the handover is completed, an 800ms verification window is started, and the average HARQ failure rate F of the whole cell and the average signal-to-noise ratio change ΔSNR of the frequency-hopping user are collected with a verification period of 100ms.

[0137] The closed-loop verification process includes,

[0138] Verification pass conditions: F is less than 2% and ΔSNR is greater than 2dB for 3 consecutive verification cycles within the verification window; when the verification pass conditions are met, the resource reallocation is determined to be successful and the reallocation state is exited.

[0139] Rollback conditions: If, within the verification window, F occurs twice consecutively, exceeding 1.2 times the average HARQ failure rate of the target communication cell before resource reallocation, or ΔSNR is less than -1dB;

[0140] If the rollback conditions are met, and resource reallocation fails, a rollback is triggered:

[0141] For user equipment that has been issued a resource reconfiguration command, send a command to return the original occupied PRB segment, determine that the resource reconfiguration has failed, and increase the weight of the user equipment's HARQ failure rate normalization value (j,k) by 15%, and the increased weight of the HARQ failure rate normalization value (j,k) is less than 1.

[0142] Specifically, if the average HARQ failure rate F of the entire cell and the average signal-to-noise ratio change ΔSNR of the frequency-hopping user do not fall into either of the two situations mentioned above, the verification period will be extended by 400ms to continue monitoring. If the extended verification period still does not fall into either of the two situations mentioned above, the resource reallocation is determined to be successful and the reallocation state is exited.

[0143] Specifically, resource reallocation is performed with a closed-loop verification and rollback mechanism. Reconfiguration commands are issued via radio resource control commands, and a verification window is activated after the handover to monitor key indicators. Verification pass and rollback conditions are set to ensure the effectiveness of resource reallocation. When reallocation fails, the system can promptly roll back to the original state and adjust the emergency transfer index calculation weights. This ensures network stability, avoids performance degradation caused by erroneous reallocation, and improves the overall system reliability.

[0144] Please see Figure 2 As shown, it is a schematic diagram of the structure of the big data-based data processing platform provided in this application, including:

[0145] The initial data stream generation unit is used to collect real-time status data of multiple user equipment in the target communication cell in an aperiodic triggering manner, and to encapsulate the real-time status data in a structured manner to form an initial data stream.

[0146] The statistical feature extraction unit is used to perform dynamic quality assessment and anomaly repair on the initial data stream, generate a cleaned data sequence, and simultaneously extract the statistical features of the cleaned data sequence.

[0147] The label setting unit is used to construct a statistical feature trend curve based on the statistical features of the cleaned data sequence, perform segmented fluctuation analysis, and set the label to be verified based on the fluctuation analysis results.

[0148] The scheme generation unit is used to perform spatiotemporal clustering analysis on the original multi-domain feature vectors of the label to be verified within the corresponding time period, identify high-load interference areas, and generate resource reallocation schemes in response to the label to be verified.

[0149] The rollback update unit is used to perform resource reallocation and execute closed-loop verification and rollback mechanisms.

[0150] The big data-based data processing platform provided in this application can execute the big data-based data processing method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the method execution.

[0151] From a hardware perspective, to implement the data processing method based on big data in a computer, this application also provides an electronic device; please refer to [link to relevant documentation]. Figure 3 As shown, it is a schematic diagram of the structure of the electronic device described in this application, including:

[0152] The system comprises a processor 1, a memory 2, a communication interface 3, and a bus 4; wherein the processor 1 and the memory 2, and the memory 2 and the communication interface 3, transmit data via the bus 4; the processor is used to process data in the memory and generate commands, the memory is used to store data, the communication interface is used to receive and send data, and the bus is used to realize data transmission between the processor, the memory, and the communication interface.

[0153] In this embodiment, the big data-based data processing method can be implemented as a runnable computer program. When the computer program is loaded into the processor or into the memory and processed by the processor via the bus, one or more steps of the big data-based data processing can be executed.

[0154] This embodiment also provides a computer-readable storage medium for storing the computer-executable instructions. The computer-readable storage medium is a tangible physical storage medium that can store the computer program and various types of data used in the program. The physical storage medium includes, but is not limited to, existing physical storage media or combinations thereof, such as random access memory, read-only memory, optical disk, and hard disk.

[0155] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A data processing method based on big data, characterized in that, include: Real-time status data of multiple user equipment in the target communication cell are collected in an aperiodic triggering manner, and the real-time status data is encapsulated in a structured manner to form an initial data stream; Dynamic quality assessment and anomaly repair are performed on the initial data stream to generate a cleaned data sequence, and the statistical characteristics of the cleaned data sequence are extracted simultaneously. A statistical characteristic trend curve is constructed based on the statistical characteristics of the purified data sequence, and a segmented fluctuation analysis is performed. A label to be verified is set based on the fluctuation analysis results. In response to the label to be verified, spatiotemporal clustering analysis is performed on the original multi-domain feature vectors within the time period corresponding to the label to be verified, high-load interference regions are identified, and resource reallocation schemes are generated.

2. The data processing method based on big data according to claim 1, wherein the dynamic quality assessment and anomaly repair of the initial data stream includes: The method involves determining the window length and stepping strategy of a sliding window, calculating a multi-dimensional local outlier factor within each sliding window, and identifying anomalous data recording units. Its key features are: The sliding window is a variable-length adaptive window, and its window length L is determined by the following formula: L=max{Lmin,floor(Nact / Mt×200)}; Where Lmin is the predefined minimum window length, Nact is the number of user devices currently active, Mt is the total number of registered users in the target communication cell, and floor() is the floor function; The stepping strategy is set as follows: the step size of each sliding window movement is equal to one-third of the current window length, and the step size is rounded up.

3. The data processing method based on big data according to claim 2, characterized in that, When calculating multi-dimensional local outlier factors and identifying anomalous data record units within each sliding window, three-dimensional local outlier indicators are set: spatial proximity outlier factor, temporal consistency outlier factor, and conversational semantic outlier factor for the i-th data record unit within the sliding window. If the spatial proximity outlier factor of the i-th data record unit within the sliding window is greater than the first outlier threshold and the temporal consistency outlier factor is greater than the second outlier threshold, or the session semantic outlier factor is greater than the third outlier threshold, then the current data record unit is determined to be an abnormal data record unit.

4. The data processing method based on big data according to claim 3, characterized in that, If neither of the two data recording units before and after the abnormal data recording unit is an abnormal data recording unit, then the Lagrange interpolation method is used to calculate the intermediate interpolation vector based on the multi-domain feature vectors of the two data recording units before and after the abnormal data recording unit to replace the multi-domain feature vector of the abnormal data recording unit. If a continuous sequence of abnormal data record units exists, all data record units in the continuous sequence of abnormal data record units are discarded and recorded as data holes. Based on the multi-domain feature vectors of the last normal data record unit before and the first normal data record unit after the data hole, linear decay weighted interpolation is performed to repair the multi-domain feature vectors of the data record units within the data hole. The data record units created after the repair are used as the purified data sequence.

5. The data processing method based on big data according to claim 4, characterized in that, When extracting statistical features from the purified data sequence, the statistical features include: mean offset and variance convergence.

6. The data processing method based on big data according to claim 5, characterized in that, Arrange the mean offset Δμ and variance convergence ρ corresponding to the continuous sliding window in time order to construct the mean offset trend curve Cμ(t) and variance convergence trend curve Cρ(t), respectively, where t is the sliding window number; Perform the following piecewise processing on Cμ(t) and Cρ(t): Cμ(t) and Cρ(t) are divided into several curve segments with a fixed time length T, and the sample variance σμ within each curve segment is calculated. 2 and σρ 2 ; Detect the peaks in each curve segment that exceed the mean ± 2 standard deviations of that curve segment, and count the proportion of abnormal peaks to the total number of peaks in that segment. Record the results as rμ and rρ. If any of the following conditions are met for each curve segment, then the time domain corresponding to that curve segment is determined to be an abnormal fluctuation period: Condition 1: σμ 2 Greater than Vμmax or σρ 2 Greater than Vρmax; Condition 2: rμ is greater than Rmax or rρ is greater than Rmax; Where Vμmax and Vρmax are the offset variance threshold and convergence variance threshold, respectively, and Rμmax is the abnormal peak ratio threshold. A verification label is set for the set of data recording units corresponding to all sliding windows during the abnormal fluctuation period, and no processing is performed on the remaining sliding windows.

7. The data processing method based on big data according to claim 6, characterized in that, The spatiotemporal clustering analysis includes: Geographic location estimates and multi-domain feature vectors for each data record unit within the corresponding time period of the label to be verified are extracted, and the DBSCAN density clustering algorithm is used to divide the data record units into several clusters, where: If the average HARQ failure rate within a cluster is greater than 5% and the average signal-to-noise ratio within a cluster is less than 10dB, the cluster is identified as a high-load interference cluster. When generating a resource reallocation scheme, all user equipment in the high-load interference cluster is added to the queue of users to be transferred, and arranged in descending order according to the emergency transfer index of the user equipment. Obtain the PRB occupancy bitmap at the current time from the base station scheduler, extract all consecutive idle PRB segments, and record the starting index, length, center frequency, and bandwidth of each consecutive idle PRB segment; For the top M user devices in the user transfer queue, sorted in descending order, select and allocate consecutive idle PRB segments that meet the needs of each user device based on the starting index, length, center frequency, and bandwidth of the consecutive idle PRB segments.

8. The data processing method based on big data according to claim 7, characterized in that, The method further includes: performing resource reallocation and executing closed-loop verification and rollback mechanisms; The system sends resource reconfiguration commands to user equipment through radio resource control command signals and resource reallocation schemes, and starts an 800ms verification window after the handover is completed. The average HARQ failure rate F of the whole cell and the average signal-to-noise ratio change ΔSNR of the frequency-hopped user are collected during the verification period. The closed-loop verification process includes: Set the verification pass and rollback conditions in the verification window; When the verification conditions are met, the resource reallocation is deemed successful and the reallocation state is exited. If the rollback conditions are met, and resource reallocation fails, a rollback is triggered: The system sends a command to the user equipment that has been issued a resource reconfiguration instruction to return the original occupied PRB segment, determines that the resource reconfiguration has failed, and adjusts the emergency transfer index calculation process for the user equipment.

9. A big data-based data processing platform, applied to the big data-based data processing method as described in any one of claims 1-8, characterized in that, include: The initial data stream generation unit is used to collect real-time status data of multiple user equipment in the target communication cell in an aperiodic triggering manner, and to encapsulate the real-time status data in a structured manner to form an initial data stream. The statistical feature extraction unit is used to perform dynamic quality assessment and anomaly repair on the initial data stream, generate a cleaned data sequence, and simultaneously extract the statistical features of the cleaned data sequence. The label setting unit is used to construct a statistical feature trend curve based on the statistical features of the cleaned data sequence, perform segmented fluctuation analysis, and set the label to be verified based on the fluctuation analysis results. The scheme generation unit is used to perform spatiotemporal clustering analysis on the original multi-domain feature vectors of the label to be verified within the corresponding time period, identify high-load interference areas, and generate resource reallocation schemes in response to the label to be verified. The rollback update unit is used to perform resource reallocation and execute closed-loop verification and rollback mechanisms.

10. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the big data-based data processing method according to any one of claims 1-8.

Images