A smart city data collection and preprocessing method and system

Abstract

The application relates to the technical field of big data processing, and discloses a smart city data collection and preprocessing method and system, which comprises the following steps: obtaining original perception data flow at an edge node; determining adaptive filtering coefficients based on environmental parameters, and performing targeted filtering processing on different types of data flow; performing noise suppression at the edge node and eliminating redundant data to generate standardized data blocks; generating data category labels according to data feature vectors and a classification strategy; generating a data transmission strategy in combination with the data category labels and the state of edge cloud resources, dynamically adjusting the data collection frequency according to the business scene type; performing quality evaluation based on data integrity, accuracy and timeliness, generating a data quality score, and implementing a differentiated processing strategy according to the score. Through dynamic resource scheduling and end cloud collaborative optimization, the application reduces network bandwidth pressure, meets response requirements, effectively improves the efficiency of smart city data collection and preprocessing, and ensures quality.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and more specifically, to a method and system for smart city data collection and preprocessing. Background Technology

[0002] As smart city construction deepens, especially in time-sensitive scenarios such as urban emergency command, real-time traffic dispatch, and public safety early warning, higher demands are placed on the timeliness and accuracy of data acquisition. First, the full uploading of unstructured data to the cloud significantly increases network bandwidth pressure, and transmission delays affect data timeliness. Second, fixed-rule preprocessing methods are ill-suited to the complex and ever-changing urban environment, and noise interference impacts subsequent data quality. Third, edge-cloud resources use a static allocation model, lacking flexibility and unable to dynamically adjust based on business load and the type of emergencies, resulting in low resource utilization. These intertwined problems form a core technological bottleneck restricting the efficiency of smart city data collection and processing, making it difficult to meet the real-time response needs of different urban scenarios.

[0003] Therefore, how to integrate edge sensing, suppress edge noise, and achieve dynamic resource scheduling and lightweight real-time preprocessing in smart city data acquisition and preprocessing systems have become urgent technical problems to be solved. Summary of the Invention

[0004] This invention provides a smart city data acquisition and preprocessing method and system, which solves the technical problems in the prior art, such as difficulty in meeting millisecond-level response requirements, the inability of fixed rule preprocessing methods to flexibly adapt to complex urban environments, noise interference affecting the quality of subsequent data, and the lack of dynamic adjustment capability in the static allocation of edge resources, resulting in low resource utilization.

[0005] This invention provides a smart city data acquisition and preprocessing method and system, comprising: Firstly, a method for smart city data collection and preprocessing includes the following steps: Acquire raw sensing data streams at edge nodes within the smart city coverage area; Based on the original sensing data stream, the computing resource utilization rate of the edge server is obtained in real time, and a dynamic resource scheduling strategy is formulated according to the current business load, network bandwidth and edge node computing capabilities. According to the dynamic resource scheduling strategy, edge node noise suppression is performed on the original sensing data stream, and redundant data is analyzed and removed in real time to retain effective feature data. The effective feature data is subjected to lightweight real-time preprocessing to generate standardized data blocks; Based on the feature vectors of the standardized data blocks and the preset classification strategy, the standardized data blocks are classified to generate at least one data category label; Generate a data transmission strategy based on the data category labels and the current edge cloud resource status; The preset upload threshold is determined according to the dynamic resource scheduling strategy, and the standardized data block is uploaded according to the data transmission strategy and the preset upload threshold. Based on the standardized data blocks, the dynamic resource scheduling strategy parameters are continuously optimized and updated, and the update results are sent to the edge nodes.

[0006] Further, the current environmental parameters are obtained, wherein the environmental parameters are selected from at least one of temperature parameters, humidity parameters, light parameters, electromagnetic interference parameters, and equipment status parameters; The adaptive filtering coefficients are determined based on the environmental parameters, wherein the adaptive filtering coefficients are obtained by inputting the environmental parameters into a preset adaptive filtering model, and the adaptive filtering model is a model built based on a neural network. The adaptive filtering coefficients are used to perform targeted filtering on different types of raw sensing data streams; the raw sensing data streams include video data, image data, audio data, environmental sensing data, and radar data. Specifically, the video stream data is subjected to inter-frame difference operations to extract valid video frames and remove redundant frame data; the audio data is subjected to time-domain and frequency-domain conversion to extract feature parameters; the environmental sensor data is subjected to outlier detection and correction; the radar data is subjected to point cloud filtering and range Doppler processing to extract valid target point clouds and remove ground clutter and static background interference; and the processed data is subjected to data format standardization, data filtering and data dimensionality reduction operations to generate edge preprocessing data.

[0007] Further, determining the adaptive filtering coefficients based on the environmental parameters includes: The environmental parameters are compared with a preset environmental parameter threshold range; When the environmental parameters are within the normal preset threshold range, a first preset filtering mechanism is used to filter the sensing data stream. The first preset filtering mechanism is designed with lightweight and low power consumption as its goals, and specifically includes: using Gaussian spatial domain smoothing filtering combined with inter-frame difference to remove slight jitter noise from video stream data; using a low-order Wiener filter to smooth the spectrum of audio data to eliminate environmental white noise; using a first-order low-pass IIR filter to smooth the temporal sequence of environmental sensing data; using statistical mean filtering to remove outliers from radar point cloud data; and using Kalman filtering to perform state fusion estimation on the multi-source processing results, outputting a smooth sensing data stream while ensuring real-time performance. When the environmental parameters are within an abnormal preset threshold range, a second preset filtering mechanism is used to filter the sensing data stream. The filtering strength of the second preset filtering mechanism is greater than that of the first preset filtering mechanism. The second preset filtering mechanism is designed with strong robustness and multi-cascade suppression as its goals. Specifically, it includes: using a combination of bilateral filtering and morphological filtering on video stream data to suppress strong noise while preserving image edge features; using a joint algorithm of spectral subtraction and non-negative matrix factorization on audio data to separate aliasing noise components and recover effective audio signals; using median filtering combined with the 3σ criterion for dual outlier removal on environmental sensing data; using a constant false alarm rate (CFAR) detection algorithm on radar point cloud data to suppress strong clutter backgrounds and extract effective target echoes; and performing deep denoising on various types of data through adaptive wavelet multi-scale decomposition to cope with severe weather or strong electromagnetic interference scenarios. The filtered data is evaluated, and the evaluation results are fed back to the adaptive filtering model to update the adaptive filtering coefficients.

[0008] Furthermore, the at least one data category label and the current edge resource state generate a data transmission strategy, including: Obtain the current edge-cloud resource status information, wherein the edge-cloud resource status information includes the computing resource utilization rate, storage resource utilization rate, network bandwidth utilization rate of the edge server, and computing resource utilization rate of the cloud server; The edge cloud resource status information is compared with a preset resource status threshold, and the comparison result is obtained by combining the data transmission priority corresponding to the data category label. The data transmission mode is determined based on the comparison results. When the edge cloud resource status meets the first preset condition, the real-time transmission mode is adopted. The first preset condition means that the network bandwidth utilization rate is lower than the first bandwidth threshold, the edge server computing resource utilization rate is lower than the first computing threshold, and the data transmission priority corresponding to the data category label is high priority, that is, the data comes from the city emergency command or public safety early warning scenario and needs to be uploaded immediately to meet the millisecond-level (latency less than 100ms) response requirements. When the edge cloud resource status meets the second preset condition, the batch transmission mode is adopted; the second preset condition is: the network bandwidth utilization rate is between the first bandwidth threshold and the second bandwidth threshold or the edge server computing resource utilization rate is between the first computing threshold and the second computing threshold, and the data transmission priority corresponding to the data category label is medium priority, that is, the data comes from real-time traffic scheduling or urban environmental monitoring scenarios, and is allowed to be cached locally on the edge node and then uploaded uniformly at a preset batch interval; When the edge cloud resource status meets the third preset condition, a delayed transmission mode is adopted. The third preset condition is: the network bandwidth utilization rate is higher than the second bandwidth threshold, or the edge server storage resource utilization rate exceeds the storage warning threshold, or the cloud server computing resource utilization rate exceeds the cloud load threshold, and the data transmission priority corresponding to the data category label is low priority, that is, the data is non-real-time archived data. The data is cached in the local storage of the edge node, and the retransmission is triggered after the resource status recovers to below the second preset condition.

[0009] Furthermore, a real-time transmission mode is adopted, including: The edge preprocessed data is compressed according to the data compression ratio determined in the data transmission strategy. The compressed edge preprocessing data is divided into packets according to the preset data packet size to generate multiple data sub-packets; According to the preset sending interval, the multiple data sub-packets are sequentially sent to the cloud server along the data transmission path determined by the data transmission strategy; Each transmitted data sub-packet is marked with a transmission status, wherein the transmission status mark includes at least one of transmitted, in transmission, and received acknowledgment.

[0010] Furthermore, a bulk transfer mode is adopted, including: The edge preprocessed data is grouped according to the data category label, and data of the same category is written into the corresponding local cache queue and arranged in order by timestamp; Real-time monitoring of the data volume of each cache queue; when the data volume of any cache queue reaches the preset batch capacity threshold or the cache duration reaches the preset batch time interval, a batch packaging operation is triggered, and the data in the queue is compressed and packaged into batch data packets according to the data compression ratio determined in the data transmission strategy. The batch data packets are sent to the cloud server along the data transmission path determined by the data transmission strategy, and the integrity of the batch data packets is verified. If the verification fails, a retransmission mechanism is triggered according to the transmission priority corresponding to the data category label, and high-priority data packets are retransmitted first.

[0011] Furthermore, a delayed transmission mode is adopted, including: The edge preprocessed data that meets the delayed transmission conditions is written to the local persistent storage queue of the edge node according to the data category label and timestamp, and the storage duration of each data entry is recorded. Real-time monitoring of the current edge cloud resource status; when the edge cloud resource status recovers from the third preset condition to below the second preset condition, the retransmission process is automatically triggered. Data to be transmitted is retrieved from the local persistent storage queue in descending order of data transmission priority. The data to be transmitted is then deduplicated and its integrity is checked, and expired data exceeding the preset maximum storage time is removed. The verified data to be transmitted is compressed according to the data compression ratio determined by the data transmission strategy and then sent to the cloud server. After the retransmission is completed, the corresponding data entry in the local persistent storage queue is cleared.

[0012] Furthermore, the data collection frequency is dynamically adjusted according to the type of business scenario, wherein the type of business scenario is selected from at least one of urban emergency command scenario, real-time traffic dispatch scenario, public safety early warning scenario, and urban environmental monitoring scenario; A mapping relationship is established between business scenario types and data collection frequencies. When the business scenario type is an urban emergency command scenario, the corresponding data collection frequency is the first frequency; when the business scenario type is a real-time traffic dispatch scenario, the corresponding data collection frequency is the second frequency; and when the business scenario type is a public safety early warning scenario, the corresponding data collection frequency is the third frequency. The dynamic scheduling function of the edge computing nodes adjusts the preprocessing priority and upload threshold in conjunction with the data collection frequency corresponding to the business scenario type; the resource pool management of the cloud server dynamically expands or shrinks each computing node according to the resource allocation strategy output by the scheduling strategy generation stage, so as to support the collaborative adaptation of collection, processing and transmission under different business scenarios.

[0013] Furthermore, the perceived data stream is subjected to quality assessment to generate a data quality score, wherein the data quality assessment is determined based on data integrity, data accuracy, and data timeliness; Data processing priority is determined based on the data quality score, wherein the highest processing priority is assigned when the data quality score is greater than a first preset score; the second highest processing priority is assigned when the data quality score is greater than a second preset score and less than or equal to the first preset score; and the normal processing priority is assigned when the data quality score is less than or equal to the second preset score. The data quality score is fed back to the cloud server for performance evaluation to update the noise feature learning model parameters; and differentiated preprocessing strategies are implemented for data with different processing priorities based on the data category labels. The updated classification model parameters are then distributed to edge computing nodes, forming a quality-driven end-cloud collaborative optimization closed loop.

[0014] Secondly, a smart city data acquisition and preprocessing system includes: Data acquisition module: used to acquire raw sensing data streams at edge nodes in the smart city coverage area; Resource scheduling module: used to obtain the computing resource utilization rate of the edge server in real time based on the original sensing data stream, and formulate dynamic resource scheduling strategies according to the current business load, network bandwidth and edge node computing capabilities; Data processing module: used to perform edge node noise suppression on the original sensing data stream according to the dynamic resource scheduling strategy, and to perform real-time analysis and remove redundant data to retain effective feature data; to perform lightweight real-time preprocessing on the effective feature data to generate standardized data blocks; Data transmission module: used to classify the standardized data blocks based on the feature vectors of the standardized data blocks and a preset classification strategy, and generate at least one data category label; and generate a data transmission strategy based on the data category label and the current edge cloud resource status; Data update module: used to determine a preset upload threshold according to the dynamic resource scheduling strategy, upload the standardized data block according to the data transmission strategy and the preset upload threshold; based on the standardized data block, continuously optimize and update the parameters of the dynamic resource scheduling strategy, and send the update results to the edge node.

[0015] The beneficial effects of this invention are as follows: By acquiring and preprocessing the raw sensing data stream at edge nodes, and delegating noise suppression and redundant data removal operations to the edge side, this invention reduces the amount of data transmitted in the cloud, effectively alleviates network bandwidth pressure, and significantly reduces transmission latency, meeting the millisecond-level (latency less than 100ms) response requirements. Through an adaptive filtering coefficient dynamic adjustment mechanism based on environmental parameters, filtering parameters are calculated in real time according to environmental factors such as temperature, humidity, light, and electromagnetic interference. A switching strategy between a first and second preset filtering mechanism is adopted to address both normal and abnormal environmental conditions, resolving the lack of flexibility in fixed-rule preprocessing methods. Noise interference in complex and variable urban environments is effectively suppressed, improving subsequent data quality. By acquiring edge server computing resource utilization, current business load, network bandwidth, and edge node computing capabilities in real time, a dynamic resource scheduling strategy is formulated, transforming the static allocation mode of edge-cloud resources into a dynamic adaptive adjustment mode. This fundamentally solves the technical problem of low resource utilization and enhances real-time response capabilities in different urban scenarios.

[0016] By establishing a mapping relationship between environmental parameters and an adaptive filtering model, and using a neural network model as the core for calculating adaptive filtering coefficients, the filtering parameters are automatically learned and dynamically updated according to environmental changes, enhancing environmental adaptability. Kalman filtering is used to perform state fusion estimation of multi-source processing results, ensuring smooth output of the perceived data stream under real-time conditions and improving fusion accuracy. The first preset filtering mechanism employs a lightweight, low-power design, while the second preset filtering mechanism adopts a robust multi-cascade suppression design. The switching boundary between the two mechanisms is clear and well-defined, optimizing the balance between processing performance and resource consumption.

[0017] By comparing edge cloud resource status information with preset resource status thresholds and combining the data transmission priority corresponding to data category tags, the data transmission mode is comprehensively determined. The switching logic of the three modes—real-time transmission mode, batch transmission mode, and delayed transmission mode—is complete, covering all the requirements of high-priority real-time scenarios, medium-priority scheduling scenarios, and low-priority archiving scenarios, thus balancing the timeliness and reliability of data transmission. Through packet processing and transmission status marking mechanisms in real-time transmission mode, cache queue management and integrity verification mechanisms in batch transmission mode, and local persistent storage and retransmission triggering mechanisms in delayed transmission mode, the reliability and traceability of data transmission are guaranteed.

[0018] By establishing a mapping relationship between business scenario types and data collection frequencies, the dynamic scheduling function of edge computing nodes adjusts preprocessing priorities and upload thresholds in conjunction with business scenario types. The resource pool management of cloud servers dynamically expands or shrinks according to resource allocation strategies. The linkage mechanism between business scenario changes and resource adjustments is complete, and the collaborative adaptation capability of collection, processing, and transmission under different business scenarios is enhanced. Through a data quality assessment mechanism, data quality scores are generated based on data integrity, accuracy, and timeliness. Processing priorities are determined based on the scores, and differentiated preprocessing strategies are implemented. A quality-driven edge-cloud collaborative optimization closed loop is formed, and an iterative optimization mechanism for model parameters is established. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a smart city data collection and preprocessing method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a smart city data acquisition and preprocessing system module provided in an embodiment of the present invention. Detailed Implementation

[0020] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0021] At least one embodiment of the present invention discloses a smart city data acquisition and preprocessing method and system, including: like Figure 1 As shown, a smart city data collection and preprocessing method includes the following steps: Step 1: Acquire raw sensing data streams at edge nodes in the smart city coverage area; Step 2: Based on the original sensing data stream, obtain the computing resource utilization rate of the edge server in real time, and formulate a dynamic resource scheduling strategy according to the current business load, network bandwidth and edge node computing capabilities; Step 3: According to the dynamic resource scheduling strategy, perform edge node noise suppression on the original sensing data stream, and perform real-time analysis and remove redundant data to retain effective feature data; perform lightweight real-time preprocessing on the effective feature data to generate standardized data blocks; Step 4: Based on the feature vector of the standardized data block and the preset classification strategy, classify the standardized data block to generate at least one data category label; generate a data transmission strategy according to the data category label and the current edge cloud resource status. Step 5: Determine the preset upload threshold according to the dynamic resource scheduling strategy, and upload the standardized data block according to the data transmission strategy and the preset upload threshold; based on the standardized data block, continuously optimize and update the parameters of the dynamic resource scheduling strategy, and send the update results to the edge node.

[0022] In this embodiment, data acquisition involves deploying multimodal sensing devices at various edge nodes within the smart city coverage area to obtain raw sensing data streams. Data acquisition connects to various sensing devices deployed on the edge nodes, including video surveillance cameras, image acquisition devices, audio acquisition microphones, environmental sensors, and radar equipment. Data acquisition receives raw data streams from these sensing devices in real time through a standardized data acquisition interface and transmits this data to resource scheduling for subsequent processing. Simultaneously with acquiring the raw sensing data streams, the system also acquires real-time environmental parameters, including temperature, humidity, illumination, electromagnetic interference, and device status parameters, through environmental monitoring deployed at the edge nodes. These environmental parameters are transmitted to an adaptive filtering model via the data acquisition's built-in interface for subsequent determination of adaptive filtering coefficients.

[0023] Resource scheduling is the core scheduling center of the entire system, deployed and running on edge servers. It acquires real-time data on the edge servers' computing resource utilization, storage resource utilization, and network bandwidth utilization, while also receiving cloud resource status information from cloud servers. Resource scheduling also receives current business load information, obtained by analyzing the characteristics of the raw sensor data streams collected from data acquisition. Based on this information, and considering the current business load, network bandwidth, and edge node computing capabilities, resource scheduling formulates a dynamic resource scheduling strategy. This dynamic resource scheduling strategy includes preprocessing priority allocation, filtering strength parameter settings, upload threshold determination, and data transmission path planning. The resource scheduling system then distributes the formulated dynamic resource scheduling strategy to data processing and data analysis to guide subsequent data processing tasks.

[0024] The data processing module performs edge node noise suppression on the raw sensor data stream according to a dynamic resource scheduling strategy. Internally, the data processing module incorporates adaptive filtering. It first receives environmental parameters from the data acquisition process and inputs these parameters into a pre-defined adaptive filtering model. This adaptive filtering model is a deep learning model built on a neural network, containing multiple convolutional layers, fully connected layers, and activation functions. It outputs corresponding adaptive filtering coefficients based on the input environmental parameters. In practice, the adaptive filtering model is pre-trained with a large number of samples to accurately establish the mapping relationship between environmental parameters and filtering coefficients.

[0025] After obtaining the adaptive filtering coefficients, these coefficients are applied to different types of raw sensing data streams for targeted filtering. The raw sensing data streams include various types such as video data, image data, audio data, environmental sensing data, and radar data. For video stream data, data processing performs inter-frame difference operations to extract valid video frames while removing redundant frames. Inter-frame difference operations compare the pixel grayscale differences between adjacent frames; if the grayscale difference exceeds a preset threshold of 20, it is considered a valid frame; otherwise, it is considered a redundant frame and removed. For audio data, data processing performs time-domain to frequency-domain conversion, converting the time-domain signal to the frequency domain and extracting feature parameters, including spectral energy distribution, fundamental frequency estimation, and harmonic components. For environmental sensing data, data processing performs outlier detection and correction, using statistical methods to identify and correct outliers in the sensor-acquired data. For radar data, data processing performs point cloud filtering and range Doppler processing to extract valid target point clouds while removing ground clutter and static background interference.

[0026] After completing the targeted filtering process described above, the data processing performs a unified data format standardization operation on the processed data, converting data from different sources into a unified internal data format. Subsequently, data filtering is performed to remove residual noise, and data dimensionality reduction is executed to reduce the data dimensionality, generating edge preprocessed data.

[0027] During adaptive filtering, data processing also requires comparing environmental parameters with preset environmental parameter threshold ranges to determine whether to use a first or second preset filtering mechanism. The preset environmental parameter threshold ranges are as follows: temperature: normal range -10℃ to 50℃, abnormal range below -10℃ or above 50℃; humidity: normal range 20% to 85%, abnormal range below 20% or above 85%; illumination: normal range 50 lux to 50,000 lux, abnormal range below 50 lux or above 50,000 lux; electromagnetic interference: normal range below 40 dBμV / m, abnormal range 40 dBμV / m and above. When environmental parameters are within the normal preset threshold range, the first preset filtering mechanism is used for data processing. The first preset filtering mechanism is designed for lightweight and low power consumption. Its specific processing methods include: for video stream data, Gaussian spatial domain smoothing filtering combined with inter-frame differencing is used to remove minor jitter noise; the Gaussian filter kernel size is set to 3×3 and the standard deviation to 0.5. For audio data, a low-order Wiener filter is used for spectral smoothing to eliminate environmental white noise; the Wiener filter order is set to 2nd. For environmental sensor data, a first-order low-pass IIR filter is used for temporal smoothing; the filter cutoff frequency is set to 5Hz. For radar point cloud data, statistical mean filtering is used to remove outliers; the statistical window is set to 5 points. After processing each type of data individually, the data processing uses Kalman filtering to perform state fusion estimation on the multi-source processing results, outputting a smooth sensing data stream while ensuring real-time performance.

[0028] When environmental parameters are within the pre-set threshold range for abnormalities, the data processing employs a second pre-set filtering mechanism. The filtering strength of the second pre-set filtering mechanism is significantly greater than that of the first pre-set filtering mechanism, designed for strong robustness and multi-cascade suppression. Specific processing methods include: for video stream data, a combination of bilateral filtering and morphological filtering is used, with the spatial domain parameter of the bilateral filter set to a 5×5 kernel and the grayscale difference parameter set to 50; the morphological filtering uses a combination of opening and closing operations. For audio data, a joint algorithm of spectral subtraction and non-negative matrix factorization is used to separate aliasing noise components to recover the effective audio signal; the noise reduction coefficient of spectral subtraction is set to 0.5, and the number of basis vectors in the non-negative matrix factorization is set to 20. For environmental sensor data, median filtering combined with the 3σ criterion is used for dual outlier removal; the median filtering window is set to 5, and the confidence interval of the 3σ criterion is set to 99.7%. For radar point cloud data, a constant false alarm rate (CFAR) detection algorithm is used to suppress strong clutter backgrounds; the CFAR is set to 0.001. After processing each type of data separately, the data processing performs deep denoising on each type of data through adaptive wavelet multi-scale decomposition to cope with severe weather or strong electromagnetic interference scenarios.

[0029] After filtering, the filtered data needs to be evaluated. The evaluation calculates metrics such as the improvement in signal-to-noise ratio, outlier detection rate, and data integrity, and feeds the evaluation results back to the adaptive filtering model to update the adaptive filtering coefficients. This feedback mechanism allows the adaptive filtering model to continuously learn and optimize, constantly improving the filtering effect.

[0030] After completing the effectiveness evaluation, a comprehensive quality assessment is further conducted on the perceived data stream to generate a data quality score. The data quality assessment is determined based on three dimensions: data integrity, data accuracy, and data timeliness. Data integrity is assessed by calculating the ratio of the actual amount of data collected to the theoretically required amount. Data accuracy is assessed by comparing the deviation between the sensor-collected data and standard reference values. Data timeliness is assessed by calculating the time delay from data collection to processing. The combined assessment results from these three dimensions generate a data quality score ranging from 0 to 100. Data processing priorities are determined based on the data quality score: when the data quality score is greater than the first preset score of 80, it is assigned the highest processing priority, and the system will prioritize processing this data using the highest level of transmission mode; when the data quality score is greater than the second preset score of 60 and less than or equal to the first preset score of 80, it is assigned the second highest processing priority; when the data quality score is less than or equal to the second preset score of 60, it is assigned a normal processing priority. The data quality score and processing priority are transmitted to the data transmission stage along with the edge preprocessed data, serving as an important reference for data classification and transmission strategy generation.

[0031] The data transmission receives standardized data blocks from the data processing unit and classifies them based on their feature vectors and a pre-defined classification strategy, generating at least one data category label. The pre-defined classification strategy is implemented using a pre-trained classification model that can determine the business category of a data block based on its feature vectors. Data category labels include three types: high-priority, medium-priority, and low-priority. High-priority labels indicate data originating from urban emergency command or public safety early warning scenarios, requiring immediate upload to meet millisecond-level (latency less than 100ms) response requirements; medium-priority labels indicate data originating from real-time traffic dispatch or urban environmental monitoring scenarios, allowing for delayed uploads; low-priority labels indicate non-real-time archived data, which can be uploaded in batches or with a delay.

[0032] After generating data category labels, the current edge resource status information is obtained. This information includes the computing resource utilization rate of edge servers, storage resource utilization rate, network bandwidth utilization rate, and computing resource utilization rate of cloud servers. Data transmission compares this edge resource status information with preset resource status thresholds, and combines this with the data transmission priority corresponding to the data category labels to obtain the comparison result. The preset resource status thresholds are as follows: the first bandwidth threshold is set to 60%, and the second bandwidth threshold is set to 85%; the first computing threshold is set to 70%, and the second computing threshold is set to 85%; the storage warning threshold is set to 80%; and the cloud load threshold is set to 85%. Based on the comparison result, the data transmission mode is determined.

[0033] When the edge resource status meets the first preset condition, data transmission adopts a real-time transmission mode. The first preset condition indicates that the network bandwidth utilization is lower than 60% of the first bandwidth threshold, the edge server computing resource utilization is lower than 70% of the first computing threshold, and the data transmission priority corresponding to the data category label is high priority. In this case, data transmission first compresses the edge preprocessed data according to the data compression ratio determined in the data transmission strategy. The compression algorithm adopts the H.264 video compression algorithm or the corresponding audio and text compression algorithm. After compression, data transmission divides the compressed edge preprocessed data into packets according to the preset data packet size, generating multiple data sub-packets. The size of each data sub-packet is set to 64KB to meet the requirements of network transmission. Subsequently, data transmission sequentially sends multiple data sub-packets to the cloud server along the data transmission path determined by the data transmission strategy according to the preset sending interval. The sending interval is set to 10 milliseconds to ensure the real-time performance of the transmission. During the transmission process, data transmission marks the transmission status of each sent data sub-packet, including three states: sent, in transmission, and received confirmation. The "Sent" status indicates that the data packet has been successfully sent to the network transport layer; the "Transmitting" status indicates that the data packet is being transmitted in the network channel; and the "Received and Acknowledged" status indicates that the cloud server has successfully received and acknowledged the data packet.

[0034] When the edge resource status meets the second preset condition, data transmission adopts a batch transmission mode. The second preset condition is that the network bandwidth utilization rate is between 60% and 85% of the first bandwidth threshold or the edge server computing resource utilization rate is between 70% and 85% of the first computing threshold, and the data transmission priority corresponding to the data category label is medium priority. In this case, data transmission first groups the edge preprocessed data according to the data category label, writes the data of the same category into the corresponding local cache queue, and arranges them in order by timestamp. The local cache queue is set in the memory of the edge server, and the maximum capacity of each queue is set to 100MB. Data transmission monitors the data volume of each cache queue in real time. When the data volume of any cache queue reaches the preset batch capacity threshold or the cache duration reaches the preset batch time interval, a batch packaging operation is triggered. The batch capacity threshold is set to 50MB, and the batch time interval is set to 5 seconds. The batch packaging operation compresses the data in the queue according to the data compression ratio determined in the data transmission strategy and encapsulates it into batch data packets. After encapsulation, data transmission sends the batch data packets to the cloud server along the data transmission path determined by the data transmission strategy and performs integrity verification on the batch data packets. Integrity verification uses the MD5 hash algorithm to calculate the hash value of the data packet and compares it with a pre-stored hash value. If the verification fails, a retransmission mechanism is triggered according to the transmission priority corresponding to the data category label, prioritizing the retransmission of higher-priority data packets.

[0035] When the edge resource status meets the third preset condition, data transmission adopts a delayed transmission mode. The third preset condition is that the network bandwidth utilization rate is higher than the second bandwidth threshold of 85%, or the edge server storage resource utilization rate exceeds the storage warning threshold of 80%, or the cloud server computing resource utilization rate exceeds the cloud load threshold of 85%, and the data transmission priority corresponding to the data category label is low priority. In this case, the data transmission writes the edge preprocessed data that meets the delayed transmission condition to the local persistent storage queue of the edge node according to the data category label and timestamp, and records the storage duration of each data entry. The local persistent storage uses SSD storage devices, and the maximum capacity of the storage queue is set to 1TB. Data transmission monitors the current edge-cloud resource status in real time. When the edge-cloud resource status recovers from the third preset condition to below the second preset condition, the retransmission process is automatically triggered. The retransmission process retrieves the data to be transmitted from the local persistent storage queue in descending order of data transmission priority, performs deduplication verification and integrity checks on the data to be transmitted, and removes expired data that exceeds the preset maximum storage duration of 24 hours. The data to be transmitted after verification is compressed according to the data compression ratio determined by the data transmission strategy and sent to the cloud server. After the retransmission is completed, the corresponding data entries in the local persistent storage queue are cleared.

[0036] The data update unit is deployed on a cloud server to determine a preset upload threshold based on a dynamic resource scheduling strategy, and then uploads standardized data blocks according to the data transmission strategy and the preset upload threshold. The data update also continuously optimizes and updates the dynamic resource scheduling strategy parameters based on the standardized data blocks, and distributes the update results to edge nodes. The update process employs reinforcement learning, using data processing effectiveness and transmission efficiency as reward functions, and iteratively optimizes the scheduling strategy parameters. After the update is complete, the data update sends the updated scheduling strategy parameters to the edge server via a secure communication channel. Upon receiving the updated parameters, the edge server immediately applies them to subsequent data processing flows.

[0037] Data processing performs quality assessment on the sensed data stream, generating a data quality score. The data quality assessment is determined based on three dimensions: data integrity, data accuracy, and data timeliness. Data integrity assessment calculates the ratio of the actual amount of data collected to the theoretically required amount; data accuracy assessment compares the deviation of the sensor-collected data from standard reference values; and data timeliness assessment calculates the time delay from data acquisition to processing. The results of the combined assessment across these three dimensions generate a data quality score ranging from 0 to 100.

[0038] Data processing priorities are determined based on data quality scores. Data with a score greater than the first preset score of 80 is assigned the highest processing priority, meaning the system will process this data first and use the highest-level transmission mode. Data with a score greater than the second preset score of 60 but less than or equal to the first preset score of 80 is assigned the next highest processing priority. Data with a score less than or equal to the second preset score of 60 is assigned a normal processing priority. Data updates dynamically adjust the data upload strategy based on processing priorities, ensuring that high-priority data arrives at the cloud server first, while low-priority data is appropriately cached or delayed for transmission, achieving a balance between data processing and transmission efficiency.

[0039] This invention also provides a specific implementation method based on business scenario linkage and resource scheduling. The system dynamically adjusts the data acquisition frequency according to the business scenario type, which is selected from urban emergency command scenario, real-time traffic dispatch scenario, public safety early warning scenario, and urban environmental monitoring scenario. The system pre-establishes a mapping relationship between business scenario type and data acquisition frequency. When the business scenario type is urban emergency command scenario, the corresponding data acquisition frequency is the first frequency, set to 100Hz, that is, 100 data acquisitions per second, to meet the high-frequency data acquisition requirements of emergency command scenario; when the business scenario type is real-time traffic dispatch scenario, the corresponding data acquisition frequency is the second frequency, set to 50Hz, that is, 50 data acquisitions per second; when the business scenario type is public safety early warning scenario, the corresponding data acquisition frequency is the third frequency, set to 30Hz, that is, 30 data acquisitions per second; when the business scenario type is urban environmental monitoring scenario, the corresponding data acquisition frequency is the fourth frequency, set to 10Hz, that is, 10 data acquisitions per second.

[0040] The dynamic scheduling function of edge computing nodes adjusts preprocessing priority and upload thresholds in conjunction with the data collection frequency corresponding to the business scenario type. In urban emergency command scenarios, edge computing nodes set the preprocessing priority to the highest level and lower the upload threshold to accelerate data transmission speed; in urban environmental monitoring scenarios, edge computing nodes set the preprocessing priority to a normal level and increase the upload threshold to reduce data transmission volume. The resource pool management component of the cloud server resource pool dynamically expands or shrinks each computing node based on the resource allocation strategy output by the scheduling strategy generation component. When an increase in business load is detected, the cloud server resource pool automatically increases the number of computing instances; when a decrease in business load is detected, the cloud server resource pool automatically reduces the number of computing instances to achieve efficient resource utilization.

[0041] This invention also provides specific implementation methods for data security and privacy protection. During data acquisition, sensitive data is anonymized by using data masking technology to conceal personal privacy information when collecting raw sensor data. During data transmission, the TLS 1.3 encryption protocol is used to ensure the security of the transmission channel, and sensitive data payloads are encrypted using AES-256. When storing data on a cloud server, a partitioned storage strategy is used to separate sensitive and non-sensitive data, and stricter access control policies are implemented for sensitive data storage areas.

[0042] This invention also provides specific implementation methods for system fault tolerance and recovery. Edge nodes are equipped with a primary / backup dual-machine hot standby mechanism. When the primary edge server fails, the backup edge server automatically switches over and takes over data processing tasks, with a switching time not exceeding 500 milliseconds. Data transmission is equipped with a transmission interruption resumption mechanism. When network interruption causes data transmission failure, the system automatically records the transmission interruption point and resumes transmission from the interruption point after network recovery, avoiding duplicate data uploads or data loss. The cloud server resource pool adopts a multi-availability zone deployment strategy. When a single availability zone fails, the system automatically switches the workload to other availability zones to ensure service continuity.

[0043] In summary, by using an adaptive filtering mechanism to dynamically adjust the filtering strategy based on environmental parameters, a business scenario linkage mechanism to dynamically adjust the data collection frequency and resource allocation according to different scenarios, and a data quality assessment mechanism to score the quality of processed data and dynamically adjust the processing priority based on the score, the processing efficiency and transmission reliability of the smart city data collection and preprocessing system are guaranteed from multiple dimensions.

[0044] like Figure 2 As shown, a smart city data acquisition and preprocessing system includes: Data acquisition module: used to acquire raw sensing data streams at edge nodes in the smart city coverage area; Resource scheduling module: used to obtain the computing resource utilization rate of the edge server in real time based on the original sensing data stream, and formulate dynamic resource scheduling strategies according to the current business load, network bandwidth and edge node computing capabilities; Data processing module: used to perform edge node noise suppression on the original sensing data stream according to the dynamic resource scheduling strategy, and to perform real-time analysis and remove redundant data to retain effective feature data; to perform lightweight real-time preprocessing on the effective feature data to generate standardized data blocks; Data transmission module: used to classify the standardized data blocks based on the feature vectors of the standardized data blocks and a preset classification strategy, and generate at least one data category label; and generate a data transmission strategy based on the data category label and the current edge cloud resource status; Data update module: used to determine a preset upload threshold according to the dynamic resource scheduling strategy, upload the standardized data block according to the data transmission strategy and the preset upload threshold; based on the standardized data block, continuously optimize and update the parameters of the dynamic resource scheduling strategy, and send the update results to the edge node.

[0045] Application scenario: Emergency response in urban emergency command scenarios Suppose a sudden fire occurs in the core commercial area of ​​a city. The city's emergency command center needs to immediately acquire multi-dimensional sensing data from the scene to support emergency decision-making and deployment. The complete workflow of the system of this invention in this scenario is as follows.

[0046] Step 1: When a sudden event triggers the data acquisition module, the module deployed at the edge node of the core commercial area immediately initiates a multi-source data synchronous acquisition mechanism. The data acquisition module establishes a real-time connection with the video surveillance cameras covering the area, acquiring live video stream data at a frequency of 100 frames per second through a standardized data acquisition interface. Simultaneously, the module also receives environmental sound data from on-site audio acquisition microphones, as well as environmental parameter data such as temperature and smoke concentration collected in real-time by surrounding environmental sensors. Furthermore, millimeter-wave radar equipment deployed in the area simultaneously acquires point cloud data of moving personnel. While acquiring the aforementioned raw sensing data stream, the data acquisition module simultaneously collects current environmental parameters through its built-in environmental monitoring system, including a temperature of 35℃ (within the normal range of -10℃ to 50℃), humidity of 60% (within the normal range of 20% to 85%), electromagnetic interference within the normal range (below 40dBμV / m), and normal equipment operating status. These environmental parameters are transmitted in real-time to the adaptive filtering model through the data acquisition module's internal interface for the dynamic determination of subsequent adaptive filtering coefficients.

[0047] Step Two: After receiving the raw sensing data stream and environmental parameters transmitted by the data acquisition module, the resource scheduling module promptly performs business load analysis and formulates a dynamic resource scheduling strategy for emergency scenarios. The resource scheduling module first obtains the current computing resource utilization rate of the edge server (45%, below the first computing threshold of 70%), storage resource utilization rate (30%, below the storage warning threshold of 80%), and network bandwidth utilization rate (20%, below the first bandwidth threshold of 60%). Simultaneously, it receives cloud resource status information from the cloud server, showing a cloud computing resource utilization rate of 40% (below the cloud load threshold of 85%). Since the current scenario is an urban emergency command scenario, the business load information analysis results indicate a high-priority business load. Based on the above information, the resource scheduling module generates a dynamic resource scheduling strategy according to the emergency scenario's scheduling strategy template. This strategy includes setting the preprocessing priority to the highest level, setting the filtering intensity parameter to standard mode, setting the upload threshold to a low threshold to accelerate the data transmission module speed, and planning the data transmission module path to the optimal path. The resource scheduling module then distributes the completed dynamic resource scheduling strategy to the data processing module through the internal communication interface to guide the next stage of noise suppression processing.

[0048] Step 3: After receiving the strategy from the dynamic resource scheduling module, the data processing module initiates adaptive filtering to perform targeted noise suppression on the raw sensing data stream. Upon receiving environmental parameters, the adaptive filter inputs them into a preset adaptive filtering model. Since all environmental parameters are within the normal preset threshold range, the filter coefficients output by the adaptive filtering model are adapted to the processing parameters of the first preset filtering mechanism. The data processing module performs inter-frame difference operations on the video stream data, extracting valid video frames by comparing pixel differences between adjacent frames. Frames with a grayscale difference exceeding a preset threshold of 20 are considered valid and retained, while redundant frames below this threshold are discarded, significantly reducing the amount of video data that needs to be transmitted. The module performs time-domain to frequency-domain conversion on the audio data, converting the time-domain signal to the frequency domain using a Fast Fourier Transform to extract spectral energy distribution characteristics and identify and label audio segments containing human voice characteristics and abnormal sounds. The module performs outlier detection on the environmental sensor data, using the Raida criterion from statistical methods to identify and correct outliers in temperature and smoke concentration data. Finally, the module performs point cloud filtering on the radar data, extracting valid moving target point clouds through range Doppler processing while removing ground clutter and static background interference.

[0049] After performing targeted filtering on various types of data, the data processing module standardizes the processed data, converting video stream data, audio data, environmental sensor data, and radar data into an internal standard data format. Subsequently, it performs dimensionality reduction, using principal component analysis to project high-dimensional feature data into a low-dimensional space, generating edge preprocessing data. The data processing module then performs a comprehensive quality assessment on this edge preprocessing data, calculating a quality score based on three dimensions: data completeness, data accuracy, and data timeliness. The current assessment result is 92 points, exceeding the first preset score of 80 points, thus assigning it the highest processing priority. This edge preprocessing data, along with its quality score and processing priority, is transmitted via the data bus to the data transmission module for further processing.

[0050] Step 4: After receiving the standardized data blocks, the data transmission module classifies the data blocks based on a preset classification strategy and generates data category labels. The classification model inside the data transmission module analyzes the feature vectors extracted from the standardized data blocks. Since the current scenario is an urban emergency command scenario, the data category labels output by the classification model are high-priority labels. After generating the data category labels, the data transmission module obtains the current edge-cloud resource status information for comparison: the edge server's computing resource utilization rate is 45% (lower than the first computing threshold of 70%), storage resource utilization rate is 30% (lower than the storage warning threshold of 80%), and network bandwidth utilization rate is 20% (lower than the first bandwidth threshold of 60%). The cloud server's computing resource utilization rate is 40% (lower than the cloud load threshold of 85%). The current status fully meets the first preset condition, therefore the data transmission module determines to use real-time transmission mode for data upload.

[0051] In real-time transmission mode, the data transmission module first compresses the edge preprocessing data according to the data compression ratio determined in the data transmission module strategy: H.264 video compression algorithm is used for video stream data, with a compression ratio of 1:20; AAC audio compression algorithm is used for audio data, with a compression ratio of 1:10; and corresponding text compression algorithms are used for environmental sensor data and radar data. After compression, the data transmission module divides the compressed data into packets according to the preset data packet size, with each data sub-packet being 64KB. After packetization, the data transmission module sequentially sends multiple data sub-packets to the cloud server along the data transmission path determined by the data transmission module strategy at a preset sending interval of 10 milliseconds. During the sending process, the data transmission module marks the transmission status of each sent data sub-packet, including three states: sent, in transmission, and received confirmation, ensuring full traceability of the transmission process. When the cloud server successfully receives and confirms all data sub-packets, the data transmission module task is completed.

[0052] Through the above process, the system of this invention can achieve end-to-end latency control within 100 milliseconds from raw data acquisition to cloud reception in urban emergency command scenarios, fully meeting the millisecond-level (latency less than 100ms) response requirements. The redundant frame removal rate of video data can reach over 60%, effectively reducing network bandwidth pressure. Adaptive filtering dynamically adjusts the filtering coefficients according to environmental parameters, ensuring good noise suppression under different environmental conditions, with a data quality score consistently above 85 points.

[0053] Step 5: After receiving data from the cloud server, the data update module initiates the optimization and update process of the dynamic resource scheduling module strategy. The data update module evaluates the received data, analyzing metrics across two dimensions: data processing module performance and transmission efficiency. The data processing module performance evaluation includes metrics such as signal-to-noise ratio (SNR) improvement and outlier detection rate; the current evaluation results show an SNR improvement of 15dB and an outlier detection rate of 95%. The transmission efficiency evaluation includes metrics such as end-to-end latency and data integrity rate; the current evaluation results show an end-to-end latency of 95 milliseconds and a data integrity rate of 99.8%. Using the data processing module performance and transmission efficiency as reward functions, the data update module iteratively optimizes the current dynamic resource scheduling module strategy using reinforcement learning. It adjusts parameters such as preprocessing priority allocation, filtering strength, and upload threshold, generating optimized scheduling strategy parameters. After optimization, the data update module sends the updated scheduling strategy parameters to the edge server via a secure communication channel. Upon receiving the updated parameters, the edge server immediately applies them to subsequent data processing module processes.

[0054] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A method for data collection and preprocessing in smart cities, characterized in that, include: Acquire raw sensing data streams at edge nodes within the smart city coverage area; Based on the original sensing data stream, the computing resource utilization rate of the edge server is obtained in real time, and a dynamic resource scheduling strategy is formulated according to the current business load, network bandwidth and edge node computing capabilities. According to the dynamic resource scheduling strategy, edge node noise suppression is performed on the original sensing data stream, and redundant data is analyzed and removed in real time to retain effective feature data. The effective feature data is subjected to lightweight real-time preprocessing to generate standardized data blocks; Based on the feature vectors of the standardized data blocks and the preset classification strategy, the standardized data blocks are classified to generate at least one data category label; Generate a data transmission strategy based on the data category labels and the current edge cloud resource status; The preset upload threshold is determined according to the dynamic resource scheduling strategy, and the standardized data block is uploaded according to the data transmission strategy and the preset upload threshold. Based on the standardized data blocks, the dynamic resource scheduling strategy parameters are continuously optimized and updated, and the update results are sent to the edge nodes.

2. The smart city data acquisition and preprocessing method according to claim 1, characterized in that, Obtain current environmental parameters, wherein the environmental parameters are selected from at least one of temperature parameters, humidity parameters, light intensity parameters, electromagnetic interference parameters, and equipment status parameters; The adaptive filtering coefficients are determined based on the environmental parameters, wherein the adaptive filtering coefficients are obtained by inputting the environmental parameters into a preset adaptive filtering model, and the adaptive filtering model is a model built based on a neural network. The adaptive filtering coefficients are used to perform targeted filtering on different types of raw sensing data streams; the raw sensing data streams include video data, image data, audio data, environmental sensing data, and radar data. Specifically, the video stream data is subjected to inter-frame difference operations to extract valid video frames and remove redundant frame data; the audio data is subjected to time-domain and frequency-domain conversion to extract feature parameters; the environmental sensor data is subjected to outlier detection and correction; the radar data is subjected to point cloud filtering and range Doppler processing to extract valid target point clouds and remove ground clutter and static background interference; and the processed data is subjected to data format standardization, data filtering and data dimensionality reduction operations to generate edge preprocessing data.

3. The smart city data acquisition and preprocessing method according to claim 2, characterized in that, Determining the adaptive filtering coefficients based on the environmental parameters includes: The environmental parameters are compared with a preset environmental parameter threshold range; When the environmental parameters are within the normal preset threshold range, the first preset filtering mechanism is used to filter the sensing data stream; When the environmental parameters are within an abnormal preset threshold range, a second preset filtering mechanism is used to filter the sensing data stream, wherein the filtering strength of the second preset filtering mechanism is greater than the filtering strength of the first preset filtering mechanism. The filtered data is evaluated, and the evaluation results are fed back to the adaptive filtering model to update the adaptive filtering coefficients.

4. The smart city data acquisition and preprocessing method according to claim 2, characterized in that, The at least one data category label and the current edge cloud resource status generate a data transmission strategy, including: Obtain the current edge-cloud resource status information, wherein the edge-cloud resource status information includes the computing resource utilization rate, storage resource utilization rate, network bandwidth utilization rate of the edge server, and computing resource utilization rate of the cloud server; The edge cloud resource status information is compared with a preset resource status threshold, and the comparison result is obtained by combining the data transmission priority corresponding to the data category label. The data transmission mode is determined based on the comparison results. When the edge cloud resource status meets the first preset condition, the real-time transmission mode is adopted. The first preset condition means that the network bandwidth utilization rate is lower than the first bandwidth threshold, the edge server computing resource utilization rate is lower than the first computing threshold, and the data transmission priority corresponding to the data category label is high priority, and it has been determined that the data comes from urban emergency command or public safety early warning scenarios. When the edge cloud resource status meets the second preset condition, the batch transmission mode is adopted; the second preset condition is: the network bandwidth utilization rate is between the first bandwidth threshold and the second bandwidth threshold or the edge server computing resource utilization rate is between the first computing threshold and the second computing threshold, and the data transmission priority corresponding to the data category label is medium priority, so as to determine that the data comes from real-time traffic scheduling or urban environmental monitoring scenarios. When the edge cloud resource status meets the third preset condition, a delayed transmission mode is adopted; the third preset condition is: the network bandwidth utilization rate is higher than the second bandwidth threshold, or the edge server storage resource utilization rate exceeds the storage warning threshold, or the cloud server computing resource utilization rate exceeds the cloud load threshold, and the data transmission priority corresponding to the data category label is low priority, the data has been determined to be non-real-time archived data, and the data is cached to the local storage of the edge node.

5. The smart city data acquisition and preprocessing method according to claim 4, characterized in that, Real-time transmission mode is adopted, including: The edge preprocessed data is compressed according to the data compression ratio determined in the data transmission strategy. The compressed edge preprocessed data is divided into packets according to the preset data packet size to generate multiple data sub-packets; According to the preset sending interval, the multiple data sub-packets are sequentially sent to the cloud server along the data transmission path determined by the data transmission strategy; Each transmitted data sub-packet is marked with a transmission status, wherein the transmission status mark includes at least one of transmitted, in transmission, and received acknowledgment.

6. The smart city data acquisition and preprocessing method according to claim 4, characterized in that, Batch transfer mode is used, including: The edge preprocessed data is grouped according to the data category label, and data of the same category is written into the corresponding local cache queue and arranged in order by timestamp; Real-time monitoring of the data volume of each cache queue; when the data volume of any cache queue reaches the preset batch capacity threshold or the cache duration reaches the preset batch time interval, a batch packaging operation is triggered, and the data in the queue is compressed and packaged into batch data packets according to the data compression ratio determined in the data transmission strategy. The batch data packets are sent to the cloud server along the data transmission path determined by the data transmission strategy, and the integrity of the batch data packets is verified. If the verification fails, a retransmission mechanism is triggered according to the transmission priority corresponding to the data category label, and high-priority data packets are retransmitted first.

7. The smart city data acquisition and preprocessing method according to claim 4, characterized in that, The delayed transmission mode includes: The edge preprocessed data that meets the delayed transmission conditions is written to the local persistent storage queue of the edge node according to the data category label and timestamp, and the storage duration of each data entry is recorded. Real-time monitoring of the current edge cloud resource status; when the edge cloud resource status recovers from the third preset condition to below the second preset condition, the retransmission process is automatically triggered. Data to be transmitted is retrieved from the local persistent storage queue in descending order of data transmission priority. The data to be transmitted is then deduplicated and its integrity is checked, and expired data exceeding the preset maximum storage time is removed. The verified data to be transmitted is compressed according to the data compression ratio determined by the data transmission strategy and then sent to the cloud server. After the retransmission is completed, the corresponding data entry in the local persistent storage queue is cleared.

8. The smart city data acquisition and preprocessing method according to claim 1, characterized in that, The data collection frequency is dynamically adjusted according to the type of business scenario, wherein the type of business scenario is selected from at least one of urban emergency command scenario, real-time traffic dispatch scenario, public safety early warning scenario and urban environmental monitoring scenario; A mapping relationship is established between business scenario types and data collection frequencies. When the business scenario type is an urban emergency command scenario, the corresponding data collection frequency is the first frequency; when the business scenario type is a real-time traffic dispatch scenario, the corresponding data collection frequency is the second frequency; and when the business scenario type is a public safety early warning scenario, the corresponding data collection frequency is the third frequency. The dynamic scheduling function of the edge computing nodes adjusts the preprocessing priority and upload threshold in conjunction with the data collection frequency corresponding to the business scenario type; the resource pool management of the cloud server dynamically expands or shrinks each computing node according to the resource allocation strategy output by the scheduling strategy generation stage, so as to support the collaborative adaptation of collection, processing and transmission under different business scenarios.

9. The smart city data acquisition and preprocessing method according to claim 1, characterized in that, The perceived data stream is subjected to quality assessment to generate a data quality score, wherein the data quality assessment is determined based on data integrity, data accuracy, and data timeliness; Data processing priority is determined based on the data quality score, wherein the highest processing priority is assigned when the data quality score is greater than a first preset score; the second highest processing priority is assigned when the data quality score is greater than a second preset score and less than or equal to the first preset score; and the normal processing priority is assigned when the data quality score is less than or equal to the second preset score. The data quality score is fed back to the cloud server for performance evaluation to update the noise feature learning model parameters; and differentiated preprocessing strategies are implemented for data with different processing priorities based on the data category labels. The updated classification model parameters are then distributed to edge computing nodes to form an edge-cloud collaborative optimization closed loop.

10. A smart city data acquisition and preprocessing system, used to execute a smart city data acquisition and preprocessing method as described in any one of claims 1-9, characterized in that, include: Data acquisition module: used to acquire raw sensing data streams at edge nodes in the smart city coverage area; Resource scheduling module: used to obtain the computing resource utilization rate of the edge server in real time based on the original sensing data stream, and formulate dynamic resource scheduling strategies according to the current business load, network bandwidth and edge node computing capabilities; Data processing module: used to perform edge node noise suppression on the original sensing data stream according to the dynamic resource scheduling strategy, and to perform real-time analysis and remove redundant data to retain effective feature data; to perform lightweight real-time preprocessing on the effective feature data to generate standardized data blocks; Data transmission module: used to classify the standardized data block based on the feature vector of the standardized data block and a preset classification strategy, and generate at least one data category label; Generate a data transmission strategy based on the data category labels and the current edge cloud resource status; Data update module: used to determine a preset upload threshold according to the dynamic resource scheduling strategy, and upload the standardized data block according to the data transmission strategy and the preset upload threshold; Based on the standardized data blocks, the dynamic resource scheduling strategy parameters are continuously optimized and updated, and the update results are sent to the edge nodes.

Images