A data processing method and device, electronic equipment and storage medium

Abstract

The application discloses a data processing method, a data processing device, an electronic device and a computer readable storage medium. The method comprises the following steps: obtaining a target data matrix; selecting part of eigenvalues with a principal component contribution rate satisfying a contribution rate condition from principal component eigenvalues corresponding to the target data matrix, performing weighted processing on eigenvectors of the part of eigenvalues to obtain weighted eigenvectors; determining a projection transformation matrix based on the weighted eigenvectors and eigenvectors of remaining eigenvalues in the principal component eigenvalues; and performing data compression processing on second data collected by all detection devices based on the projection transformation matrix to obtain compressed second data. In this way, the data after compression processing is used as data to be transmitted, the network load is reduced, the data transmission efficiency is improved, the consumption of computing resources by massive sensor data is saved, and the influence of the external environment on the original data is reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, a data processing apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the advent of the Internet of Things (IoT) era, an increasing number of smart hardware devices rely on sensors to realize intelligent operations. In the process of realizing intelligent operations, sensors collect a large amount of data. Therefore, this data needs to be processed to improve resource utilization. Summary of the Invention

[0003] This application provides a data processing method, a data processing apparatus, an electronic device, and a computer-readable storage medium, which solves the problem in related technologies that directly use raw data generated by various sensors for network transmission and data processing, resulting in excessive network load and low utilization of computer resources.

[0004] The technical solution of this application is implemented as follows:

[0005] A data processing method, the method comprising:

[0006] Obtain a target data matrix; wherein the target data matrix is ​​obtained by preprocessing the first data collected by the detection device connected to the Internet of Things;

[0007] From the principal component eigenvalues ​​corresponding to the target data matrix, select some eigenvalues ​​whose principal component contribution rates satisfy the contribution rate condition, and perform weighted processing on the eigenvectors of the selected eigenvalues ​​to obtain the weighted eigenvectors.

[0008] Based on the weighted eigenvectors and the eigenvectors of the remaining eigenvalues ​​in the principal component eigenvalues, a projection transformation matrix is ​​determined; wherein the remaining eigenvalues ​​and the partial eigenvalues ​​constitute the principal component eigenvalues.

[0009] Based on the projection transformation matrix, the second data collected by all the detection devices is compressed to obtain compressed second data.

[0010] A data processing apparatus, the processing apparatus comprising:

[0011] An acquisition unit is used to acquire a target data matrix; wherein the target data matrix is ​​obtained by preprocessing the first data collected by the detection device connected to the Internet of Things.

[0012] The processing unit is configured to select a subset of eigenvalues ​​whose principal component contribution rates satisfy the contribution rate condition from the principal component eigenvalues ​​corresponding to the target data matrix, and to perform weighted processing on the eigenvectors of the subset of eigenvalues ​​to obtain a weighted eigenvector.

[0013] The processing unit is further configured to determine a projection transformation matrix based on the weighted feature vector and the feature vector of the remaining feature values ​​in the principal component feature values; wherein the remaining feature values ​​and the partial feature values ​​constitute the principal component feature values;

[0014] The processing unit is further configured to perform data compression processing on the second data collected by all the detection devices based on the projection transformation matrix, so as to obtain compressed second data.

[0015] An electronic device, the electronic device comprising:

[0016] Memory, used to store executable instructions;

[0017] The processor, when executing executable instructions stored in the memory, implements the steps of the data processing method described above.

[0018] A computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the data processing method described above.

[0019] The data processing method, data processing apparatus, electronic device, and computer-readable storage medium provided in this application acquire a target data matrix. The target data matrix is ​​obtained by preprocessing first data collected by a detection device connected to the Internet of Things (IoT). From the principal component eigenvalues ​​corresponding to the target data matrix, a subset of eigenvalues ​​whose contribution rates satisfy the contribution rate condition are selected, and the eigenvectors of these subsets are weighted to obtain a weighted eigenvector. Based on the weighted eigenvector and the eigenvectors of the remaining eigenvalues ​​in the principal component eigenvalues, a projection transformation matrix is ​​determined. The remaining eigenvalues ​​and the subset of eigenvalues ​​constitute the principal component eigenvalues. By using a projection transformation matrix, the second data collected by all detection devices is compressed to obtain compressed second data. In other words, this application uses a projection transformation matrix to compress the original data when all detection devices have collected the original data, and uses the compressed data as the data to be transmitted. This solves the problem in related technologies where the original data generated by various sensors is directly used for network transmission and data processing, resulting in excessive network load and low utilization of computer resources. It improves the efficiency of data transmission, reduces network load, and saves computing resources consumed when processing massive amounts of sensor data, thus reducing computing time. Attached Figure Description

[0020] Figure 1 A schematic diagram of the network architecture for implementing the data processing method provided in this application;

[0021] Figure 2 A flowchart illustrating an optional data processing method provided in this application;

[0022] Figure 3 A flowchart illustrating an optional data processing method provided in this application;

[0023] Figure 4 A flowchart illustrating an optional data processing method provided in this application;

[0024] Figure 5 A schematic diagram of the structure of a data processing device provided in this application;

[0025] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0026] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0027] See Figure 1 , Figure 1 This is a schematic diagram of the network architecture for implementing the data processing method provided in this application. One end of the data processing device 200 is connected to the detection device 100, and the other end is connected to the network 300. This enables the data collected by the detection device 100 to be sent to the server 400. The server 400 then performs logical control of the business based on the data reported by the data processing device 200. For example, the detection device 100 detects the measured information and can transform the detected information into an electrical signal or other required form of information output according to a certain rule to meet the requirements of information transmission, processing, storage, display, recording, and control. The detection device 100 includes, but is not limited to, piezoresistive sensors, weighing sensors, variable frequency power sensors, and Hall effect sensors. The data processing device 200 is a device in the Internet of Things (IoT) that connects the sensor network layer and the transmission network layer, enabling data collection and transmission to the network layer. The data processing device 200 includes, but is not limited to, smartphones, tablets, personal digital assistants (PDAs), laptops, and desktop computers. The network 300 includes, but is not limited to, local area networks (LANs), metropolitan area networks (MANs), and wide area networks (WANs). Server 400 can be a single server, or a server cluster or cloud computing center consisting of multiple servers.

[0028] See Figure 2 , Figure 2 This is a flowchart illustrating the data processing method provided in this application, which is applied to... Figure 1 The data processing apparatus 200 shown; the data processing method includes the following steps:

[0029] Step 201: Obtain the target data matrix.

[0030] The target data matrix is ​​obtained by preprocessing the first data collected by the detection devices connected to the Internet of Things.

[0031] In this embodiment of the application, in the IoT detection scenario, the data processing device first obtains the first data collected by all detection devices within the target time period, and preprocesses the first data to obtain the target data matrix.

[0032] In some embodiments, the data processing device acquires multiple historical data collected by all detection devices over multiple historical time periods; and divides each historical time period into multiple target time periods according to a preset number. It should be noted that the preset number is a positive integer greater than 1, and the user can modify the value of the preset number according to actual needs. For example, if each historical time period is 24 hours and the preset number is 24, then the historical time periods are divided according to the preset number to obtain multiple target time periods, each target time period being 1 hour.

[0033] In this embodiment, the first data record includes data collected by all detection devices, such as m frequency conversion power sensors, within a target time period, for example, 8:00-9:00. It should be noted that the target time period can be a day, an hour, or a minute, etc. For example, a target time period can be one day, or a day can include 24 target time periods, each of which is one hour. The target time period can be set according to actual conditions, and this application does not impose specific limitations on it.

[0034] In this embodiment, in the IoT detection scenario, all detection devices can connect to the cloud using various network access methods, including Wireless-Fidelity (Wi-Fi), Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), 3G, 4G, 5G, and Narrow Band Internet of Things (NBIoT). After connecting to the cloud, each detection device within the IoT can periodically upload data or upload data to the cloud in real time. All detection devices in the IoT detection scenario can interact with the cloud using the standard IoT protocol, Message Queuing Telemetry Transport (MQTT), to exchange control information for business data.

[0035] Step 202: From the principal component eigenvalues ​​corresponding to the target data matrix, select some eigenvalues ​​whose principal component contribution rates meet the contribution rate conditions, and perform weighted processing on the eigenvectors of these eigenvalues ​​to obtain the weighted eigenvectors.

[0036] In this embodiment, after obtaining the target data matrix, for example, a real matrix C of size m×n, the data processing device obtains the eigenvalues ​​of the target data matrix C. For example, the data processing device uses Singular Value Decomposition (SVD) to decompose the target data matrix C into the form shown in formula (1):

[0037] C=U∑V T (1)

[0038] Where U is an m×m identity orthogonal matrix, called the left singular matrix; V is an n×n identity orthogonal matrix, called the right singular matrix. The left singular matrix and its transpose satisfy UU T Given the condition I, the right singular matrix and its transpose satisfy VV. T=I; I is an identity matrix with 1s on the main diagonal and 0s for all other elements. ∑ is a diagonal matrix with non-zero elements only on the main diagonal. It should be noted that each non-zero element on the main diagonal of ∑ is an eigenvalue of the target data matrix C.

[0039] It should be noted that the data processing device uses SVD to decompose the target data matrix C in the following specific process:

[0040] The first step is for the data processing device to calculate the target data matrix C and the transpose C of the target data matrix, respectively. T The first matrix CC after multiplication T ; and the transpose matrix C of the target data matrix. T The second matrix C obtained by multiplying with the target data matrix C T C. It should be noted that the first matrix and the second matrix satisfy the properties in formulas (2) and (3), and the first matrix CC T Second matrix C T C is a matrix with the same number of rows and columns.

[0041] CC T =U∑V T V∑ T U T =U∑∑ T U T (2)

[0042] C T C=V∑ T U T U∑V T =V∑ T ∑V T (3)

[0043] It should be noted that ∑∑ T and ∑ T From a matrix perspective, these are not equal because they have different dimensions, i.e., ∑∑ T For an m×m matrix, ∑ T ∑ are n×n matrices; however, their singular values ​​on the main diagonal are equal.

[0044] Step 2: The data processing device processes the first matrix CC. T Eigenvalue decomposition yields the left singular matrix U.

[0045] Step 3: The data processing device processes the second matrix C. T Eigenvalue decomposition of C yields the right singular matrix V.

[0046] Step 4: The data processing device processes ∑ T ∑ or ∑∑ TBy taking the square root of the eigenvalues ​​in the matrix, we can obtain all the singular values, i.e., the eigenvalues, of the target data matrix C.

[0047] In this embodiment of the application, after the data processing device obtains all the eigenvalues ​​of the target data matrix, for example, all the eigenvalues ​​are λ1, λ2, λ3…λ n Sort all eigenvalues ​​in descending order. For example, if all eigenvalues ​​are sorted in descending order, then all eigenvalues ​​are λ2, λ3, λ1…λ. n Select the first s eigenvalues ​​from all eigenvalues ​​after sorting in descending order, and calculate the weight of the first s eigenvalues ​​to all eigenvalues. The calculation formula is as follows:

[0048]

[0049] Here, γ1 represents the weight of the first s eigenvalues ​​relative to all eigenvalues. When γ1 satisfies formula (5), the selected first s eigenvalues ​​are principal component eigenvalues. That is, the principal component eigenvalues ​​include all eigenvalues ​​arranged in descending order that satisfy the principal component threshold. It should be noted that the data processing device selects principal component eigenvalues ​​that satisfy the principal component threshold from all eigenvalues, and uses the low-dimensional principal component eigenvalues ​​to represent the high-dimensional all eigenvalues, thus completing the dimensionality reduction of the first data.

[0050] γ1≥γ (5)

[0051] Wherein, γ is the principal component threshold. In actual implementation, the principal component threshold can be determined through multiple experiments based on the actual situation. After determining the principal component threshold, the number of principal component eigenvalues ​​can be increased from 1 to n (n is the total number of eigenvalues ​​of the target data matrix) until the proportion of the selected eigenvalue to all eigenvalues ​​satisfies formula (5), then the currently selected eigenvalue is recorded as the principal component eigenvalue. For example, in the scenario of data compression, the data processing device sets the principal component candidate thresholds to 80%, 90%, and 95%, respectively, and performs data compression with these three thresholds, selecting the principal component candidate threshold with the best compression effect as the principal component threshold.

[0052] In this embodiment of the application, after the data processing device selects principal component eigenvalues ​​from all eigenvalues, it obtains the eigenvectors of some eigenvalues ​​in the principal component eigenvalues ​​corresponding to the target data matrix, performs normalization processing on the eigenvectors of some eigenvalues ​​to obtain normalized eigenvectors of some eigenvalues, and performs weighted processing on the normalized eigenvectors to obtain weighted eigenvectors.

[0053] In this embodiment of the application, after the data processing device obtains the principal component eigenvalues ​​corresponding to the target data matrix, it obtains the relationship between the principal component contribution rate and the contribution rate threshold of each eigenvalue, and determines that the eigenvalues ​​with a principal component contribution rate greater than the contribution rate threshold are partial eigenvalues.

[0054] It should be noted that the principal component contribution rate is used to measure the explanatory power of a component for the variability of the original variable.

[0055] Step 203: Determine the projection transformation matrix based on the weighted eigenvectors and the eigenvectors of the remaining eigenvalues ​​in the principal component eigenvalues.

[0056] Among them, the remaining eigenvalues ​​and some eigenvalues ​​constitute the principal component eigenvalues.

[0057] In this embodiment, the data processing device obtains the eigenvectors of the remaining eigenvalues ​​excluding some of the principal component eigenvalues, and normalizes these eigenvectors to obtain normalized eigenvectors of the remaining eigenvalues. Based on the normalized weighted eigenvectors and the normalized eigenvectors of the remaining eigenvalues, a projection transformation matrix is ​​determined. In the data compression and recovery scenario, the projection transformation matrix serves as a template for the data compression and data recovery compression matrix.

[0058] Step 204: Based on the projection transformation matrix, perform data compression processing on the second data collected by all detection devices to obtain compressed second data.

[0059] In this embodiment, the data processing device uses a projection transformation matrix to compress the second data collected by all detection devices to obtain compressed second data. It should be noted that the first data and the second data can be time-series data collected by the detection devices at multiple moments within the same time period; or they can be time-series data collected by the detection devices at multiple moments within different time periods.

[0060] The data processing method provided in this application includes: obtaining a target data matrix; wherein the target data matrix is ​​obtained by preprocessing first data collected by detection devices connected to the Internet of Things; selecting some eigenvalues ​​whose principal component contribution rates satisfy the contribution rate condition from the principal component eigenvalues ​​corresponding to the target data matrix, and weighting the eigenvectors of the eigenvalues ​​to obtain weighted eigenvectors; determining a projection transformation matrix based on the weighted eigenvectors and the eigenvectors of the remaining eigenvalues ​​in the principal component eigenvalues; wherein the remaining eigenvalues ​​and some eigenvalues ​​constitute the principal component eigenvalues; and performing data compression processing on second data collected by all detection devices based on the projection transformation matrix to obtain compressed second data. In other words, this application uses a projection transformation matrix to compress the original data when all detection devices have collected it, and then uses the compressed data as the data to be transmitted. This solves the problem in related technologies where the original data generated by various sensors is directly used for network transmission and data processing, resulting in excessive network load and low utilization of computer resources. It improves the efficiency of data transmission, reduces network load, saves computing resources consumed when processing massive amounts of sensor data, reduces computing time, and also reduces the impact of the external environment on the original data.

[0061] In other embodiments of this application, step 201, obtaining the target data matrix, can be achieved through the following steps:

[0062] The first step is to acquire the first time-series data collected by all detection devices connected to the Internet of Things at multiple moments within the target time period.

[0063] The second step is to arrange the first data collected by each detection device at different times into an array, with each time point as a different dimension, to obtain an initial data matrix.

[0064] In this embodiment, the data processing device takes the first data collected by each detection device at different times, uses the type of detection device corresponding to the first data as the row variable, the different times at which the first data was collected by at least two detection devices as column variables, and the first data as the element, and arranges the first data in an array with different times as different dimensions to obtain an initial data matrix. For example, if there are m detection devices and n times, the initial data matrix is ​​a matrix of size m×n.

[0065] The third step is to remove the mean from the data in each dimension of the initial data matrix to obtain the target data matrix.

[0066] In this embodiment of the application, in order to prevent overfitting of the data, after obtaining the initial data matrix, the data processing device calculates the average value of the data collected by all detection devices at each time moment, that is, the average value of each row of the initial data matrix; then, the data processing device subtracts the average value of the current time from the data collected by all detection devices at each time moment, that is, subtracts the average value of each row of the initial data matrix from the data of each row, to obtain the mean-free initial data matrix.

[0067] In this embodiment, the data processing device obtains the covariance matrix of the mean-free initial data matrix, i.e., the target data matrix, based on the mean-free initial data matrix and the number of all detection devices. For example, in an IoT detection scenario with m detection devices and a mean-free initial data matrix X, the target data matrix C is... Among them, X T This is the transpose of the initial data matrix after mean reduction.

[0068] See Figure 3 , Figure 3 This is a flowchart illustrating the data processing method provided in this application, which is applied to... Figure 1 The data processing apparatus 200 shown; the data processing method includes the following steps:

[0069] Step 301: Obtain the target data matrix.

[0070] The target data matrix is ​​obtained by preprocessing the first data collected by the detection devices connected to the Internet of Things.

[0071] Step 302: Select some eigenvalues ​​from the principal component eigenvalues ​​whose contribution rates satisfy the contribution rate condition.

[0072] Step 303: Determine the weight of the eigenvector of each eigenvalue in the partial eigenvalues.

[0073] In this embodiment, the data processing device determines the weight of the eigenvector of each eigenvalue based on the principal component contribution rate corresponding to each eigenvalue.

[0074] It should be noted that there is a negative correlation between the principal component contribution rate corresponding to each eigenvalue and the weight of the eigenvector of each eigenvalue; that is, the higher the influence of the principal component contribution rate corresponding to each eigenvalue, the lower the weight of the eigenvector of each eigenvalue. This application determines the weight of some eigenvalues ​​whose principal component contribution rate meets the contribution rate condition, and multiplies the eigenvector of each eigenvalue in the partial eigenvalues ​​by the corresponding weight. In this way, the difference between each eigenvalue in the partial eigenvalues ​​can be effectively reduced, the influence of the eigenvalues ​​most susceptible to environmental pollution on the data can be reduced, and the pollution caused by the external environment to the sensor data can be reduced.

[0075] In some embodiments, there is a positive correlation between the principal component contribution rate corresponding to each feature value and the weight of the feature vector of each feature value; that is, the higher the influence of the principal component contribution rate corresponding to each feature value, the higher the weight of the feature vector of each feature value. This application determines the weight of some feature values ​​whose principal component contribution rate meets the contribution rate condition, and multiplies the feature vector of each feature value in the partial feature values ​​by the corresponding weight. In this way, for scenarios that only consider a certain type of component, the recognition of the corresponding component by increasing the proportion of a certain type of component is improved.

[0076] Step 304: Based on the eigenvector of each eigenvalue and the weight of each eigenvector, perform weighted processing to obtain the weighted eigenvector.

[0077] Step 305: Determine the projection transformation matrix based on the weighted eigenvectors and the eigenvectors of the remaining eigenvalues ​​in the principal component eigenvalues.

[0078] Among them, the remaining eigenvalues ​​and some eigenvalues ​​constitute the principal component eigenvalues.

[0079] Step 306: Based on the projection transformation matrix, perform data compression processing on the second data collected by all detection devices to obtain compressed second data.

[0080] Step 307: Obtain the inverse matrix of the projection transformation matrix.

[0081] Step 308: Based on the inverse matrix, perform data decompression on the compressed second data to obtain the decompressed second data.

[0082] In this embodiment, for compressed data, if a scenario requires the reproduction of the original data, the data processing device obtains the inverse matrix of the projection transformation matrix and, based on the inverse matrix, performs data recovery (decompression) on the compressed second data to obtain the data before compression. When the user needs to process the compressed original data, the inverse matrix of the projection transformation matrix is ​​used to effectively restore the data, effectively meeting the needs of the actual scenario.

[0083] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions of other embodiments, and will not be repeated here.

[0084] See Figure 4 , Figure 4 This is a flowchart illustrating the data processing method provided in this application, which is applied to... Figure 1 The data processing apparatus 200 shown; the data processing method includes the following steps:

[0085] Step 401: Obtain the target data matrix.

[0086] The target data matrix is ​​obtained by preprocessing the first data collected by the detection device connected to the Internet of Things.

[0087] Step 402: From the principal component eigenvalues ​​corresponding to the target data matrix, select some eigenvalues ​​whose principal component contribution rates meet the contribution rate conditions, and perform weighted processing on the eigenvectors of these eigenvalues ​​to obtain the weighted eigenvectors.

[0088] Step 403: Determine the projection transformation matrix based on the weighted eigenvectors and the eigenvectors of the remaining eigenvalues ​​in the principal component eigenvalues.

[0089] Among them, the remaining eigenvalues ​​and some eigenvalues ​​constitute the principal component eigenvalues.

[0090] Step 404: If the third data collected by all detection devices in the Internet of Things is obtained, and the correlation between the third data and the first data collected by all detection devices corresponding to the projection transformation matrix conforms to the target correlation, the projection transformation matrix is ​​updated based on the third data collected by all detection devices to obtain the updated projection transformation matrix.

[0091] In this embodiment of the application, in an IoT detection scenario, if the data processing device detects that the change between the first data collected by all detection devices and the third data collected by all detection devices exceeds a preset fluctuation range, the data processing device updates the projection transformation matrix based on the third data collected by all detection devices to obtain an updated projection transformation matrix. Here, the third data is the data collected after all detection devices have collected the first data. That is to say, when compressing the original data collected by the detection devices using the projection transformation matrix, this application also considers the changes in the original data collected by the detection devices. When the changes in the original data collected by the detection devices are large, the projection transformation matrix is ​​updated, so that the compressed data can better represent the original data, effectively improving the compression effect of the data processing device when compressing the original data collected by the detection devices.

[0092] In this embodiment of the application, in an IoT detection scenario, if the data processing device detects that the correlation between the first data collected by all detection devices and the third data collected by all detection devices corresponding to the projection transformation matrix is ​​less than a correlation threshold, the data processing device updates the projection transformation matrix based on the third data collected by all detection devices to obtain the updated projection transformation matrix. Here, the correlation threshold is used to characterize the changes between data; the higher the correlation threshold, the smaller the degree of change.

[0093] In IoT detection scenarios, the optimal principal components may undergo subtle changes over time. In subsequent time intervals, the data compression template can be updated periodically using the data compression template calculated in a time interval close to the current time interval.

[0094] Step 405: Based on the updated projection transformation matrix, perform data compression processing on the third data collected by all detection devices to obtain compressed third data.

[0095] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0096] This application provides a data processing apparatus that can be applied to... Figures 2 to 4 In the provided test data generation method, refer to Figure 5 As shown, the data processing device 200 includes:

[0097] The acquisition unit 501 is used to acquire the target data matrix; wherein, the target data matrix is ​​obtained by preprocessing the first data collected by the detection device connected to the Internet of Things;

[0098] The processing unit 502 is used to select some eigenvalues ​​whose contribution rate satisfies the contribution rate condition from the principal component eigenvalues ​​corresponding to the target data matrix, perform weighted processing on the eigenvectors of the partial eigenvalues ​​to obtain weighted eigenvectors; determine the projection transformation matrix based on the weighted eigenvectors and the eigenvectors of the remaining eigenvalues ​​in the principal component eigenvalues; wherein the remaining eigenvalues ​​and partial eigenvalues ​​constitute the principal component eigenvalues; and perform data compression processing on the second data collected by all detection devices based on the projection transformation matrix to obtain compressed second data.

[0099] In other embodiments of this application, the data processing apparatus 200 further includes:

[0100] The processing unit 502 is used to determine the weight of the eigenvector of each eigenvalue in the partial eigenvalues; and to perform weighted processing based on the eigenvector of each eigenvalue and the weight of each eigenvector to obtain the weighted eigenvector.

[0101] In other embodiments of this application, the data processing apparatus 200 further includes:

[0102] The processing unit 502 is used to determine the weight of the eigenvector of each eigenvalue based on the principal component contribution rate corresponding to each eigenvalue; wherein there is a negative correlation between the principal component contribution rate corresponding to each eigenvalue and the weight of the eigenvector of each eigenvalue.

[0103] In other embodiments of this application, the data processing apparatus 200 further includes:

[0104] Acquisition unit 501 is used to obtain the inverse matrix of the projection transformation matrix;

[0105] The processing unit 502 is used to perform data decompression processing on the compressed second data based on the inverse matrix to obtain the decompressed second data.

[0106] In other embodiments of this application, the data processing apparatus 200 further includes:

[0107] The processing unit 502 is used to update the projection transformation matrix based on the third data collected by all detection devices connected to the Internet of Things if the correlation between the third data collected by all detection devices and the first data collected by all detection devices corresponding to the projection transformation matrix conforms to the target correlation relationship, and to obtain the updated projection transformation matrix.

[0108] The processing unit 502 is used to perform data compression processing on the third data collected by all detection devices based on the updated projection transformation matrix to obtain compressed third data.

[0109] In other embodiments of this application, the data processing apparatus 200 further includes:

[0110] The acquisition unit 501 is used to acquire first data with time series collected by all detection devices in the Internet of Things at multiple moments within a target time period;

[0111] The processing unit 502 is used to arrange the first data collected by each detection device at different times in an array according to different dimensions to obtain an initial data matrix; and to perform mean-averaging on the data in each dimension of the initial data matrix to obtain a target data matrix.

[0112] The data processing apparatus provided in this application acquires a target data matrix. The target data matrix is ​​obtained by preprocessing first data collected by detection devices connected to the Internet of Things. From the principal component eigenvalues ​​corresponding to the target data matrix, a subset of eigenvalues ​​whose contribution rates satisfy the contribution rate condition are selected. The eigenvectors of these subsets are then weighted to obtain a weighted eigenvector. Based on the weighted eigenvector and the eigenvectors of the remaining eigenvalues ​​in the principal component eigenvalues, a projection transformation matrix is ​​determined. The remaining eigenvalues ​​and the subset of eigenvalues ​​constitute the principal component eigenvalues. Based on the projection transformation matrix, second data collected by all detection devices is compressed to obtain compressed second data. In other words, this application uses a projection transformation matrix to compress the original data when all detection devices have collected it, and then uses the compressed data as the data to be transmitted. This solves the problem in related technologies where the original data generated by various sensors is directly used for network transmission and data processing, resulting in excessive network load and low utilization of computer resources. It improves the efficiency of data transmission, reduces network load, saves computing resources consumed when processing massive amounts of sensor data, reduces computing time, and also reduces the impact of the external environment on the original data.

[0113] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0114] This application provides an electronic device that can be applied to... Figures 2 to 4 In the provided test data generation method, refer to Figure 6 As shown, the electronic device 6 (corresponding to the data processing device 200) includes: a memory 601 and a processor 602, and a communication bus 603, wherein:

[0115] The communication bus 603 is used to realize the communication connection between the processor 602 and the memory 601.

[0116] The processor 602 is used to execute the data processing program stored in the memory 601 to perform the following steps:

[0117] Obtain the target data matrix; wherein, the target data matrix is ​​obtained by preprocessing the first data collected by the detection device connected to the Internet of Things;

[0118] From the principal component eigenvalues ​​corresponding to the target data matrix, select some eigenvalues ​​whose contribution rates satisfy the contribution rate condition, and perform weighted processing on the eigenvectors of some eigenvalues ​​to obtain weighted eigenvectors; based on the weighted eigenvectors and the eigenvectors of the remaining eigenvalues ​​in the principal component eigenvalues, determine the projection transformation matrix; where the remaining eigenvalues ​​and some eigenvalues ​​constitute the principal component eigenvalues;

[0119] Based on the projection transformation matrix, the second data collected by all detection devices is compressed to obtain the compressed second data.

[0120] The processor 602 is used to execute the data processing program stored in the memory 601 to perform the following steps:

[0121] Determine the weight of the eigenvector for each eigenvalue in the subset of eigenvalues;

[0122] Based on the eigenvector of each eigenvalue and the weight of each eigenvector, a weighted eigenvector is obtained.

[0123] The processor 602 is used to execute the data processing program stored in the memory 601 to perform the following steps:

[0124] The weight of the eigenvector of each eigenvalue is determined based on the principal component contribution rate corresponding to each eigenvalue; there is a negative correlation between the principal component contribution rate corresponding to each eigenvalue and the weight of the eigenvector of each eigenvalue.

[0125] The processor 602 is used to execute the data processing program stored in the memory 601 to perform the following steps:

[0126] Obtain the inverse of the projection transformation matrix;

[0127] Based on the inverse matrix, the compressed second data is decompressed to obtain the decompressed second data.

[0128] The processor 602 is used to execute the data processing program stored in the memory 601 to perform the following steps:

[0129] If the correlation between the third data collected by all detection devices in the obtained IoT detection scenario and the first data collected by all detection devices corresponding to the projection transformation matrix conforms to the target correlation relationship, the projection transformation matrix is ​​updated based on the third data collected by all detection devices to obtain the updated projection transformation matrix.

[0130] Based on the updated projection transformation matrix, the third data collected by all detection devices is compressed to obtain compressed third data.

[0131] The processor 602 is used to execute the data processing program stored in the memory 601 to perform the following steps:

[0132] Acquire first time-series data collected by at least two detection devices in an IoT detection scenario at multiple moments within a target time period;

[0133] The first data collected by each detection device at different times are arranged in an array according to different dimensions of different times to obtain an initial data matrix;

[0134] The data in each dimension of the initial data matrix is ​​demeaned to obtain the target data matrix.

[0135] The electronic device provided in this application acquires a target data matrix; wherein the target data matrix is ​​a matrix obtained by preprocessing the first data collected by the detection devices connected to the Internet of Things; from the principal component eigenvalues ​​corresponding to the target data matrix, select some eigenvalues ​​whose principal component contribution rates meet the contribution rate conditions, and perform weighted processing on the eigenvectors of some eigenvalues ​​to obtain weighted eigenvectors; based on the weighted eigenvectors and the eigenvectors of the remaining eigenvalues ​​in the principal component eigenvalues, determine the projection transformation matrix; wherein the remaining eigenvalues ​​and some eigenvalues ​​constitute the principal component eigenvalues; based on the projection transformation matrix, perform data compression processing on the second data collected by all detection devices to obtain compressed second data. In other words, this application uses a projection transformation matrix to compress the original data when all detection devices have collected it, and then uses the compressed data as the data to be transmitted. This solves the problem in related technologies where the original data generated by various sensors is directly used for network transmission and data processing, resulting in excessive network load and low utilization of computer resources. It improves the efficiency of data transmission, reduces network load, saves computing resources consumed when processing massive amounts of sensor data, reduces computing time, and also reduces the impact of the external environment on the original data.

[0136] It should be noted that the specific implementation process of the steps executed by the processor 602 in this embodiment can be referred to Figures 2 to 4 The implementation process of the provided data processing method will not be elaborated here.

[0137] Based on the foregoing embodiments, this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to perform the following steps:

[0138] Obtain the target data matrix; wherein, the target data matrix is ​​obtained by preprocessing the first data collected by the detection device connected to the Internet of Things;

[0139] From the principal component eigenvalues ​​corresponding to the target data matrix, select some eigenvalues ​​whose principal component contribution rates satisfy the contribution rate condition, and perform weighted processing on the eigenvectors of some eigenvalues ​​to obtain the weighted eigenvectors.

[0140] The projection transformation matrix is ​​determined based on the weighted eigenvectors and the eigenvectors of the remaining eigenvalues ​​in the principal component eigenvalues; where the remaining eigenvalues ​​and some eigenvalues ​​constitute the principal component eigenvalues.

[0141] Based on the projection transformation matrix, the second data collected by all detection devices is compressed to obtain the compressed second data.

[0142] In other embodiments of this application, the one or more programs may be executed by one or more processors, and may also perform the following steps:

[0143] Determine the weight of the eigenvector for each eigenvalue in the subset of eigenvalues;

[0144] Based on the eigenvector of each eigenvalue and the weight of each eigenvector, a weighted eigenvector is obtained.

[0145] In other embodiments of this application, the one or more programs may be executed by one or more processors, and may also perform the following steps:

[0146] The weight of the eigenvector of each eigenvalue is determined based on the principal component contribution rate corresponding to each eigenvalue; there is a negative correlation between the principal component contribution rate corresponding to each eigenvalue and the weight of the eigenvector of each eigenvalue.

[0147] In other embodiments of this application, the one or more programs may be executed by one or more processors, and may also perform the following steps:

[0148] Obtain the inverse of the projection transformation matrix;

[0149] Based on the inverse matrix, the compressed second data is decompressed to obtain the decompressed second data.

[0150] In other embodiments of this application, the one or more programs may be executed by one or more processors, and may also perform the following steps:

[0151] If the correlation between the third data collected by all detection devices in the obtained IoT detection scenario and the first data collected by all detection devices corresponding to the projection transformation matrix conforms to the target correlation relationship, the projection transformation matrix is ​​updated based on the third data collected by all detection devices to obtain the updated projection transformation matrix.

[0152] Based on the updated projection transformation matrix, the third data collected by all detection devices is compressed to obtain compressed third data.

[0153] In other embodiments of this application, the one or more programs may be executed by one or more processors, and may also perform the following steps:

[0154] Acquire the first time-series data collected by all detection devices connected to the Internet of Things at multiple moments within a target time period;

[0155] The first data collected by each detection device at different times are arranged in an array according to different dimensions of different times to obtain an initial data matrix;

[0156] The data in each dimension of the initial data matrix is ​​demeaned to obtain the target data matrix.

[0157] The computer-readable storage medium provided in this application acquires a target data matrix. The target data matrix is ​​obtained by preprocessing first data collected by detection devices connected to the Internet of Things. From the principal component eigenvalues ​​corresponding to the target data matrix, some eigenvalues ​​whose principal component contribution rates satisfy the contribution rate condition are selected, and the eigenvectors of these eigenvalues ​​are weighted to obtain a weighted eigenvector. Based on the weighted eigenvector and the eigenvectors of the remaining eigenvalues ​​in the principal component eigenvalues, a projection transformation matrix is ​​determined. The remaining eigenvalues ​​and some eigenvalues ​​constitute the principal component eigenvalues. Based on the projection transformation matrix, second data collected by all detection devices is compressed to obtain compressed second data. In other words, this application uses a projection transformation matrix to compress the original data when all detection devices have collected it, and then uses the compressed data as the data to be transmitted. This solves the problem in related technologies where the original data generated by various sensors is directly used for network transmission and data processing, resulting in excessive network load and low utilization of computer resources. It improves the efficiency of data transmission, reduces network load, saves computing resources consumed when processing massive amounts of sensor data, reduces computing time, and also reduces the impact of the external environment on the original data.

[0158] It should be noted that the specific implementation process of the steps executed by the processor in this embodiment can be referred to Figures 2 to 4 The implementation process of the provided data processing method will not be elaborated here.

[0159] It should be noted that the aforementioned computer storage media / memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; it can also be various terminals that include one or any combination of the above-mentioned memory, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0160] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0161] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0162] Furthermore, in the various embodiments of this application, all functional units can be integrated into one processing module, or each unit can be a separate unit, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units. Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0163] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0164] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0165] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0166] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the target data matrix; wherein, the target data matrix is ​​obtained by preprocessing the first data collected by the sensing devices connected to the Internet of Things during the target time period; From the principal component eigenvalues ​​corresponding to the target data matrix, select some eigenvalues ​​whose principal component contribution rates satisfy the contribution rate condition, and perform weighted processing on the eigenvectors of the selected eigenvalues ​​to obtain the weighted eigenvectors. Based on the weighted eigenvectors and the eigenvectors of the remaining eigenvalues ​​in the principal component eigenvalues, a projection transformation matrix is ​​determined; wherein the remaining eigenvalues ​​and the partial eigenvalues ​​constitute the principal component eigenvalues. Based on the projection transformation matrix, the second data collected by all sensing devices is compressed to obtain compressed second data. Obtain the inverse of the projection transformation matrix; Based on the inverse matrix, the compressed second data is decompressed to obtain the decompressed second data; The compressed second data is used as the detection data of the sensing device for network transmission and data processing. If the third data collected by all the sensors in the Internet of Things is obtained, and the correlation between the third data and the first data collected by all the sensors corresponding to the projection transformation matrix conforms to the target correlation relationship, the projection transformation matrix is ​​updated based on the third data collected by all the sensors to obtain the updated projection transformation matrix. Based on the updated projection transformation matrix, the third data collected by all the sensing devices is compressed to obtain compressed third data.

2. The data processing method according to claim 1, characterized in that, The weighted processing of the eigenvectors of the aforementioned eigenvalues ​​to obtain weighted eigenvectors includes: Determine the weight of the eigenvector for each eigenvalue in the subset of eigenvalues; Based on the eigenvector of each eigenvalue and the weight of each eigenvector, a weighted eigenvector is obtained.

3. The data processing method according to claim 2, characterized in that, Determining the weight of the eigenvector of each eigenvalue in the subset of eigenvalues ​​includes: The weight of the eigenvector of each eigenvalue is determined based on the principal component contribution rate corresponding to each eigenvalue.

4. The data processing method according to claim 3, characterized in that, There is a negative correlation between the principal component contribution rate corresponding to each feature value and the weight of the feature vector of each feature value.

5. The data processing method according to any one of claims 1 to 4, characterized in that, The acquisition of the target data matrix includes: Acquire the first time-series data collected by all sensing devices connected to the Internet of Things at multiple moments within a target time period; The first data collected by each sensor at different times are arranged in an array with the different times as different dimensions to obtain an initial data matrix. The target data matrix is ​​obtained by removing the mean from the data in each dimension of the initial data matrix.

6. A data processing apparatus, characterized in that, The processing device includes: An acquisition unit is used to acquire a target data matrix; wherein the target data matrix is ​​obtained by preprocessing the first data collected by the sensing devices connected to the Internet of Things during a target time period; The processing unit is configured to select a subset of eigenvalues ​​whose principal component contribution rates satisfy the contribution rate condition from the principal component eigenvalues ​​corresponding to the target data matrix, and to perform weighted processing on the eigenvectors of the subset of eigenvalues ​​to obtain a weighted eigenvector. The processing unit is further configured to determine a projection transformation matrix based on the weighted feature vector and the feature vector of the remaining feature values ​​in the principal component feature values; wherein the remaining feature values ​​and the partial feature values ​​constitute the principal component feature values; The processing unit is also used to perform data compression processing on the second data collected by all sensing devices based on the projection transformation matrix to obtain compressed second data. The acquisition unit is further configured to acquire the inverse matrix of the projection transformation matrix; The processing unit is further configured to perform data decompression processing on the compressed second data based on the inverse matrix to obtain the decompressed second data; The processing unit is also used to use the compressed second data as the detection data of the sensing device for network transmission and data processing. The acquisition unit is further configured to, if the third data collected by all the sensing devices in the Internet of Things is acquired and the correlation between the third data and the first data collected by all the sensing devices corresponding to the projection transformation matrix conforms to the target correlation relationship, update the projection transformation matrix based on the third data collected by all the sensing devices to obtain the updated projection transformation matrix. The acquisition unit is further configured to perform data compression processing on the third data collected by all the sensing devices based on the updated projection transformation matrix, so as to obtain compressed third data.

7. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the data processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the data processing method as described in any one of claims 1 to 5.

Images