A data processing method and apparatus

Abstract

This application discloses a data processing method and apparatus, belonging to the field of computer science. The method first acquires first operational data of a target device, then labels the first operational data based on preset tags to obtain second operational data carrying metadata tags corresponding to the target device. Based on the metadata tags and the compression factor corresponding to the target device, the sampling frequency of the second operational data is compressed to obtain first-level compressed data. Second-level compressed data is obtained by frequency domain processing of the first-level compressed data. Joint feature data is constructed based on the second-level compressed data and the metadata tags. Finally, the joint feature data is input into a pre-trained fault identification model to obtain the fault identification result of the target device. This method can overcome the limitation of the context length of the large model, preserving the interpretability and physical meaning of the data while compressing it, thereby improving the accuracy and timeliness of fault identification.

Description

Technical Field

[0001] This application belongs to the field of computer technology, specifically relating to a data processing method and apparatus. Background Technology

[0002] With the development of artificial intelligence technology, the application of large language models is becoming increasingly widespread. Taking factory system operation data as an example, large language models can solve problems such as limitations of human experience, blind spots, and low efficiency. They can assist factory maintenance personnel in electronics factories to quickly uncover hidden system correlations and capture subtle early signs of equipment sub-health from massive amounts of data. However, factory system data is usually primarily time-series data. Large language models, limited by the length of the context during inference, find it difficult to directly perform coherent analysis of the entire lifecycle and all dimensions of raw data. Therefore, how to process the data input into the large language model is a crucial issue.

[0003] Currently, there are two common data processing methods. The first is to convert time-series data into images, that is, to transform the analysis problem of time-series data into the problem of image classification or object detection in computer vision. The second is to use automatic summarization technology to generate semantic representation information, that is, to use an encoder or summarization model to compress long-series data into a highly compact semantic vector or symbolic representation that retains key information.

[0004] However, the first data processing method often leads to the loss of the physical meaning of data, such as precise numerical values ​​and timestamps. Large language models struggle to understand and reason about the relationships between these physical quantities, resulting in poor fault identification accuracy. The second data processing method typically leads to poor interpretability of large models, making it difficult to understand the specific meaning of the generated semantic vectors. Especially when large models make incorrect identifications, the poor interpretability makes debugging and attribution of errors extremely difficult, also leading to poor fault identification accuracy. Therefore, there is a need to provide a data processing method that can overcome the contextual limitations of large language models while improving the accuracy of fault identification. Summary of the Invention

[0005] The purpose of this application is to provide a data processing method and apparatus that can solve the problem of poor fault identification accuracy in large language models caused by data processing methods in related technologies.

[0006] This application provides a data processing method, including: Acquire the first operating data of the target device, wherein the first operating data is time-series data; The first running data is labeled with preset tags to obtain second running data carrying metadata tags corresponding to the target device; Based on the metadata tag and the compression factor corresponding to the target device, the sampling frequency of the second running data is compressed to obtain first-level compressed data. Second-level compressed data is obtained by frequency domain processing of the first-level compressed data. Based on the second-level compressed data and the metadata tag, joint feature data corresponding to the first running data of the target device is constructed. The joint feature data is input into a pre-trained fault identification model to obtain the fault identification result of the target device. The fault identification model is iteratively trained by adjusting the compression factor of the device.

[0007] In this application, the preset tags include: preset metadata tags and / or tags generated according to preset data labeling standards.

[0008] In this application, before annotating the first running data based on preset tags to obtain the second running data carrying metadata tags corresponding to the target device, the method further includes: Based on the timestamp of the first running data, the missing data is filled in to obtain the first running data after completion.

[0009] In this application, the method for completing missing data based on the timestamp of the first running data includes: Based on the timestamp of the first running data, determine the two sets of data adjacent to and before the missing data; The missing data is determined based on the two sets of data adjacent to the missing data before and after it. Based on the missing data and the first running data, the completed first running data is determined.

[0010] In this application, before compressing the sampling frequency of the second operating data to obtain first-level compressed data based on the metadata tag and the compression factor corresponding to the target device, and obtaining second-level compressed data by frequency domain processing of the first-level compressed data, and before constructing joint feature data corresponding to the first operating data of the target device based on the second-level compressed data and the metadata tag, the method further includes: Convert the timestamps of the first running data into a continuous timeline; The process of obtaining secondary compressed data by frequency domain processing of the primary compressed data includes: The first-level compressed data is subjected to Fourier transform processing to obtain the second-level compressed data.

[0011] In this application, the compression factor corresponding to the target device is determined based on the historical fault data of the target device, and the compression factor is negatively correlated with the fault mechanism data of the target device.

[0012] In this application, the model training method for the large-scale fault identification model includes: Multiple historical first running data from multiple devices are acquired, and the multiple historical first running data of each device are labeled to obtain historical second running data carrying historical metadata tags corresponding to each device, wherein the historical first running data is time-series data; Based on the historical metadata tags and the compression factor corresponding to each device, the sampling frequency of the historical second operating data is compressed to obtain historical first-level compressed data. The historical first-level compressed data is frequency domain-processed to obtain historical second-level compressed data. Based on the historical second-level compressed data and the historical metadata tags, a joint feature data sample corresponding to the historical first operating data of each device is constructed. The joint feature data samples are input into the fault identification large model. By adjusting the compression factor of each device, the fault identification large model is trained using a preset loss function until the proportion of the number of fault identification failure samples to the number of test samples in the joint feature data samples is less than or equal to a preset proportion, or the difference between the fault identification large model recognition accuracy of the next iteration cycle and the fault identification large model recognition accuracy of the previous iteration cycle is less than a preset difference. The model training is then terminated, and the trained fault identification large model is obtained.

[0013] In this application, the metadata tag includes: a device metadata tag for the target device and data point metadata tags for multiple data points related to the target device. The step of constructing joint feature data corresponding to the first operational data of the target device based on the secondary compressed data and the metadata tags includes: Based on the secondary compressed data, the total energy of multiple data points related to the target device, the fault status code of the target device, and the frequency corresponding to a preset number of peak values ​​are determined. Based on the total energy of the determined multiple data points, the fault status code of the target device, the frequency corresponding to a preset number of peak values, the device metadata tag of the target device, and the data point metadata tags of the multiple data points related to the target device, joint feature data corresponding to the first operating data of the target device is constructed.

[0014] This application provides a data processing apparatus, the apparatus comprising: The first operational data acquisition module is used to acquire the first operational data of the target device, wherein the first operational data is time-series data. The second operation data acquisition module is used to label the first operation data based on preset tags to obtain second operation data carrying metadata tags corresponding to the target device; The data compression module is used to compress the sampling frequency of the second running data based on the metadata tag and the compression factor corresponding to the target device to obtain first-level compressed data, to obtain second-level compressed data by frequency domain processing of the first-level compressed data, and to construct joint feature data corresponding to the first running data of the target device based on the second-level compressed data and the metadata tag. The fault identification module is used to input the joint feature data into a pre-trained fault identification model to obtain the fault identification result of the target device. The fault identification model is trained iteratively by adjusting the compression factor of the device.

[0015] In this application, the preset tags in the second running data acquisition module include: preset metadata tags and / or tags generated according to preset data labeling standards.

[0016] In this application, the device further includes: a data completion module, used to complete the missing data according to the timestamp of the first running data to obtain the completed first running data.

[0017] In this application, the data completion module includes: The adjacent data determination unit is used to determine two sets of data adjacent to the missing data, one before and one after, based on the timestamp of the first running data. The missing data determination unit is used to determine the missing data based on the two sets of data adjacent to the missing data before and after it. The first running data determination unit is used to determine the completed first running data based on the missing data and the first running data.

[0018] In this application, the device further includes: a timeline conversion module, used to convert the timestamps of the first running data into a continuous timeline; The data compression module is further configured to obtain secondary compressed data by performing Fourier transform processing on the primary compressed data.

[0019] In this application, the compression factor corresponding to the target device in the data compression module is determined based on the historical fault data of the target device, and the compression factor is negatively correlated with the fault mechanism data of the target device.

[0020] In this application, the apparatus further includes: a model training module, configured to acquire multiple historical first operating data from multiple devices, and label the multiple historical first operating data of each device to obtain historical second operating data carrying historical metadata tags corresponding to each device, wherein the historical first operating data is time-series data; based on the historical metadata tags and the compression factor corresponding to each device, compress the sampling frequency of the historical second operating data to obtain historical first-level compressed data, and perform frequency domain processing on the historical first-level compressed data to obtain historical second-level compressed data, and construct joint feature data samples corresponding to the historical first operating data of each device based on the historical second-level compressed data and the historical metadata tags; input the joint feature data samples into a fault identification large model, and train the fault identification large model using a preset loss function by adjusting the compression factor of each device until the proportion of the number of fault identification failure samples to the number of test samples in the joint feature data samples is less than or equal to a preset proportion, or the difference between the fault identification large model recognition accuracy of the later iteration cycle and the fault identification large model recognition accuracy of the previous iteration cycle is less than a preset difference, at which point the model training terminates, and the trained fault identification large model is obtained.

[0021] In this application, the metadata tags in the second running data acquisition module include: the device metadata tag of the target device and the data point metadata tags of multiple data points related to the target device; The data compression module is further configured to: determine the total energy of multiple data points related to the target device, the fault status code of the target device, and the frequency corresponding to a preset number of peak values ​​based on the secondary compressed data; and construct joint feature data corresponding to the first operating data of the target device based on the determined total energy of multiple data points, the fault status code of the target device, the frequency corresponding to a preset number of peak values, the device metadata tag of the target device, and the data point metadata tags of the multiple data points related to the target device.

[0022] This application provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the data processing method provided above.

[0023] The technical solution provided in this application may include the following beneficial effects: This application provides a data processing method. After acquiring the first operational data of the target device, it labels the data with preset tags to obtain the second operational data. This method automatically labels time-series data based on metadata tags, giving the collected data semantic information and making it identifiable, traceable, and associative. This preserves the contextual information of the data, overcoming the limitations of large model context length and providing a basis for understanding the logical relationships between target devices and between target devices and data points in large-scale fault identification models. By compressing the sampling frequency of the second operational data to obtain first-level compressed data, the amount of collected data can be significantly reduced. This compression process is based on metadata tags and compression factors corresponding to the target device, preserving the interpretability and physical meaning of the data while compressing it. Different compression factors correspond to different device types and device health states, enabling adaptive data compression and greatly reducing the probability of fault information being buried. By performing frequency domain processing on the first-level compressed data to obtain second-level compressed data, data storage and computational overhead can be significantly reduced, saving resources. The method of constructing joint feature data based on secondary compressed data and metadata tags enables the fault identification large-scale model to understand the logical relationships between devices and between devices and data points by combining contextual information, thereby improving the identification accuracy and timeliness of the fault identification large-scale model. Furthermore, the fault identification large-scale model in this invention is obtained by adjusting the compression factor of the devices during model training. That is, the compression factor can be adjusted in a timely manner according to the accuracy of the model identification during the model training process, rather than using a fixed compression factor. This approach also helps to improve the accuracy and efficiency of fault identification. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a data processing method disclosed in an embodiment of this application; Figure 2 This is a schematic diagram of the process for annotating the first running data as disclosed in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the model training principle of the fault identification large model disclosed in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of a data processing device disclosed in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application.

[0025] Explanation of reference numerals in the attached drawings: 210: First operating data acquisition module; 220: Second operating data acquisition module; 230: Data compression module; 240: Fault identification module; 301: Processor; 302: Memory; 303: Power supply; 304: Wired or wireless network interface; 305: Input / output interface; 306: Keyboard. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0027] The technical solutions disclosed in the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0028] Large Language Model (LLM) can address the limitations of human experience, blind spots, and low efficiency, assisting factory maintenance personnel in quickly uncovering hidden system correlations and capturing subtle precursors to equipment health issues from massive amounts of data. However, current AI capabilities still face challenges in meeting the demands of system data diagnostic scenarios: First, LLM's processing capacity is limited. System data is primarily time-series, and LLM's limitation on context length restricts diagnostic scenarios involving long-term, multi-dimensional data such as load change trends, significantly reducing the accuracy of trend prediction analysis. Second, there is a lack of understanding of equipment-level logic. Systems often have clear spatial affiliations and complex upstream and downstream relationships, and LLM lacks accurate understanding of this architecture, resulting in insufficient ability to analyze the root causes of abnormal operations. Third, the technology is costly to implement. Many cross-system, cross-device, and multi-parameter intelligent diagnostic scenarios have high requirements for model parameters and hardware. These issues mean that LLM technology currently focuses primarily on single-device operation diagnostics, making it difficult to apply from a system perspective to complex scenarios such as load coordination optimization in decision support systems, root cause analysis of system operation anomalies, and multi-dimensional parameter influence weight analysis.

[0029] Plant operation systems often involve high sampling frequencies and numerous correlated variables in their equipment or facility operation data. Equipment failures are frequently hidden within data trends spanning weeks or months. Large models, limited by data context length during inference, struggle to directly and coherently analyze the entire lifecycle and all dimensions of raw data. This hinders the accurate capture of fault symptoms embedded in long-term data, impacting the accuracy and timeliness of fault diagnosis. This invention compresses plant operation data while preserving its physical meaning and trends. It overcomes the limitations of large models on data context length, improving the accuracy and timeliness of large model identification, without increasing computational overhead after data compression.

[0030] Example 1 like Figure 1As shown in the embodiments of this specification, a data processing method is provided. The execution subject of this method can be a terminal device or a server. The terminal device can be a mobile phone, tablet computer, or a computer device such as a laptop or desktop computer. The server can be a single independent server or a server cluster composed of multiple servers. Specifically, the method may include the following steps: In step S1, the first operating data of the target device is obtained.

[0031] In this invention, the first operating data refers to the original operating data of equipment or facilities in the plant system, also known as plant system operating data, which is time-series data. Taking an electronics factory as an example, the target equipment in the plant system can be a low-temperature chilled water primary pump, and the corresponding first operating data can be current value, frequency value, and equipment status value; the target equipment in the plant system can also be a pure water supply tank, and the corresponding first operating data can be pressure value, liquid level value, and inlet regulating valve opening value. In the embodiments of this specification, there can be one or more target devices.

[0032] In step S2, the first running data is labeled based on preset tags to obtain the second running data carrying metadata tags corresponding to the target device.

[0033] The operational data of plant system equipment and facilities (i.e., primary operational data) often suffers from a lack of standardized naming conventions and inconsistent formats. Data variables, relying solely on their names, fail to accurately reflect their physical meaning or define the logical relationships and coupling between variables. Using large-scale models for equipment diagnosis and data analysis under these conditions severely reduces the accuracy and reliability of the results. Secondary operational data, obtained by labeling the acquired primary operational data with pre-defined tags, carries metadata tags, giving it semantic information and thus enabling it to be identifiable, traceable, and associative.

[0034] In this invention, the preset tags can be preset metadata tags, such as tags in the target device metadata tag library; or tags generated according to preset data annotation standards, wherein the preset data annotation standards can be industry data annotation standards for the target device's industry; or other... Figure 2 As shown, the tags are constructed by combining preset metadata tags and tags generated based on preset data annotations. Figure 2 In this context, the first operational data refers to the original operational data of the equipment and facilities, while the second operational data refers to the semantic operational data of the equipment and facilities after annotation processing.

[0035] In step S3, based on the metadata tag and the compression factor corresponding to the target device, the sampling frequency of the second running data is compressed to obtain first-level compressed data. Second-level compressed data is obtained by frequency domain processing of the first-level compressed data. Based on the second-level compressed data and the metadata tag, joint feature data corresponding to the first running data of the target device is constructed.

[0036] In implementation, step S3 includes the following process: In step S31, based on the metadata tags and the compression factor corresponding to the target device, the sampling frequency of the second running data is compressed to obtain first-level compressed data.

[0037] In this embodiment, the compression factor corresponding to the target device is a parameter reflecting the device type and health status, which can be determined based on the historical fault data of the target device. Furthermore, the compression factor is negatively correlated with the fault mechanism data of the target device. Since the compression factor corresponding to the target device is determined based on the historical fault data of the target device, this application can achieve dynamic adaptive compression based on the historical data of the target device. There can be multiple historical fault data entries; each fault occurrence of a device generates one fault data entry. Fault mechanism data refers to the values ​​of equipment location data related to the fault. For example, if the water pump current abnormally decreases, the associated fault mechanism data includes the water pump's current value, frequency value, and equipment status value.

[0038] In implementation, according to step S31, metadata can be automatically extracted from the second running data, and a target device metadata tag vector can be defined. Where te represents the total number of target devices and the data point label vector. tp represents the total number of data points. A target device can contain one or more data points. A data point is a specific data item on the target device that can be monitored and controlled. For example, a low-temperature chilled water primary pump device contains data points such as current, frequency, and device status; a pure water supply tank device contains data points such as pressure, liquid level, and inlet regulating valve opening. The first operating data is the specific value of the data point.

[0039] definition Let i = 1, 2, ..., te, representing the tag vector corresponding to the i-th target device. j=1,2,...,tp represents the label vector corresponding to the j-th data point. The compression factor of a specific data point contained in any device. , The compression factor is a positive integer, determined based on historical fault data and negatively correlated with fault mechanism data. If the original data sampling frequency (i.e., the sampling frequency of the second operating data) is... The sampling frequency after the first-level compression processing (i.e., compressing the sampling frequency of the second running data based on metadata tags and the compression factor corresponding to the target device) is: ,but Therefore, through the first compression process, the second running data was compressed. -1 times, resulting in Level 1 compressed data.

[0040] In step S32, secondary compressed data is obtained by frequency domain processing of the primary compressed data.

[0041] The first-level compressed data is time-domain data. By performing frequency-domain processing on it, the second-level compressed data is frequency-domain data, which can significantly reduce data storage and computational overhead. In implementation, wavelet transform, Fourier transform, and other methods can be used to perform frequency-domain processing on the first-level compressed data; this embodiment does not limit this approach.

[0042] In step S33, joint feature data corresponding to the first operating data of the target device is constructed based on the secondary compressed data and metadata tags.

[0043] The metadata tags include: the device metadata tag for the target device and the data point metadata tags for multiple data points related to the target device. The secondary compressed data, obtained by frequency domain processing of the primary compressed data, includes the total energy of multiple data points, the fault status code of the target device, and multiple peak values. The total energy is a general signal analysis indicator.

[0044] In implementation, a preset number of peak values ​​can be determined for multiple peak values ​​in the secondary compressed data, in descending order of peak value, and the frequency corresponding to each peak value in the preset number of peak values ​​can be determined.

[0045] In practice, the method of constructing joint feature data based on secondary compressed data and metadata tags can be constructed directly by splicing, or it can be constructed by assigning weights according to the importance of different data features. This implementation does not limit this method.

[0046] In step S4, the joint feature data is input into the pre-trained fault identification model to obtain the fault identification result of the target device. The fault identification model is iteratively trained by adjusting the compression factor of the device.

[0047] In this implementation, the acquired joint feature data is constructed based on secondary compressed data and metadata tags. This combined data includes the total energy of multiple data points related to the target device, the fault status code of the target device, the frequency corresponding to multiple peaks, the device metadata tag of the target device, and the data point metadata tags of multiple data points. This joint feature data expands the first running data in the form of time-series data into a combined data form of time-series data and context information. Inputting this joint feature data into the fault identification large model can preserve data characteristics while including data context information, thereby breaking the constraints of the large model context window and effectively improving the accuracy and timeliness of model identification.

[0048] Furthermore, prior to step S3, the data processing method in this embodiment also includes step S5: converting the timestamp of the first running data into a continuous time axis.

[0049] Since the metadata tags in the second running data imply the data sampling frequency, in implementation, the time of the first data point in the second running data can be recorded as 0, and the time of subsequent data points as... Where i represents the row number of the data. The data sampling frequency in the metadata tag is then... By converting timestamps into continuous timelines, a data foundation can be provided for subsequent secondary compression processing (i.e., frequency domain processing) of the primary compressed data, which helps to improve the efficiency of data processing.

[0050] Accordingly, the processing in step S32 above can be varied. The following provides an optional processing method, which can be found in the following step S321: In step S321, the first-level compressed data is subjected to Fourier transform processing to obtain the second-level compressed data.

[0051] In implementation, the first-level compressed data can be processed using Fast Fourier Transform (FFT), decomposing multiple parameters into independent FFT operations on each column of values. This can be expressed by the formula: Where dc[n] is the time series data after first-level compression (i.e., first-level compressed data), n is the number of data points contained, n=0,1,2...N-1, N is the number of first-level compressed data, and k is the frequency index, k=0,1,2...N-1. It is the imaginary unit.

[0052] Furthermore, windowing can be applied to the Fast Fourier Transform process to obtain... ,in, By adding windows, the data processing process can be made smoother and the data processing results more accurate.

[0053] After performing Fourier transform processing on the first-level compressed data, the total energy E and amplitude spectrum of multiple data points can also be calculated. ,but .

[0054] After performing a Fourier transform on the first-level compressed data, multiple peaks can be identified. A preset number of peaks are determined according to their values ​​from highest to lowest, and the frequency corresponding to each peak within this preset number is also determined. The preset number is at least three.

[0055] In implementation, the preset quantity can be set to 3, that is, 3 peak values ​​are determined in descending order of peak value, and the frequency corresponding to each of the 3 peak values ​​is determined. The preset quantity of 3 peak values ​​can ensure that the amount of data in the secondary compressed data is sufficient, improve the accuracy of data processing, and at the same time reduce the amount of calculation and computing resources, which is conducive to improving the efficiency of data processing.

[0056] Taking the extraction of 3 peak values ​​p[i] (i=1,2,3) as an example, the corresponding peak frequencies are: The amplitude is This can be expressed as a formula: , , .

[0057] After two-stage compression (i.e., frequency domain processing of the first-stage compressed data), the data compression ratio is: Where N is the number of primary compressed data. The sampling frequency after the first-stage compression is denoted as i, and the number of peak values ​​is a preset value. In summary, the second-stage compression process can significantly reduce data storage and computational overhead.

[0058] Furthermore, the processing in step S33 above can take many forms. The following provides an optional processing method, which can be found in the following steps S331-S332: In step S331, the total energy of multiple data points related to the target device, the fault status code of the target device, and the frequency corresponding to a preset number of peak values ​​are determined based on the secondary compression data. In step S332, based on the total energy of the determined multiple data points, the fault status code of the target device, the frequency corresponding to the preset number of peak values, the device metadata tag of the target device, and the data point metadata tags of the multiple data points related to the target device, joint feature data corresponding to the first operating data of the target device is constructed.

[0059] The data structure of the joint feature data can be represented as: (tag, v, c), where: tag: Represents metadata [("unit", "°C"), ("equipRef", "AHU-2"), ("floor", 2), ("zone", "2-1"), (discharge temp)], v: Represents the feature vector after secondary compression. v = [E, f1, A1, f2, A2, f3, A3] , c: indicates fault code [000001].

[0060] Furthermore, the data processing method also includes step S6: based on the timestamp of the first running data, the missing data is filled in to obtain the filled first running data.

[0061] In practice, step S6 can be performed before step S2 or before step S3. It is preferable to perform data completion processing before step S2. This method can perform one-time annotation processing on the first running data (including the first running data and the missing data) after completion processing to obtain the second running data corresponding to the first running data after completion processing, reducing the number of annotation processing and helping to further improve data processing efficiency.

[0062] Furthermore, the processing in step S6 above can take many forms. The following provides an optional processing method, which can be found in the following steps S61-S63: In step S61, based on the timestamp of the first running data, determine the two sets of data adjacent to the missing data, one before and one after. In step S62, the missing data is determined based on the two sets of data adjacent to the missing data before and after it. In step S63, the completed first running data is determined based on the missing data and the first running data.

[0063] Based on steps S61-S63 above, the coordinates corresponding to the missing data are defined as ( , Based on the timestamp, the coordinates of the two preceding data points adjacent to the missing data are ( , ), ( , ), the coordinates of the two data points adjacent to the missing data ( , ), ( , The missing data can then be calculated by averaging. : , , , .

[0064] Furthermore, the training method for the fault identification large model in step S4 above can be varied. The following provides an optional model training method, which can be found in the following steps S41-S43: In step S41, multiple historical first running data of multiple devices are obtained, and the multiple historical first running data of each device are labeled to obtain historical second running data with historical metadata tags corresponding to each device. The historical first running data is time-series data.

[0065] In step S42, based on the historical metadata tags and the compression factor corresponding to each device, the sampling frequency of the historical second running data is compressed to obtain historical first-level compressed data. The historical first-level compressed data is frequency domain-processed to obtain historical second-level compressed data. Based on the historical second-level compressed data and the historical metadata tags, a joint feature data sample corresponding to the historical first running data of each device is constructed.

[0066] The processing methods for steps S41-S42 above can be referred to the processing methods for steps S1-S3 above, and will not be repeated here.

[0067] In step S43, the joint feature data samples are input into the fault identification large model. By adjusting the compression factor of each device, the fault identification large model is trained using a preset loss function until the proportion of the number of fault identification failure samples to the number of test samples in the joint feature data samples is less than or equal to a preset proportion, or the difference between the fault identification accuracy of the large model in the next iteration cycle and the fault identification accuracy of the large model in the previous iteration cycle is less than a preset difference. The model training is then terminated, and the trained fault identification large model is obtained.

[0068] In implementation, the preset percentage can be set to 0.5, and the preset difference can be set to 0.01. According to step S43, when the proportion of the number of failed fault identification samples to the number of test samples in the joint feature data samples is greater than the preset percentage, it is determined that the compression factor of the corresponding device is too large. The compression factor is adjusted by reducing the compression factor, thereby reducing the magnitude of the first-level compression processing. For example, the initial compression factor is subtracted by 1 to obtain the adjusted compression factor, and based on the adjusted compression factor, the process returns to step S42 to re-compress the sampling frequency of the historical second running data.

[0069] After obtaining the joint feature data samples, they can be divided into training and testing samples in a 4:1 ratio. These two types of data samples are then used to train and evaluate the large-scale fault identification model. Definition For the training sample set, Given a test sample set, with training samples and test samples in a 4:1 ratio, then according to the method in step S43, ... , , , , in, This represents the set of samples where fault identification failed. Let x represent the prediction function for sample x. This represents the actual fault code of sample x, and r represents the proportion of data in the category to which x belongs (i.e., the preset proportion). Taking a preset proportion of 0.5 as an example, when... The compression factor of the original data corresponding to the category to which sample x belongs is changed from... Updated to -1. p represents the difference in the accuracy of the large model before and after the update (i.e., the accuracy of the fault identification large model in the next iteration cycle). Fault identification accuracy compared to the previous iteration's large model The difference), taking a preset percentage of 0.5 and a preset difference of 0.01 as an example, then when and When the failure rate of fault identification is relatively high and there is significant room for improvement in the model's accuracy after iterative training, the initial compression factor is reduced, and iterative training of the large fault identification model continues until... or At this point, model training terminates, and the trained fault identification large model is obtained.

[0070] Figure 3 This is a schematic diagram illustrating the training principle of the fault identification large model in this application embodiment. Figure 3 Data preprocessing includes at least labeling multiple historical first-run datasets, and may also include completing missing data based on the timestamps of the historical first-run datasets, and / or converting the timestamps of the historical first-run datasets into a continuous timeline. Data compression refers to a two-stage compression process. Figure 3 It can be seen that the compression factor can be adjusted in a timely manner during model training, thereby achieving adaptive compression.

[0071] In this embodiment's model training method, after acquiring multiple historical first-run data sets, they are not directly used for model training. Instead, they are labeled and subjected to two-stage compression to obtain joint feature data samples. The fault identification model is then trained based on these joint feature data samples. This approach effectively improves the accuracy and efficiency of model training. Furthermore, during model training, the compression factor of the corresponding device is adjusted in a timely manner based on the proportion of the number of failed fault identification samples to the number of test samples in the joint feature data samples, achieving adaptive compression and preventing fault information from being overwhelmed. In addition, the model training termination condition in this embodiment is: the proportion of the number of failed fault identification samples to the number of test samples in the joint feature data samples is less than or equal to a preset proportion; or, the difference between the fault identification accuracy of the large-scale fault identification model in the later iteration cycle and the fault identification accuracy of the large-scale fault identification model in the previous iteration cycle is less than a preset difference. That is, model training stops when the proportion of failed fault identification decreases to a certain range or when the improvement in pattern recognition accuracy decreases significantly during iteration. This method fully utilizes the compression factor to improve the accuracy of the large-scale fault identification model and enhances the efficiency of model training.

[0072] Example 2 The above describes the data processing method provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a data processing device, such as... Figure 4 As shown.

[0073] The data processing device includes: a first operating data acquisition module 210, a second operating data acquisition module 220, a data compression module 230, and a fault identification module 240, wherein: The first operational data acquisition module 210 is used to acquire the first operational data of the target device, which is time-series data. The second operation data acquisition module 220 is used to label the first operation data based on preset tags to obtain the second operation data carrying metadata tags corresponding to the target device; The data compression module 230 is used to compress the sampling frequency of the second running data based on the metadata tag and the compression factor corresponding to the target device to obtain first-level compressed data, to obtain second-level compressed data by frequency domain processing of the first-level compressed data, and to construct joint feature data corresponding to the first running data of the target device based on the second-level compressed data and the metadata tag. The fault identification module 240 is used to input joint feature data into a pre-trained fault identification model to obtain the fault identification result of the target device. The fault identification model is iteratively trained by adjusting the compression factor of the device.

[0074] In this embodiment of the specification, the preset tags in the second running data acquisition module 220 include: preset metadata tags and / or tags generated according to preset data labeling standards.

[0075] In this embodiment of the specification, the data processing device further includes a data completion module, which is used to complete the missing data according to the timestamp of the first running data to obtain the completed first running data.

[0076] In the embodiments of this specification, the data completion module includes: an adjacent data determination unit, a missing data determination unit, and a first running data determination unit, wherein: The adjacent data determination unit is used to determine two sets of data adjacent to the missing data, one before and one after, based on the timestamp of the first running data. The missing data determination unit is used to determine the missing data based on the two sets of data adjacent to the missing data before and after it. The first running data determination unit is used to determine the completed first running data based on the missing data and the first running data.

[0077] In this embodiment of the specification, the data processing device further includes: a time axis conversion module, used to convert the timestamps of the first running data into a continuous time axis; The data compression module 230 is also used to obtain secondary compressed data by performing Fourier transform processing on the primary compressed data.

[0078] In the embodiments of this specification, the compression factor corresponding to the target device in the data compression module 230 is determined based on the historical fault data of the target device, and the compression factor is negatively correlated with the fault mechanism data of the target device.

[0079] In this embodiment of the specification, the data processing device further includes: a model training module, used to acquire multiple historical first operating data from multiple devices, and to label the multiple historical first operating data of each device to obtain historical second operating data carrying historical metadata tags corresponding to each device, wherein the historical first operating data is time-series data; based on the historical metadata tags and the compression factor corresponding to each device, to compress the sampling frequency of the historical second operating data to obtain historical first-level compressed data, to obtain historical second-level compressed data by frequency domain processing of the historical first-level compressed data, and to construct joint feature data samples corresponding to the historical first operating data of each device based on the historical second-level compressed data and the historical metadata tags; to input the joint feature data samples into the fault identification large model, and to train the fault identification large model using a preset loss function by adjusting the compression factor of each device until the proportion of the number of failed fault identification samples to the number of test samples in the joint feature data samples is less than or equal to a preset proportion, or the difference between the fault identification large model recognition accuracy of the later iteration cycle and the fault identification large model recognition accuracy of the previous iteration cycle is less than a preset difference, the model training terminates, and the trained fault identification large model is obtained.

[0080] In this embodiment of the specification, the metadata tags in the second running data acquisition module 220 include: the device metadata tag of the target device and the data point metadata tags of multiple data points related to the target device; The data compression module 230 is also used to determine the total energy of multiple data points related to the target device, the fault status code of the target device, and the frequency corresponding to a preset number of peaks based on the secondary compressed data; and to construct joint feature data corresponding to the first operating data of the target device based on the determined total energy of multiple data points, the fault status code of the target device, the frequency corresponding to a preset number of peaks, the device metadata tag of the target device, and the data point metadata tags of multiple data points related to the target device.

[0081] Example 3 The above describes the data processing apparatus provided in the embodiments of this specification. Based on the same concept, the embodiments of this specification also provide an electronic device, such as... Figure 5As shown. Electronic devices can vary considerably due to differences in configuration or performance, and may include one or more processors 301 and memory 302. Memory 302 may store one or more application programs or data. Memory 302 may be temporary or persistent storage. The application programs stored in memory 302 may include one or more modules (not shown), each module may include a series of computer-executable instructions for the electronic device. Furthermore, processor 301 may be configured to communicate with memory 302 and execute the series of computer-executable instructions in memory 302 on the electronic device. The electronic device may also include one or more power supplies 303, one or more wired or wireless network interfaces 304, one or more input / output interfaces 305, and one or more keyboards 306.

[0082] Specifically, in this embodiment, the electronic device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for use in the electronic device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following: Acquire the first operational data of the target device, which is time-series data; The first running data is labeled based on preset tags to obtain the second running data carrying metadata tags corresponding to the target device; Based on the metadata tags and the compression factor corresponding to the target device, the sampling frequency of the second running data is compressed to obtain the first-level compressed data. The first-level compressed data is then frequency-domain processed to obtain the second-level compressed data. Based on the second-level compressed data and the metadata tags, joint feature data corresponding to the first running data of the target device is constructed. The joint feature data is input into a pre-trained fault identification model to obtain the fault identification result of the target device. The fault identification model is iteratively trained by adjusting the compression factor of the device.

[0083] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Acquire the first operating data of the target device, wherein the first operating data is time-series data; The first running data is labeled based on preset tags to obtain second running data carrying metadata tags corresponding to the target device; Based on the metadata tag and the compression factor corresponding to the target device, the sampling frequency of the second running data is compressed to obtain first-level compressed data. Second-level compressed data is obtained by frequency domain processing of the first-level compressed data. Based on the second-level compressed data and the metadata tag, joint feature data corresponding to the first running data of the target device is constructed. The joint feature data is input into a pre-trained fault identification model to obtain the fault identification result of the target device. The fault identification model is iteratively trained by adjusting the compression factor of the device.

2. The method according to claim 1, characterized in that, The preset tags include: preset metadata tags and / or tags generated according to preset data labeling standards.

3. The method according to claim 1, characterized in that, Before annotating the first operational data with preset tags to obtain the second operational data carrying metadata tags corresponding to the target device, the method further includes: Based on the timestamp of the first running data, the missing data is filled in to obtain the first running data after completion.

4. The method according to claim 3, characterized in that, The method for completing missing data based on the timestamp of the first running data includes: Based on the timestamp of the first running data, determine the two sets of data adjacent to and before the missing data; The missing data is determined based on the two sets of data adjacent to the missing data before and after it. Based on the missing data and the first running data, the completed first running data is determined.

5. The method according to claim 1, characterized in that, Based on the metadata tags and the compression factor corresponding to the target device, the sampling frequency of the second operating data is compressed to obtain first-level compressed data. Second-level compressed data is obtained by frequency domain processing of the first-level compressed data. Before constructing joint feature data corresponding to the first operating data of the target device based on the second-level compressed data and the metadata tags, the method further includes: Convert the timestamps of the first running data into a continuous timeline; The process of obtaining secondary compressed data by frequency domain processing of the primary compressed data includes: The first-level compressed data is subjected to Fourier transform processing to obtain the second-level compressed data.

6. The method according to claim 1, characterized in that, The compression factor corresponding to the target device is determined based on the historical fault data of the target device, and the compression factor is negatively correlated with the fault mechanism data of the target device.

7. The method according to claim 1, characterized in that, The model training method for the large-scale fault identification model includes: Multiple historical first running data from multiple devices are acquired, and the multiple historical first running data of each device are labeled to obtain historical second running data carrying historical metadata tags corresponding to each device, wherein the historical first running data is time-series data; Based on the historical metadata tags and the compression factor corresponding to each device, the sampling frequency of the historical second operating data is compressed to obtain historical first-level compressed data. The historical first-level compressed data is frequency domain-processed to obtain historical second-level compressed data. Based on the historical second-level compressed data and the historical metadata tags, a joint feature data sample corresponding to the historical first operating data of each device is constructed. The joint feature data samples are input into the fault identification large model. By adjusting the compression factor of each device, the fault identification large model is trained using a preset loss function until the proportion of the number of fault identification failure samples to the number of test samples in the joint feature data samples is less than or equal to a preset proportion, or the difference between the fault identification large model recognition accuracy of the next iteration cycle and the fault identification large model recognition accuracy of the previous iteration cycle is less than a preset difference. The model training is then terminated, and the trained fault identification large model is obtained.

8. The method according to claim 1, characterized in that, The metadata tags include: a device metadata tag for the target device and data point metadata tags for multiple data points related to the target device. The step of constructing joint feature data corresponding to the first operational data of the target device based on the secondary compressed data and the metadata tags includes: Based on the secondary compressed data, the total energy of multiple data points related to the target device, the fault status code of the target device, and the frequency corresponding to a preset number of peak values ​​are determined. Based on the total energy of the determined multiple data points, the fault status code of the target device, the frequency corresponding to a preset number of peak values, the device metadata tag of the target device, and the data point metadata tags of the multiple data points related to the target device, joint feature data corresponding to the first operating data of the target device is constructed.

9. A data processing apparatus, characterized in that, The device includes: The first operational data acquisition module is used to acquire the first operational data of the target device, wherein the first operational data is time-series data. The second operation data acquisition module is used to label the first operation data based on preset tags to obtain second operation data carrying metadata tags corresponding to the target device; The data compression module is used to compress the sampling frequency of the second running data based on the metadata tag and the compression factor corresponding to the target device to obtain first-level compressed data, to obtain second-level compressed data by frequency domain processing of the first-level compressed data, and to construct joint feature data corresponding to the first running data of the target device based on the second-level compressed data and the metadata tag. The fault identification module is used to input the joint feature data into a pre-trained fault identification model to obtain the fault identification result of the target device. The fault identification model is trained iteratively by adjusting the compression factor of the device.

10. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data processing method as described in any one of claims 1-8.

Images