Internet of Things data processing method and system based on large model

Abstract

The invention relates to the technical field of computers, in particular to an Internet of Things data processing method and system based on a large model, and the method comprises the steps: obtaining multi-source Internet of Things data through an edge node, processing the Internet of Things data through a multi-source heterogeneous data processing method of converting Box-Cox into Z-score, and integrating the processed Internet of Things data; performing pruning and quantitative compression on a preset large model to remove redundant or unimportant parameters in the large model so as to reduce the size and calculation amount of the large model; obtaining a problem description provided by a user, and quickly generating an ILP model framework through the pruned and quantized compressed large model according to the problem description; selecting a relay node for data transmission according to the constraint condition and the target function, and generating an optimal transmission path; and transmitting the integrated Internet of Things data to the cloud according to the optimal transmission path. According to the invention, the data processing and transmission efficiency of the Internet of Things can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and system for Internet of Things (IoT) data processing based on a large model. Background Technology

[0002] With the widespread application of Internet of Things (IoT) devices, the amount of data generated is growing exponentially, placing higher demands on data processing and transmission. Traditional cloud computing models face problems such as high latency and insufficient bandwidth when processing massive amounts of data, especially in scenarios with high real-time requirements, such as intelligent transportation, industrial automation, and telemedicine. This model relies on uploading data to remote data centers for processing, but with the surge in data volume, this centralized processing method has gradually exposed its limitations, particularly in terms of data transmission speed and response time.

[0003] Therefore, how to efficiently process and transmit IoT data has become one of the key research focuses. Summary of the Invention

[0004] Therefore, it is necessary to provide an IoT data processing method and system based on a large model that can improve the efficiency of IoT data processing and transmission, addressing the aforementioned technical problems.

[0005] In a first aspect, this application provides an IoT data processing method based on a large model, the method comprising: The IoT data is acquired from multiple sources through edge nodes, and processed using a multi-source heterogeneous data processing method based on Box-Cox transformation Z-score. The processed IoT data is then integrated. The pre-defined large model is pruned and quantized to remove redundant or unimportant parameters, thereby reducing the size and computational cost of the large model. Obtain the problem description provided by the user, and based on the problem description, quickly generate the ILP model framework by pruning and quantizing the large model. The ILP model framework consists of constraints and objective functions. Based on the constraints and the objective function, select the relay node for data transmission and generate the optimal transmission path; Based on the optimal transmission path, the integrated IoT data will be transmitted to the cloud.

[0006] In one embodiment, the pruning and quantization compression of the preset large model includes: An initial large model is obtained and trained. During the training process, a soft masking strategy and a sparse factor cosine decay are introduced to gradually prune the large model. After training, a pre-defined large model is obtained; Through quantization compression, the floating-point weights of the preset large model are converted into low-precision integers or half-precision floating-point numbers.

[0007] In one embodiment, the step of obtaining a user-provided problem description and, based on the problem description, quickly generating an ILP model framework through a large model after pruning and quantization compression includes: Obtain the problem description provided by the user, and based on the problem description, transform the problem description into a structured logical expression through a large model that has been pruned and quantized; Based on the logical expression, logical rules are generated, which are used to construct constraints and objective functions. Based on the aforementioned logical rules, the ILP model framework is generated quickly.

[0008] In one embodiment, the step of obtaining a user-provided problem description and, based on the problem description, transforming the problem description into a structured logical expression using a large model after pruning and quantization compression includes: Obtain the problem description provided by the user, and based on the problem description, automatically define structured symbol vocabulary and relation templates through a large model that has been pruned and quantized; Based on the structured symbol vocabulary and the relation template, a structured logical expression is obtained.

[0009] In one embodiment, the step of acquiring multi-source IoT data through edge nodes, processing the IoT data using a multi-source heterogeneous data processing method with Box-Cox transformation Z-score, and integrating the processed IoT data includes: The IoT data is acquired from multiple sources through edge nodes, and then transformed into a form that is closer to a normal distribution through Box-Cox transformation to obtain the first transformed data, thereby reducing the IoT data offset problem. By standardizing with Z-score, the first transformed data is converted into a normal distribution with a mean of 0 and a standard deviation of 1 to obtain the second transformed data, so as to ensure the consistency of the multi-source IoT data in terms of dimensions and magnitude. The second transformed data is then integrated.

[0010] In one embodiment, the step of acquiring multi-source IoT data through edge nodes, processing the IoT data using a multi-source heterogeneous data processing method with Box-Cox transformation Z-score, and integrating the processed IoT data further includes: The SM4 algorithm is used to encrypt the integrated IoT data to improve the security of data transmission.

[0011] In one embodiment, selecting a relay node for data transmission and generating an optimal transmission path based on the constraints and the objective function includes: Obtain the computing resources of the edge nodes, and based on the computing resources, determine whether the integrated IoT data needs to be transmitted through the relay nodes; If so, then based on the constraints and the objective function, select the relay node for data transmission and generate the optimal transmission path; If not, the integrated IoT data is processed through the edge node.

[0012] Secondly, this application also provides an IoT data processing device based on a large-scale model. The device includes: The data integration module is used to acquire multi-source IoT data through edge nodes, process the IoT data using the Box-Cox transformation Z-score multi-source heterogeneous data processing method, and integrate the processed IoT data. The model compression module is used to prune and quantize a preset large model to remove redundant or unimportant parameters in the large model, thereby reducing the size and computational load of the large model. The model framework generation module is used to obtain the problem description provided by the user, and based on the problem description, quickly generate the ILP model framework through the large model after pruning and quantization compression. The ILP model framework consists of constraints and objective functions. The optimal path generation module is used to select relay nodes for data transmission and generate the optimal transmission path based on the constraints and the objective function. The data transmission module is used to transmit the integrated IoT data to the cloud according to the optimal transmission path.

[0013] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps: The IoT data is acquired from multiple sources through edge nodes, and processed using a multi-source heterogeneous data processing method based on Box-Cox transformation Z-score. The processed IoT data is then integrated. The pre-defined large model is pruned and quantized to remove redundant or unimportant parameters, thereby reducing the size and computational cost of the large model. Obtain the problem description provided by the user, and based on the problem description, quickly generate the ILP model framework by pruning and quantizing the large model. The ILP model framework consists of constraints and objective functions. Based on the constraints and the objective function, select the relay node for data transmission and generate the optimal transmission path; Based on the optimal transmission path, the integrated IoT data will be transmitted to the cloud.

[0014] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps: The IoT data is acquired from multiple sources through edge nodes, and processed using a multi-source heterogeneous data processing method based on Box-Cox transformation Z-score. The processed IoT data is then integrated. The pre-defined large model is pruned and quantized to remove redundant or unimportant parameters, thereby reducing the size and computational cost of the large model. Obtain the problem description provided by the user, and based on the problem description, quickly generate the ILP model framework by pruning and quantizing the large model. The ILP model framework consists of constraints and objective functions. Based on the constraints and the objective function, select the relay node for data transmission and generate the optimal transmission path; Based on the optimal transmission path, the integrated IoT data will be transmitted to the cloud.

[0015] In summary, this application includes the following beneficial technical effects: By acquiring multi-source IoT data through edge nodes and processing the data using a combination of Box-Cox transformation and Z-score, data optimization and integration are achieved, improving data consistency. A pre-defined large model is pruned and quantized to remove redundant or unimportant parameters, reducing its size and computational load. This allows the large model to run efficiently on resource-constrained edge nodes, enabling rapid decision-making and response. Furthermore, by quickly generating an ILP model framework, the optimal transmission path is determined. The ILP model, composed of constraints and an objective function, can select the optimal relay node based on actual needs, thereby optimizing data transmission efficiency and path selection. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating an IoT data processing method based on a large model in one embodiment. Figure 2This is a flowchart illustrating an IoT data processing method based on a large model, as described in another embodiment. Figure 3 This is a structural block diagram of an IoT data processing device based on a large model in one embodiment; Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0017] This invention provides an IoT data processing method and system based on a large model.

[0018] The embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0019] In the description of the embodiments disclosed in this invention, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0020] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the IoT data processing method based on a large model in this invention includes: The S100 acquires multi-source IoT data through edge nodes, processes the IoT data using the Box-Cox transformation Z-score multi-source heterogeneous data processing method, and integrates the processed IoT data.

[0021] Specifically, edge nodes are key computing devices in an IoT system, typically deployed near IoT devices. They are responsible for data acquisition, preprocessing, analysis, and local control. They connect to IoT devices via wireless communication technologies (such as Wi-Fi, Bluetooth, NB-IoT, etc.) and upload processed data to the cloud. Edge nodes first collect IoT data from various sources (such as sensors, actuators, etc.), which may include structured, semi-structured, or unstructured data. To address the differences in dimensions and magnitudes of multi-source heterogeneous data, a method combining Box-Cox transformation and Z-score normalization is introduced to process IoT data and integrate the processed data. The Box-Cox transformation and Z-score multi-source heterogeneous data processing method is an efficient data preprocessing technique that combines data normalization and nonlinear transformation. This method effectively solves the inconsistency problems caused by differences in format, dimension, data type, and magnitude during the acquisition, transmission, and fusion of multi-source heterogeneous IoT data. Before processing IoT data, it is necessary to clean the data, including missing value imputation, outlier correction, and duplicate data removal.

[0022] In this embodiment, multi-source IoT data is acquired through edge nodes, and the multi-source heterogeneous data processing method using Box-Cox transformation Z-score can effectively solve the problems of storage, fusion, and analysis of multi-source heterogeneous data, thereby improving the efficiency and accuracy of IoT data processing.

[0023] S200 prunes and quantizes the preset large model to remove redundant or unimportant parameters, thereby reducing the size and computational cost of the large model.

[0024] Specifically, large models, as powerful language understanding and generation tools, can provide high-quality hints and examples for ILP (Integer Linear Programming), thereby generating programs or rule sets that better conform to logical rules. Large models can generate a series of intermediate reasoning steps, significantly improving their performance in complex reasoning tasks. This capability is particularly important in ILP, as ILP itself relies on inductively deducing logical rules from data, and the reasoning ability of large models can help generate more accurate and reasonable rules. First, the pre-defined large model needs to be compressed, including pruning and quantization. Pruning is a technique to reduce the model size by removing unimportant or redundant parameters; it identifies and removes weights or neurons that contribute little to model performance, thus reducing the model size while maintaining performance. Quantization is a technique to reduce model size and computational complexity by lowering the precision of model parameters; it converts floating-point numbers to low-precision integers or fixed-point numbers, thereby reducing the model's storage space and computational resource requirements.

[0025] S300 obtains the problem description provided by the user and, based on the problem description, quickly generates the ILP model framework from the large model after pruning and quantization compression.

[0026] Specifically, in traditional ILP modeling methods, users typically define constraints and objective functions manually. To quickly generate an ILP model framework, a large model (such as GPT-3 or LLaMA) can be used to assist in defining constraints and objective functions. First, the user describes the problem in natural language, for example, "I need an ILP model to select relay nodes to optimize the optimal path." The large model generates the ILP model framework based on the natural language input, including constraints and objective functions. Furthermore, based on the problem description, the problem can be decomposed into multiple subproblems. Each subproblem adds new constraints to the previous subproblem. For example, the first relay node to be found can be the first subproblem, the next relay node the second, and so on. Specifically, the first subproblem is relaxed into a linear programming (LP) problem, and its relaxed solution is solved. The relaxed solution can serve as the starting point for subsequent solutions and provides upper and lower bound information. Based on the results of the relaxed solution, it is determined whether new constraints need to be added. This process is repeated until all constraints are found. In each iteration, the complexity of the problem gradually increases, but by adding constraints step by step, an overly complex problem can be avoided from being generated all at once.

[0027] In this embodiment, the rapid generation of the ILP model framework through a large model can significantly improve the efficiency and flexibility of ILP model construction.

[0028] S400 selects relay nodes for data transmission and generates the optimal transmission path based on constraints and objective function.

[0029] Specifically, the objective function is typically to minimize cost or delay. In data transmission problems, the objective function can be the minimization of the total cost of the transmission path or the end-to-end delay. Constraints include data transmission time, bandwidth limitations, node capacity, etc. Using the ILP model framework, based on the constraints and the objective function, relay nodes for data transmission are selected, and the optimal transmission path that satisfies the constraints of delay, bandwidth, and transmission power is generated.

[0030] The S500 transmits the integrated IoT data to the cloud via the optimal transmission path.

[0031] Specifically, edge nodes transmit the integrated IoT data to the cloud based on the optimal transmission path. The cloud then provides powerful computing capabilities and storage resources to process more complex analysis tasks based on the transmitted IoT data.

[0032] In one embodiment, pruning and quantizing a pre-defined large model includes: Obtain an initial large model and train it. During the training process, a soft masking strategy and sparse factor cosine decay are introduced to gradually prune the large model. After training, a preset large model is obtained. Through quantization compression, the floating-point weights of the preset large model are converted into low-precision integers or half-precision floating-point numbers.

[0033] Specifically, soft masking and cosine decay of sparsity factor are introduced during the initial large model training process to achieve pruning. Soft masking allows the model to be pruned gradually and finely during training, rather than removing a large number of parameters all at once. This strategy uses a differentiable mask to control which connections can be pruned, thus preserving model performance during training. Unlike traditional binary masks, soft masks allow the model to dynamically adjust its structure during training, avoiding performance degradation caused by pruning. Cosine decay of sparsity factor is a mechanism for controlling model sparsity. It gradually reduces the proportion of unused connections in the model using a cosine function, thus gradually compressing the model to the target sparsity during training. This method avoids the need to manually set the pruning ratio and the number of fine-tuning rounds required in traditional pruning methods, making the model compression process more automated and efficient. After pruning, the large model is quantized, converting the floating-point weights of the large model to low-precision integers or half-precision floating-point numbers to reduce the storage space and computational overhead of the large model.

[0034] In this embodiment, by introducing a soft masking strategy and sparse factor cosine decay during training, the effect of pruning without fine-tuning is achieved, enabling the model to automatically identify and retain important connections during the training phase, thereby achieving efficient model compression without relying on subsequent fine-tuning.

[0035] In one embodiment, such as Figure 2 As shown, S300 includes: S320: Obtain the problem description provided by the user, and based on the problem description, transform the problem description into a structured logical expression through a large model that has been pruned and quantized. S340, Generate logical rules based on logical expressions; S360 quickly generates ILP model frameworks based on logical rules.

[0036] Specifically, firstly, the business requirements or problem descriptions provided by users need to be transformed into structured logical expressions. This process is accomplished through Large Model (LLM). The problem descriptions provided by users guide the model to understand the task context and transform it into structured logical expressions. Specifically, it is necessary to identify the key elements in the problem description, decompose the problem, and construct the logical expression. After constructing the logical expression, logical rules are generated. These rules will be used to construct the constraints and objective functions in the ILP model. After generating the logical rules, these rules need to be transformed into the ILP model. This process requires mapping the logical rules into mathematical expressions and ensuring that they conform to the syntax requirements of ILP.

[0037] In one embodiment, obtaining a user-provided problem description and, based on the problem description, transforming it into a structured logical expression using a large model that has undergone pruning and quantization compression includes: The system obtains a problem description provided by the user and, based on the problem description, automatically defines a structured vocabulary of symbols and relational templates from a large model that has been pruned and quantized; and obtains a structured logical expression based on the structured vocabulary of symbols and relational templates.

[0038] Specifically, a large model is used to automatically define structured symbol lexicons (predicates) and relation templates from text data. Predicates are symbols representing relations or attributes, typically used to describe the state or behavior of the subject. For example, in predicate logic, predicate symbols are interpreted as domain relations, used to express the relationship between the subject and the object. For instance, "greater than," "belongs to," and "is" can all be used as predicates. The large model automatically generates structured symbol lexicons (predicates) to overcome the dependence on predefined symbol structures and sensitivity to noise inherent in traditional methods. Relation templates (such as "entity X → attribute Y") are used to describe the structured expression of semantic relationships between entities. They typically consist of a predicate (verb or noun phrase) and its corresponding arguments, representing the logical relationship between events, actions, or states. For example, "generate" can be a predicate, while "ILP model framework" and "transmission path" can be its arguments, forming a relation template like "ILP model framework generates transmission path."

[0039] In this embodiment, a large model is used to automatically define structured symbol vocabulary and relation templates from text data, thereby overcoming the dependence of traditional ILP on predefined symbol structures and the sensitivity of pure LLM methods to noise. This automated symbol grounding process enables the system to convert natural language into logical facts more efficiently, providing a solid foundation for subsequent rule learning.

[0040] In one embodiment, multi-source IoT data is acquired through edge nodes, and the IoT data is processed using a multi-source heterogeneous data processing method based on Box-Cox transformation Z-score. The processed IoT data is then integrated, including: The system acquires multi-source IoT data through edge nodes and transforms the IoT data into a form that is closer to a normal distribution through Box-Cox transformation to obtain the first transformed data, thereby reducing the IoT data offset problem. The first transformed data is then transformed into a normal distribution with a mean of 0 and a standard deviation of 1 through Z-score standardization to obtain the second transformed data, thereby ensuring the consistency of the multi-source IoT data in terms of dimensions and magnitude. The second transformed data is then integrated.

[0041] Specifically, the multi-source heterogeneous data processing method based on Box-Cox transform and Z-score is a method that improves data processability by addressing data bias during the normalization process. This method combines Box-Cox transform and Z-score standardization. In the data processing stage, the IoT data is first subjected to noise filtering and feature analysis. Then, the Box-Cox transform is applied. By adjusting parameters, the Box-Cox transform makes the data distribution closer to a normal distribution, thereby reducing data bias. The transformed IoT data is then subjected to Z-score standardization, converting the data into a normal distribution with a mean of 0 and a standard deviation of 1. This process ensures consistency in units and magnitude across different data sources, laying the foundation for subsequent data analysis.

[0042] In this embodiment, the Box-Cox transformation can significantly improve the distribution characteristics of the data, making the IoT data closer to a normal distribution, thereby improving the accuracy of Z-score standardization. Through Z-score standardization, the consistency of multi-source IoT data in terms of dimensions and magnitude is guaranteed.

[0043] In one embodiment, multi-source IoT data is acquired through edge nodes, and the IoT data is processed using a multi-source heterogeneous data processing method based on Box-Cox transformation Z-score. After integrating the processed IoT data, the method further includes: The SM4 algorithm is used to encrypt the integrated IoT data to improve the security of data transmission.

[0044] Specifically, the SM4 algorithm is a block cipher algorithm with a key length and block length of 128 bits. It has high security and efficiency. Encrypting integrated IoT data using the SM4 algorithm can effectively improve the security of data transmission.

[0045] In one embodiment, selecting relay nodes for data transmission and generating the optimal transmission path based on constraints and the objective function includes: The system acquires the computing resources of the edge nodes and determines whether the integrated IoT data should be transmitted through relay nodes based on these resources. If so, it selects the relay nodes for data transmission based on constraints and the objective function, and generates the optimal transmission path. If not, it processes the integrated IoT data through the edge nodes.

[0046] Computing resources refer to the total amount of hardware and software facilities required to perform computing tasks. These resources include, but are not limited to, processors (CPU), memory, storage devices, and network bandwidth.

[0047] Specifically, the choice of data transmission path depends on whether the edge nodes have sufficient computing resources. If the edge nodes' computing power is insufficient to process the integrated data, the data needs to be transmitted to a relay node or the cloud for further processing. After determining that data needs to be transmitted through a relay node, the optimal relay node is selected and the optimal transmission path is generated based on the constraints and objective function in the ILP model framework. If the edge nodes have sufficient computing resources, the integrated data can be processed directly and transmitted to the cloud through the edge nodes.

[0048] In this embodiment, edge nodes can determine whether to transmit data to the cloud for processing through relay nodes based on their own computing resources, thus avoiding resource waste. For tasks that require relay transmission, the optimal transmission path can be generated based on constraints and objective functions, thereby improving the overall task processing efficiency.

[0049] In one embodiment, such as Figure 3 As shown, an IoT data processing device based on a large model is provided, including: a data integration module 10, a model compression module 20, a model framework generation module 30, an optimal path generation module 40, and a data transmission module 50, wherein: The data integration module 10 is used to acquire multi-source IoT data through edge nodes, process the IoT data using the Box-Cox transformation Z-score multi-source heterogeneous data processing method, and integrate the processed IoT data. The model compression module 20 is used to prune and quantize the preset large model to remove redundant or unimportant parameters in the large model, thereby reducing the size and computational load of the large model. The model framework generation module 30 is used to obtain the problem description provided by the user, and based on the problem description, quickly generate the ILP model framework through the large model after pruning and quantization compression. The ILP model framework consists of constraints and objective functions. The optimal path generation module 40 is used to select relay nodes for data transmission and generate the optimal transmission path based on constraints and objective functions. The data transmission module 50 is used to transmit the integrated IoT data to the cloud according to the optimal transmission path.

[0050] In one embodiment, the model compression module 20 is also used to obtain an initial large model and train the initial large model. During the training process, a soft masking strategy and sparse factor cosine decay are introduced to gradually prune the large model. After training, a preset large model is obtained. Through quantization compression, the floating-point weights of the preset large model are converted into low-precision integers or half-precision floating-point numbers.

[0051] In one embodiment, the model framework generation module 30 is also used to obtain a problem description provided by the user, and based on the problem description, transform the problem description into a structured logical expression through a large model after pruning and quantization compression; generate logical rules based on the logical expression, and use the logical rules to construct constraints and objective functions; and quickly generate an ILP model framework based on the logical rules.

[0052] In one embodiment, the model framework generation module 30 is also used to obtain a problem description provided by the user, and based on the problem description, automatically define a structured symbol vocabulary and relation template through a large model after pruning and quantization compression; and obtain a structured logical expression based on the structured symbol vocabulary and relation template.

[0053] In one embodiment, the data integration module 10 is further configured to acquire multi-source IoT data through edge nodes, and convert the IoT data into a form closer to a normal distribution through Box-Cox transformation to obtain first transformed data, thereby reducing IoT data offset issues; convert the first transformed data into a normal distribution with a mean of 0 and a standard deviation of 1 through Z-score standardization to obtain second transformed data, thereby ensuring consistency of multi-source IoT data in terms of dimensions and magnitude; and integrate the second transformed data.

[0054] In one embodiment, the IoT data processing device based on a large model further includes a data encryption module for encrypting the integrated IoT data using the SM4 algorithm to improve the security of data transmission.

[0055] In one embodiment, the optimal path generation module 40 is further configured to acquire the computing resources of the edge nodes and, based on the computing resources, determine whether the integrated IoT data should be transmitted through relay nodes; if so, the relay nodes for data transmission are selected according to the constraints and objective function, and the optimal transmission path is generated; if not, the integrated IoT data is processed through the edge nodes.

[0056] The various modules in the aforementioned large-scale IoT data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0057] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores infrared image data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a large-scale Internet of Things (IoT) data processing method.

[0058] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0059] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for IoT data processing based on a large model, characterized in that, include: The IoT data is acquired from multiple sources through edge nodes, and processed using a multi-source heterogeneous data processing method based on Box-Cox transformation Z-score. The processed IoT data is then integrated. The pre-defined large model is pruned and quantized to remove redundant or unimportant parameters, thereby reducing the size and computational cost of the large model. Obtain the problem description provided by the user, and based on the problem description, quickly generate the ILP model framework by pruning and quantizing the large model. The ILP model framework consists of constraints and objective functions. Based on the constraints and the objective function, select the relay node for data transmission and generate the optimal transmission path; Based on the optimal transmission path, the integrated IoT data will be transmitted to the cloud.

2. The IoT data processing method based on a large model according to claim 1, characterized in that, The step of pruning and quantizing the preset large model includes: An initial large model is obtained and trained. During the training process, a soft masking strategy and a sparse factor cosine decay are introduced to gradually prune the large model. After training, a pre-defined large model is obtained; Through quantization compression, the floating-point weights of the preset large model are converted into low-precision integers or half-precision floating-point numbers.

3. The IoT data processing method based on a large model according to claim 1, characterized in that, The process of obtaining a user-provided problem description and, based on that description, rapidly generating an ILP model framework from a large model after pruning and quantization compression includes: Obtain the problem description provided by the user, and based on the problem description, transform the problem description into a structured logical expression through a large model that has been pruned and quantized; Based on the logical expression, logical rules are generated, which are used to construct constraints and objective functions. Based on the aforementioned logical rules, the ILP model framework is generated quickly.

4. The IoT data processing method based on a large model according to claim 3, characterized in that, The process of obtaining a user-provided problem description and, based on that description, transforming it into a structured logical expression using a large model that has undergone pruning and quantization compression includes: Obtain the problem description provided by the user, and based on the problem description, automatically define structured symbol vocabulary and relation templates through a large model that has been pruned and quantized; Based on the structured symbol vocabulary and the relation template, a structured logical expression is obtained.

5. The IoT data processing method based on a large model according to claim 1, characterized in that, The method of acquiring multi-source IoT data through edge nodes, processing the IoT data using the Box-Cox transformation Z-score multi-source heterogeneous data processing method, and integrating the processed IoT data includes: The IoT data is acquired from multiple sources through edge nodes, and then transformed into a form that is closer to a normal distribution through Box-Cox transformation to obtain the first transformed data, thereby reducing the IoT data offset problem. By standardizing with Z-score, the first transformed data is converted into a normal distribution with a mean of 0 and a standard deviation of 1 to obtain the second transformed data, so as to ensure the consistency of the multi-source IoT data in terms of dimensions and magnitude. The second transformed data is then integrated.

6. The IoT data processing method based on a large model according to claim 1, characterized in that, The method of acquiring multi-source IoT data through edge nodes and processing the IoT data using a Box-Cox transform Z-score multi-source heterogeneous data processing method, and then integrating the processed IoT data, further includes: The SM4 algorithm is used to encrypt the integrated IoT data to improve the security of data transmission.

7. The IoT data processing method based on a large model according to claim 1, characterized in that, The step of selecting relay nodes for data transmission and generating the optimal transmission path based on the constraints and the objective function includes: Obtain the computing resources of the edge nodes, and based on the computing resources, determine whether the integrated IoT data needs to be transmitted through the relay nodes; If so, then based on the constraints and the objective function, select the relay node for data transmission and generate the optimal transmission path; If not, the integrated IoT data is processed through the edge node.

8. An Internet of Things (IoT) data processing device based on a large model, characterized in that, include: The data integration module is used to acquire multi-source IoT data through edge nodes, process the IoT data using the Box-Cox transformation Z-score multi-source heterogeneous data processing method, and integrate the processed IoT data. The model compression module is used to prune and quantize a preset large model to remove redundant or unimportant parameters in the large model, thereby reducing the size and computational load of the large model. The model framework generation module is used to obtain the problem description provided by the user, and based on the problem description, quickly generate the ILP model framework through the large model after pruning and quantization compression. The ILP model framework consists of constraints and objective functions. The optimal path generation module is used to select relay nodes for data transmission and generate the optimal transmission path based on the constraints and the objective function. The data transmission module is used to transmit the integrated IoT data to the cloud according to the optimal transmission path.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images