IoT protocol analysis method, equipment and media based on multimodal AI

Abstract

This application discloses a multimodal AI-based IoT protocol parsing method, device, and medium, relating to the field of industrial IoT. The method includes: collecting protocol byte streams and associated multimodal physical data of industrial equipment; parsing the protocol byte streams and multimodal data through multimodal AI, extracting multimodal features, and fusing them; inputting them into a pre-trained protocol syntax generation model, outputting a protocol syntax tree corresponding to the communication protocol of the industrial equipment, semantically decoding the protocol syntax tree, and obtaining structured protocol data; obtaining multimodal physical measured values ​​of the industrial equipment, verifying the structured protocol data based on the multimodal physical measured values, and calculating the comprehensive confidence level; and calibrating the structured protocol data based on the comprehensive confidence level. Multimodal AI automatically identifies and parses various industrial protocol messages, eliminating the need for manual configuration of protocol parameters, reducing human errors, and improving data acquisition speed and efficiency.

Description

Technical Field

[0001] This application relates to the field of industrial Internet of Things, and specifically to Internet of Things protocol parsing methods, equipment and media based on multimodal AI. Background Art

[0002] In the Industrial Internet of Things (IIoT), the equipment and systems involved in industrial production may come from different manufacturers, making direct communication and data sharing difficult between these different systems. Industrial communication protocols, such as Modbus, OPC UA, and MQTT, are often used to enable communication between industrial devices. These communication protocols differ significantly in data formats, transmission mechanisms, and semantic definitions, necessitating protocol parsing during communication. However, the message formats of industrial communication protocols are complex and diverse, and parameters such as data types, data addresses, and register lengths vary across devices, making the parsing process particularly complex.

[0003] Traditional protocol parsing methods often require manual configuration and intervention. For example, data collection at industrial sites often requires manual reading of device screens, which can be prone to human error and inefficiency. Some data collection vendors often rely on protocol content for data extraction and matching. However, if the protocol has limitations, it can be difficult to quickly retrieve the relevant data.

[0004] To address these issues, existing protocol parsing methods include two approaches. One involves processing unknown protocol messages through digital conversion and clipping, then identifying high similarity with known protocol messages to achieve intelligent detection of IIoT protocol fields. However, this approach relies on template matching to identify the corresponding protocol and relies on the content of known protocol messages. This approach is incapable of handling new or custom protocols.

[0005] Another method is to preprocess the initial Industrial Internet of Things protocol message sequence sample set, build an LSTM neural network model based on the format characteristics of the protocol, input the training data set into the model for parameter training, and finally use the trained LSTM network model to predict unknown Industrial Internet of Things protocol message fields. This method lacks a real-time data calibration and matching process and is prone to data errors. Summary of the Invention

[0006] To solve the above problems, this application proposes an IoT protocol parsing method based on multimodal AI, including:

[0007] Collecting protocol byte streams and associated multimodal physical data of industrial equipment; the multimodal physical data includes physical signal data, equipment status visual data, and network topology data;

[0008] Parsing the protocol byte stream and the multimodal data through multimodal AI, extracting and fusing multimodal features;

[0009] Inputting the fused multimodal features into a pre-trained protocol grammar generation model to output a protocol grammar tree corresponding to the communication protocol of the industrial equipment;

[0010] Performing semantic decoding on the protocol syntax tree to obtain structured protocol data;

[0011] Acquiring multimodal physical measured values ​​of the industrial equipment, verifying the structured protocol data based on the multimodal physical measured values, and calculating a comprehensive confidence level;

[0012] The structured protocol data is calibrated based on the comprehensive confidence.

[0013] On the other hand, this application also proposes an IoT protocol parsing device based on multimodal AI, comprising:

[0014] at least one processor; and,

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal AI-based Internet of Things protocol parsing method as described in the above example.

[0017] On the other hand, the present application also proposes a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as: the multimodal AI-based Internet of Things protocol parsing method described in the above example.

[0018] The IoT protocol parsing method based on multimodal AI proposed in this application can bring the following beneficial effects:

[0019] Through multimodal AI, various industrial protocol messages can be automatically identified and parsed, eliminating the need for manual reading of device screen information or manual configuration of protocol parameters. This reduces human errors, improves data collection speed and efficiency, and meets the high real-time requirements of industrial IoT environments.

[0020] By integrating multimodal data with a pre-trained protocol grammar generation model, this method breaks through the traditional method's reliance on known protocol templates and can adaptively parse industrial communication protocols of different formats, including new or custom protocols, improving the method's versatility and adaptability. Furthermore, the introduction of multimodal physical measurement verification and comprehensive confidence calibration mechanisms addresses the parsing errors caused by the lack of real-time data calibration in traditional methods. By verifying and dynamically calibrating structured protocol data, the method ensures the consistency and accuracy of parsing results and reduces the risk of data errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 This is a flow chart of a multimodal AI-based IoT protocol parsing method in an embodiment of the present application;

[0023] Figure 2 This is a schematic diagram of an IoT protocol parsing device based on multimodal AI in an embodiment of the present application. DETAILED DESCRIPTION

[0024] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0026] like Figure 1 As shown, the embodiment of the present application provides an IoT protocol parsing method based on multimodal AI, including:

[0027] S101: Collecting protocol byte streams and associated multimodal physical data of industrial equipment; the multimodal physical data includes physical signal data, equipment status visual data, and network topology data.

[0028] Specifically, a binary sequence is obtained from the communication interface of the industrial equipment through physical layer bypass monitoring, which is the protocol byte stream of the industrial equipment. It contains the complete protocol frame structure sent by the industrial equipment, including the frame header, function code, data segment, check bit, frame tail, etc., to record the original communication instructions and status information sent or received by the device.

[0029] High-precision current sensing devices are deployed in series or clamp-type installation on the power lines of equipment associated with industrial equipment. The current sensing devices collect current waveform data in the power supply line, convert it into a digital signal through an analog-to-digital converter (ADC), and continuously collect data at a preset sampling rate to generate physical signal data containing current amplitude, frequency, waveform distortion rate, and time series change characteristics.

[0030] It should be noted that the change in the current waveform directly reflects the operating status of the device, forming a causal relationship with the control instructions in the protocol byte stream, and providing a physical layer basis for subsequent verification of the authenticity of the protocol parsing results.

[0031] At the same time, the image data of the device operation panel or actuator is captured by a camera to obtain the visual information of the device's main state; the network topology data is collected through the device communication port to obtain the topological relationship of the industrial network to which the device is connected.

[0032] It should be noted that in the embodiments of this application, multimodal data of the target device during its historical operation is acquired from at least two different dimensions through a variety of pre-defined data collection methods, such as capturing protocol messages via network communication, capturing device screen images with industrial cameras, reading device manuals or configuration files, and manually entering key parameters. This data includes, but is not limited to, device communication messages, parameter images displayed on the screen, protocol documents, etc., to form an original data set covering the device's operating rules and historical status.

[0033] Specifically, determine to establish communication with the target industrial equipment, obtain network communication messages through the network interface, capture the equipment information image through the camera device, the equipment information image is the equipment screen image or the equipment protocol file image, obtain the equipment protocol electronic file, extract the protocol text data in the electronic file, and obtain the manually entered data input by the user through the user interaction interface, the manually entered data is the equipment display data of the target industrial equipment or the text data input by the user.

[0034] For the network communication protocol message capture method, it directly establishes communication with industrial equipment through IP, port, and network port, reads the message data sent by the equipment, and supports standard communication protocols (such as Modbus, OPC UA, etc.). Network communication collection is the main data source of the system and can provide real-time and accurate equipment status and operation data.

[0035] Regarding manual entry of key parameters, when a device doesn't support network communication or requires manual data verification, users can manually enter device screen data through the system's user interface. The system records these inputs for subsequent processing. Manual entry offers flexibility but can introduce human error, necessitating cross-verification with other data sources.

[0036] Aiming at the method of capturing device screen images with industrial cameras, this paper uses the camera to capture images of the device screen or protocol files, and extracts text data from the images using the Tesseract OCR algorithm.

[0037] For reading device manuals or configuration files, import files in formats such as Word, PDF, or Excel to directly read the device's protocol description file and obtain the device's data format and structure information. These files typically contain key information such as the device's register map, data type, and verification method, which is crucial for correctly parsing and mapping data.

[0038] For example, network communication captures message data sent by devices, including device addresses, data packets, and checksums. Protocol file parsing captures data format and structure information from protocol description files, such as register mapping, data types, and checksum methods. Image recognition captures text data from screens or file images, which may include device display values, status information, and text content from protocol files. Manual entry captures user-entered device screen data, which may include values, status parameters, and so on.

[0039] S102: Parsing the protocol byte stream and the multimodal data through multimodal AI, extracting and fusing multimodal features.

[0040] Specifically, the protocol byte stream and multimodal physical data are time-aligned based on the acquisition timestamp. The aligned protocol byte stream is then fed into the protocol parsing engine. The byte embedding layer converts the binary protocol frame into a high-dimensional vector, capturing the byte-level features of the fields. A multi-head self-attention mechanism then mines the dependencies between fields. The pooling layer then outputs a fixed-dimensional protocol feature vector containing abstract features such as protocol type, field semantics, and interaction logic.

[0041] At the same time, the physical signal data is input into the physical layer parsing engine, which performs a wavelet transform on the current waveform, decomposing it into different frequency components. Time domain features (such as peak value, mean, and variance) and frequency domain features (such as main frequency and harmonic components) are extracted. The TCN's multi-layer one-dimensional convolution kernel captures the local mutation characteristics of the waveform, and the time domain features, frequency domain features, and mutation features are integrated to generate a physical layer feature vector that reflects the electrical operating status of the device.

[0042] The protocol modal parsing engine is invoked for the time-aligned protocol byte stream, extracting protocol features through layered processing. Specifically, frame boundaries are initially located using preset frame header or frame footer identifiers. Within the identified frame structure, fields are divided using field delimiters or length offsets, and field attributes are identified. A field dependency graph is constructed using a graph neural network (GNN), where nodes represent fields and edge weights represent the strength of association between fields. Data address fields and numeric fields have mapping dependencies. Graph pooling operations are used to compress the dependency graph into a fixed-dimensional protocol feature vector, which contains frame structure patterns, field type distribution, and logical association features.

[0043] Physical layer features are extracted from current waveform data through time-frequency analysis. Specifically, wavelet transform is used to perform time-frequency decomposition on the current waveform time series data, converting the one-dimensional time domain signal into a two-dimensional spectrogram. Multi-scale feature extraction is performed on the generated spectrogram. In the time domain, the energy mean, peak factor, and kurtosis within each time window are calculated to capture the instantaneous fluctuation characteristics of the current. In the frequency domain, the main frequency ratio and harmonic distortion rate are extracted to reflect the spectral distribution pattern of the current waveform. The time and frequency domain features are spliced ​​together to form a physical layer feature vector, which represents the electrical operating status of the device.

[0044] For image data, the image modal analysis engine is called to convert visual information into structured state features. Specifically, the device panel image is preprocessed. First, the screen area is located through edge detection, and then the screen tilt is corrected through perspective transformation to eliminate the distortion caused by the shooting angle. Finally, the text recognition algorithm is used to extract the numerical values, status identifiers and symbols displayed on the screen. The continuous image extraction results of the same device are sorted by timestamp to form a device status data sequence. For device status data in different states, the corresponding features are extracted. For the device status data sequence of numerical states, the mean, trend slope and fluctuation variance in the sliding window are calculated; for discrete states, the state duration and switching frequency are extracted. The two types of features are combined to generate a device status feature vector to reflect the change law of the device's operating state.

[0045] Based on the network topology data, a connection diagram is constructed and network structure features are extracted. Specifically, a weighted directed graph is constructed, using the devices in the network topology data as nodes and the communication links between devices as directed edges. In this application, node features can include device type, IP / MAC address, and supported protocols; edge weights are set based on communication frequency, with higher frequencies receiving greater weights.

[0046] The graph attention network (GAT) is used to learn the features of the connection relationship graph, extract node features and global graph features, including node centrality features, shortest path features, and protocol type distribution features of statistical edges. The node features are fused with the global graph features to generate a network structure feature vector to characterize the topological stability and communication capability of the industrial network.

[0047] Furthermore, an attention mechanism with protocol features as the main query is established, and the attention weights of physical signal features, visual features and topological features relative to protocol features are calculated respectively. The multimodal features are dynamically weighted and fused according to the weights to generate a joint feature vector.

[0048] It should be noted that the protocol language understanding module adopts a neural network based on the Transformer architecture, which acquires semantic understanding capabilities through pre-training on industrial protocol corpus; the temporal convolutional network contains multiple layers of one-dimensional convolution kernels to capture local patterns in signal waveforms; the three-dimensional convolutional network extracts spatial-temporal joint features from device status videos; and the attention weight calculation adopts a query-key-value matching mechanism.

[0049] It should be noted that when the data is in image mode, optical character recognition is performed through a neural network to extract text characters in the data, and the text characters are parsed through the parsing engine corresponding to the text mode; when the data is in text mode, document structure analysis is performed on the data to obtain semantic relationships, and key parameters in the data are identified based on the semantic relationships; when the data is in numerical form, it is determined whether the data has a frame structure. If so, the data is determined to be a network communication message, and the network communication message is parsed through the communication protocol.

[0050] For image data, edge line detection is performed on the image data, the rotation compensation angle is calculated based on the detected text line edge line, the image data is rotationally corrected based on the rotation compensation angle, a horizontally corrected image is generated, the grayscale threshold of the horizontally corrected image is calculated, the image is converted into a binary image according to the grayscale threshold, the binary image is input into the neural network model, and the text characters output by the neural network model are obtained.

[0051] In the embodiments of the present application, the Tesseract OCR algorithm is used to extract text data from images. Specifically, displayed data is extracted from device screenshots, text information is extracted from protocol file images, and device information is extracted from device labels and logos.

[0052] For image preprocessing, the RGB image is converted into a grayscale image to reduce the computational complexity. The formula is: , where R, G, B are the red, green, and blue channel values. The optimal threshold T is automatically determined by the Otsu algorithm (maximum inter-class variance method), based on the formula: ,in, They are: foreground and background pixel ratios, Respectively: foreground and background pixel ratio. Based on the formula: Generate the final binary image. Use median filtering to reduce noise, based on the formula: , where W is a sliding window. Detect the text tilt angle θ through Hough transform: , and correct the image by -θ rotation. Edge detection is achieved through the Canny algorithm, steps: Gaussian smoothing: , calculate the gradient magnitude and direction: , Tesseract uses connected component analysis (CCA) to locate text blocks and minimize the energy function: ,in is a text block, and λ is the overlap penalty coefficient. Character recognition is achieved through LSTM neural network, and the output of time step t is: , the loss function is: , where z is the true label sequence.

[0053] Furthermore, the generated sequence tags are spelled corrected and the minimum edit distance D is calculated: , where II is the indicator function.

[0054] For protocol text data, recurrent neural networks (RNN) or LSTM are used to identify and analyze the hierarchical structure in the document. ,in, is the hidden state of the LSTM unit, W is the weight matrix, and b is the bias term. Named Entity Recognition (NER) technology is used to extract key information from documents (such as register addresses, data names, and data formats). Regular expressions or template-based matching methods are used to extract specific fields. A table detection algorithm is used to extract register mapping tables from documents. Data is extracted based on specific register patterns (such as register name, address, and size). Data types (such as integer, floating-point number, and string) are obtained from the register mapping table. Pattern matching is used to identify common data types such as numbers, dates, and strings. Standardized data structure definitions are generated.

[0055] S103: Input the fused multimodal features into a pre-trained protocol syntax generation model, output a protocol syntax tree corresponding to the communication protocol of the industrial equipment, perform semantic decoding on the protocol syntax tree, and obtain structured protocol data.

[0056] Specifically, the multimodal features that have undergone time alignment and weighted fusion are input into the pre-trained protocol grammar generation model. The model performs semantic mapping on the fused features through a multi-layer Transformer decoder to generate a protocol syntax tree corresponding to the communication protocol.

[0057] Furthermore, the protocol syntax tree is semantically decoded. By traversing the hierarchical nodes of the syntax tree, meta-information such as the type identifier, semantic description, and numerical range of each field is extracted. Combined with the context association in the multimodal features, the abstract syntax structure is mapped into specific structured protocol data.

[0058] For example, when parsing the "Pressure Value" field under the "Data Segment" node in the syntax tree, the binary value is converted into a key-value pair such as "Pressure: 1.2 MPa" by combining it with the measured characteristics of the pressure sensor in the physical signal data. When parsing the "Status Code" field, the indicator light recognition results in the device status visual data are linked to convert the code value "0x01" into the descriptive information "Operational Status: Normal." The resulting structured protocol data is presented in a standardized format (such as JSON or XML) and contains complete information such as the device's real-time parameters, operating status, control commands, and communication properties, providing a structured foundation for subsequent verification and calibration.

[0059] It should be noted that the protocol syntax tree clearly presents the complete logic of the protocol in a hierarchical structure. The root node is the protocol type. The first-level child nodes include core components such as the frame header, data segment, and check bit. The second-level child nodes are refined to the specific attributes of each field. At the same time, the dependency logic between fields is reflected through edge connection relationships.

[0060] Before inputting the fused multimodal features into a pre-trained protocol grammar generation model and outputting the protocol syntax tree for the communication protocol corresponding to the industrial equipment, the process also includes building and training the protocol grammar generation model. Specifically, the protocol grammar generation model is first pre-trained on a general grammar using a public set of standard communication data samples for industrial protocols. Key fields in the protocol byte stream are partially masked, and a general protocol training set is constructed to train the network to predict the protocol function type of the masked fields based on the context. The generated protocol grammar structure is then verified to be consistent with the standard protocol documentation.

[0061] Secondly, the general protocol training set is input into the protocol grammar generation model. The encoder in the protocol grammar generation model converts the masked protocol byte stream into a context feature vector. The decoder predicts the protocol type of the masked field based on the context feature vector and generates a complete protocol grammar structure based on the predicted field type.

[0062] Again, based on the public standard protocol documents, the degree of conformity between the predicted protocol syntax structure and the standard structure is calculated, including the accuracy of field types, the arrangement order of fields within the frame, the proportion of nested relationships consistent with the standard, the proportion of predicted field lengths consistent with the standard definition, etc.

[0063] Finally, the conformance is determined to meet a preset threshold. If not, the model parameters are adjusted using a backpropagation algorithm to optimize the weights of the self-attention layer and the decoder's output projection layer. After each iteration, new masked samples are used for training until the model's conformance on the validation set stabilizes above the threshold. Ultimately, a pre-trained protocol grammar generation model capable of parsing general protocol grammar is obtained.

[0064] S104: Acquire multimodal physical measured values ​​of the industrial equipment, verify the structured protocol data based on the multimodal physical measured values, and calculate a comprehensive confidence level.

[0065] Specifically, the system acquires multimodal physical measurements of industrial equipment, including physical layer measurements, equipment physical status, and real-time topological paths. It then parses structured protocol data to extract numerical data, equipment status information, and equipment control instructions. The system then verifies the structured protocol data based on these multimodal physical measurements, including verification of data authenticity, equipment status consistency, and instruction reachability. A comprehensive confidence score is generated based on the verification results.

[0066] Among them, the collection process of multi-modal physical measured values ​​includes: collecting measurement data generated by direct sampling of industrial equipment communication lines or sensor output ends to obtain physical layer measured values; obtaining the physical status of the equipment through equipment body images or video streams collected by independent industrial cameras; and obtaining the real-time topology path by scanning the equipment connection status generated by real-time network probes.

[0067] Furthermore, the process of verifying structured protocol data based on multimodal physical measured values ​​includes: comparing the deviation of the numerical data parsed from the protocol with the measured values ​​of the physical layer, and marking it as credible when the deviation does not exceed the industrial tolerance threshold defined in the equipment technical specification; matching the status flag parsed from the protocol with the independently collected physical status of the device, and marking it as credible when the status description is consistent with the image analysis result; checking whether the target device of the control instruction is reachable according to the real-time topology path, and marking it as reachable when the target device is within the valid communication path and the response delay is less than the protocol requirement.

[0068] Perform weighted aggregation on the data value trust mark, state trust mark and instruction reachability mark, and output a comprehensive confidence score in the range of 0-1.

[0069] S105: Calibrate the structured protocol data based on the comprehensive confidence.

[0070] Specifically, when the comprehensive confidence is not lower than the first preset threshold, the structured protocol data is marked as trusted data and output directly; when the comprehensive confidence is lower than the first preset threshold and not lower than the second preset threshold, the structured protocol data is corrected based on the multimodal physical measured values; when the comprehensive confidence is lower than the second preset threshold, the structured protocol data is marked as untrustworthy data, and the manual review process is triggered.

[0071] This application uses multimodal AI to automatically identify and parse various industrial protocol messages, eliminating the need for manual reading of device screen information or manual configuration of protocol parameters. This reduces human errors, improves data collection speed and efficiency, and meets the high real-time requirements of industrial Internet of Things environments.

[0072] By integrating multimodal data with a pre-trained protocol grammar generation model, this method breaks through the traditional method's reliance on known protocol templates and can adaptively parse industrial communication protocols of different formats, including new or custom protocols, improving the method's versatility and adaptability. Furthermore, the introduction of multimodal physical measurement verification and comprehensive confidence calibration mechanisms addresses the parsing errors caused by the lack of real-time data calibration in traditional methods. By verifying and dynamically calibrating structured protocol data, the method ensures the consistency and accuracy of parsing results and reduces the risk of data errors.

[0073] like Figure 2 As shown, the embodiment of the present application also proposes an IoT protocol parsing device based on multimodal AI, including:

[0074] at least one processor; and,

[0075] a memory communicatively connected to the at least one processor; wherein,

[0076] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal AI-based Internet of Things protocol parsing method as described in any of the above embodiments.

[0077] An embodiment of the present application also provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as: the multimodal AI-based Internet of Things protocol parsing method described in any of the above embodiments.

[0078] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device and medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0079] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0080] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0082] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0084] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0085] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0086] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0087] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0088] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A multimodal AI-based IoT protocol parsing method, characterized in that: include: Collecting protocol byte streams and associated multimodal physical data of industrial equipment; the multimodal physical data includes physical signal data, equipment status visual data, and network topology data; Parsing the protocol byte stream and the multimodal data through multimodal AI, extracting and fusing multimodal features; The method further comprises: Obtain a sample set of standard communication data of open industrial protocols; Partially masking key fields in the standard communication data sample set to construct a general protocol training set; Inputting the general protocol training set into the protocol grammar generation model, predicting the protocol type of the masked field according to the protocol context, and outputting the predicted protocol grammar structure; Verifying the predicted conformity of the predicted protocol grammar structure based on the standard protocol document, and adjusting the protocol grammar generation model according to the predicted conformity to obtain a pre-trained protocol grammar generation model; Collect historical multimodal physical data of industrial equipment associated with the Internet of Things protocol, extract multimodal feature vectors from the historical multimodal physical data through multimodal AI, and construct a multimodal feature training set; Based on a multimodal interface, the multimodal feature training set is input into the pre-trained protocol grammar generation model to construct a multi-objective optimization function; Solving the multi-objective optimization function and adjusting network parameters of the pre-trained protocol grammar generation model; Inputting the fused multimodal features into a pre-trained protocol syntax generation model, outputting a protocol syntax tree corresponding to the communication protocol of the industrial equipment, and semantically decoding the protocol syntax tree to obtain structured protocol data; Acquiring multimodal physical measured values ​​of the industrial equipment, verifying the structured protocol data based on the multimodal physical measured values, and calculating a comprehensive confidence level; The structured protocol data is calibrated based on the comprehensive confidence.

2. The method for analyzing Internet of Things protocols based on multimodal AI according to claim 1, characterized in that: The acquisition of the protocol byte stream and associated multimodal physical data of the industrial equipment specifically includes: Monitor the physical layer signal of the industrial equipment through the communication interface of the industrial equipment to obtain a protocol byte stream; Using a sensor device, collecting current waveform data of a device power line associated with the industrial device to obtain physical signal data; Using a camera device, collecting image data of a device panel associated with the industrial device to obtain visual data of the device status; The connection topology relationship of the industrial equipment is collected through the equipment communication port to obtain network topology data.

3. The method for analyzing Internet of Things protocols based on multimodal AI according to claim 2, characterized in that: The parsing of the protocol byte stream and the multimodal data by multimodal AI, extracting and fusing multimodal features, specifically includes: Based on the acquisition timestamp, timing aligning the protocol byte stream and the multimodal physical data; The time-aligned protocol byte stream and multimodal physical data are input into the multimodal AI. The corresponding data modality parsing engine is used to extract the feature vectors corresponding to each modality. The feature vectors include protocol features, physical layer features, device status features, and network structure features. The feature vectors are weightedly fused according to the attention weights corresponding to the feature vectors to generate a fused feature vector.

4. The method for analyzing Internet of Things protocols based on multimodal AI according to claim 3, characterized in that: The extraction of the feature vector corresponding to each modality by the corresponding data modality parsing engine specifically includes: Parsing the protocol byte stream, identifying a protocol frame structure, determining fields in the protocol byte stream, and generating protocol features based on dependency relationships between the fields; Performing time-frequency transformation on the current waveform data to generate a spectrum graph, extracting time sequence features of the spectrum graph, and obtaining physical layer features; Extracting screen display data from the image data, constructing a device status data sequence, and extracting device status features from the device status data sequence; Based on the network topology data, a device connection relationship diagram is constructed to extract network structure features.

5. The method for analyzing Internet of Things protocols based on multimodal AI according to claim 1, characterized in that: The verifying of the structured protocol data based on the multimodal physical measured values ​​and calculating the comprehensive confidence level specifically includes: Parsing the structured protocol data to extract numerical data, device status information, and device control instructions; Based on the multimodal physical measured value, performing deviation comparison with the numerical data, the device status information and the device control instruction respectively to obtain corresponding deviation values; The deviation values ​​are weighted to calculate the comprehensive confidence of the structured protocol data.

6. The method for analyzing Internet of Things protocols based on multimodal AI according to claim 5, characterized in that: The calibrating the structured protocol data based on the comprehensive confidence level specifically includes: When the comprehensive confidence level is not lower than a first preset threshold, marking the structured protocol data as credible data and directly outputting it; When the comprehensive confidence is lower than the first preset threshold and not lower than a second preset threshold, correcting the structured protocol data based on the multimodal physical measured values; When the comprehensive confidence is lower than the second preset threshold, the structured protocol data is marked as untrustworthy data, and a manual review process is triggered.

7. An IoT protocol parsing device based on multimodal AI, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the following operations: Collecting protocol byte streams and associated multimodal physical data of industrial equipment; the multimodal physical data includes physical signal data, equipment status visual data, and network topology data; Parsing the protocol byte stream and the multimodal data through multimodal AI, extracting and fusing multimodal features; Also includes: Obtain a sample set of standard communication data of open industrial protocols; Partially masking key fields in the standard communication data sample set to construct a general protocol training set; Inputting the general protocol training set into the protocol grammar generation model, predicting the protocol type of the masked field according to the protocol context, and outputting the predicted protocol grammar structure; Verifying the predicted conformity of the predicted protocol grammar structure based on the standard protocol document, and adjusting the protocol grammar generation model according to the predicted conformity to obtain a pre-trained protocol grammar generation model; Collecting historical multimodal physical data of industrial equipment associated with the IoT protocol, extracting multimodal feature vectors from the historical multimodal physical data through multimodal AI, and constructing a multimodal feature training set; Based on a multimodal interface, the multimodal feature training set is input into the pre-trained protocol grammar generation model to construct a multi-objective optimization function; Solving the multi-objective optimization function and adjusting network parameters of the pre-trained protocol grammar generation model; Inputting the fused multimodal features into a pre-trained protocol syntax generation model, outputting a protocol syntax tree corresponding to the communication protocol of the industrial equipment, and semantically decoding the protocol syntax tree to obtain structured protocol data; Acquiring multimodal physical measured values ​​of the industrial equipment, verifying the structured protocol data based on the multimodal physical measured values, and calculating a comprehensive confidence level; The structured protocol data is calibrated based on the comprehensive confidence.

8. A non-volatile computer storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are configured to: Collecting protocol byte streams and associated multimodal physical data of industrial equipment; the multimodal physical data includes physical signal data, equipment status visual data, and network topology data; Parsing the protocol byte stream and the multimodal data through multimodal AI, extracting and fusing multimodal features; Also includes: Obtain a sample set of standard communication data of open industrial protocols; Partially masking key fields in the standard communication data sample set to construct a general protocol training set; Inputting the general protocol training set into the protocol grammar generation model, predicting the protocol type of the masked field according to the protocol context, and outputting the predicted protocol grammar structure; Verifying the predicted conformity of the predicted protocol grammar structure based on the standard protocol document, and adjusting the protocol grammar generation model according to the predicted conformity to obtain a pre-trained protocol grammar generation model; Collect historical multimodal physical data of industrial equipment associated with the Internet of Things protocol, extract multimodal feature vectors from the historical multimodal physical data through multimodal AI, and construct a multimodal feature training set; Based on a multimodal interface, the multimodal feature training set is input into the pre-trained protocol grammar generation model to construct a multi-objective optimization function; Solving the multi-objective optimization function and adjusting network parameters of the pre-trained protocol grammar generation model; Inputting the fused multimodal features into a pre-trained protocol syntax generation model, outputting a protocol syntax tree corresponding to the communication protocol of the industrial equipment, and semantically decoding the protocol syntax tree to obtain structured protocol data; Acquiring multimodal physical measured values ​​of the industrial equipment, verifying the structured protocol data based on the multimodal physical measured values, and calculating a comprehensive confidence level; The structured protocol data is calibrated based on the comprehensive confidence.

Images