Multimodal AI edge terminal integrating language interaction, visual analysis and logical reasoning

Abstract

The invention discloses a multi-modal AI edge terminal integrating language interaction, visual analysis and logical reasoning, which belongs to the crossing field of edge calculation and multi-modal AI technology and comprises a language interaction module, a visual analysis module and a combined logical reasoning module. The method does not need to manually operate a terminal interface or adapt to different communication protocols, greatly improves the on-site operation convenience, greatly improves the operation and maintenance efficiency and intelligent interaction experience of the power distribution station house, reduces the hardware purchase, wiring and maintenance cost, remarkably improves the accuracy of anomaly detection and decision making, and is suitable for large-scale popularization and application. And the intelligence and automation level of an industrial scene is comprehensively improved.

Description

Technical Field

[0001] This invention relates to the intersection of edge computing and multimodal AI technologies, and particularly to a multimodal AI edge terminal that integrates language interaction, visual analysis, and logical reasoning. Background Technology

[0002] In the field of industrial monitoring, especially power monitoring, traditional edge terminals have three major pain points: (1) Dispersed interaction entry points: The physical interfaces (RS485, Ethernet, LoRa, etc.) and transmission protocols (Modbus, DL / T 645, IEC 61850, etc.) of various edge terminals (inspection terminals, data acquisition terminals, fault diagnosis terminals, gateways, etc.) are not unified, and maintenance personnel need to carry multiple devices or repeatedly adapt interfaces, resulting in low operation efficiency.

[0003] (2) Complex sensing methods: It relies on power sensors (temperature, humidity, smoke, water level, access control) to collect data. The hardware cost is high, the sensors are unreliable, false alarms and offline problems occur frequently, the sensor protocol compatibility is poor, it is not easy to review the problems, and the hardware needs to be replaced when the sensor is damaged, which increases the workload of sensor maintenance.

[0004] (3) Limitations in reasoning ability: It relies heavily on simple local threshold judgments and cannot integrate natural language, visual features, and historical device data for joint analysis, making it difficult to meet the needs of accurate decision-making in complex scenarios.

[0005] In current technologies, single-modal edge terminals (such as those that only support voice interaction or visual recognition) cannot solve the problem of multi-information collaboration; while multimodal inference that relies entirely on the cloud suffers from problems such as high data transmission latency and privacy and security risks.

[0006] Therefore, there is an urgent need for multimodal AI edge terminals that integrate language interaction, visual analysis, and logical reasoning. These terminals can unify the operation entry point through language interaction, expand the perception dimension through visual analysis, and integrate multi-source information through joint logical reasoning. At the same time, they can combine edge-cloud collaboration to balance real-time performance and computing power requirements, thus adapting to the complex monitoring needs of industrial scenarios such as power. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by proposing a multimodal AI edge terminal that integrates language interaction, visual analysis, and logical reasoning.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal AI edge terminal integrating language interaction, visual analysis, and logical reasoning, including a language interaction module, a visual analysis module, and a joint logical reasoning module; The language interaction module is used to receive wake-up commands and activate the system, then recognize the collected voice signals and generate voice text, while converting the recognition results into standard speech. The visual analysis module is used to train a visual model for power scenarios and to perform image preprocessing and visual feature extraction on the edge terminal, while only uploading the feature data to the cloud. The joint logic reasoning module is used to perform structured processing of multimodal data and joint analysis of multiple information. The language interaction module includes a microphone, an industrial-grade sound card, and an industrial-grade wake-up unit.

[0009] As a further aspect of the present invention, the detailed steps for the language interaction module to receive the wake-up command and activate the system are as follows: S1.1: In low-power standby mode, the voice interaction module continuously listens for wake-up words through an industrial-grade microphone. When a suspected voice signal is detected, the industrial-grade wake-up unit extracts the acoustic features of MFCC features and voiceprint features through the embedded DSP and performs real-time matching with each wake-up template stored locally. S1.2: If the matching degree between the received acoustic features and any wake-up template exceeds the set threshold, the language interaction module confirms that the command source is valid and switches the system from standby mode to working mode. At the same time, the system outputs a voice response to prompt the maintenance personnel to continue to issue operation commands.

[0010] As a further aspect of the present invention, the language interaction module identifies the collected speech signal, generates speech text, and converts the recognition result into standard speech in the following specific steps: S2.1: The microphone collects the voice commands of the staff, performs analog-to-digital conversion through an industrial-grade sound card to generate corresponding digital signals, performs environmental noise filtering through end-side algorithms and automatically adjusts the gain, then divides the voice signal into frames with a frame length of 20-30ms, and extracts the feature vector of each voice signal frame by frame. S2.2: Input the extracted feature vectors into the ASR model and perform decoding analysis through the ASR model. At the same time, convert the decoding results into corresponding pinyin or word sequences. Combine the language model of the power operation and maintenance field to perform context correction on the generated pinyin or word sequences and generate accurate text results. S2.3: The corrected text results are processed by word segmentation, noise reduction and synonym normalization. Then, the text type is determined by the intent classification model, and the text results are structured into standard operation units. The processing results are then input into the joint logic reasoning module, and the text results fed back by the joint logic reasoning module are received. The text results fed back by the joint logic reasoning module are then processed by sentence normalization and pronunciation optimization. S2.4: Call the power text-to-speech unit to convert the optimized text result into standard speech, which is then played through the speaker. If the user continues to ask questions, the system maintains the conversation state and waits for new speech input.

[0011] As a further aspect of the present invention, the specific steps for the visual analysis module to train the power scene visual algorithm are as follows: S3.1: Real-time high-definition images are collected from various environments in real power distribution rooms and substations. At the same time, the shooting angle covers multiple angles, including front, side and top views, and the lighting conditions cover daytime, nighttime and low-light environments. The quality of each collected image data is reviewed, and the image data that meets the preset conditions is retained. Then, the annotation tool is used to annotate the preset targets in each image data with rectangular boxes to form a label file that conforms to the YOLO format. S3.2: Divide the data into training set, validation set and test set according to the preset ratio. Then adjust the pixel size of each image data in each dataset to meet the input requirements of YOLOv8 network. Then perform image enhancement on each image data. Then normalize the pixel values ​​of each enhanced image data and adjust the label coordinates at the same time. S3.3: Based on the PyTorch framework, the training set data is divided into multiple training sample sets, which are then sequentially input into the YOLOv8 network. A multi-model parallel training method is adopted, and the safety helmet recognition model, water level recognition model, and access control recognition model are trained through the YOLOv8 network respectively. After each round of training, the loss value between the output of each model and the corresponding real data is calculated, and the parameters of each model are adjusted through the Adam optimization algorithm. S3.4: After multiple rounds of training, when the change in the loss value of each model converges to the preset range, input the validation set and test set into each model and evaluate the performance of each model. If the helmet recognition accuracy is ≥1%, then retrain the helmet recognition model. If the water level recognition model has a recognition error ≥5cm, then retrain the water level recognition model. If the access control status recognition model has a recognition error ≥1%, then retrain the access control status recognition model. S3.5: Repeatedly train, validate, and test each model until the preset conditions are met. Export the three trained models in ONNX format, then convert them to an edge-compatible format. Deploy each model to the NPU of the edge terminal main chip for real-time inference, and output the visual features generated by inference in a structured form.

[0012] As a further aspect of the present invention, the specific steps of the joint logic reasoning module in performing structured processing on multimodal data are as follows: S4.1: Obtain structured speech text after ASR and semantic parsing from the language interaction module, receive the recognized visual features from the visual analysis module, including the detection target category, state attribute, confidence level and spatial location information, obtain the device historical status, alarm record and running timestamp information from the terminal database or log, add modal identifiers to each type of data, and use regular expression parsing and keyword mapping. S4.2: Extract the required semantic fields from the speech text. If a certain modality is missing some fields, it will be automatically filled with the current system time. If the speech is missing a device ID, it will be filled according to semantic context reasoning or default configuration. Then, the synonym fields of different modalities will be uniformly named and quantified. S4.3: Perform one-hot encoding on the operation type and target object in the speech text, convert the parameter or numerical field into floating-point number, unify the unit information in the speech text, encode the target category and state label in the visual features into integer or Boolean variables, standardize the spatial location information to the range of [0, 1], map the historical state value in the local state into binary variables, and convert the timestamp into relative time; S4.4: Align the encoded modal data according to the acquisition time, generate modal weight vectors based on modal confidence, and then encapsulate each modal data into a data matrix in a unified format based on the generated modal weight vectors, where each row corresponds to a modal record. Then, store the normalized data matrix in the local cache queue and upload it synchronously to the cloud logic inference interface.

[0013] As a further aspect of the present invention, the specific steps of the joint logic reasoning module in performing multi-information joint analysis are as follows: S5.1: The joint logic reasoning module uploads the generated data matrix containing voice text, visual features and local state data to the cloud. Then, the industrial multimodal large model deployed in the cloud calculates the correlation matrix between modalities through multi-head self-attention and cross-attention. S5.2: Based on the established relevance matrix, the voice intent is used as the query condition to retrieve the evidence segment with the highest relevance between visual features and local state data, and the corresponding evidence support score is calculated. The symbolic check is performed on the preset high-priority rules, and the numerical value and state are mapped to the rule premise. If the rule condition is met, a symbolic inference output is formed and the confidence of the conclusion is improved. S5.3: The results from statistical inference and symbolic rules are fused according to preset weights to generate the final conclusion confidence and recommended actions. Then, local state data is used to backtrack the historical records corresponding to the current vision or voice to identify similar events or repeated alarms within a specified time period. S5.4: Construct candidate causal chains based on retrieved evidence fragments and evaluate the strength of the links. When visual conclusions conflict with historical records, check visual confidence, historical credibility, and time difference, and use rules to determine priority or initiate a resampling request. Then, encapsulate the model conclusions, supporting evidence, confidence, and recommended actions into structured output, and automatically generate explanatory text to explain the key points of the reasoning process in natural language.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention utilizes a microphone to listen for wake-up words under low power consumption. The wake-up unit extracts MFCC / voiceprint and compares it with a local template; a match activates ASR. The microphone data is captured by a sound card, filtered, enhanced, framed, and its features extracted before being sent to ASR decoding. This is then corrected using a language model specific to the power industry and semantically parsed into structured intent. At the visual end, multi-angle, multi-lighting images are captured, labeled, enhanced, and trained using a lightweight YOLOv8 model, which is then quantized and deployed to the NPU, providing real-time output of target category, coordinates, and confidence level. At the edge, speech, text, visual features, and local history are aggregated, fields are standardized, encoded, time-aligned, and a normalized data matrix is ​​generated and uploaded to the cloud. The cloud-based multimodal large model retrieves visual / historical evidence using voice intent, combines cross-modal attention and rule-based symbolic reasoning to calculate confidence and suggestions, and outputs structured conclusions and interpretable text. Edge-based TTS synthesized speech is played back and recorded for subsequent iterations. This eliminates the need for manual terminal interface operation or adaptation to different communication protocols, significantly improving on-site operational convenience, greatly enhancing the operation and maintenance efficiency and intelligent interactive experience of power distribution stations, reducing hardware procurement, wiring, and maintenance costs, significantly improving the accuracy of anomaly detection and decision-making, and comprehensively enhancing the level of intelligence and automation in industrial scenarios. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0016] Figure 1 This is a system block diagram of the multimodal AI edge terminal integrating language interaction, visual analysis, and logical reasoning proposed in this invention. Figure 2 This is a flowchart illustrating the operation of the multimodal AI edge terminal that integrates language interaction, visual analysis, and logical reasoning proposed in this invention. Detailed Implementation

[0017] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0018] Reference Figure 1-2It is a multimodal AI edge terminal that integrates language interaction, visual analysis and logical reasoning, including a language interaction module, a visual analysis module and a joint logical reasoning module.

[0019] The language interaction module is used to receive wake-up commands and activate the system. It then recognizes the collected voice signals, generates voice text, and converts the recognition results into standard speech. The language interaction module includes a microphone, an industrial-grade sound card, and an industrial-grade wake-up unit.

[0020] Specifically, in low-power standby mode, the voice interaction module continuously listens for wake-up words through an industrial-grade microphone. When a suspected voice signal is detected, the industrial-grade wake-up unit extracts MFCC features and voiceprint features through an embedded DSP and performs real-time matching with each wake-up template stored locally. If the matching degree between the received acoustic features and any wake-up template exceeds a set threshold, the voice interaction module confirms that the command source is valid and switches the system from standby mode to working mode. At the same time, the system outputs a voice response to prompt the maintenance personnel to continue issuing operation commands.

[0021] Specifically, the microphone captures the voice commands of the staff, performs analog-to-digital conversion through an industrial-grade sound card to generate corresponding digital signals, filters environmental noise using an edge-side algorithm, and automatically adjusts the gain. The voice signal is then segmented into frames of 20-30ms, and feature vectors are extracted from each frame. These extracted feature vectors are input into an ASR model for decoding and analysis. The decoding results are converted into corresponding pinyin or word sequences. Using a language model specific to the power industry, the generated pinyin or word sequences are corrected for contextual information, resulting in accurate text. The corrected text is then segmented, denoised, and normalized for synonyms. An intent classification model determines the text type, and the text is structured into standard operation units. The processed results are then input into a joint logic inference module. The system receives text results from this module, performs sentence standardization and pronunciation optimization, and calls a power text-to-speech unit to convert the optimized text into standard speech, which is then played through a speaker. If the user continues to ask questions, the system maintains the conversation state and waits for new voice input.

[0022] The visual analysis module is used to train a visual model for power scenarios and perform image preprocessing and visual feature extraction on the edge terminal, while only uploading the feature data to the cloud.

[0023] Specifically, real-time high-definition images are acquired from various environments in real power distribution rooms and substations. The camera angles cover multiple perspectives, including front, side, and overhead views, and lighting conditions encompass daytime, nighttime, and low-light environments. The acquired image data undergoes quality review, retaining only those meeting preset conditions. Then, a labeling tool is used to annotate preset targets in each image, creating label files conforming to the YOLO format. The data is divided into training, validation, and test sets according to preset proportions. Next, the pixel size of each image in each dataset is adjusted to meet the input requirements of the YOLOv8 network. Image augmentation is then performed on each image, followed by pixel value normalization and simultaneous adjustment of label coordinates. Based on the PyTorch framework, the training set data is divided into multiple training sample sets, which are sequentially input into the YOLOv8 network. A multi-model parallel training approach is employed, and the YOLOv8 network... The safety helmet recognition model, water level recognition model, and access control recognition model are trained separately. After each training round, the loss value between the output of each model and the corresponding real data is calculated, and the parameters of each model are adjusted using the Adam optimization algorithm. When the change value of each model converges to the preset range after multiple training rounds, the validation set and test set are input into each model, and the performance of each model is evaluated. If the safety helmet recognition accuracy is ≥1%, the safety helmet recognition model is retrained. If the recognition error of the water level recognition model is ≥5cm, the water level recognition model is retrained. If the recognition error of the access control status recognition model is ≥1%, the access control status recognition model is retrained. The training, validation, and testing of each model are repeated until the preset conditions are met. The three sets of trained models are exported in ONNX format, then converted to an edge-compatible format, and then each model is deployed to the NPU of the edge terminal main chip for real-time inference. The visual features generated by the inference are output in a structured form.

[0024] The joint logic reasoning module is used to perform structured processing of multimodal data and joint analysis of multiple information.

[0025] Specifically, the system obtains structured speech text after ASR and semantic parsing from the language interaction module, receives recognized visual features from the visual analysis module, including the detected target category, state attributes, confidence level, and spatial location information, and retrieves device historical status, alarm records, and runtime timestamps from the terminal database or logs. Modal identifiers are added to each data type. Regular expression parsing and keyword mapping are used to extract necessary semantic fields from the speech text. If a modality is missing a field, it is automatically filled with the current system time. If the speech text lacks a device ID, it is filled based on semantic context reasoning or default configuration. Synonymous fields from different modalities are then uniformly named and quantified, and the operation type and target in the speech text are matched. The system performs one-hot encoding, converting parameter or numerical fields into floating-point numbers and unifying unit information in speech and text. It encodes target categories and state labels in visual features into integer or Boolean variables, standardizes spatial location information to the range [0, 1], maps historical state values ​​in local states into binary variables, converts timestamps into relative times, aligns the encoded modal data according to the acquisition time, and generates modal weight vectors based on modal confidence. Then, based on the generated modal weight vectors, the modal data is encapsulated into a data matrix in a unified format, where each row corresponds to a modal record. Finally, the normalized data matrix is ​​stored in a local cache queue and synchronously uploaded to the cloud logic inference interface.

[0026] Specifically, the joint logic reasoning module uploads the generated data matrix containing speech text, visual features, and local state data to the cloud. Then, the industrial multimodal large model deployed in the cloud calculates the correlation matrix between modalities using multi-head self-attention and cross-attention. Based on the established correlation matrix, it uses speech intent as a query condition to retrieve the evidence segment with the highest correlation between visual features and local state data, and calculates the corresponding evidence support score. It performs symbolic checks on preset high-priority rules, mapping numerical values ​​and states to rule premises. If the rule conditions are met, it constructs a symbolic-level inference output and increases the confidence of the conclusion. This incorporates data from statistical reasoning. The results of the severance and symbol rules are fused according to preset weights to generate the final conclusion confidence and recommended action. Then, local state data is used to backtrack the historical records corresponding to the current vision or speech, identify similar events or repeated alarms within a specified time period, construct candidate causal chains based on the retrieved evidence fragments, and evaluate the strength of the links. When the visual conclusion conflicts with the historical records, the visual confidence, historical credibility, and time difference are checked, and rules are used to determine the priority or initiate a supplementary sampling request. Then, the model conclusion, supporting evidence, confidence, and recommended action are encapsulated into a structured output, and explanatory text is automatically generated to explain the key points of the reasoning process in natural language.

[0027] Furthermore, it should be noted that this embodiment is deployed in a multimodal AI panoramic intelligent power distribution station. This station follows the traditional "cloud-pipe-edge-device" smart IoT architecture of intelligent power distribution stations. On the "device" side, a full-vision acquisition solution is adopted, requiring only 2 bullet cameras, 4 dome cameras, and 2 cable trench cameras per station to provide panoramic coverage of areas such as station gates, high-voltage switchgear, transformers, low-voltage switchgear, and cable trenches. This achieves panoramic perception of the station through vision, realizing a "zero-sensor" and "pure vision" end-side construction approach. On the "edge" side, multimodal AI edge terminals are deployed. These terminals integrate natural language understanding, computer vision analysis, joint logic reasoning, and speech synthesis, comprehensively applying large-scale model technology to achieve a multimodal AI intelligent upgrade of traditional edge gateways. On the "cloud" side, a panoramic monitoring platform for the power distribution station is deployed. Information perception and hazard warnings for all services of the station are presented visually, enabling operation and maintenance management as if one were actually there, all from the cloud master station. At the same time, refer to Figure 2 The specific operation process of this embodiment is as follows: Maintenance personnel call "Xiaodian" (a mobile terminal), and the multimodal AI edge terminal identifies the user through the wake-up module, thereby waking up the system. It then replies "Hello, Master" through the TTS (Text-to-Speech) module. At this time, the maintenance personnel can ask questions in natural language such as "Station status," "Current water level," "Last opening time," and "Main station connection status." The multimodal AI edge terminal converts the natural speech into text through the ASR (Automatic Speech Recognition) module, transmits the text to the information integration module, formats it, and sends it to the logic reasoning model. The logic reasoning model performs reasoning analysis based on the historical records and real-time data from the visual analysis module and returns the results to the information integration module. After parsing, the information integration module transmits the valid text portion to the TTS module, which then replies to the maintenance personnel with the answer to the question, such as: "Current station status is good, no abnormalities," or "Current water level is 50 cm, exceeding the warning line by 40 cm; this alarm has been reported to the main station."

Claims

1. A multimodal AI edge terminal integrating language interaction, visual analysis, and logical reasoning, characterized in that: It includes a language interaction module, a visual analysis module, and a joint logical reasoning module; The language interaction module is used to receive wake-up commands and activate the system, then recognize the collected voice signals and generate voice text, while converting the recognition results into standard speech. The visual analysis module is used to train a visual model for power scenarios and perform image preprocessing and visual feature extraction on the edge terminal, while only uploading the feature data to the cloud. The joint logic reasoning module is used to perform structured processing of multimodal data and joint analysis of multiple information. The language interaction module includes a microphone, an industrial-grade sound card, and an industrial-grade wake-up unit.

2. The multimodal AI edge terminal integrating language interaction, visual analysis, and logical reasoning according to claim 1, characterized in that, The detailed steps for the language interaction module to receive the wake-up command and activate the system are as follows: S1.1: In low-power standby mode, the voice interaction module continuously listens for wake-up words using an industrial-grade microphone. When a suspected voice signal is detected, the industrial-grade wake-up unit extracts MFCC features and voiceprint features, as well as other acoustic features, through an embedded DSP. It also performs real-time matching with each wake-up template stored locally; S1.2: If the matching degree between the received acoustic features and any wake-up template exceeds the set threshold, the language interaction module confirms that the command source is valid and switches the system from standby mode to working mode. At the same time, the system outputs a voice response to prompt the maintenance personnel to continue to issue operation commands.

3. The multimodal AI edge terminal integrating language interaction, visual analysis, and logical reasoning according to claim 2, characterized in that, The specific steps of the language interaction module in recognizing the collected speech signals, generating speech-to-text, and converting the recognition results into standard speech are as follows: S2.1: The microphone collects the voice commands of the staff, performs analog-to-digital conversion through an industrial-grade sound card to generate corresponding digital signals, performs environmental noise filtering through end-side algorithms and automatically adjusts the gain, then divides the voice signal into frames with a frame length of 20-30ms, and extracts the feature vector of each voice signal frame by frame. S2.2: Input the extracted feature vectors into the ASR model and perform decoding analysis through the ASR model. At the same time, convert the decoding results into corresponding pinyin or word sequences. Combine the language model of the power operation and maintenance field to perform context correction on the generated pinyin or word sequences and generate accurate text results. S2.3: The corrected text results are processed by word segmentation, noise reduction and synonym normalization. Then, the text type is determined by the intent classification model, and the text results are structured into standard operation units. The processing results are then input into the joint logic reasoning module, and the text results fed back by the joint logic reasoning module are received. The text results fed back by the joint logic reasoning module are then processed by sentence normalization and pronunciation optimization. S2.4: Call the power text-to-speech unit to convert the optimized text result into standard speech, which is then played through the speaker. If the user continues to ask questions, the system maintains the conversation state and waits for new speech input.

4. The multimodal AI edge terminal integrating language interaction, visual analysis, and logical reasoning according to claim 1, characterized in that, The specific steps for the visual analysis module to train the power scene visual algorithm are as follows: S3.1: Real-time high-definition images are collected from various environments in real power distribution rooms and substations. At the same time, the shooting angle covers multiple angles, including front, side and top views, and the lighting conditions cover daytime, nighttime and low-light environments. The quality of each collected image data is reviewed, and the image data that meets the preset conditions is retained. Then, the annotation tool is used to annotate the preset targets in each image data with rectangular boxes to form a label file that conforms to the YOLO format. S3.2: Divide the data into training set, validation set and test set according to the preset ratio, then adjust the pixel size of each image data in each dataset to meet the input requirements of YOLOv8 network, then perform image enhancement on each image data, then normalize the pixel values ​​of each enhanced image data, and adjust the label coordinates simultaneously. S3.3: Based on the PyTorch framework, the training set data is divided into multiple training sample sets, which are then sequentially input into the YOLOv8 network. A multi-model parallel training method is adopted, and the safety helmet recognition model, water level recognition model, and access control recognition model are trained through the YOLOv8 network respectively. After each round of training, the loss value between the output of each model and the corresponding real data is calculated, and the parameters of each model are adjusted through the Adam optimization algorithm. S3.4: After multiple rounds of training, when the change in the loss value of each model converges to the preset range, input the validation set and test set into each model and evaluate the performance of each model. If the helmet recognition accuracy is ≥1%, then retrain the helmet recognition model. If the water level recognition model has a recognition error ≥5cm, then retrain the water level recognition model. If the access control status recognition model has a recognition error ≥1%, then retrain the access control status recognition model. S3.5: Repeatedly train, validate, and test each model until the preset conditions are met. Export the three trained models in ONNX format, then convert them to an edge-compatible format. Deploy each model to the NPU of the edge terminal main chip for real-time inference and output the visual features generated by inference in a structured form.

5. The multimodal AI edge terminal integrating language interaction, visual analysis, and logical reasoning according to claim 1, characterized in that, The specific steps of the joint logic reasoning module in performing structured processing on multimodal data are as follows: S4.1: Obtain structured speech text after ASR and semantic parsing from the language interaction module, receive the recognized visual features from the visual analysis module, including the detection target category, state attribute, confidence level and spatial location information, obtain the device historical status, alarm record and running timestamp information from the terminal database or log, add modal identifiers to each type of data, and use regular expression parsing and keyword mapping. S4.2: Extract the required semantic fields from the speech text. If a certain modality is missing some fields, it will be automatically filled with the current system time. If the speech is missing a device ID, it will be filled according to semantic context reasoning or default configuration. Then, the synonym fields of different modalities will be uniformly named and quantified. S4.3: Perform one-hot encoding on the operation type and target object in the speech text, convert the parameter or numerical field into floating-point number, unify the unit information in the speech text, encode the target category and state label in the visual features into integer or Boolean variables, standardize the spatial location information to the range of [0, 1], map the historical state value in the local state into binary variables, and convert the timestamp into relative time; S4.4: Align the encoded modal data according to the acquisition time, generate modal weight vectors based on modal confidence, and then encapsulate each modal data into a data matrix in a unified format based on the generated modal weight vectors, where each row corresponds to a modal record. Then, store the normalized data matrix in the local cache queue and upload it synchronously to the cloud logic inference interface.

6. The multimodal AI edge terminal integrating language interaction, visual analysis, and logical reasoning according to claim 5, characterized in that, The specific steps of the joint logic reasoning module in performing multi-information joint analysis are as follows: S5.1: The joint logic reasoning module uploads the generated data matrix containing voice text, visual features and local state data to the cloud. Then, the industrial multimodal large model deployed in the cloud calculates the correlation matrix between modalities through multi-head self-attention and cross-attention. S5.2: Based on the established relevance matrix, the voice intent is used as the query condition to retrieve the evidence segment with the highest relevance between visual features and local state data, and the corresponding evidence support score is calculated. The symbolic check is performed on the preset high-priority rules, and the numerical value and state are mapped to the rule premise. If the rule condition is met, a symbolic inference output is formed and the confidence of the conclusion is improved. S5.3: The results from statistical inference and symbolic rules are fused according to preset weights to generate the final conclusion confidence and recommended actions. Then, local state data is used to backtrack the historical records corresponding to the current vision or voice to identify similar events or repeated alarms within a specified time period. S5.4: Construct candidate causal chains based on retrieved evidence fragments and evaluate the strength of the links. When visual conclusions conflict with historical records, check visual confidence, historical credibility, and time difference, and use rules to determine priority or initiate a resampling request. Then, encapsulate the model conclusions, supporting evidence, confidence, and recommended actions into structured output, and automatically generate explanatory text to explain the key points of the reasoning process in natural language.

Images