An intelligent gateway

Abstract

The present invention relates to an intelligent gateway, and belongs to the field of intelligent gateway design. The present invention integrates sensor data of the same monitored object into a video frame, obtains a recognition result of the video frame through a corresponding pre-trained model, and then displays the recognition result and the video frame integrated with the sensor data on a visual interface of the platform. The present invention integrates sensor data with video stream data and displays the recognition result of the video data, effectively integrates multi-source information, and reminds operation and maintenance personnel in a visual form.

Description

Technical Field

[0001] The invention relates to an intelligent gateway and belongs to the field of intelligent gateway design. Background Art

[0002] With the rapid development of information technology, the Internet of Things technology has been widely used in various fields, especially in the fields of smart stations, smart homes, and smart buildings. However, higher requirements are put forward for the acquisition and processing of information in these fields. As an important part of the Internet of Things, the smart gateway can realize the functions of information collection, information input, information output, centralized control, remote control, and linkage control of various sensors, network devices, cameras, and host devices in the local area network.

[0003] However, traditional intelligent gateways only process the data received from each device separately and then forward the processed data. When dealing with complex energy scenarios and multi-source data, they do not effectively integrate multi-source data, resulting in a low level of intelligence. Summary of the invention

[0004] The purpose of the present invention is to provide an intelligent gateway to solve the problem of low intelligence level caused by the failure of the intelligent gateway to fuse multi-source information.

[0005] To achieve the above object, the solution of the present invention includes: An intelligent gateway of the present invention comprises a multi-source perception module, a multi-modal module, a data processing and fusion module and a display module; the multi-source perception module comprises a database, the database is used to store sensor data and video stream data of the same monitoring object; the multi-modal module comprises a pre-trained model, the pre-trained model is used to identify the video stream data and obtain corresponding video frame recognition results; the data processing and fusion module integrates the corresponding sensor data into the video frame; the display module displays the video frame integrated with the sensor data and the video frame recognition results in a visual interface; the training set of the pre-trained model is pre-collected video frame data that has been labeled with corresponding categories.

[0006] Furthermore, the data processing and fusion module also determines whether the video frame recognition result and / or the sensor data are in an abnormal state. If so, the data processing and fusion module sends the video frame recognition result integrated with the sensor data to the display module in the form of an alarm.

[0007] Furthermore, the pre-trained model is trained based on a convolutional neural network.

[0008] Furthermore, the multimodal module stores the video frame recognition results in a database and then transmits them to the data processing and fusion module.

[0009] Furthermore, the visual interface is built using the VUE framework.

[0010] Furthermore, the data processing and fusion module also manages the sensors through corresponding configuration files.

[0011] Furthermore, the data processing and fusion module also pre-processes the video frame recognition results and the sensor data, and then fuses the pre-processed sensor data and the video frame recognition results. The pre-processing includes cleaning and denoising.

[0012] Furthermore, the database is a MYSQL database.

[0013] The beneficial effects of the present invention are as follows: the present invention is a pioneering invention, which integrates the sensor data of the same monitored object into the video frame, obtains the recognition result of the video frame through the corresponding pre-trained model, and then displays the recognition result and the video frame with the sensor data on the visual interface of the platform. The present invention integrates the sensor data with the video stream data and displays the recognition result of the video data, effectively integrates the multi-source information, and reminds the operation and maintenance personnel in a visual form. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a system architecture diagram of the intelligent gateway of the present invention; Figure 2 It is a data processing flow diagram of the network model of the present invention; Figure 3 It is a schematic diagram of the overall data processing flow of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0016] The idea of ​​the present invention is to use a modular design to implement the corresponding functions of the gateway, use a network model to identify video stream data, and fuse sensor data with video stream data, and then display the video stream data fused with sensor data and the recognition results on a visual interface.

[0017] Gateway Example: This embodiment provides an intelligent gateway. The intelligent gateway of the present invention adopts an open architecture design concept of software and hardware decoupling, that is, the functions to be implemented by the intelligent gateway are modularly designed. When a module or subsystem fails, it will not affect the operation of other modules or subsystems. Pins are used to connect different hardware modules. The system architecture of the intelligent gateway of the present invention is as follows: Figure 1As shown, it includes a multi-source perception module, a multi-modal intelligent computing module (multi-modal module), a data processing and fusion technology module (data processing and fusion module) and an interface display module (display module).

[0018] The multi-source perception module includes a MYSQL database and a model file. The MYSQL database is used to store the data information of different devices in different tables and divide them in detail to make the information of each device more organized. The model file is a physical model file established for different devices. The corresponding device data can be received by the gateway through the defined physical model file. Among them, the sensor data includes data collectors for cold, hot, electric, other, and warm, and the video stream data includes cameras that support protocols such as GB / T28181, ONVIF, RTSP, and RTMP. The objects collected by the data collector and the camera should be the same target or the same target area. Among them, the database can also use other types of databases according to needs, such as QuestDB, MongoDB, etc.

[0019] The multimodal intelligent computing module includes database reading and writing, multimodal model, recognition result storage and information update functions. Database reading and writing is to read the data stored in the MYSQL database in the multi-source perception module, and then pass the data into the multimodal model after reading. The multimodal model is dedicated to image recognition. In order to realize the training of high-performance models on small data sets, the present invention adopts a pre-trained model based on a convolutional neural network as the multimodal model in the intelligent gateway. The pre-trained model is a neural network model that has been pre-trained on a large data set. Through transfer learning, the pre-trained model is adapted to specific new tasks, which significantly reduces training time and computing resources. The training set of the pre-trained model of the present invention is the video stream data pre-collected and labeled with the corresponding category of the corresponding device, and the parameters such as the data path, the number of working processes, the size of the input image, the training cycle and the batch size are set for training, thereby obtaining a trained pre-trained model. The trained pre-trained model is then deployed to the intelligent gateway, and the process of using the model is as follows: Figure 2 As shown, intelligent analysis and processing of the input video stream data is achieved, the video frame recognition result of the video stream data is output, and the video frame recognition result is stored. The data is updated in the information update and then passed to the next module.

[0020] The data processing and fusion technology module includes input-output interaction, parameter management, sensor management, abnormal information alarm, model management and user management functions. Input-output interaction is used to receive data transmitted by the multimodal intelligent computing module, that is, to receive the corresponding video frames, read the sensor data in the database and integrate the sensor data into the received video frames, and then send the video frames and video frame recognition results after the fusion of sensor data to the cloud platform, and display the data in the visual interface (interface display module) of the cloud platform. Parameter management is mainly to adjust the parameters of the algorithm of the gateway's built-in model, and different algorithm parameters can be selected for different scenarios; sensor management manages the sensor devices connected to the gateway, and performs parameter configuration, on-off management and other operations through the corresponding configuration files; abnormal information alarm is to judge whether the received video frame recognition results and / or sensor data are in an abnormal state. If they are abnormal, the video frame recognition results are sent to the cloud platform in the form of an alarm. In order to achieve this function, the corresponding recognition results and / or sensor data of the video stream data can be classified. For example, the video stream data with the recognition result of fire is in an abnormal state, and the temperature data from the temperature sensor is in an abnormal state after exceeding a certain threshold; model management can manage and configure the built-in model of the gateway, that is, choose which algorithm model to use. In order to achieve this function, the gateway has built-in multiple trained algorithm models, but the built-in algorithm models must be able to realize image recognition, and only one of them is used at the same time; user management defines the operation permissions of the gateway to set different usage permissions for different users. In order to remove interfering data, the video frame recognition results and sensor data are preprocessed before data fusion, and then fused. The preprocessing includes data cleaning and data denoising. Data cleaning uses methods such as whole case deletion, filling missing values, replacing error values, and deduplication to achieve the purpose of cleaning missing values, outliers, and duplicate values. Data denoising uses statistical methods to improve the quality and readability of data, thereby supporting more accurate data analysis and application.

[0021] The interface display module is a visual interface built through the VUE framework. It is developed based on standard HTML and CSS. It can customize the interface required by users and render the visual interface.

[0022] In summary, the overall processing process of the intelligent gateway of the present invention for multi-source information is as follows: Figure 3 shown.

[0023] The effect of the present invention is described below with simple examples: When conducting intelligent inspections on a power generation equipment in a smart building, for example, to check whether a fire has occurred. First, a camera that can record the power generation equipment is installed near the power generation equipment, and a temperature sensor is installed inside the power generation equipment. The camera sends the collected video stream data and the temperature data collected by the temperature sensor to the database in the multi-source perception module; the multimodal intelligent computing module reads the video stream data in the database, inputs the video stream data into the pre-trained model that has been trained, and obtains the corresponding recognition results. The training set used for this situation is the video stream marked with the fire situation and the video stream without the fire situation. The obtained video frame recognition results are stored in the database and the information is updated, and then passed to the data processing and fusion technology module. Now assume that a fire has just occurred around the power generation equipment, but the power generation equipment has not been burned. Therefore, the video frame recognition result is that a fire has occurred. At the same time, temperature data from the temperature sensor is received. The temperature data is the temperature of the power generation equipment itself when there is no fire. The temperature data is then integrated into the video frame where the fire has occurred. Since the video frame with the recognition result of a fire is an abnormal state at this time, the video frame recognition result is sent to the cloud platform in the form of an alarm (for example, a flashing red triangle), and the video frame with the temperature data is sent to the cloud platform. The cloud platform's visual interface displays the alarm information and the video frame with the temperature data to remind the staff to deal with it in time.

[0024] Taking smart buildings as an example, by deploying smart gateways on smart buildings, real-time monitoring and early warning of the operating status of various sensor devices in the building can be achieved. When abnormal conditions occur in sensor devices, the smart gateway can detect and issue an alarm in time to remind staff to handle them. At the same time, the smart gateway can also perform intelligent analysis and processing of collected data, provide optimization suggestions and decision support, and help enterprises improve security efficiency and reduce operating costs.

[0025] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An intelligent gateway, characterized in that: The intelligent gateway includes a multi-source perception module, a multi-modal module, a data processing and fusion module and a display module; the multi-source perception module includes a database, which is used to store sensor data and video stream data of the same monitored object; the multi-modal module includes a pre-trained model, which is used to identify video stream data and obtain corresponding video frame recognition results; the data processing and fusion module integrates the corresponding sensor data into the video frame; the display module displays the video frame integrated with the sensor data and the video frame recognition results in a visual interface; the training set of the pre-trained model is pre-collected video frame data that has been labeled with the corresponding category.

2. The intelligent gateway according to claim 1, characterized in that: The data processing and fusion module also determines whether the video frame recognition result and / or the sensor data are in an abnormal state. If so, the data processing and fusion module sends the video frame recognition result integrated with the sensor data to the display module in the form of an alarm.

3. The intelligent gateway according to claim 1, characterized in that: The pre-trained model is trained based on a convolutional neural network.

4. The intelligent gateway according to claim 1, characterized in that: The multimodal module stores the video frame recognition results in a database and then transmits them to the data processing and fusion module.

5. The intelligent gateway according to claim 1, characterized in that: The visual interface is constructed using the VUE framework.

6. The intelligent gateway according to claim 1, characterized in that: The data processing and fusion module also manages the sensors through corresponding configuration files.

7. The intelligent gateway according to claim 1, characterized in that: The data processing and fusion module also preprocesses the video frame recognition results and the sensor data, and then fuses the preprocessed sensor data and the video frame recognition results, wherein the preprocessing includes cleaning and denoising.

8. The intelligent gateway according to claim 1, characterized in that: The database is a MYSQL database.

Images