Indoor intelligent monitoring system and method

Abstract

The present application provides an indoor intelligent monitoring system and method, including: a data acquisition terminal for collecting and uploading terminal data, wherein the terminal data includes at least image data and sensor data; a 5G network communication system for sending the terminal data to a human-computer interaction system; and a human-computer interaction system for completing the real-time display of terminal data through a visual interface. The data acquisition terminal includes a main control chip, and a 5G module, a camera module, a light intensity sensor, a smoke gas sensor, and a buzzer respectively connected to the main control chip. The 5G module and the camera module are connected to the main control chip via a USB interface, and the light intensity sensor and the smoke gas sensor are connected to the main control chip via an external DuPont line. The present application has good system stability and image data transmission performance, can uniformly and intelligently monitor smoke and combustible gases, manage electricity consumption by monitoring lights, and effectively and timely monitor indoor conditions through cameras.

Description

Technical Field

[0001] The present application relates to the field of intelligent monitoring technology, and in particular to an indoor intelligent monitoring system and method. Background Art

[0002] Indoor intelligent monitoring refers to networking and intelligentizing multiple objects within a certain range using sensor technology, communication technology, sensor network technology, and graphic display technology, so as to achieve specific functions such as remote real-time monitoring and timely warning on mobile platforms or computer platforms. The current indoor monitoring faces the following requirements:

[0003] (1) Various high-concentration smoke gases or flammable gases are often volatilized during indoor experiments. Reasonable detection of them can not only help improve the indoor environment, but also prevent laboratory hidden dangers. Smoke gas sensors can be used to obtain the current indoor gas (such as flammable gas) concentration status.

[0004] (2) When there is no one on duty, if the laboratory lighting is still in working state, it will cause a waste of resources. The working state of the light can be judged by the light value during a specific time period (such as the early morning stage). The current light value in the room can be obtained through the light intensity sensor.

[0005] (3) The flow of personnel in the laboratory is complex, and it is easy to break in or lose items. It is also difficult to remotely grasp the real-time progress of the equipment in operation. Remote images are used to capture the working status of indoor personnel or equipment in real time, so that managers can accurately grasp the indoor conditions at any time. The image data can be obtained through the camera module.

[0006] The current indoor monitoring system can often only meet one of the above monitoring requirements and cannot be integrated and intelligently complete the monitoring task in one system. Summary of the invention

[0007] In view of this, the purpose of this application is to propose an indoor intelligent monitoring system and method, which can solve the existing problems in a targeted manner.

[0008] Based on the above objectives, according to the first aspect of the present invention, the present application proposes an indoor intelligent monitoring system, comprising:

[0009] A data acquisition terminal, used to collect and upload terminal data, wherein the terminal data includes at least image data and sensor data;

[0010] 5G network communication system, used to send the terminal data to the human-computer interaction system;

[0011] The human-computer interaction system is used to display the terminal data in real time through a visual interface.

[0012] Furthermore, the data acquisition terminal includes a main control chip, and a 5G communication module, a camera module, a light intensity sensor, a smoke sensor, and a buzzer respectively connected to the main control chip.

[0013] Furthermore, the 5G communication module and the camera module are connected to the main control chip via a USB interface, and the light intensity sensor and the smoke gas sensor are connected to the main control chip via an external DuPont cable.

[0014] Furthermore, the image data collected by the data collection terminal is transmitted to the human-computer interaction system via UDP, and the sensor data is transmitted to the human-computer interaction system via TCP.

[0015] Based on the above purpose, according to the second aspect of the present invention, the present application proposes an indoor intelligent monitoring method according to the system of the first aspect, comprising:

[0016] Initialize the mutex;

[0017] Determine whether the creation of the acquisition and camera running threads is successful. If not, the operation fails, all threads exit, and the applied resources are released. If yes, the main thread is blocked and the data acquisition thread and camera running thread are entered;

[0018] The data collection thread is locked while running, and then the 5G communication module starts dialing;

[0019] After the 5G communication module dials successfully, it is unlocked and resources are released;

[0020] The sensor collects data, and the camera running thread starts running after obtaining resources.

[0021] Furthermore, the sensor sends a frame of sensor data every 30 seconds.

[0022] Furthermore, a timer event is enabled, by which the number of received image frames is counted per second and displayed in the image control of the human-computer interaction system.

[0023] Furthermore, the data acquisition information of the human-computer interaction system consists of basic data information and sensor data information. The basic data information includes the manufacturer of the system, server IP, data acquisition port, image data transmission port, reset times, upload interval, device type, signal strength, software version, registered network type, and alarm times; the sensor data information includes chip temperature, light intensity, buzzer status, and combustible gas information.

[0024] Furthermore, the human-computer interaction system includes a system alarm service, and the alarm modes include email alarm and short message alarm.

[0025] Furthermore, the email alarm is based on the email sending class. After specifying the email sending protocol and the recipient, setting the email sending content and completing the corresponding format transcoding, it is automatically sent when an alarm occurs; the short message alarm is based on the short message service of the 5G communication module. When the alarm condition is triggered, the human-computer interaction system software sends back a frame of data carrying the alarm information. After receiving the frame data, the data acquisition terminal extracts the valid information, and then the 5G communication module assembles the valid information into a new AT command statement and sends it out.

[0026] In general, the advantages of this application and the experience it brings to users are: this application has good system stability and image data transmission performance, can uniformly and intelligently monitor smoke and flammable gases, manage electricity consumption by monitoring lights, and effectively and timely monitor indoor conditions through cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0028] Figure 1 A schematic diagram showing the system architecture principle of the present application is shown.

[0029] Figure 2 The following is a flowchart showing the execution of the main program of the system terminal software according to an embodiment of the present application.

[0030] Figure 3 A camera thread execution flow chart according to an embodiment of the present application is shown.

[0031] Figure 4 A flow chart of sensor data collection according to an embodiment of the present application is shown.

[0032] Figure 5 A schematic diagram of the data display interface of the real-time data monitoring module is shown.

[0033] Figure 6 The figure shows the historical data query and management interface diagram of the human-computer interaction system of the present application.

[0034] Figure 7 A statistical diagram of the operating status of the system of this application is shown.

[0035] Figure 8 A schematic diagram of the statistics of the frame rate of received images based on different communication modes of the present application is shown. DETAILED DESCRIPTION

[0036] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0037] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] Figure 1 A schematic diagram of the system architecture principle of the present application is shown. In the embodiment of the present application, it includes: a data acquisition terminal, a 5G network communication system and a human-computer interaction system. Among them, the data acquisition terminal mainly completes the collection and upload of terminal data; the 5G network communication system is responsible for the transmission of remote data, and the network operator is responsible for the construction and maintenance of this part of the hardware and software platform; the human-computer interaction system uses a visual interface to complete the real-time display of terminal data, which is mainly achieved through the 5G network communication system. The two-way communication process with the data acquisition terminal, that is, the image data collected by the data acquisition terminal is transmitted to the human-computer interaction system through UDP, and the sensor or other data is uploaded using TCP, and the data transmission process is also the same.

[0039] Among them, the data acquisition terminal includes a main control chip, and a 5G module, a camera module, a light intensity sensor, a smoke sensor, and a buzzer respectively connected to the main control chip.

[0040] STM32MP157 and FM150-AE are selected as the main control chip and 5G communication module of the indoor intelligent monitoring system terminal. Accordingly, in order to meet the sensor data and image data collection requirements of the system, corresponding camera modules and monitoring sensors need to be added. The USB camera model MY-CAM002U launched by Mir Electronics is selected for the image input of the system. This camera module is designed based on the OV9712 chip and supports direct connection with the main control chip via USB. After completing the corresponding configuration, the image data collected by the module can be obtained. The monitoring sensors used are mainly: light intensity sensor, smoke gas sensor and buzzer. Table 1 shows the hardware used in this system and its characteristics.

[0041] Table 1 Main hardware modules used in system terminals

[0042]

[0043] In order to ensure the normal use of each module of the terminal hardware, it is necessary to test it. This application takes the light intensity sensor test as an example. The test method is: write a test program that collects data every 1 second and collects AD values ​​through the serial port output, and then simulates different light intensities by blocking the sensor module. It is found that the AD value of its output decreases with the increase of light intensity. This shows that the sensor module can meet the needs of light intensity acquisition of this system, and the test of other modules can also be carried out in the same way as this test process. Finally, under the premise of ensuring that each hardware module functions normally, the terminal hardware used in this system is connected in combination with the contents described in Table 1. Among them, the MPU (Microprocessor Unit) containing the main control chip part expands multiple USB interfaces through a hub for the camera module and the 5G communication module to connect to it respectively; and various monitoring sensors are connected to the MPU through external DuPont lines on the basis of multiplexing pins, thereby completing the information interaction process of the entire system. At this point, the hardware platform of this system has been built.

[0044] Compared with the existing common engineering framework for 5G development (such as 5GDC-EDP software, which refers to the name of the general embedded development platform used in this application, 5G Data Communication Embedded Development Platform (5G Data Communication Embedded Development Platform, 5GDC-EDP), 5GDC-EDP itself is a graphical software written in C# language and needs to run on Windows system), the development of the terminal software part of this system only needs to add and modify some content. The specific content that needs to be added and modified is shown in Table 2.

[0045] Table 2 Comparison between the terminal software of this system and the common engineering framework for 5G development

[0046]

[0047]

[0048] It can be seen from the contents in the table that, thanks to the component-based software engineering framework design, the terminal software development of this system only needs to add a camera application component to the 5G development common engineering framework and modify some file contents. The camera component is mainly designed for the operation of the camera module, and each sensor module is based on the corresponding pin information defined in the device tree file (i.e., the stm32mp157.dts file). After completing the device tree update operation, the data can be directly read and sent through the encapsulated basic component function. The main basic components used in this system are gpio and adc components. The application component design method of this application is a standard and unified external interface function for camera module encapsulation. Some of its calling interfaces and instructions are shown in Table 3.

[0049] Table 3 Description of some interface functions of camera component

[0050]

[0051] Software Process Design

[0052] In addition to adding a new camera component, the development of the terminal software of this system also requires the modification of some files. The modifications to the macro definition (.h) file are generally: including the newly added component name, adding the corresponding character device driver, and the modification of the main.c file often requires the design of the entire system software process. What needs to be noted in the design of the terminal software process of this system is that the image data collected by the camera can only be sent under the premise that the MPU is connected to the Internet, that is, before the image is captured, the 5G communication module must first complete the dialing work, and the image data collection and transmission itself requires an infinite loop operation to maintain, which involves the issue of module working sequence and resource sharing. In this regard, this application adopts a multi-threaded approach to design the terminal main program, and uses mutex locks to orderly plan the startup sequence of each module to ensure the stable operation of the entire program. The execution process of the terminal software main program of this system is as follows Figure 2 As shown, it includes: initializing the mutex lock; judging whether the creation of the acquisition and camera running threads is successful, if not, the operation fails, all threads exit, and the applied resources are released; if yes, the main thread is blocked, and the data acquisition thread and the camera running thread are entered; the data acquisition thread is locked while running, and then the 5G communication module starts dialing; after the 5G communication module dials successfully, it is unlocked and resources are released; the sensor performs data acquisition, and the camera running thread starts running after obtaining resources.

[0053] Figure 3 and Figure 4 The terminal camera thread execution process and sensor data collection flow chart of this system are shown respectively.

[0054] After the main program starts executing, the mutex is initialized through the pthread_mutex_init function. After completing the relevant property settings, the pthread_create function can be used to create the data collection thread collectDataThread and the camera running thread cameraThread. collectDataThread is responsible for the 5G module dialing, the timing collection and transmission of sensor data, and cameraThread is responsible for the initialization of the camera module, the collection and transmission of image data. The two run in an orderly manner under the guidance of the mutex mechanism. The internal hardware modules complete the system's specific functions such as dial-up networking, alarm, sensor and image data collection based on the call of the corresponding component functions.

[0055] Human-computer interaction system design

[0056] The human-computer interaction system of this system is designed by using the cloud server listening software framework. The framework has realized the single-step sending and receiving of basic data. On this basis, a multi-threading mechanism is added to complete the sending and receiving of image and sensor data, system alarm and other functions. As the functional support software of the indoor intelligent monitoring system, the human-computer interaction system mainly provides remote image display, data acquisition information display and system alarm and other functions. Its software interface is as follows Figure 5 As shown. Remote image display mainly displays the image data collected by the terminal camera device in real time in this module. Corresponding to the terminal software, the human-computer interaction system software also opens up two threads for real-time reception of image data and other data. Among them, image data is transmitted based on UDP, the port used is 64000, and thread 1 is responsible for listening; other data is transmitted based on TCP, the port used is 28806, and thread 2 is responsible for listening. The number of frames per second (Frames Per Second, FPS) is used as a standard for judging the speed of image transmission. The implementation method is: enable a timer event, which counts the number of image frames received per second and displays it in the image control PictureBox. This application will use this feature to test the image transmission performance in the subsequent part. The data acquisition information consists of basic data information and sensor data information. The basic data information includes the manufacturer of this system, server IP, data acquisition port, image data transmission port, reset times, upload interval, device type, signal strength, software version, registered network type, alarm times, etc.; while the sensor data information includes chip temperature, light intensity, buzzer status and combustible gas information.

[0057] Correspondingly, in order to cooperate with the early warning measures of this system, the human-computer interaction system also provides a system alarm service, and the alarm methods are mainly email and short message. Among them, the email alarm is based on the email sending class MailMessage. By specifying the email sending protocol and the recipient, setting the email sending content and completing the corresponding format transcoding, it can be automatically sent when the alarm is triggered. The short message alarm is based on the short messaging service (SMS) service of the 5G communication module. When the alarm condition is triggered, the human-computer interaction system software will send back a frame of data carrying the alarm information. After receiving the frame data, the terminal will complete the extraction of valid information, and then the 5G communication module will assemble it into a new AT command statement and send it out.

[0058] Take the 5G module sending a real short message to the mobile phone as an example under the premise of completing the enabling of the Protocol Data Unit (PDU) mode. The content of the message is: the light is not turned off. Then the key command information to be sent by the 5G module is: 0891683108502105F011000D915161425291F000080008706F672A5173. The corresponding meanings of each part of the information are shown in Table 4.

[0059] Table 4 Send short message command format in PDU mode

[0060]

[0061] System Testing

[0062] The following will be based on the indoor intelligent monitoring system built earlier. Under the premise that all hardware modules of the terminal are operating normally and the 5G base station signal is good, the system performance test will be carried out. The test includes system stability test and image data transmission performance test.

[0063] 1. System stability test

[0064] In this experiment, the frequency of terminal data collection and upload of the indoor intelligent monitoring system is set to transmit one frame every 30 seconds. After the human-computer interaction system software detects the terminal data, it will display it in real time and store it in the database. The software interface for historical data query and management is as follows: Figure 6 shown.

[0065] The test scenario is an embedded laboratory welding room. The indoor intelligent monitoring system terminal is deployed somewhere in the welding room. The human-computer interaction system is used to record the overall operation of the system in real time. The operation information is as follows: Figure 7Among them, "Monitoring start time" indicates the time when the software is first opened; "Latest communication time" indicates the time when the software receives the latest frame of data; "Normal communication duration" indicates the total duration of the software receiving terminal data normally, such as: 17.06:21:34 means that it has run for 17 days, 6 hours, 21 minutes and 34 seconds. The specific data information received can be obtained through Figure 6 The historical data query and management page is shown. It can be seen that the indoor intelligent monitoring system can run stably for about 17 days from the beginning of deployment, which fully proves that the system has good stability.

[0066] 2 Image transmission performance test

[0067] The FPS value can reflect the changes in the number of image frames received per second by the indoor intelligent monitoring system. To verify the performance of the image transmission of this system, the terminal camera module is deployed somewhere in the laboratory and the average FPS changes within 5 days are counted under the premise that the image acquisition rate is 30 frames per second and the image acquisition resolution is 320×240. The statistical frequency is 12 hours / time. At the same time, based on the same conditions, the average FPS values ​​of this system under the network supply modes such as 4G communication module L610, wired broadband and local area network are compared. Taking a certain day as the starting time, the average FPS value of the human-computer interaction system software is counted every 12 hours, and the following can be obtained: Figure 8 The average frame rate change statistics are shown.

[0068] Depend on Figure 8 It can be seen that when the terminal captures images with a resolution of 320×240 and a capture rate of 30 frames per second, compared with the average FPS of the 4G network, the average image frame rate reception under the 5G network can basically be maintained at 24 to 27 frames per second, and the change in the average FPS value is as stable as that of the local area network and wired network. This proves that the system has the performance advantage of using 5G communication technology to transmit images, has good real-time performance, and meets the expected effects of the system.

[0069] summary

[0070] This application mainly starts with the system's demand analysis and input-output analysis, proposes the corresponding system overall framework, and then explains the system construction process from three aspects: terminal hardware platform construction, terminal software design, and human-computer interaction system design. Finally, the corresponding system test is given to verify the feasibility of the system.

[0071] It should be noted that:

[0072] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems may also be used together with the teachings based thereon. It is apparent from the above description that the structure required for constructing such systems is not intended for any particular programming language. In addition, the present application is not intended for any particular programming language either. It should be understood that the content of the present application described herein may be implemented using various programming languages, and the description of the particular language above is intended to disclose the best mode of implementation of the present application.

[0073] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0074] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0075] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0076] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0077] The various component embodiments of the present application can be implemented in hardware, or implemented in software modules running on one or more processors, or implemented in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in the creation system of the virtual machine according to the embodiment of the present application. The application can also be implemented as a device or system program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0078] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of appropriately programmed computers. In a unit claim that lists several systems, several of these systems may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0079] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An indoor intelligent monitoring method according to an indoor intelligent monitoring system, the system include: A data acquisition terminal is used to collect and upload terminal data, wherein the terminal data includes at least image data and sensor data; a 5G network communication system is used to send the terminal data to a human-computer interaction system; and the human-computer interaction system is used to complete the real-time display of the terminal data through a visual interface, and is characterized in that it includes: Initialize the mutex; Determine whether the creation of the acquisition and camera running threads is successful. If not, the operation fails, all threads exit, and the applied resources are released. If yes, the main thread is blocked and the data acquisition thread and camera running thread are entered; The data collection thread is locked while running, and then the 5G communication module starts dialing; After the 5G communication module dials successfully, it is unlocked and resources are released; The sensor collects data, and the camera running thread starts running after obtaining resources.

2. The method according to claim 1, It is characterized in that The data acquisition terminal includes a main control chip, and a 5G communication module, a camera module, a light intensity sensor, a smoke and gas sensor, and a buzzer respectively connected to the main control chip.

3. The method according to claim 2, It is characterized in that The 5G communication module and the camera module are connected to the main control chip via a USB interface, and the light intensity sensor and the smoke gas sensor are connected to the main control chip via an external DuPont cable.

4. The method according to claim 3, It is characterized in that The image data collected by the data collection terminal is transmitted to the human-computer interaction system via UDP, and the sensor data is transmitted to the human-computer interaction system via TCP.

5. The method according to claim 1, It is characterized in that The sensor sends one frame of sensor data every 30 seconds.

6. The method according to claim 1 or 5, It is characterized in that A timer event is enabled, which counts the number of received image frames per second and displays them in the image control of the human-computer interaction system.

7. The method according to claim 1 or 5, It is characterized in that The data acquisition information of the human-computer interaction system consists of basic data information and sensor data information. The basic data information includes the manufacturer of the system, server IP, data acquisition port, image data transmission port, reset times, upload interval, device type, signal strength, software version, registered network type, and alarm times; the sensor data information includes chip temperature, light intensity, buzzer status, and combustible gas information.

8. The method according to claim 1 or 5, It is characterized in that The human-computer interaction system includes system alarm service, and the alarm modes include email alarm and short message alarm.

9. The method according to claim 8, It is characterized in that The email alarm is based on the email sending class. It specifies the email sending protocol and the recipient, sets the email sending content and completes the corresponding format transcoding, and then automatically sends it when an alarm occurs. The short message alarm uses the short message service that comes with the 5G communication module. When the alarm condition is triggered, the human-computer interaction system software sends back a frame of data carrying the alarm information. After receiving the frame data, the data acquisition terminal extracts the valid information, and then the 5G communication module assembles the valid information into a new AT command statement and sends it out.

Images