Abnormal condition monitoring method and device based on multiple sensors and computer readable medium

Abstract

The abnormal condition monitoring method based on a plurality of sensors includes: processing a sensing signal and outputting sensing data obtained by converting the sensing signal into a digital signal, the sensing signal being received from a plurality of sensors configured to detect different objects; generating input data, the input data being obtained by variably adjusting the size of the sensing data, the input data being used for being input to a neural network model; the neural network model is used for executing operation on the input data in a simulation domain, operation data are output, and the operation data are obtained by converting an operation result into digital data; and outputting a monitoring result, wherein the monitoring result is obtained by classifying abnormal conditions according to the operation data.

Description

Technical Field

[0001] The present disclosure relates to a method, a device and a computer-readable medium for monitoring abnormal conditions based on multiple sensors. Background Art

[0002] With the advancement of electronic and communication technologies, various Internet of Things (IoT) sensor devices have been applied in many fields. These IoT sensor devices are installed in various environments and locations, such as smart homes, factories, public places, and roads, to sense various external environments.

[0003] IoT sensor devices sense the external environment through built-in sensors, processors, and communication chipsets, and process the sensed data or transmit the sensed data to the outside through communication chipsets.

[0004] IoT systems can receive and process sensing data from multiple IoT sensors. In this case, when information is processed by connecting one sensor to one circuit, in order to obtain one piece of information through the information of several sensors, an increase in the size of the device is inevitable, and thus the design cost may increase. Therefore, it is necessary to study a method of processing signals so that multiple sensors can be processed in one circuit.

[0005] The above-mentioned background technology is the technical information obtained by the inventor in order to arrive at the present invention or in the process of arriving at the present invention, and may not necessarily be regarded as the known technology disclosed to the public before submitting the present application. Summary of the invention

[0006] According to some embodiments of the present disclosure, a method, device and computer-readable medium for monitoring abnormal conditions based on multiple sensors are provided. The tasks to be solved by the present invention are not limited to the above tasks, and other tasks not mentioned herein and the advantages of the present invention can be understood through the following description and can be more clearly understood through the embodiments of the present invention. In addition, it can be seen that the tasks and advantages to be solved by the present invention can be achieved by the means and combinations thereof indicated in the patent claims.

[0007] As a technical means to achieve the above-mentioned technical tasks, the first aspect of the present disclosure is a method for monitoring abnormal conditions based on multiple sensors, including: processing a sensing signal and outputting sensing data, wherein the sensing data is obtained by converting the sensing signal into a digital signal, wherein the sensing signal is received from multiple sensors, and the multiple sensors are configured to detect different objects; generating input data, wherein the input data is obtained by variably adjusting the size of the sensing data, and the input data is used to input into a neural network model; using the neural network model to perform operations on the input data in an analog domain, and outputting operation data, wherein the operation data is obtained by converting the operation result into digital data; and outputting a monitoring result, wherein the monitoring result is obtained by classifying the abnormal condition according to the operation data.

[0008] A second aspect of the present disclosure is an abnormal condition monitoring device based on multiple sensors, comprising:

[0009] a memory configured to store a program;

[0010] A processor configured to execute the program, wherein the processor is configured to:

[0011] Processing a sensing signal and outputting sensing data, wherein the sensing signal is obtained by converting the sensing signal into a digital signal, wherein the sensing signal is received from a plurality of sensors, and the plurality of sensors are configured to detect different objects; generating input data, wherein the input data is obtained by variably adjusting the size of the sensing data, and the input data is used to be input into a neural network model; performing an operation on the input data in an analog domain using the neural network model, and outputting operation data, wherein the operation data is obtained by converting the operation result into digital data; and outputting a monitoring result, wherein the monitoring result is obtained by classifying an abnormal condition according to the operation data.

[0012] A third aspect of the present disclosure is a computer-readable storage medium storing a program for executing the method of the first aspect on a computer.

[0013] In addition, other methods and other systems for implementing the present invention, as well as computer-readable storage media storing a computer program for executing the method may also be provided.

[0014] In addition to the foregoing description, other aspects, features, and advantages will become apparent from the following drawings, claims, and detailed description of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other objects, features and other advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0016] Figure 1 is a schematic diagram of a monitoring system including a sensor device according to an embodiment;

[0017] Figure 2 is a block diagram showing an example of the internal structure of a signal processing circuit according to an embodiment;

[0018] 3 is a block diagram showing an example of the internal structure of a power supply and a sensor signal processor according to an embodiment;

[0019] Figure 4 is a block diagram of an example of the internal structure of an analog PIM module according to an embodiment;

[0020] FIG5 is a schematic diagram of the internal structure and operation of an analog PIM according to an embodiment;

[0021] Figure 6 is a schematic diagram of an example of an analog PIM controller according to an embodiment;

[0022] Figure 7 is a flowchart of an exemplary control process for determining an output matching any sensor and calculating a resolution according to an embodiment;

[0023] Figure 8 is a schematic diagram of an example of a system for monitoring abnormal conditions based on multiple sensors according to an embodiment;

[0024] Fig. 9 is a schematic diagram of an example of the internal structure of a monitoring device according to an embodiment;

[0025] Fig.10 is a schematic diagram of another example of the internal structure of a monitoring device according to an embodiment;

[0026] Fig.11 is a flowchart of an example of a method for monitoring abnormal conditions based on multiple sensors according to an embodiment;

[0027] Fig.12 is a schematic diagram of an example of sensing data according to an embodiment;

[0028] Fig.13 is a schematic diagram of an example of input data according to an embodiment;

[0029] Fig.14 is a flowchart of another example of a method for monitoring abnormal conditions based on multiple sensors according to an embodiment;

[0030] Fig.15 is a schematic diagram of another example of sensing data according to an embodiment. DETAILED DESCRIPTION

[0031] By referring to the embodiments described in conjunction with the accompanying drawings, the advantages and features of the present invention and the methods for achieving these advantages and features will become clear. However, the present invention is not limited to the embodiments shown below, and can be implemented in various different forms, and it should be understood that the present invention includes all conversions, equivalents and substitutes within the spirit and technical scope of the present invention. The embodiments proposed below are to ensure that the disclosure of the present invention is complete and to enable those skilled in the art to fully understand the scope of the present invention. When describing the present invention, when it is determined that the detailed description of the relevant known technology may obscure the main points of the present invention, the detailed description will be omitted.

[0032] The terms used in this application are only used to describe specific embodiments and are not intended to limit the invention. Unless the context clearly indicates otherwise, singular expressions include plural expressions. In this application, it should be understood that terms such as "including" or "having" are intended to indicate that the features, numbers, steps, operations, elements, parts or combinations thereof described in the specification exist, and do not exclude the possibility of one or more other features, numbers, steps, operations, elements, parts or combinations thereof. In some embodiments, the two terms "operation" and "operation" can be replaced with each other.

[0033] Some embodiments of the present disclosure can be represented by function block configuration and various processing steps. All or part of these function blocks can be implemented with various numbers of hardware and / or software configurations for performing specific functions. For example, the function blocks of the present disclosure can be implemented by one or more microprocessors or circuits configured for certain functions. In addition, for example, the function blocks of the present disclosure can be implemented in various programming or scripting languages. The function blocks can be implemented as algorithms executed on one or more processors. In addition, the present disclosure can adopt conventional techniques for electronic environment settings, signal processing and / or data processing. Terms such as "mechanism", "element", "component" and "construction" can be widely used and are not limited to mechanical and physical components.

[0034] In addition, the connecting lines or connecting members between the components shown in the drawings are only used to illustrate functional connections and / or physical or circuit connections. In actual devices, the connections between the components can be represented by various alternative or additional functional connections, physical connections or circuit connections.

[0035] In this specification, "neural network model" may refer to a deep learning model based on an artificial neural network (ANN). The deep learning model has a deep structure with multiple hidden layers in the artificial neural network, through which complex patterns can be learned. For example, the deep learning model may include an input layer, a hidden layer, and an output layer, and each layer may include multiple neurons. In addition, each neuron can calculate the output value by multiplying the input value by the weight applied to the activation function, each layer passes the signal to the next layer, and the signal can be adjusted by weights and biases. In other words, the deep learning model optimizes weights and biases using training data, thereby allowing the complex relationship between input and output to be learned. At the same time, the deep learning model can be implemented by various types of models, such as multilayer perceptron (MLP), convolutional neural network (CNN), and recurrent neural network (RNN). In addition, the deep learning model can be implemented as various models according to methods such as configuring activation functions, loss functions, optimization algorithms, etc., and is not limited to a specific type of model.

[0036] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. However, the embodiments may be implemented in various different forms and are not limited to the examples described herein.

[0037] Figure 1 is a schematic diagram of a monitoring system including a sensor device 1 according to an embodiment.

[0038] The sensor device 1 according to the present invention includes a signal processing circuit 10, a sensor 30 and an energy harvesting circuit 20. The sensor 30 is a device for sensing the external environment of the location where the sensor device 1 is installed, for example, sensing temperature, humidity or image. The sensor 30 is connected to the signal processing circuit 10, operates according to the power provided by the signal processing circuit 10, and outputs a sensing signal to the signal processing circuit 10. The sensor 30 connected to the signal processing circuit 10 can be various types of sensors.

[0039] The energy collection circuit 20 collects energy from the environment outside the sensor device 1. The energy collection circuit 20 includes an RF (radio frequency) antenna, a resonant coil, a solar cell and / or an electrostatic collection circuit, and can collect energy (electricity) from RF signals, resonant signals, solar thermal energy and / or static electricity as an energy source.

[0040] The signal processing circuit 10 generates and stores power through the energy harvesting circuit 20, and supplies the generated power to the sensor 30. In addition, the signal processing circuit 10 can receive a sensing signal from the sensor 30 and process the sensing signal. For example, the signal processing circuit 10 can determine the condition of the sensor 30 or the condition outside the sensor device 1 by processing the sensing signal.

[0041] The signal processing circuit 10 according to the present invention applies CNN (convolutional neural network) algorithm technology to generate optimal power from various power sources (RF signals, resonant signals, solar energy and / or static electricity) and process sensing signals, etc. According to the present invention, the signal processing circuit 10 includes an expandable analog PIM module 1000 (or analog PIM 1100) that can dynamically increase or decrease the calculation resolution. The signal processing circuit 10 is configured to be able to perform convolution operations in the analog domain.

[0042] The signal processing circuit 10 including the expandable analog PIM module 1000 (or analog PIM 1100) can provide the required variable processing performance for various types of sensors 30 by adjusting the calculation resolution, thereby achieving optimal power consumption efficiency. The signal processing circuit 10 is configured to be universally applicable to various sensors 30. The signal processing circuit 10 includes a self-generating circuit using various power sources, so it can be free from environmental restrictions related to external power supply. The signal processing circuit 10 is preferably configured as an integrated circuit. The signal processing circuit 10 can be integrated into a semiconductor packaged in plastic or metal and mounted on a board of the sensor device 1. The signal processing circuit 10 will be in Figure 2 And detailed description later.

[0043] The control terminal 2 of the monitoring system is a terminal for controlling the sensor device 1 (specifically, the signal processing circuit 10). The control terminal 2 (which can be configured as a personal computer, a cloud server, etc.) updates the weights (filter or kernel weights) of the CNN algorithm processed by the signal processing circuit 10 or sets parameters for setting the accuracy, efficiency, etc. of the CNN algorithm or other processing. The control terminal 2 and the sensor device 1 can be connected by wire (e.g., USB, etc.) or wirelessly (e.g., Wi-Fi or ZigBee, etc.) so as to update or change the settings, parameters, weights, etc. of the sensor device 1 under the control of the control terminal 2.

[0044] Figure 2 is a block diagram showing the internal structure of the signal processing circuit 10.

[0045] according to Figure 2 The signal processing circuit 10 includes a CNN control unit 100, a power supply 200, a sensor signal processor 300, and an energy storage unit 400. The signal processing circuit 10 may also include Figure 2 Modules other than the hardware modules shown in .

[0046] The power supply 200 generates power from a signal outside the sensor device 1. The power supply 200 is connected to the energy harvesting circuit 20 of the sensor device 1, and generates and outputs power from the RF signal, the resonance signal, the solar thermal energy and / or the electrostatic signal of the energy harvesting circuit 20 under the control of the CNN control unit 100. The power supply 200 will be further investigated with reference to FIG. 3.

[0047] The sensor signal processor 300 processes a sensing signal from the sensor 30. The sensor signal processor 300 processes a sensing signal from any sensor 30 under the control of the CNN control unit 100 and outputs processed sensing data.

[0048] The energy storage unit 400 stores the power generated by the power supply 200. The energy storage unit 400 may include a known storage element for storing power, or may include a capacitor. The power stored in the energy storage unit 400 (and the power generated by the current power supply 200) is supplied to other blocks (CNN control unit 100, sensor signal processor 300 and power supply 200).

[0049] The CNN control unit 100 controls the power supply 200 and the sensor signal processor 300 according to the power workload data generated by the power supply 200 and the sensing data generated by the sensor signal processor 300. The CNN control unit 100 includes a CNN controller 2000 and an expandable analog PIM module 1000. The workload data may be data indicating the amount of current to generate power.

[0050] The analog PIM module 1000 performs a convolution operation between weights and input data (workload data and / or sensing data) in an analog signal domain, converts it into digital convolution data, and outputs the digital convolution data as a result signal. The analog PIM module 1000 operates according to the control (control signal) of the CNN controller 2000, and preferably performs a convolution operation by variably setting the weight size and the input data size.

[0051] In other words, the analog PIM module 1000 can reduce the power consumption required for analog operation and improve the operation speed by optimizing the effective operation size for processing input data. The data resolution required for processing the sensing data from various sensors 30 (such as temperature, humidity, and image) may be different for each sensor 30. According to the present invention, the analog PIM module 1000 is configured to be able to dynamically perform efficient operations or detect input resolutions according to the data characteristics of the combined sensor 30, and perform analog convolution operations at the corresponding operations or input resolutions.

[0052] The CNN controller 2000 in the CNN control unit 100 executes a CNN artificial intelligence algorithm (AI) to control the power supply 200 and / or the sensor signal processor 300 by controlling the analog PIM module 1000. The CNN controller 2000 may include an MCU (e.g., a low power MCU), a memory, and a circuit or logic for extracting and classifying features.

[0053] The CNN control unit 100 can determine the status of the sensor 30 by executing the CNN AI algorithm using the scalable analog PIM module 1000. In addition, the CNN control unit 100 can utilize the scalable analog PIM module 1000 to process the workload data and the sensing data according to the CNN AI algorithm technology, generate a power control signal for controlling the power supply 200, and a gain control signal for adjusting the gain of the sensor signal processor 300, and output the power control signal to the power supply 200 and the sensor signal processor 300.

[0054] The CNN control unit 100 may process subsequent layers after one layer of input data is processed according to the CNN AI algorithm to generate a power control signal and a gain control signal according to the classification, and may determine various other situations.

[0055] FIG. 3 is a block diagram of a configuration example of the power supply 200 and the sensor signal processor 300 .

[0056] Fig. 3A is an example of a hardware block diagram of the power supply 200, and Fig. 3B is an example of a hardware block diagram of the sensor signal processor 300. The example of Fig. 3 is a configuration diagram shown for understanding the present invention, and the sensor device 1 may have various or different configurations of the power supply 200 and the sensor signal processor 300 that are known or will be known.

[0057] When briefly examining the power supply 200 and the sensor signal processor 300 with reference to FIG3 , the power supply 200 includes an adaptive matching network 210, one or more rectifiers 250, and one or more DC-DC converters 290. The adaptive matching network 210 is configured to optimally match one or more of the RF signal, the resonant signal, the solar signal, and / or the electrostatic signal according to the received power control signal. The CNN control unit 100 can output the power control signal to enable the adaptive matching network 210 to match a specific power supply.

[0058] The rectifier 250 is connected to the adaptive matching network 210 to generate direct current (DC) power according to the signal from the adaptive matching network 210. The rectifier 250 can be, for example, a full-bridge rectifier or a half-bridge rectifier. One or more rectifiers 250 operate according to the power control signal received from the CNN control unit 100 to generate and output DC power.

[0059] The DC-DC converter 290 is connected to the rectifier 250 and increases or decreases the DC power of the rectifier 250 according to a set voltage. For example, the DC-DC converter 290 may be a buck-boost converter. The DC-DC converter 290 may operate according to a power control signal received from the CNN control unit 100 and output DC power at a set level.

[0060] Here, the power control signal received from the CNN control unit 100 is a signal that controls the adaptive matching network 210, one or more rectifiers 250, and one or more DC-DC converters 290 by executing the CNN AI algorithm on the workload data (current data) of the DC power output by the power supply 200. The CNN control unit 100 generates and outputs the power control signal according to the currently input workload data (further using the sensing data) to optimally control the power supply 200.

[0061] The sensor signal processor 300 includes a first MUX 310 (multiplexer), a TIA (transimpedance amplifier) ​​320, an LNA (low noise amplifier) ​​330, a second MUX 350, a VGA (variable gain amplifier) ​​370, and an SD ADC (sensor device analog-to-digital converter) 390. The first MUX 310 outputs a signal of one of a plurality of sensor signal lines according to an applied selection signal. For example, the first MUX 310 may output a signal of a sensor signal line of one of four pins of the sensor 30 that may be connected to the integrated circuit (i.e., the signal processing circuit 10) according to a signal applied to a selection signal pin.

[0062] The TIA 320 is connected to the first MUX 310, amplifies the signal from the first MUX 310, and outputs the amplified signal. The TIA 320 amplifies and outputs the input signal according to the gain control signal received from the CNN control unit 100. For example, the TIA 320 may convert a current signal into a voltage signal, and amplify and output the converted voltage signal according to the applied gain control signal.

[0063] The LNA 330 is connected to the first MUX 310 to amplify the signal from the first MUX 310 and output the amplified signal. The LNA 330 may amplify the input voltage signal according to the gain control signal received from the CNN control unit 100 and output the amplified voltage signal.

[0064] The second MUX 350 is connected to the TIA 320 and the LNA 330 to select and output the amplified signal from the TIA 320 or the amplified signal from the LNA 330. The second MUX 350 may output a signal according to the same selection signal as the selection signal pin of the first MUX 310, or may output a signal according to a selection signal received from the CNN control unit 100.

[0065] The VGA 370 is connected to the second MUX 350 and amplifies the signal from the second MUX 350 according to the gain control signal.

[0066] The SD ADC 390 is connected to the VGA 370, converts an analog signal from the VGA 370 into a digital signal, and outputs the converted sensing data. The SD ADC 390 may be a successive approximation ADC including a SAR logic, a CDAC, and a comparator.

[0067] The SD ADC 390 has a resolution of up to 8 bits, 16 bits, or 32 bits, and outputs the converted digital sensing data within the maximum resolution range. The SD ADC 390 receives an enable signal from the analog PIM module 1000 or the CNN control unit 100, which specifies the number of bits enabled within the maximum resolution, and outputs digital sensing data having an output resolution corresponding to the number of bits enabled within the maximum resolution according to the enable signal.

[0068] For example, when the maximum bit resolution of the SD ADC 390 is 16 bits and the SD ADC 390 receives an 8-bit enable signal, an operation of scaling the data to 8 bits to output digital sensing data will be performed. When the SD ADC 390 receives an enable signal requesting an increase to 10 bits when outputting 8-bit sensing data, 10-bit sensing data can be generated and output by controlling the CDAC.

[0069] Here, the gain control signal received from the CNN control unit 100 is generated by executing the CNN AI algorithm on the sensing data output by the sensor signal processor 300. The CNN control unit 100 uses the sensing data output from the sensor signal processor 300 and further workload data as input to generate and output the gain control signal. The gain control signal is used to control the gain of the TIA 320, the LNA 330 and / or the VGA 370.

[0070] Figure 4 FIG. 5 is a block diagram of the structure of the analog PIM module 1000, and FIG. 6 is a diagram describing the internal structure and operation of the analog PIM.

[0071] According to the present invention, Figure 4The analog PIM module 1000 in the embodiment is configured to be able to perform convolution operations in the analog signal domain, and is configured to be able to dynamically adjust (enlarge or reduce) the resolution of the convolution operation. By using the analog PIM module 1000 that can adjust the calculation resolution in a scalable manner, the signal processing circuit 10 can dynamically perform the optimal convolution calculation and accordingly apply the optimal CNN algorithm, regardless of the type or kind of the sensor 30.

[0072] In addition, the signal processing circuit 10 can perform optimal convolution calculations by adjusting the calculation resolution, thereby reducing power consumption and increasing calculation speed. Figure 4 The analog PIM module 1000 in the embodiment may be configured as an analog PIM circuit and preferably integrated in an integrated circuit.

[0073] according to Figure 4 , the analog PIM module 1000 includes an analog PIM 1100 and an analog PIM controller 1500 . Figure 4 The analog PIM module 1000 is preferably integrated into the signal processing circuit 10 of the sensor device 1, which is an integrated circuit.

[0074] The analog PIM 1100 is configured to perform a convolution operation between a weight matrix of variable size (e.g., 8*8, 4*4, 3*3, 2*2, 1*1, etc.) and an input data matrix (e.g., 88, 44, 33, 22, 11, etc.) in the analog signal domain.

[0075] The analog PIM 1100 includes a plurality (e.g., 8, 16, 64, etc., hereinafter referred to as a "first group") of SRAM operators 1130. The first group of SRAM operators 1130 are interconnected (interconnected in rows, see Figure 4 ①), a convolution operation is performed in the analog domain using the SRAM 1131 and the capacitor 1133 therein, and an analog convolution result signal is generated according to the convolution operation.

[0076] In the first group of SRAM operators 1130, the enabled number (e.g., 1, 4, 8, 16, etc., hereinafter referred to as the "second number") of SRAM operators 1130 perform convolution operations under the control of the analog PIM controller 1500. In the first group of SRAM operators 1130, the remaining non-enabled SRAM operators 1130 suspend the execution of the analog convolution operation. According to the disable signal, the power supply to the SRAM operator 1130 can be suspended, or the capacitor 1133 of each SRAM operator 1130 can be turned on to suspend the execution of the convolution operation.

[0077] Each operator in the first group of SRAM operators 1130 (temporarily) stores one weight (weight of 1 or 0) in the weight matrix in SRAM 1131, and stores one input data in the input data matrix as an analog signal in capacitor 1133. The SRAM operator 1130 performs a product operation in the analog domain between the weight of SRAM 1131 and the input data stored in capacitor 1133. And, the first group of SRAM operators 1130 performs an averaging operation on the product operation using the (total capacity) of the provided capacitor 1133 to generate an analog convolution result signal.

[0078] The analog PIM 1100 further includes a first set of DACs 1150. Each DAC 1150 in the first set of DACs 1150 is connected to the first set of SRAM operators 1130, and the DAC 1150 converts digital input data (eg, workload data or sensing data) into analog input data and outputs the analog input data to the SRAM operators 1130.

[0079] According to the present invention, each SRAM operator in the first group of SRAM operators 1130 ( Figure 4 1131 and one or more (preferably two) capacitors 1133, and one or more skip switches 1135 (see Figure 5A The structure of the SRAM operator 1130 is configured to control the storage of analog input data in one or more capacitors 1133 through the DAC 1150.

[0080] The SRAM 1131 including 10 transistors or the like stores the weight of a binary signal 1 or 0, and the capacitor 1133 stores analog input data via the connected DAC 1150. The skip switch 1135 allows the SRAM operator 1130 to store the analog input data in the capacitor 1133 according to an on control signal (zero skip (ZS) control signal) received from the analog PIM controller 1500, and also allows the SRAM operator 1130 not to store the analog input data in the capacitor 1133 according to a received off control signal.

[0081] In addition, the analog PIM 1100 further includes one or more ADCs 1110. When the first group of SRAM operators 1130 in the analog PIM module 1000 is L SRAM operators 1130 (L is an integer of 1 or greater, for example, Figure 4 As shown in 4), the analog PIM 1100 may have four ADCs 1110.

[0082] The ADC 1110 is connected to the first group of SRAM operators 1130 of a specific row, converts the analog convolution result signal calculated by the first group of SRAM operators 1130 into digital convolution data, and outputs the converted digital convolution data.

[0083] The analog PIM controller 1500 of the analog PIM module 1000 controls the analog PIM 1100 to perform a convolution operation in the analog domain. The analog PIM controller 1500 includes gates through a logic circuit (e.g., a circuit generated according to an RTL code), and controls the analog PIM 1100 to perform a convolution operation according to a control signal received from the CNN control unit 100 of the signal processing circuit 10.

[0084] The analog PIM controller 1500 is preferably configured to perform the convolution operation in the analog domain by controlling the second number of SRAM operators 1130. The second number of SRAM operators are the SRAM operators enabled in the first group of SRAM operators 1130.

[0085] The analog PIM controller 1500 stores the weight in the SRAM 1131 of the enabled second number of SRAM operators 1130, and stores the input data in the capacitor 1133. The analog PIM controller 1500 stores the weight and the input data through the DAC 1150 connected to the second number of SRAM operators 1130. When the weight value stored in the weight storage process is a specified value (e.g., logic (bit) 0), the analog PIM controller 1500 outputs an on control signal to the SRAM operator 1130 to turn on the skip switch 1135.

[0086] By outputting the off control signal to the SRAM operator 1130, the analog PIM controller 1500 omits the analog operation (product operation) performed in the corresponding SRAM operator 1130. When the stored weight is a different value (e.g., logic (bit) 1), the analog PIM controller 1500 outputs an on control signal to turn on (connect) the skip switch 1135 to the SRAM operator 1130.

[0087] By further including control of the skip switch 1135 in the SRAM operator 1130, the analog product operation in the SRAM operator 1130 may be omitted when the weight is 0, thereby reducing power consumption of the analog PIM 1100 and improving performance speed of the convolution operation.

[0088] 5B shows an example of a weight matrix of a 4*4 size (4*4, the second number is 16). As shown in the example of FIG5B , when the weight of (the element of) a specific row and column in the weight matrix is ​​0, the skip switch 1135 of the corresponding SRAM operator 1130 is closed (opened), and the corresponding SRAM operator 1130 omits the product operation.

[0089] Based on the convolution data output from ADC 1110, the analog PIM controller 1500 outputs an enable control signal to the first group of SRAM operators 1130 to activate a second number of SRAM operators 1130 that is less than or equal to the first group of SRAM operators 1130 (the second number is 1 or more and is less than or equal to the number of the first group of SRAM operators 1130).

[0090] The analog PIM controller 1500 outputs an enable control signal to the second number of SRAM operators 1130 in the first group of SRAM operators 1130 to activate the second number of SRAM operators 1130 , and outputs a disable control signal to the remaining SRAM operators 1130 in the first group of SRAM operators 1130 to deactivate the remaining SRAM operators 1130 .

[0091] Each disable signal suspends the analog operation (product operation) performed in each remaining SRAM operator 1130. For example, the disable signal may block the power supply to the SRAM operator 1130, or open the capacitor 1133 connected to the ground through an additionally provided switch, which is separated from the read / write line (see Figure 5A ).

[0092] 5C shows an example formula and process of a convolution operation performed by the analog PIM 1100, which is converted from the digital domain to the analog domain. The input data (workload data and / or sensing data) and weights are converted to analog input data and weights through the DAC 1150 and stored in a second number (e.g., 16) of SRAM operators 1130 in the first group of SRAM operators 1130 (see ① in FIG. 5C ).

[0093] The second number (e.g., 16) of SRAM operators 1130 in the first group of SRAM operators 1130 perform a product operation between SRAM 1131 and capacitor 1133, and average the signal (e.g., voltage signal) according to the total capacity of the capacitor 1133 in the second number (e.g., 16) of SRAM operators 1130 to generate an analog convolution result signal (see ② of FIG. 5C ). The ADC 1110 connected to the enabled second number of SRAM operators 1130 converts the digital convolution data according to the size of the convolution result signal (see ③ of FIG. 5C ).

[0094] The converted digital convolution data is output to the CNN controller 2000 of the CNN control unit 100, and the CNN controller 2000 can extract and classify features according to the CNN AI algorithm.

[0095] The analog PIM controller 1500 is preferably configured to dynamically change the data used in the convolution operation and the resolution of the operation. The analog PIM controller 1500 is preferably configured to be able to adjust the input data resolution of the SD ADC 390 and / or the resolution of the analog convolution operation (the second number of SRAM operators 1130) based on the input data output by the SD ADC 390 connected to the analog PIM module 1000 and / or the convolution data output by the ADC 1110 connected to the analog PIM 1100.

[0096] For example, when the input data output by the SD ADC 390 of the sensor signal processor 300 connected to the analog PIM module 1000 reaches a set number of times or more, the analog PIM controller 1500 increases the output resolution of the SD ADC 390. And after the output resolution of the SD ADC 390 is increased, a new second number of SRAM operators 1130 to be enabled in the first group of SRAM operators 1130 is determined according to the convolution data output by the ADC 1110.

[0097] In addition, the analog PIM controller 1500 is connected to the analog PIM 1100 interface, outputs the sensing data from the sensor signal processor 300 or the workload data of the power supply to the DAC 1150, and the DAC 1150 is connected to the second number of SRAM operators 1130 to which the input data is selected. And outputs the weight from the CNN controller 2000 to the DAC 1150 connected to the second number of SRAM operators 1130. In addition, the analog PIM controller 1500 can output the convolution data output by the ADC 1110 to the CNN controller 2000.

[0098] The analog PIM controller 1500 (or CNN controller 2000) samples the input data periodically according to the sampling period (and event) according to the determined second quantity, stores the sampled input data in the analog PIM 1100, and then allows the analog convolution operation to be performed. In this way, the analog PIM controller 1500 outputs the adjusted input data to the changed second number of SRAM operators 1130. The adjusted input data is adjusted according to the changed second number. In addition, the analog PIM controller 1500 receives the weight data adjusted according to the changed second number from the CNN controller 2000, and outputs the adjusted weight to the second number of SRAM operators 1130.

[0099] Figure 6 is an example block diagram of an analog PIM controller 1500 .

[0100] according to Figure 6 , the analog PIM controller 1500 includes an ADC output detector 1501 , a comparator 1505 , a memory 1503 and an ADC / SRAM enable controller 1509 . Figure 6 The analog PIM controller 1500 in FIG. 1 includes a logic circuit (eg, a circuit generated according to an RTL code) and is implemented as an integrated circuit through a plurality of gates.

[0101] The ADC output detector 1501 detects the convolution data from the ADC 1110 of the analog PIM 1100, and further detects the input data (sensing data) output from the SD ADC 390 of the sensor signal processor 300 connected to the analog PIM module 1000. The ADC output detector 1501 may detect the data upon receiving a signal representing the output data from the CNN controller 2000, other control logic of the analog PIM controller 1500, and / or the sensor signal processor 300. The ADC output detector 1501 outputs the detected data (convolution data or input data).

[0102] The memory 1503 stores previous input data and previous convolution data output to the ADC output detector 1501.

[0103] The comparator 1505 digitally compares the convolution data output by the ADC output detector 1501 with the convolution data outputted before (immediately before), and outputs a comparison result signal. In addition, the comparator 1505 further digitally compares the input data from the ADC output detector 1501 with the input data outputted according to the previous (immediately before) detection, and outputs a comparison result signal.

[0104] The comparator 1505 outputs a comparison result signal by comparing each convolution data with each input data, and the comparison result signal indicates whether the data in the memory 1503 and the data output by the ADC output detector 1501 are the same.

[0105] According to the comparison result in the comparator 1505, the ADC / SRAM enable controller 1509 determines the number of bits of the SD ADC 390 and the second number of the first group of SRAM operators 1130 for the analog convolution operation. The ADC / SRAM enable controller 1509 generates an enable (control) signal according to the comparison between the current data (input data or convolution data) and the previous data, and outputs the enable (control) signal to the SD ADC 390 to increase the resolution of the SD ADC 390. And, generates an enable (control) signal and outputs the enable (control) signal to the second number of SRAM operators 1130 of the analog PIM 1100 to increase the resolution of the convolution operation.

[0106] For example, when the ADC / SRAM enable controller 1509 continuously receives the same comparison result signal through the comparator 1505 for the number of times that reaches the number of times set for the convolution data (e.g., 4 times, 5 times, etc.) or more, the ADC / SRAM enable controller 1509 determines a new second number that is greater than the currently set second number to increase the resolution of the analog operation of the first group of SRAM operators 1130. For example, when there are 64 SRAM operators 1130 in total and 4 of them are currently used, the ADC / SRAM enable controller 1509 increases the second number from 4 to 5 (or 8, 9, etc.). The set number of times can be set by the control terminal 2.

[0107] The ADC / SRAM enable controller 1509 outputs an enable control signal to the first group of SRAM operators 1130 to enable the added second number of SRAM operators 1130. The ADC / SRAM enable controller 1509 outputs an enable signal to the second number of SRAM operators 1130 to enable the added second number of SRAM operators 1130 in the first group of SRAM operators 1130, and outputs a disable signal to the remaining SRAM operators 1130 to disable the remaining SRAM operators 1130.

[0108] Through this process, the analog PIM module 1000 can perform a convolution operation that matches the sensor 30 or is optimized for the state of the sensor 30. At the same time, when the output resolution of the convolution operation of the input data is low, the power consumption is reduced by increasing the analog calculation resolution in real time.

[0109] In addition, according to the comparison result of the input data of the comparator 1505, the ADC / SRAM enable controller 1509 increases the output resolution of the SDADC 390. When the ADC / SRAM enable controller 1509 continuously receives the same comparison result signal for a number of times that reaches the number of times set for the input data (e.g., 4 times, 5 times, etc.) or more, the ADC / SRAM enable controller 1509 outputs an enable control signal to the SD ADC 390 to make the output resolution higher than the currently set output resolution. The ADC / SRAM enable controller 1509 outputs an enable control signal to the SD ADC 390 to enable a number of bits greater than the currently set number of bits of the SD ADC 390 (e.g., increase by 1 bit). The set number of times can be set by the control terminal 2, and the set number of times for the input data and the set number of times for the convolution data can be different.

[0110] The SD ADC 390 , upon receiving the enable control signal, controls an internal CDAC, etc., and outputs digital sensing data having a resolution with an increased number of bits.

[0111] As the number of output bits of the SD ADC 390 increases, the ADC / SRAM enable controller 1509 re-determines the second number enabled in the first group of SRAM operators 1130 using the convolution data output by the ADC 1110 of the analog PIM 1100. When the ADC / SRAM enable controller 1509 continuously receives the same comparison result signal for the convolution data a set number of times (e.g., 4 times, 5 times, etc.) or more through the comparator 1505 starting from the minimum number of times (e.g., 1), the ADC / SRAM enable controller 1509 determines the second number greater than the currently set second number to increase the analog operation resolution of the first group of SRAM operators 1130.

[0112] The ADC / SRAM enabling controller 1509 repeatedly or simultaneously (in parallel) executes the process of determining a second number when the output resolution of the SD ADC 390 changes to detect the optimal resolution of the convolution operation, thereby allowing an optimized CNN AI algorithm to be executed based on the connected sensor 30 or the conditions of the sensor 30.

[0113] The ADC / SRAM enable controller 1509 outputs an enable control signal to the first group of SRAM operators 1130 to enable the second number of SRAM operators 1130, and outputs an enable control signal to activate the second number of DACs 1150 connected to the second number of SRAM operators 1130. In addition, the ADC / SRAM enable controller 1509 may output a disable control signal to the remaining DACs 1150 in the first group of DACs 1150 to deactivate the remaining DACs 1150.

[0114] The resolution determination process performed in the ADC / SRAM enable controller 1509 will refer to Figure 7 Conduct a more detailed investigation.

[0115] Figure 7 is a flow chart describing an exemplary control flow for determining output and calculation resolution matching with any sensor 30 .

[0116] Figure 7 The control flow is used to control the analog PIM 1100, which is executed in the signal processing circuit 10, and is preferably executed by the control of the CNN control unit 100 and the analog PIM module 1000 (the analog PIM controller 1500). Figures 1 to 6 A detailed investigation was conducted, so the control process will be investigated intensively and in detail below.

[0117] First, the signal processing circuit 10 enables the sensor 30 (S101). The signal processing circuit 10 can power the sensor 30 according to a POR (Power On Reset) signal applied from the outside or generated according to a power supply, and start or initialize the operation of the sensor 30. The signal processing circuit 10 also initializes the internal circuits 100, 200, and 300.

[0118] Thereafter, the signal processing circuit 10 enables the SD ADC 390 of the sensor signal processor 300 (S103). For example, the signal processing circuit 10 enables the SD ADC 390 so that the SD ADC 390 has an output resolution of all bits.

[0119] The signal processing circuit 10 samples the input data (sensing data) output from the SD ADC 390 of the sensor signal processor 300 ( S105 ), and determines the resolution of the SD ADC 390 according to the sampled input data ( S107 ).

[0120] The signal processing circuit 10 samples the input data multiple times and determines the initial resolution of the SD ADC 390 using the comparator 1505. The initial resolution is the number of bits that changes less than the set number of times from the minimum number of bits (e.g., 1 bit) to the maximum number of bits (e.g., the number of bits of the maximum resolution that can be output by the SDADC 390, such as 16 bits or 32 bits). The signal processing circuit 10 can determine its initial resolution by performing processing on each bit in turn, or by performing parallel processing on all bits.

[0121] The signal processing circuit 10 outputs an enable control signal to the SD ADC 390 to enable the number of bits determined according to the initial resolution, and the SD ADC 390 outputs digital sensing data according to the set resolution.

[0122] After determining the initial (first) resolution, the signal processing circuit 10 enables the analog PIM 1100 ( S107 ). The CNN controller 2000 may initialize the analog PIM 1100 by controlling the analog PIM controller 1500 .

[0123] After enabling the analog PIM 1100 , the signal processing circuit 10 samples the convolution data output from the ADC 1110 of the analog PIM 1100 ( S109 ), and determines the resolution of the analog PIM based on the sampled convolution data ( S111 ).

[0124] Under the control of the CNN controller 2000, the signal processing circuit 10 can perform comparison tests on the convolution data in sequence from the minimum resolution (the second number is 1) to the maximum resolution (the second number is the total number of the first group), and determine the calculation resolution of the first (initial) analog PIM 1100 based on the comparison test.

[0125] The signal processing circuit 10 can perform a convolution operation on the weight and input data (e.g., sensing data) at a currently set resolution, and the set resolution is between the minimum resolution and the maximum resolution. When the result of the convolution operation, i.e., the digital convolution data, is the same number of times continuously reaches a set number of times or more, the currently set resolution is increased. And by repeatedly comparing the current convolution data with the previous convolution data at the increased resolution, the initial optimal calculation resolution is determined. Through this test process, the signal processing circuit 10 determines the initial second quantity.

[0126] After determining the number of bits and the second number of the SD ADC 390 according to the initialization process (see S101 to S111), the signal processing circuit 10 dynamically adjusts the number of bits and the second number of the SD ADC 390 according to the output of the SD ADC 390 or the ADC 1110 of the analog PIM 1100 to maintain the optimal resolution required for the convolution operation in real time and optimize power consumption.

[0127] More specifically, after determining the number of bits and the second number of the SD ADC 390 (see S101 to S111 ), the signal processing circuit 10 performs the sensor operation under the control of the CNN control unit 100 ( S113 ).

[0128] The signal processing circuit 10 performs analog operations on the weights and input data (e.g., sensing data) by analog PIM 1100 through a second number of SRAM operators 1130 (the second number of SRAM operators 1130 are one or more enabled SRAM operators in the first group of SRAM operators 1130) to generate an analog convolution result signal. The generated convolution result signal is converted into digital convolution data through the ADC 1110 of the analog PIM 1100. The signal processing circuit 10 can use the converted convolution data to execute the CNN AI algorithm in real time.

[0129] The process of generating the convolution result signal is configured to omit the analog operation (product operation) by skipping the switch 1135 according to the shutdown control signal received from the analog PIM controller 1500. The shutdown control signal is set according to the value of the weight in the one or more second number SRAM operators 1130.

[0130] After the sensor operation, the signal processing circuit 10 determines whether the digital input data output by the sensor signal processor 300 is continuously the same for a set number of times or more (S115). When the digital input data is continuously the same for a set number of times or more, the signal processing circuit 10 determines that the output resolution of the sensor signal processor 300 is low, and increases the output resolution of the input data (S117).

[0131] The signal processing circuit 10 increases the number of output bits currently set by the SD ADC 390 by 1 bit or more, and outputs an enable control signal instructing the increase in the number of bits to the SD ADC 390 .

[0132] As the output resolution of the input data increases (according to the increase in the number of bits of the SD ADC 390), the signal processing circuit 10 re-executes S109 and S111 to gradually increase the second number of enabled SRAM operators 1130 between the minimum number and the number of the first group, and re-determines the second number based on comparing the convolution data output by the ADC 1110 of the increased analog PIM 1100 with the convolution data of the same second number before. Through this process, when the output bit number of the SD ADC 390 changes, the resolution of the optimal analog operation is dynamically calculated to optimize the performance of the CNN AI algorithm.

[0133] In addition, the signal processing circuit 10 dynamically determines a second number of SRAM operators 1130 to be enabled in the first group of SRAM operators 1130 according to the convolution data output by the analog PIM 1100 (S119 and S121). When the convolution data output by the analog PIM 1100 is continuously the same set number of times or more (S119), the signal processing circuit 10 increases the second number to improve the calculation resolution of the first group of SRAM operators 1130 (S121).

[0134] The signal processing circuit 10 determines a second number greater than the currently determined second number, enables the determined second number of SRAM operators 1130 (and the corresponding DAC 1150), and performs subsequent sensor operations accordingly. The signal processing circuit 10 can output an enable control signal to the first group of SRAM operators 1130 (and the corresponding second number of DACs 1150) to enable the changed second number of SRAM operators 1130 (and the corresponding second number of DACs 1150), and output the convolution result of the calculation resolution adjusted by the activated SRAM operators 1130 and DACs 1150.

[0135] Through the above control flow, the signal processing circuit 10 can be universally combined with the sensor 30 regardless of the type of the sensor 30, and has dynamically efficient convolution operation processing and optimized power consumption according to the type or condition of the sensor 30.

[0136] Another embodiment of a monitoring system will be described below. The system includes a plurality of sensors and monitors abnormal conditions based on the plurality of sensors.

[0137] Figure 8 is a schematic diagram of a system for monitoring abnormal conditions based on multiple sensors according to an embodiment.

[0138] Reference Figure 8 , a system 800 for monitoring abnormal conditions based on multiple sensors (hereinafter referred to as a monitoring system) may include multiple sensors 811, 812, and 813, a monitoring device 820, a server 830, and one or more user terminals 840 and 850. However, in addition to this, the monitoring system 800 may further include at least one external device. The various components included in the monitoring system 800 may be connected via wired or wireless communication to send and receive data to each other.

[0139] The plurality of sensors 811, 812, and 813 may be sensors for building an Internet of Things (IoT) system. For example, the plurality of sensors 811, 812, and 813 may detect various objects or phenomena in various environments and situations in the area where the IoT system is built. For example, each of the plurality of sensors 811, 812, and 813 may be any one of a temperature sensor, a humidity sensor, a motion sensor, a gas sensor, a vibration sensor, a light sensor, a noise sensor, a distance sensor, a pressure sensor, a proximity sensor, a pH sensor, an ultraviolet sensor, a magnetic sensor, an image sensor, a sound sensor, a wind speed sensor, a wind direction sensor, a current sensor, a voltage sensor, and a flow sensor. In addition, the sensors of the monitoring system 800 may include sensors configured to detect objects associated with the IoT system.

[0140] about Figure 8 , the object detected by each sensor may be an object related to the abnormal condition monitored by the monitoring system 800. Here, the abnormal condition monitored by the monitoring system 800 may refer to a condition that is beyond the normal range due to a change in the operating environment of the IoT system or the monitoring system 800. For example, the monitoring system 800 may monitor whether an abnormal condition such as a fire or a gas leak occurs around the area constructed by the monitoring system 800 (or the area constructed by the IoT system) based on the changes in temperature, pressure, and gas detected by the temperature, pressure sensor, and gas sensor, respectively.

[0141] At the same time, the area constructed by the monitoring system 800 can be a target area for the monitoring system 800 to monitor whether an abnormal condition occurs. In other words, multiple sensors 811, 812, and 813 can be installed in the target area for monitoring abnormal conditions to detect objects related to the abnormal conditions. At the same time, although multiple sensors 811, 812, and 813 are installed in the target area, the detection range can vary depending on the type of object being detected. Therefore, depending on the type of object detected by the sensor, the monitoring system 800 may include multiple sensors that detect the same object. For example, the monitoring system 800 may include a first gas sensor, a second gas sensor, and a temperature sensor. However, for the convenience of description, even if multiple sensors that detect the same object are installed in the target area, the sensors that detect the object will be combined and described as one sensor.

[0142] In the present disclosure, multiple sensors 811, 812, and 813 may not be limited to sensors that detect one object. For example, multiple sensors 811, 812, and 813 may each be a sensor that selectively detects one object or a sensor that detects multiple objects simultaneously. In other words, among the sensors included in the monitoring system 800, each sensor that detects a specific object may be implemented as an independent device, or as a device that can simultaneously perform the function of detecting one object and the function of detecting another object. As an example, the monitoring system 800 may include a gas sensor, a pressure sensor, and a temperature sensor that detect gas, pressure, and temperature, respectively. As another example, the monitoring system 800 may include a first sensor configured to detect gas and a second sensor configured to detect pressure and temperature.

[0143] At the same time, each of the plurality of sensors 811, 812 and 813 may be Figure 1 The sensor 30 in the monitoring system 800. For example, the multiple sensors 811, 812, and 813 may be devices installed in the target area of ​​the monitoring system 800 to sense the external environment. The multiple sensors 811, 812, and 813 may operate according to the supplied power to detect objects related to abnormal conditions and output sensing signals related to the detection results.

[0144] The monitoring device 820 may be connected to the plurality of sensors 811, 812, and 813 included in the monitoring system 800 to receive sensing signals from the plurality of sensors 811, 812, and 813. In other words, the monitoring device 820 may receive sensing signals for detecting different types of objects. For example, in the present disclosure, a first sensing signal associated with a first object detected by a first sensor and a second sensing signal associated with a second object detected by a second sensor may be processed by the monitoring device 820 instead of being received and processed by an independent device.

[0145] The monitoring device 820 may output a monitoring result related to whether an abnormal condition occurs in the target area based on processing the received sensing signal. In more detail, the monitoring device 820 may convert the sensing signal into input data to be input into the neural network model, and operate the input data using the neural network model to obtain a monitoring result related to the classification of the abnormal condition. Figures 9 to 15 A specific operation example of the monitoring device 820 of the present disclosure is described.

[0146] The server 830 may be a device configured to communicate with an external device, which may include a monitoring device 820 and user terminals 840 and 850. For example, the server 830 may receive a monitoring result from the monitoring device 820 and store the monitoring result. In addition, the server 830 may also transmit the stored monitoring result to the user terminal 840 of the manager 861 or the user terminal 850 of the IoT system user 862.

[0147] The user terminals 840 and 850 may be computing devices each having a display device and including a memory and a processor. For example, the display device may be implemented as a touch screen to receive user input. Alternatively, each user terminal 840 and 850 may also include a separate device (e.g., a keyboard, a mouse, etc.) for receiving user input. For example, the user terminals 840 and 850 may include, but are not limited to, a smartphone, a laptop computer, a desktop computer, a portable computer, a tablet computer, etc.

[0148] The user terminal 840 of the administrator 861 who manages the monitoring system 800 (or IoT system) can output the status information of the IoT system based on the monitoring results received from the server 830. For example, through the user terminal 840, the administrator 861 can verify whether the environment surrounding the current IoT system is in a normal state, or whether an abnormal condition has occurred for a predetermined object, or receive an alarm about the occurrence of an abnormal condition. Similarly, the user terminal 850 of the user 862 who uses the IoT system can output the status information of the IoT system based on the monitoring results received from the server 830. Similarly, through the user terminal 850, the user 862 can verify whether the environment surrounding the current IoT system is in a normal condition, or whether an abnormal condition has occurred for a predetermined object, or receive an alarm about the occurrence of an abnormal condition. At the same time, refer to Figure 8 , user terminals 840 and 850 are shown as receiving monitoring results from server 830. However, the present invention is not limited thereto. User terminals 840 and 850 may receive monitoring results directly from monitoring device 820, or may receive monitoring results from another device (not shown).

[0149] Fig. 9 is a schematic diagram of an example of the internal structure of a monitoring device according to an embodiment.

[0150] refer to Fig. 9 , the monitoring device 900 may include a processor 910, a memory 920, an input / output interface 930 and a communication module 940. For the convenience of description, Fig. 9 Only components relevant to the present invention are shown. Fig. 9 In addition to the components shown in FIG. 9 , the monitoring device 900 may also include other common components. In addition, it is obvious to those skilled in the art that the present invention is related to the present invention. Fig. 9 The processor 910, the memory 920, the input / output interface 930, and the communication module 940 shown in FIG. 9 may be implemented as independent devices.

[0151] The processor 910 can process computer program instructions by performing basic arithmetic, logic, and input / output operations. Here, the instructions can come from the memory 920 or an external device (e.g., an external server, etc.). In addition, the processor 910 can generally control the operation of other components included in the monitoring device 900.

[0152] The processor 910 may determine an operation mode based on the status information of the target area where the abnormal condition is detected.

[0153] For example, the processor 910 may determine any one of a first mode and a second mode as the operation mode. The first mode detects different objects simultaneously. The second mode detects objects related to the status information in real time.

[0154] (First Operation Mode)

[0155] The processor 910 may receive sensing signals from a plurality of sensors installed in the target area in response to the first mode of simultaneously detecting different objects being determined as the operation mode.

[0156] Here, the plurality of sensors may include at least one of a gas sensor, a pressure sensor, and a temperature sensor.

[0157] The processor 910 may process the sensing signal and output first sensing data obtained by converting the sensing signal into a digital signal.

[0158] For example, the processor 910 may select one of the sensing signals based on the first selection control signal, amplify the selected sensing signal according to the amplification control signal and the gain control signal, and digitally convert the amplified sensing signal to output sensing data.

[0159] According to an embodiment, the first selection control signal may be a signal configured to perform a control operation to select the sensing signal in time series.

[0160] According to an embodiment, the first selection control signal may be a signal configured to perform a control operation to select one of the sensing signals based on a predetermined period.

[0161] The processor 910 may generate first input data to be input to the first neural network model by variably adjusting the size of the first sensing data.

[0162] For example, the processor 910 may accumulate the first sensing data of the corresponding number of bits and generate the first input data having an output resolution corresponding to the number of bits determined based on an enable signal for specifying the number of bits enabled in the maximum resolution.

[0163] The processor 910 may perform an operation on the first input data in the analog domain using the first neural network model, and output first operation data obtained by converting the operation result into digital data.

[0164] For example, the processor 910 may convert the first input data into an input value in the analog domain, use a plurality of SRAM operators to perform a convolution operation on the input value in the analog domain, and output first operation data. The first operation data is obtained by converting an analog convolution result into data in the digital domain, and the analog convolution result is an output value of the convolution operation.

[0165] The processor 910 may output a first monitoring result obtained by classifying the abnormal condition based on the first operation data.

[0166] For example, the processor 910 may classify the abnormal condition of each object detected by the plurality of sensors, and output a monitoring result according to the classification result of the abnormal condition of each object.

[0167] According to an embodiment, the processor 910 may determine the category of the abnormal condition based on the abnormal condition classification result of each object, and output the monitoring result by classifying the risk level within the determined abnormal condition category.

[0168] According to an embodiment, the processor 910 may assign a weight to any object detected by the plurality of sensors, and assign a weight to the abnormal condition classification result of the weighted object.

[0169] According to an embodiment, the processor 910 may update the status information of the target area based on the first monitoring result.

[0170] (Second Operation Mode)

[0171] The processor 910 may select a target sensor from among a plurality of sensors installed in a target area in response to the second mode of detecting an object related to the situation information in real time being determined as the operation mode.

[0172] Here, the plurality of sensors may include at least one of a gas sensor, a pressure sensor, and a temperature sensor.

[0173] For example, the processor 910 may select a sensor that detects a primary object related to the situation information as a primary target sensor from among a plurality of sensors, and select a sensor that detects a secondary object related to the primary object as a secondary target sensor.

[0174] The processor 910 may process an object signal received from the object sensor and output second sensing data obtained by converting the object signal into a digital signal.

[0175] For example, the processor 910 may receive a target signal from the target sensor based on a second selection control signal in response to selecting the target sensor, amplify the target signal according to an amplification control signal and a gain control signal, and digitally convert the amplified target signal to output second sensing data.

[0176] The processor 910 may generate second input data to be input to the second neural network model by variably adjusting the size of the second sensing data.

[0177] For example, the processor 910 may accumulate the second sensing data of the corresponding number of bits and generate the second input data having an output resolution corresponding to the number of bits. The number of bits is determined based on an enable signal for specifying the number of bits enabled in the maximum resolution.

[0178] Here, the second neural network model is a model used in response to the second mode being determined as the operation mode. The second neural network model can be described as a model that performs an operation by considering input data at a time before the operation is performed.

[0179] The processor 910 may perform an operation on the second input data in the analog domain using the second neural network model, and output second operation data obtained by converting the operation result into digital data.

[0180] For example, the processor 910 may convert the second input data into a second input value in the analog domain, use a plurality of SRAM operators to perform a convolution operation on the second input value in the analog domain, and output second operation data. The second operation data is obtained by converting the second analog convolution result into data in the digital domain, and the second analog convolution result is the output value of the convolution operation.

[0181] The processor 910 may output a second monitoring result obtained by classifying the abnormal condition based on the second operation data.

[0182] For example, the processor 910 may determine the category of the abnormal condition in response to the target sensor detecting the object, and output the second monitoring result by classifying the risk level within the determined category of the abnormal condition.

[0183] According to an embodiment, the processor 910 may determine the category of the abnormal condition based on each abnormal condition classification result of the primary object and the secondary object, and output the second monitoring result considering the risk level classification of the primary object and the secondary object within the determined abnormal condition category.

[0184] According to an embodiment, the processor 910 may update the status information of the target area based on the second monitoring result.

[0185] The processor 910 may be implemented as an array of multiple logic gates, or may be implemented as a combination of a general-purpose microprocessor and a memory storing executable programs on the microprocessor. For example, the processor 910 may include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some cases, the processor 910 may include an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. For example, the processor 910 may refer to a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors and a DSP core, or a combination of processing devices, such as any other such configurations.

[0186] The memory 920 may include any non-temporary computer-readable storage medium. As an example, the memory 920 may include a random access memory (RAM) and a permanent mass storage device, such as a read-only memory (ROM), a disk drive, a solid-state drive (SSD), or a flash memory. As another example, a permanent mass storage device, such as a ROM, an SSD, a flash memory, or a disk drive, may be a separate persistent storage device different from the memory. In addition, an operating system (OS) and at least one program code (e.g., for execution by the processor 910, which will be referred to later) may be included. Figures 11 to 15 The code for the operation) may be stored in the memory 920.

[0187] These software components may be loaded from a computer-readable storage medium that is separate from the memory 920. The separate computer-readable storage medium may be a storage medium that is directly connected to the monitoring device 900. For example, the storage medium may include a computer-readable storage medium such as a floppy disk drive, a disk, a tape, a DVD / CD-ROM drive, or a memory card. Alternatively, the software components may be loaded into the memory 920 via the communication module 940 instead of a computer-readable storage medium. For example, at least one program may be based on a computer program that is loaded into the memory 920 (e.g., for execution by the processor 910, as described later). Figures 11 to 15 The computer program is a computer program installed by a file provided by a developer or a file distribution system through the communication module 940.

[0188] The communication module 930 may provide a configuration or function so that the plurality of sensors 811, 812, and 813 and the server 830 communicate with the monitoring device 900 through a network. In addition, the communication module 940 may provide a configuration or function so that the monitoring device 900 communicates with other external devices (e.g., user terminals 840 and 850). For example, control signals, commands, data, etc. provided under the control of the processor 910 may be transmitted to the server 830 and / or external devices through the communication module 940 and the network.

[0189] Meanwhile, according to the embodiment, although Fig. 9 Not shown, the monitoring device 900 may also include a display device. Alternatively, the monitoring device 900 may be connected to an independent display device via wired or wireless communication to send and receive data between them. For example, the display device may display the status information of the target area according to the monitoring results.

[0190] According to another embodiment, the monitoring device 900 may further include an input / output interface. The input / output interface may be a component that interfaces with an input or output device (e.g., a keyboard, a mouse, etc.) that may be connected to the monitoring device 900 or included in the monitoring device 900. The input / output interface may be implemented as an element configured separately from the processor 910, or may be configured to be included in the processor 910.

[0191] Fig.10 is a schematic diagram of another example of the internal structure of a monitoring device according to an embodiment.

[0192] refer to Fig.10 , the monitoring device may include a sensor signal processing unit 1020 , a microcontroller unit (MCU) 1030 , and an analog PIM unit 1040 .

[0193] According to an embodiment, the sensor signal processing unit 1020 may correspond to Figures 1 to 7 The sensor signal processor 300 described in Figures 1 to 7 A portion of the operations performed by the sensor signal processor 300 described in can be performed by the sensor signal processing unit 1020.

[0194] For example, the sensor signal processing unit 1020 may include a first MUX 1021 (multiplexer), a TIA 1022, an LNA 1023, a second MUX 1024, a VGA 1025, and a SDADC 1026. These may correspond to the first MUX 310 (multiplexer), the TIA 320, the LNA 330, the second MUX 350, the VGA 370, and the SDADC 390 of the sensor signal processor 300, respectively.

[0195] For example, the first MUX 1021 and the second MUX 1024 may receive a signal and select a signal. The TIA 1022 may convert a sensing signal in the form of a current into a voltage signal based on a first amplification control signal. The LNA 1023 may amplify the sensing signal in the form of a voltage based on a second amplification control signal to minimize noise. The VGA 1025 may amplify the voltage signal based on a dynamically adjusted gain control signal. The SDADC 1026 may digitally convert the amplified sensing signal and output sensing data. For example, the SDADC 1026 may convert an analog sensing signal into a digital signal and output the converted sensing data.

[0196] According to an embodiment, the sensor signal processing unit 1020 may receive sensing signals from a plurality of sensors installed in the target area when a first mode of simultaneously detecting different objects is determined as the operation mode.

[0197] According to an embodiment, the sensor signal processing unit 1020 may select a target sensor from among a plurality of sensors installed in a target area in response to the second mode of detecting an object related to the situation information in real time being determined as the operation mode.

[0198] Here, the plurality of sensors may include at least one of a gas sensor, a pressure sensor, and a temperature sensor.

[0199] According to an embodiment, the sensor signal processing unit 1020 may select a sensing signal based on a first selection control signal, amplify the selected sensing signal according to an amplification control signal and a gain control signal, and digitally convert the amplified sensing signal to output sensing data.

[0200] According to an embodiment, the first selection control signal may be a signal configured to perform a control operation to select the sensing signal in time series.

[0201] According to an embodiment, the first selection control signal may be a signal configured to perform a control operation to select any one of the sensing signals based on a predetermined period.

[0202] According to an embodiment, the sensor signal processing unit 1020 may select a sensor that detects a main object related to the situation information as a main target sensor, and select a sensor that detects a secondary object related to the main object as a secondary target sensor. The sensor signal processing unit 1020 may process a target signal from the target sensor and obtain second sensing data by converting the target signal into a digital signal.

[0203] According to an embodiment, the sensor signal processing unit 1020 may receive a target signal from the target sensor based on a second selection control signal in response to selecting a target sensor, amplify the target signal according to an amplification control signal and a gain control signal, and digitally convert the amplified target signal to output second sensing data.

[0204] According to an embodiment, the sensor signal processing unit 1020 may generate first input data to be input to the first neural network model by variably adjusting the size of the first sensing data. Alternatively, the sensor signal processing unit 1020 may generate second input data to be input to the second neural network model by variably adjusting the size of the second sensing data. For example, the sensor signal processing unit 1020 may accumulate the first sensing data (or the second sensing data) of the corresponding number of bits, and generate the first input data (or the second input data) having an output resolution corresponding to the number of bits. The number of bits is determined based on an enable signal, which is used to specify the number of bits enabled in the maximum resolution.

[0205] According to an embodiment, the MCU 1030 may correspond to Figures 1 to 7 CNN controller 2000 described in. Figures 1 to 7 A portion of the operations performed by the CNN controller 2000 described in the accompanying drawings may be performed by the MCU 1030.

[0206] According to an embodiment, the MCU 1030 may determine an operation mode based on the status information of the target area where the abnormal condition is detected. For example, the MCU 1030 may use any one of the first mode and the second mode as the operation mode. The first mode detects different objects at the same time. The second mode detects objects related to the status information in real time.

[0207] According to an embodiment, the MCU 1030 may output a first monitoring result obtained by classifying the abnormal condition based on the first operation data.

[0208] For example, the MCU 1030 may classify the abnormal condition of each object detected by the plurality of sensors, and output the monitoring result according to the classification result of the abnormal condition of each object.

[0209] According to an embodiment, the MCU 1030 may determine the category of the abnormal condition based on the abnormal condition classification result of each object, and output the monitoring result by classifying the risk level within the determined abnormal condition category.

[0210] According to an embodiment, the MCU 1030 may assign a weight to any object detected by the plurality of sensors, and assign a weight to an abnormal condition classification result of the weighted object.

[0211] According to an embodiment, the MCU 1030 may output a second monitoring result obtained by classifying the abnormal condition based on the second operation data.

[0212] For example, the MCU 1030 may determine the category of the abnormal condition in response to the target sensor detecting the object, and output the second monitoring result by classifying the risk level within the determined category of the abnormal condition.

[0213] According to an embodiment, the MCU 1030 may determine the category of the abnormal condition based on each abnormal condition classification result of the primary object and the secondary object, and output the second monitoring result considering the risk level classification of the primary object and the secondary object within the determined abnormal condition category.

[0214] According to an embodiment, the MCU 1030 may update the status information of the target area according to the first monitoring result or the second monitoring result.

[0215] The analog PIM unit 1040 may include an analog PIM (analog processor in memory) 1041 and a digital controller 1042. Here, the analog PIM 1041 may correspond to Figures 1 to 7 The analog PIM 1100 described in. Figures 1 to 7 A portion of the operations performed by the analog PIM 1100 described in the above may be performed by the analog PIM 1041. In addition, the digital controller 1042 may correspond to Figures 1 to 7 The analog PIM controller 1500 described in. Figures 1 to 7 A portion of the operations performed by the analog PIM controller 1500 described in can be performed by the digital controller 1042.

[0216] According to an embodiment, the analog PIM unit 1040 may perform an operation on the first input data in the analog domain using the first neural network model, and output first operation data obtained by converting the operation result into digital data.

[0217] For example, the analog PIM unit 1040 may convert the first input data into an input value in the analog domain, perform a convolution operation on the input value in the analog domain using a plurality of SRAM operators, and output first operation data, wherein the first operation data is obtained by converting the analog convolution result into data in the digital domain, and the analog convolution result is the output value of the convolution operation.

[0218] According to an embodiment, the analog PIM unit 1040 may perform an operation on the second input data in the analog domain using the second neural network model, and output second operation data obtained by converting the operation result into digital data.

[0219] For example, the analog PIM unit 1040 may convert the second input data into an input value in the analog domain, perform a convolution operation on the input value in the analog domain using a plurality of SRAM operators, and output second operation data. The second operation data is obtained by converting the second analog convolution result into data in the digital domain, and the second analog convolution result is the output value of the convolution operation.

[0220] Fig.11 is a flowchart describing an example of a method for monitoring abnormal conditions based on multiple sensors according to an embodiment.

[0221] In step 111 , the monitoring device may process a sensing signal and output sensing data, wherein the sensing data is obtained by converting the sensing signal into a digital signal, wherein the sensing signal is received from a plurality of sensors, and the plurality of sensors detect different objects.

[0222] The plurality of sensors may be sensors for detecting objects related to the IoT system, as described above. Figure 8 In addition, the multiple sensors detect multiple objects respectively, and the multiple objects may be objects related to the abnormal conditions monitored by the monitoring system.

[0223] For example, the monitoring device may receive sensing signals from a gas sensor, a pressure sensor, and a temperature sensor. In other words, the plurality of sensors may include at least one of a gas sensor, a pressure sensor, and a temperature sensor.

[0224] Meanwhile, although one sensor is described as detecting a specific object for the convenience of description, the present invention is not limited thereto. For example, when the monitoring device receives a sensing signal related to a gas from a gas sensor, the monitoring device may receive sensing signals related to the gas from a plurality of gas sensors. In other words, the monitoring system may include a plurality of gas sensors, each of which is configured to detect a gas.

[0225] The monitoring device can perform signal processing on the sensing signal and output the processing result as sensing data. The signal processing process includes selecting one of the sensing signals, amplifying the selected sensing signal and performing digital conversion on the amplified sensing signal.

[0226] In more detail, the monitoring device can select one of the received sensing signals based on the selection control signal. In this regard, the method for the monitoring device to select the sensing signal will be described in Fig.12 is described in .

[0227] Fig.12 is a schematic diagram for describing an example of sensing data according to an embodiment.

[0228] refer to Fig.12In the first signal processing example 1210, the second signal processing example 1220, and the third signal processing example 1230 shown in FIG. 1 , the monitoring device may receive a sensing signal A, a sensing signal B, and a sensing signal C from a plurality of sensors that detect different objects.

[0229] In the present disclosure, since the monitoring device monitors a plurality of sensors simultaneously, ie, performs signal processing on sensing signals received from the plurality of sensors simultaneously, it may be necessary to assign priorities to determine which received sensing signals will be processed first.

[0230] According to an embodiment, the selection control signal may be a signal configured to perform a control operation to select a sensing signal in a time series. In other words, the monitoring device may select one of the sensing signals over time. For example, when the monitoring device receives sensing signals from a plurality of sensors within a predetermined time, the monitoring device may divide the predetermined time and select a sensing signal to be received during each time interval.

[0231] According to an embodiment, the selection control signal may be a signal configured to perform a control operation to select one of the sensing signals based on a predetermined period. For example, the predetermined period may be a value preset by an administrator or user of the monitoring system. For example, in the first signal processing example 1210, the monitoring device may select one of the received sensing signals based on the selection control signal, and the selection control signal is configured to perform a control operation to select the sensing signal based on a period of 10 milliseconds. Therefore, the monitoring device may select sensing signal A 1211 from 0 seconds to 10 milliseconds, sensing signal B 1212 from 10 milliseconds to 20 milliseconds, and sensing signal C 1213 from 20 milliseconds to 30 milliseconds.

[0232] Here, the selection control signal may be a signal that controls the selection of the sensing signal so as to adjust the frequency at which each sensing signal is selected. In more detail, the frequency at which each sensing signal is selected may be determined based on the weight assigned to each sensing signal. For example, in the second signal processing example 1220, the monitoring device may assign the maximum weight to the sensing signal A 1221 in the received sensing signal, and select the sensing signal based on the selection control signal that controls the selection of the sensing signal, so that the sensing signal A 1221 is selected at the highest frequency. Therefore, in the second signal processing example 1220, the monitoring device may select the sensing signal A 1211 from 0 seconds to 10 milliseconds, select the sensing signal B 1212 from 10 milliseconds to 20 milliseconds, select the sensing signal A 1211 again from 20 seconds to 30 milliseconds, and select the sensing signal C 1213 from 30 milliseconds to 40 milliseconds.

[0233] Meanwhile, the weight of each sensing signal may be assigned according to the importance of the object of the sensing signal, i.e., the object detected by the sensor and related to the abnormal condition. For example, when the importance of monitoring temperature is high, a high weight may be assigned to the temperature sensor or the sensing signal received from the temperature sensor. As another example, when the importance of monitoring gas is high, a high weight may be assigned to the gas sensor or the sensing signal received from the gas sensor.

[0234] According to another embodiment, the selection control signal may be a signal selected by the control signal so as to adjust the time interval of the received signal for each sensing signal. More specifically, the time interval in which each sensing signal is received may be determined based on the weight assigned to each sensing signal. For example, in the third signal processing example 1230, the monitoring device may assign the maximum weight to the sensing signal B 1232 in the received sensing signal, and thus may select the sensing signal based on the selection control signal selected by the control signal so that the time interval in which the sensing signal B 1232 is received is the longest. Therefore, in the third signal processing example 1230, the monitoring device may select the sensing signal A 1231 from 0 seconds to 5 milliseconds, select the sensing signal B 1232 from 5 milliseconds to 25 milliseconds, and select the sensing signal C 1233 from 25 milliseconds to 30 milliseconds.

[0235] at the same time, Fig.12 The sensing signals 1211, 1212, 1213, etc. shown in the first signal processing example 1210, the second signal processing example 1220, and the third signal processing example 1230 may refer to sensing data converted into digital signals. For example, in the first signal processing example 1210, the sensing data A may be sensing data obtained by amplifying the sensing signal A based on the amplification control and gain control signals, and performing digital conversion on the amplified sensing signal. Similarly, the sensing data B and the sensing data C may also be sensing data in digital form, respectively. That is, in the first signal processing example 1210, the sequence data connected in time series may be data obtained by connecting digital sensing data. The monitoring device may use the corresponding sequence data to generate input data to be input into the neural network model. In this regard, it will be described again below.

[0236] At the same time, the monitoring device can adjust the signal level of the sensing signal (which is an analog signal) before converting the received sensing signal into a digital signal. For example, since the analog sensing signal received from the gas sensor and the analog sensing signal received from the temperature sensor have different signal levels (or signal amplitudes, etc.), the monitoring device can adjust the signal level to meet the input conditions for converting each sensing signal into a digital signal. To this end, the monitoring device can amplify the selected sensing signal based on the amplification control signal and the gain control signal. As an example, the monitoring device can convert the sensing signal in the form of current into a voltage signal based on the first amplification control signal, and amplify the converted voltage signal based on the dynamically adjusted gain control signal. As another example, the monitoring device can amplify the sensing signal in the form of voltage based on the second amplification control signal to minimize noise, and amplify the sensing signal based on the dynamically adjusted gain control signal.

[0237] The monitoring device may output the sensing data by digitally converting the amplified sensing signal. For example, the monitoring device may convert the analog sensing signal into a digital signal and output the converted sensing data.

[0238] Reference again Fig.11 In step 113, the monitoring device may generate input data, wherein the input data is obtained by variably adjusting the size of the sensing data, and the input data is used to be input into the neural network model.

[0239] Regarding step 113, Fig.13 A more detailed description is given.

[0240] Fig.13 is a schematic diagram describing an example of input data according to an embodiment.

[0241] As described above, the sensing data refers to data obtained by converting the sensing signal into a digital signal. For example, the monitoring device may generate sequence data by connecting the sensing data and adjusting the size of the sequence data to correspond to the size of the input of the neural network model, thereby generating input data.

[0242] For example, the monitoring device can accumulate sensing data of a corresponding number of bits and generate input data having an output resolution corresponding to the number of bits. The number of bits is determined based on an enable signal, which is used to specify the number of bits enabled in the maximum resolution. For example, the monitoring device can generate input data using digital sensing data, and the maximum resolution of the digital sensing data is 8 bits, 16 bits, or 32 bits.

[0243] As an example, the monitoring device may generate the first input data 1310 by accumulating (or connecting to the sequence data) the sensing data 1321 based on the first enable signal 1311. Here, the first enable signal 1311 may be an enable signal that specifies the number of bits (maximum number of bits) corresponding to the maximum resolution. In other words, the monitoring device may generate the first input data 1310 by accumulating the sensing data 1321 until the maximum number of bits.

[0244] As another example, the monitoring device may generate the second input data 1320 by accumulating (or connecting to the sequence data) the sensing data 1322 based on the second enable signal 1312. Here, the second enable signal 1312 may be an enable signal that specifies a number of bits lower than the maximum resolution. In other words, the monitoring device may generate the second input data 1320 by accumulating the sensing data 1322 to a predetermined number of bits.

[0245] Reference again Fig.11 In step 115, the monitoring device may use a neural network model to perform an operation on the input data in an analog domain and output operation data, wherein the operation data is obtained by converting the operation result into digital data.

[0246] For example, the monitoring device may convert the input data into an input value in the analog domain. In other words, the monitoring device may convert the digital input data into an input value (which is analog input data) and output the converted input value to the SRAM operator.

[0247] In addition, the monitoring device can use multiple SRAM operators to perform convolution operations on input values ​​in the analog domain. Here, the structure and operation method of multiple SRAM operators have been Figure 4 Described in detail in.

[0248] In addition, the monitoring device can output operation data, which is obtained by converting the analog convolution result into data in the digital domain, and the analog convolution result is the output value of the convolution operation. For example, the monitoring device can convert the analog convolution result signal calculated by the SRAM operator into operation data, that is, digital convolution data, and output the converted operation data.

[0249] Meanwhile, according to an embodiment, the monitoring device may include an analog PIM unit 1040. Fig.10 described, at least in part, in Figures 1 to 7 The operations described in hereinafter and performed by analog PIM module 1000, analog PIM 1100, and analog PIM controller 1500 are performed by a monitoring device.

[0250] Reference again Fig.11In step 117, the monitoring device may output a monitoring result, wherein the monitoring result is obtained by classifying the abnormal condition according to the operation data.

[0251] For example, the monitoring device may classify the abnormal condition of each object detected by multiple sensors, and output the monitoring result according to the classification result of the abnormal condition of each object.

[0252] An example of outputting monitoring results based on sensing signals received from a gas sensor, a temperature sensor, and a pressure sensor will be described below according to Table 1.

[0253] Table 1

[0254]

[0255] For example, referring to Table 1, the monitoring device can classify each abnormal condition of gas, temperature and pressure detected by the gas sensor, temperature sensor and pressure sensor as normal, abnormality detected and risk. In addition, the monitoring device can output monitoring results such as fire risk based on the abnormal condition classification results of each object.

[0256] Meanwhile, according to an embodiment, the monitoring device may determine the category of the abnormal condition based on the abnormal condition classification result of each object. For example, the monitoring device may determine the category of the abnormal condition as a category related to fire in response to the abnormal condition classification result of fire being any one of abnormality or risk detected.

[0257] In addition, the monitoring device may output the monitoring result by classifying the risk level in the determined abnormal condition category. For example, the monitoring device may classify the risk level as an initial fire risk in response to the abnormal condition classification result of fire as abnormality detected, or may classify the risk level as a fire risk in response to the abnormal condition classification result of fire as risk.

[0258] According to another embodiment, the monitoring device may assign a weight to any of the objects detected by the multiple sensors, and assign a weight to the abnormal condition classification results of the weighted objects. For example, the monitoring device may comprehensively consider the abnormal condition classification results of the temperature detected by the temperature sensor and the abnormal condition classification results of the gas detected by the gas sensor to determine that the fire is the category of the abnormal condition. Or comprehensively consider the abnormal condition classification results of the gas detected by the gas sensor and the abnormal condition classification results of the pressure detected by the pressure sensor to determine that the gas leak is the category of the abnormal condition. However, the abnormal condition associated with the fire and the abnormal condition associated with the gas leak have a strong relationship with the temperature (or the abnormal condition classification result of the temperature) and the gas (or the abnormal condition classification result of the gas), respectively. Therefore, the monitoring device can assign a weight to the corresponding object (or the abnormal condition classification result of the object). For example, the monitoring device can determine the category of the abnormal condition or classify the risk level of the abnormal condition by further considering the weight applied to the object (or the abnormal condition classification result of the object).

[0259] Fig.14 is a flowchart describing another example of a method for monitoring abnormal conditions based on multiple sensors according to an embodiment.

[0260] refer to Fig.14 In step 1410, the monitoring device may determine an operation mode based on the status information of the target area where the abnormal condition is detected.

[0261] Here, the condition information of the target area refers to condition information related to the area where the abnormal condition is monitored by multiple sensors. For example, when any object detected by the multiple sensors deviates from the normal condition, the condition information of the target area may be in a state where the abnormal condition is detected. Specifically, the condition information of the target area may be a monitoring result, including the category, risk level, etc. of the object deviating from the normal condition.

[0262] Alternatively, when all objects detected by the multiple sensors are in a normal state, the state information of the target area may be in a normal state. Here, the normal state of the object may refer to a normal state, and the deviation of the object from the normal state may mean that the risk level of the object is an abnormal state or a risk state is detected.

[0263] For example, the monitoring device may determine the first mode as the operation mode in response to the target area being in a normal state, and the first mode detects different objects simultaneously. As another example, the monitoring device may determine the second mode as the operation mode in response to the target area being in an abnormal state, and the second mode detects objects related to the condition information in real time.

[0264] Hereinafter, steps 1421, 1431, 1441, 1451, 1461 and 1471 show an example of the operation flow of the monitoring device operating in the first mode, and steps 1422, 1432, 1442, 1452, 1462 and 1472 show an example of the operation flow of the monitoring device operating in the second mode. Meanwhile, in the following, the operation of the monitoring device operating in the first mode may include Fig.11 The monitoring equipment described in this section operates as part of the Fig.11 The operation of the monitoring device described in Fig.14 The monitoring device operates in the first mode.

[0265] In step 1421 , the monitoring device may receive sensing signals from a plurality of sensors installed in the target area in response to the first mode being determined as the operation mode.

[0266] In step 1431 , the monitoring device may process the sensing signal and output first sensing data, where the first sensing data is obtained by converting the sensing signal into a digital signal.

[0267] In step 1441, the monitoring device may generate first input data, where the first input data is obtained by variably adjusting the size of the first sensing data, and the first input data is used to input into a first neural network model.

[0268] In step 1451, the monitoring device may use the first neural network model to perform an operation on first input data in the analog domain and output first operation data, where the first operation data is obtained by converting the operation result into digital data.

[0269] In step 1461, the monitoring device may output a first monitoring result, where the first monitoring result is obtained by classifying the abnormal condition according to the first operating data.

[0270] In step 1471, according to an embodiment, the monitoring device may update the status information of the target area based on the first monitoring result. For example, when the status information of the target area is updated to (maintain) a normal state, the monitoring device may operate in the first operation mode. Alternatively, when the status information of the target area is updated to (or changed to) a state in which an abnormality is detected, the monitoring device may operate in the second operation mode.

[0271] Likewise, in step 1422, the monitoring device may select a target sensor from a plurality of sensors installed in the target area in response to the second mode being determined as the operation mode, the second mode detecting an object related to the condition information in real time.

[0272] That is, the target sensor may be a sensor that detects an object related to the condition information. For example, when the condition information of the target area is an initial fire risk state, the monitoring device may determine that the second mode is the operation mode and select the temperature sensor as the target sensor to monitor the temperature sensor related to the fire in real time.

[0273] Meanwhile, according to an embodiment, the monitoring device may select a sensor that detects a main object related to the condition information as a main target sensor from among a plurality of sensors, and select a secondary target sensor that detects a secondary object related to the main object.

[0274] For example, when the status information of the target area is an initial fire risk state, the monitoring device can determine the temperature as the primary object to select the temperature sensor as the primary target sensor. And determine the gas related to the temperature as the secondary object to select the gas sensor as the secondary target sensor. In other words, according to the embodiment, even when operating in the second mode, the monitoring device can detect the object that finds the abnormal condition in real time and continuously monitor the objects associated with the object.

[0275] Thereafter, in step 1432 , the monitoring device may process the target signal received from the target sensor and output second sensing data, where the second sensing data is obtained by converting the target signal into a digital signal.

[0276] Meanwhile, the processing process of outputting the second sensing data based on the target signal can be Fig.11 The processing of outputting sensing data based on sensing signals described in the above is the same as that described in the above. Therefore, repeated descriptions will be omitted. For example, the monitoring device can be Fig.11 In the same manner as described in , the target signal is amplified based on the amplification control signal and the gain control signal, or the amplified target signal is digitally converted to output second sensing data.

[0277] According to an embodiment, the monitoring device may receive a target signal from the target sensor based on a second selection control signal in response to the target sensor being selected. Fig.15 Give a description.

[0278] Fig.15 is a schematic diagram for describing other examples of sensing data according to an embodiment.

[0279] Reference Fig.15 In the first signal processing example 1510 and the second signal processing example 1520 shown in FIG. 1 , the monitoring device may receive a target signal from a target sensor. Here, target signals A 1511 and 1521 may be primary target signals received from a primary target sensor, and target signal B 1522 and target signal C 1523 may be secondary target signals received from a secondary target sensor.

[0280] Fig.11 The description of the selection control signal in the above may be applied to the second selection control signal without change. For example, the second selection control signal may be a signal configured to perform a control operation to select the target signal in a time series. As another example, the second selection control signal may be a signal configured to perform a control operation to select the target signal based on a predetermined period.

[0281] According to another embodiment, the second selection control signal may be a signal selected by the control signal, and is used to adjust the time intervals for receiving the primary target signal and the secondary target signal. For example, in the first signal processing example 1510, the monitoring device may select the target signal according to the second selection control signal, so that the time interval for selecting the target signal A 1511 as the primary target signal is longer than the time interval for selecting the target signal B 1512 as the secondary target signal. Therefore, the monitoring device may select the target signal A 1511 from 0 milliseconds to 15 milliseconds, and select the target signal B 1512 from 15 milliseconds to 20 milliseconds.

[0282] According to another embodiment, the second selection control signal may be a signal for controlling the selection of the target signal to adjust the frequency of selecting the main target signal and the secondary target signal. For example, in the second signal processing example 1520, the monitoring device may select the target signal according to the second selection control signal so that the frequency of selecting the target signal A 1521 as the main target signal is higher than the frequency of selecting the target signal B 1522 and the target signal C 1523 as the secondary target signal. Therefore, the monitoring device may select the target signal A 1521 from 0 milliseconds to 10 milliseconds, select the target signal B 1522 from 10 milliseconds to 20 milliseconds, select the target signal A 1521 again from 20 milliseconds to 30 milliseconds, and select the target signal C 1523 from 30 milliseconds to 40 milliseconds.

[0283] Reference again Fig.14 In step 1442, the monitoring device may generate second input data, where the second input data is obtained by variably adjusting the size of the second sensing data, and the second input data is used to input into a second neural network model.

[0284] At the same time, the process of generating input data based on sensor data can be combined with Fig.11 The process described in is the same, so repeated description will be omitted. For example, according to an enable signal specifying the number of bits enabled within the maximum resolution, the monitoring device can accumulate second sensing data corresponding to the number of bits and generate second data having an output resolution corresponding to the number of bits.

[0285] According to an embodiment, in response to the second mode being determined as the operation mode, the monitoring device may change the neural network model used from the first neural network model used in the first mode to the second neural network model. Here, the second neural network model may be described as a model that performs an operation by considering input data at a time before the operation is performed. That is, according to the embodiment, the second neural network model is a neural network model with a cyclic structure, and when the second neural network model is used, temporal continuity or sequence may be considered, so the second neural network may be more suitable for the second operation mode of intensively monitoring the target sensor.

[0286] For example, the first neural network model may be a CNN model, and the second neural network model may be an RNN model. At the same time, the example of the second neural network model is not limited to the RNN model, and any model with a recurrent neural network structure, such as a long short-term memory (LSTM) model or a gated recurrent unit (GRU) model, may correspond to the second neural network model.

[0287] Then, in step 1452, the monitoring device may use the second neural network model to perform an operation on the second input data in the analog domain, and output second operation data obtained by converting the operation result into digital data. Similarly, the process of operating the second input data using the second neural network model may be similar to Fig.11 The operation process described in is the same, so repeated description will be omitted.

[0288] For example, the monitoring device can convert the second input data into a second input value in the analog domain, use multiple SRAM operators to operate the second input value in the analog domain, and output second operation data, where the second operation data is obtained by converting the second analog convolution result (i.e., the output value of the convolution operation) into digital domain data.

[0289] Then, in step 1462, the monitoring device may output a second monitoring result, wherein the second monitoring result is obtained by classifying the abnormal condition according to the second operation data. Similarly, the output process of the second monitoring result of the target sensor may be similar to that of Fig.11 The output process of the first monitoring result described in is the same, so the repeated description will be omitted.

[0290] For example, the monitoring device may determine the category of the abnormal condition according to the object detected by the target sensor, and output the second monitoring result by classifying the risk level within the determined abnormal condition category. For example, when the target sensor is a temperature sensor, the monitoring device may determine a fire related to temperature as the category of the abnormal condition. In addition, the monitoring device may classify the risk level related to the fire into safety, initial risk level, risk level, etc. according to processing the target signal received from the temperature sensor.

[0291] According to an embodiment, the monitoring device can determine the category of the abnormal condition based on the abnormal condition classification result of each object in the main object related to the condition information and the secondary object related to the main object. For example, when the main object is temperature and the secondary object is gas, the monitoring device can determine fire, etc. as the category of the abnormal condition.

[0292] In addition, the monitoring device may consider the risk level classification of the main object and the risk level classification of the secondary object within the abnormal condition category to output the second monitoring result. More specifically, the monitoring device may assign a higher weight to the risk level classification of the main object than to the risk level classification of the secondary object. For example, when the risk level of the main object and the risk level of the secondary object are both safety levels, the monitoring device may output information indicating that the target area is in a normal state as the second monitoring result. As another example, when monitoring abnormal conditions related to fire, when the risk level associated with temperature (main object) is an initial risk and the risk level associated with gas (secondary object) is a risk, the monitoring device may output information indicating that the target area has an initial fire risk as the second monitoring result. As another example, when the risk level associated with temperature (main object) is a risk and the risk level associated with gas (secondary object) is an initial risk, the monitoring device may output information indicating that the target area is at risk of fire as the second monitoring result.

[0293] Reference again Fig.14 In step 1472, the monitoring device may update the status information of the target area according to the second monitoring result.

[0294] For example, when the status information of the target area is updated to (changed to) a normal state, the monitoring device can operate in the first operation mode. In addition, when the status information of the target area is updated to (or maintained to) a state where an abnormality is detected, the monitoring device can operate in the second operation mode.

[0295] Unless there is a clear order description or contrary description for the steps included in the method according to the present invention, the steps can be performed in any suitable order. The present invention is not necessarily limited by the order of the steps. Any embodiments or illustrative terms (such as, etc.) in the present invention are only used to describe the present invention in detail, and unless limited by the claims, the scope of the present invention is not limited by the embodiments or illustrative terms. In addition, those skilled in the art will recognize that various modifications, combinations and changes can be made according to design conditions and factors within the scope of the attached claims or their equivalents. .

[0296] Therefore, the spirit of the present invention should not be limited to the embodiments described above, but rather to the following claims. All equivalent or equivalent variations to the claims belong to the scope of the spirit of the present invention.

Claims

1. A method for monitoring abnormal conditions based on multiple sensors, the method comprising: Processing a sensing signal and outputting sensing data, wherein the sensing data is obtained by converting the sensing signal into a digital signal, wherein the sensing signal is received from a plurality of sensors, and the plurality of sensors are configured to detect different objects; generating input data, wherein the input data is obtained by variably adjusting the size of the sensing data, and the input data is used to be input into a neural network model; Using the neural network model to perform operations on the input data in the analog domain, and outputting operation data, wherein the operation data is obtained by converting the operation results into digital data; A monitoring result is output, wherein the monitoring result is obtained by classifying the abnormal condition according to the operation data.

2. The method according to claim 1, wherein: The plurality of sensors include at least one of a gas sensor, a pressure sensor, and a temperature sensor.

3. The method according to claim 1, wherein: The output sensing data includes: selecting a sense signal based on a selection control signal; amplifying the selected sensing signal based on the amplification control signal and the gain control signal; The amplified sensing signal is digitally converted to output the sensing data.

4. The method according to claim 3, wherein: The selection control signal is a signal configured to perform a control operation to select the sensing signal in time series.

5. The method according to claim 3, wherein: The selection control signal is a signal configured to perform a control operation to select any one of the sensing signals based on a predetermined period.

6. The method according to claim 1, wherein: The generating of input data comprises: According to an enable signal designating a bit number, the sensing data is accumulated to correspond to the bit number enabled within a maximum resolution, and the input data having an output resolution corresponding to the bit number is generated.

7. The method according to claim 1, wherein: The output operation data includes: converting input data into input values ​​in the simulation domain; performing a convolution operation on the input value in an analog domain using a plurality of SRAM operators; and, The operation data is output, where the operation data is obtained by converting the analog convolution result into data in the digital domain, and the analog convolution result is the output value of the convolution operation.

8. The method according to claim 1, wherein: The output monitoring result includes: Classify the abnormal condition of each object detected by multiple sensors; According to the abnormal condition classification results of each object, the monitoring results are output.

9. The method according to claim 8, wherein: The output monitoring result includes: Determine the category of the abnormal condition according to the abnormal condition classification result of each object; The monitoring result is outputted by classifying the risk level within the determined abnormal condition category.

10. The method according to claim 8, further comprising assigning a weight to any object detected by the plurality of sensors, and assigning a weight to an abnormal condition classification result of the weighted object.

11. An abnormal condition monitoring device based on multiple sensors, comprising: a memory configured to store a program; A processor configured to execute the program, wherein the processor is configured to: Processing a sensing signal and outputting sensing data, wherein the sensing data is obtained by converting the sensing signal into a digital signal, wherein the sensing signal is received from a plurality of sensors, and the plurality of sensors are configured to detect different objects; generating input data, wherein the input data is obtained by variably adjusting the size of the sensing data, and the input data is used to be input into a neural network model; Using the neural network model to perform operations on the input data in the analog domain, and outputting operation data, wherein the operation data is obtained by converting the operation results into digital data; A monitoring result is output, wherein the monitoring result is obtained by classifying the abnormal condition according to the operation data.

12. A computer-readable storage medium having stored thereon a program for executing the method according to claim 1 on a computer.

Images