Method, apparatus, and computer readable medium for monitoring abnormal conditions based on multiple sensors according to operating modes

Abstract

An abnormal condition monitoring method based on multiple sensors includes: determining an operation mode based on condition information in a target area in which an abnormal condition is monitored; in response to a first mode in which different objects are simultaneously detected being determined as the operation mode, receiving sensing signals from a plurality of sensors installed within the target area; processing the sensing signal and outputting first sensing data obtained by converting the sensing signal into a digital signal; generating first input data to be input to a first neural network model by variably adjusting a size of the first sensing data; performing an operation on the first input data in an analog domain using the first neural network model, and outputting first operation data obtained by converting an operation result into digital data; and outputting a first monitoring result obtained by classifying an abnormal condition based on the first operation data.

Description

Technical Field

[0001] The present disclosure relates to a method, apparatus, and computer-readable medium for monitoring abnormal conditions based on multiple sensors according to an operating mode. Background Art

[0002] With the advancement of electronics and communication technologies, various IoT (Internet of Things) sensor devices have been applied in various fields. IoT sensor devices are installed in various environments and locations, such as smart homes, factories, public places, and roads, to sense various external environments.

[0003] The IoT sensor device senses an external environment through a sensor, a processor, and a communication chipset included therein, and processes the sensed data or transmits the sensed data to the outside through the communication chipset.

[0004] IoT systems can receive and process sensing data from multiple IoT sensors. In this case, when information is processed by connecting one sensor to one circuit, it is inevitable that in order to obtain one piece of information through information from multiple sensors, the device size will increase, and therefore the design cost may increase. Therefore, it is necessary to study a method of processing signals so that multiple sensors can be processed in one circuit.

[0005] The above-mentioned background technology is technical information possessed by the inventor in order to obtain the present invention or obtained in the process of obtaining the present invention, and is not necessarily regarded as known technology disclosed to the public before submitting the present application. Summary of the invention

[0006] According to some embodiments of the present disclosure, a method, device and computer-readable medium for monitoring abnormal conditions based on multiple sensors according to an operating mode are provided. The tasks to be solved by the present invention are not limited to the above tasks, and other tasks and advantages of the present invention not mentioned herein can be understood by the following description and can be more clearly understood by the embodiments of the present invention. In addition, it can be seen that the tasks and advantages to be solved by the present invention can be achieved by the means shown in the claims and their combinations.

[0007] As a technical means for achieving the above-mentioned technical tasks, the first aspect of the present disclosure is a method for monitoring abnormal conditions based on multiple sensors, comprising: determining an operation mode based on condition information in a target area where the abnormal condition is monitored; in response to a first mode in which different objects are simultaneously detected being determined as the operation mode, receiving sensing signals from multiple sensors installed in the target area; processing the sensing signals and outputting first sensing data obtained by converting the sensing signals into digital signals; generating first input data to be input to a first neural network model by variably adjusting the size of the first sensing data; performing operations on the first input data in an analog domain using the first neural network model, and outputting first operation data obtained by converting the operation results into digital data; and outputting a first monitoring result obtained by classifying the abnormal condition based on the first operation data.

[0008] A second aspect of the present disclosure is an abnormal condition monitoring device based on multiple sensors, comprising a memory configured to store at least one program and a processor configured to execute the at least one program, wherein the processor is configured to: determine an operation mode based on condition information in a target area for monitoring abnormal conditions; in response to a first mode in which different objects are detected simultaneously being determined as the operation mode, process sensing signals received from multiple sensors installed in the target area; output first sensing data obtained by converting the sensing signals into digital signals; generate first input data to be input to a first neural network model by variably adjusting the size of the first sensing signal; perform an operation on the first input data in an analog domain using the first neural network model, and output first operation data obtained by converting the operation result into digital data; and output a first monitoring result obtained by classifying the abnormal condition based on the first operation data.

[0009] A third aspect of the present disclosure is a computer-readable medium on which a program for executing the method according to the first aspect on a computer is stored.

[0010] In addition, another method for implementing the present invention, another system, and a computer-readable medium storing a computer program for executing the method may also be provided.

[0011] In addition to the aspects, features, and advantages described above, further aspects, features, and advantages will become apparent from the following drawings, claims, and detailed description of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other objects, features and other advantages of the present invention will become more apparent through the following detailed description made with reference to the accompanying drawings, in which:

[0013] Figure 1 is a diagram showing a monitoring system including a sensor device according to an embodiment;

[0014] Figure 2 is a block diagram showing an example of the internal configuration of a signal processing circuit according to the embodiment;

[0015] 3 is a block diagram showing an example of an internal configuration of a power supply and a sensor signal processor according to an embodiment;

[0016] Figure 4 is a block diagram showing an example of an internal configuration of an analog PIM module according to an embodiment;

[0017] 5 is a diagram for describing the internal structure and operation of an analog PIM according to an embodiment;

[0018] Figure 6 is a diagram for describing an example of an analog PIM controller according to an embodiment;

[0019] Figure 7 is a flow chart describing an illustrative control flow for determining output matching with any sensor and calculating resolution according to an embodiment;

[0020] Figure 8 is a diagram for describing an example of a system for monitoring an abnormal condition based on a plurality of sensors according to an embodiment;

[0021] Fig. 9 is a diagram for describing an example of an internal configuration of a monitoring device according to an embodiment;

[0022] Fig.10 is a diagram for describing another example of the internal configuration of the monitoring device according to the embodiment;

[0023] Fig.11 is a flowchart for describing an example of a method for monitoring abnormal conditions based on multiple sensors according to an embodiment;

[0024] Fig.12 is a diagram for describing an example of sensing data according to an embodiment;

[0025] Fig.13 is a diagram for describing an example of input data according to an embodiment;

[0026] Fig.14 is a flowchart for describing another example of a method for monitoring abnormal conditions based on multiple sensors according to an embodiment; and

[0027] Fig.15 is a diagram for describing other examples of sensing data according to an embodiment. DETAILED DESCRIPTION

[0028] By referring to the embodiments described in detail in conjunction with the accompanying drawings, the advantages and features of the present invention and the methods for achieving these advantages and features will become clear. However, the present invention is not limited to the embodiments given below, and can be implemented in various forms, and it should be understood that the present invention includes all conversions, equivalents and substitutes included in the spirit and technical scope of the present invention. The embodiments given below are provided to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the present invention. When describing the present invention, when it is determined that the detailed description of the relevant known technology may confuse the main points of the present invention, the detailed description will be omitted.

[0029] The terms used in this application are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly stipulates otherwise. In this application, it should be understood that terms such as "including" or "having" are intended to represent the presence of features, numbers, steps, operations, elements, parts or combinations thereof described in the specification, and do not exclude the possibility of adding or having one or more other features, numbers, steps, operations, elements, parts or combinations thereof. In some embodiments, the two terms "operation" and "operation" can be replaced with each other.

[0030] Some embodiments of the present disclosure can be represented by function block configuration and various processing steps. Some or all of these function blocks can be implemented with various numbers of hardware and / or software configurations for performing specific functions. For example, the function blocks of the present disclosure can be implemented by one or more microprocessors or by circuit configurations for certain functions. In addition, for example, the function blocks of the present disclosure can be implemented with various programming or scripting languages. The function blocks can be implemented as algorithms executed on one or more processors. In addition, the present disclosure can adopt traditional techniques for electronic environment settings, signal processing and / or data processing. Terms such as "mechanism", "element", "device" and "configuration" can be widely used, but are not limited to mechanical and physical components.

[0031] In addition, the connecting lines or connecting members between the components shown in the drawings only show functional connections and / or physical or circuit connections. In actual equipment, the connections between components can be represented by various replaceable or additional functional connections, physical connections or circuit connections.

[0032] In this specification, "neural network model" may refer to a deep learning model based on an artificial neural network (ANN, ArtificialNeural Network). The deep learning model has a deep structure with multiple hidden layers in the ANN, through which complex patterns can be learned. For example, the deep learning model may include an input layer, a hidden layer, and an output layer, and each layer may include several neurons. In addition, each neuron can calculate the output value by multiplying the input value by the weight applied to the activation function, each layer transmits the signal to the next layer, and the signal can be adjusted by weights and biases. In other words, the deep learning model uses training data to optimize weights and biases, thereby allowing the complex relationship between input and output to be learned. At the same time, the deep learning model can be implemented with various types of models, such as multi-layer perceptron (MLP, Multi-Layer Perceptron), convolutional neural network (CNN, Convolutional Neural Network) and recurrent neural network (RNN, Recurrent Neural Network). In addition, the deep learning model can be implemented as various models according to the method of configuring activation function, loss function, optimization algorithm, etc., and is not limited to a specific type of model.

[0033] The embodiments will be described in detail below with reference to the accompanying drawings. However, the embodiments may be implemented in various different forms and are not limited to the examples described herein.

[0034] Figure 1 It is a diagram showing a monitoring system including the sensor device 1 according to the embodiment.

[0035] The sensor device 1 according to the present invention includes a signal processing circuit 10, a sensor 30, and an energy harvesting circuit 20. The sensor 30 is a device for sensing the external environment at the location where the sensor device 1 is installed, and senses temperature, humidity, or image, for example. The sensor 30 is connected to the signal processing circuit 10, operates according to the electric energy supplied from the signal processing circuit 10, and outputs a sensing signal to the signal processing circuit 10. The sensor 30 connected to the signal processing circuit 10 may be various types of sensors.

[0036] The energy collection circuit 20 collects energy from the environment outside the sensor device 1. The energy collection circuit 20 includes an RF antenna, a resonant coil, a solar cell and / or an electrostatic collection circuit, and can collect energy (power) from RF signals, resonant signals, solar thermal energy and / or static electricity that can be used as energy sources.

[0037] The signal processing circuit 10 generates and stores electric energy through the energy harvesting circuit 20, and provides the generated electric energy to the sensor 30. In addition, the signal processing circuit 10 may receive a sensing signal from the sensor 30 and process the sensing signal. For example, the signal processing circuit 10 may determine the condition of the sensor 30 or the state outside the sensor device 1 by processing the sensing signal.

[0038] The signal processing circuit 10 according to the present invention applies CNN (Convolutional Neural Network) algorithm technology to generate optimal power from various energy sources (RF signals, resonance signals, solar energy and / or static electricity), and processes sensing signals, etc. The signal processing circuit 10 is configured to be able to perform convolution calculations in the analog domain by including a scalable analog PIM module 1000 (or analog PIM 1100) according to the present invention that can dynamically increase or decrease the calculation resolution.

[0039] The signal processing circuit 10 including the scalable analog PIM module 1000 (or analog PIM 1100) can provide variable processing performance required by various types of sensors 30 by adjusting the calculation resolution, which leads to optimal power consumption efficiency. The signal processing circuit 10 is configured to be universally applicable to various sensors 30. The signal processing circuit 10 includes a self-generated power circuit using various energy sources, and thus may not be subject to environmental restrictions associated with an external power source. The signal processing circuit 10 is preferably configured as an integrated circuit. The signal processing circuit 10 may be integrated into a semiconductor in a package made of plastic or metal and mounted on a board of the sensor device 1. Figure 2 The signal processing circuit 10 is examined in detail in the following figures.

[0040] The control terminal 2 of the monitoring system is a terminal for controlling the sensor device 1 (specifically, the signal processing circuit 10). The control terminal 2 can be configured as a personal computer, a cloud server, etc., to update the weights (weights of filters or kernels) of the CNN algorithm processed by the signal processing circuit 10, or to set parameters for setting the accuracy, efficiency, etc. of the CNN algorithm or other processing. The control terminal 2 and the sensor device 1 can be connected by wire (such as USB, etc.) or wirelessly (such as Wi-Fi or ZigBee, etc.) to update or change the settings, parameters, weights, etc. of the sensor device 1 under the control of the control terminal 2.

[0041] Figure 2 : is a block diagram showing the internal structure of the signal processing circuit 10 .

[0042] according to Figure 2 The signal processing circuit 10 includes a CNN control unit 100, a power supply 200, a sensor signal processor 300 and an energy storage unit 400. The signal processing circuit 10 may also include Figure 2 Blocks other than the hardware blocks shown.

[0043] The power supply 200 generates electric energy according to a signal outside the sensor device 1. The power supply 200 is connected to the energy harvesting circuit 20 of the sensor device 1 to generate and output electric energy from an RF signal, a resonance signal, solar heat, and / or an electrostatic signal from the energy harvesting circuit 20 under the control of the CNN control unit 100. The power supply 200 will be further examined with reference to FIG. 3 .

[0044] The sensor signal processor 300 processes a sensing signal from the sensor 30. The sensor signal processor 300 processes a sensing signal from any sensor 30 under the control of the CNN control unit 100 and outputs processed sensing data.

[0045] The energy storage unit 400 stores the electric energy generated by the power supply 200. The energy storage unit 400 may include a known storage element for storing electric energy or may include a capacitor. The electric energy stored in the energy storage unit 400 (in addition, the electric energy generated by the current power supply 200) is provided to other blocks (CNN control unit 100, sensor signal processor 300 and power supply 200).

[0046] The CNN control unit 100 controls the power supply 200 and the sensor signal processor 300 according to the workload data of the power supply generated by the power supply 200 and the sensing data generated by the sensor signal processor 300. The CNN control unit 100 includes a CNN controller 2000 and an expandable analog PIM module 1000. The workload data may be data indicating the amount of current of the generated power.

[0047] The analog PIM module 1000 performs a convolution operation between weights and input data (workload data and / or sensing data) in an analog signal domain, performs conversion to digital convolution data, and outputs the digital convolution data as a result signal. The analog PIM module 1000 operates according to the control (control signal) of the CNN controller 2000, and can preferably perform a convolution operation by variably setting the size of the weight and the size of the input data.

[0048] That is, the analog PIM module 1000 can reduce the power consumption required for analog operation and further improve the operation speed by optimizing to the effective operation size for processing input data. The resolution of data required to process the sensing data (such as temperature, humidity, and image) from various sensors 30 may be different for each sensor 30. The analog PIM module 1000 according to the present invention is configured to be able to dynamically perform effective operation or detect input resolution according to the data characteristics of the combined sensor 30, and can perform analog convolution operation with the corresponding operation or input resolution.

[0049] The CNN controller 2000 of the CNN control unit 100 executes a CNN artificial intelligence algorithm (AI) to control the power supply 200 and / or the sensor signal processor 300 by controlling the analog PIM module 1000. The CNN controller 2000 may include an MCU (e.g., a low-power MCU), a memory, and a circuit or logic for extracting and classifying features.

[0050] The CNN control unit 100 may determine the status of the sensor 30 according to the execution of the CNN AI algorithm using the scalable analog PIM module 1000. In addition, the CNN control unit 100 generates a power control signal for controlling the power supply 200 and a gain control signal for controlling the gain of the sensor signal processor 300 from the workload data and the sensing data according to the (technique) of the CNN AI algorithm using the scalable analog PIM module 1000, and outputs the power control signal to the power supply 200 and the sensor signal processor 300.

[0051] The CNN control unit 100 may process subsequent layers after processing a layer for input data according to the CNN AI algorithm to generate a power control signal and a gain control signal according to classification, and may determine various other conditions.

[0052] FIG. 3 is a block diagram showing a configuration example of the power supply 200 and the sensor signal processor 300 .

[0053] Fig. 3A shows an example hardware block diagram of the power supply 200, and Fig. 3B shows an example hardware block diagram of the sensor signal processor 300. The example of Fig. 3 is a diagram showing a configuration for understanding the present invention, and the sensor device 1 may have various or different configurations of the power supply 200 and the sensor signal processor 300 that are known or will be known.

[0054] When briefly examining the power supply 200 and the sensor signal processor 300 with reference to FIG3 , the power supply 200 includes an adaptive matching network 210, one or more rectifiers 250, and one or more DC-DC converters 290. The adaptive matching network 210 is configured to optimally match one or more of an RF signal, a resonant signal, a solar signal, and / or an electrostatic signal according to a received power control signal. The CNN control unit 100 may output a power control signal that allows the adaptive matching network 210 to match a specific energy source.

[0055] The rectifier 250 is coupled to the adaptive matching network 210 to generate direct current (DC) power from the signal from the adaptive matching network. The rectifier 250 can be, for example, a full-bridge rectifier or a half-bridge rectifier. One or more rectifiers 250 operate according to the power control signal received from the CNN control unit 100 to generate and output direct current.

[0056] The DC-DC converter 290 is connected to the rectifier 250 and increases or decreases the DC power from the rectifier 250 according to a set voltage. For example, the DC-DC converter 290 may be a buck-boost converter. The DC-DC converter 290 may operate according to a power control signal received from the CNN control unit 100 and output DC power at a set level.

[0057] Here, the power control signal received from the CNN control unit 100 is a signal for controlling the adaptive matching network 210, one or more rectifiers 250, and one or more DC-DC converters 290 by executing the CNN AI algorithm on the workload data (current amount data) of the DC power output from the power supply 200. The CNN control unit 100 generates and outputs a power control signal for optimal control of the power supply 200 for the currently input workload data (further using the sensing data).

[0058] The sensor signal processor 300 includes a first MUX 310 (Multiplex), a TIA (TransImpedance Amp) 320, an LNA (Low Noise Amp) 330, a second MUX 350, a VGA (Variable Gain Amp) 370, and an SD ADC (Sensor Device ADC) 390. The first MUX 310 outputs a signal on one of a plurality of sensor signal lines according to an applied selection signal. For example, the first MUX 310 may output a signal of a sensor signal line that may be connected to one of four pins of the sensor 30 that is an integrated circuit of the signal processing circuit 10 according to a signal applied to the pin of the selection signal.

[0059] The TIA 320 is connected to the first MUX 310, amplifies the signal from the first MUX, and outputs the amplified signal. The TIA 320 amplifies and outputs the input signal according to the gain control signal received from the CNN control unit 100. For example, the TIA 320 may convert a current signal into a voltage signal, and amplify and output the converted voltage signal according to the applied gain control signal.

[0060] The LNA 330 is connected to the first MUX 310 to amplify the signal from the first MUX and output the amplified signal. The LNA 330 may amplify the input voltage signal according to the gain control signal received from the CNN control unit 100 and output the amplified voltage signal.

[0061] The second MUX 350 is connected to the TIA 320 and the LNA 330 to select and output the amplified signal from the TIA 320 or the amplified signal from the LNA 330. The second MUX 350 may output a signal according to the same selection signal as the pin of the selection signal of the first MUX 310, or may output a signal according to the selection signal received from the CNN control unit 100.

[0062] The VGA 370 is connected to the second MUX 350 to amplify the signal from the second MUX 350 according to the gain control signal.

[0063] The SD ADC 390 is connected to the VGA 370, converts an analog signal from the VGA 370 into a digital signal, and outputs the converted sensing data. The SD ADC 390 may be a successive approximation ADC including a SAR logic, a CDAC, and a comparator.

[0064] The SD ADC 390 has a maximum resolution of 8 bits, 16 bits, or 32 bits, and outputs the converted digital sensing data within the maximum resolution. The SD ADC 390 receives an enable signal specifying the number of bits enabled within the maximum resolution from the analog PIM module 1000 or the CNN control unit 100, and outputs digital sensing data having an output resolution corresponding to the number of bits enabled within the maximum resolution according to the enable signal.

[0065] For example, when the maximum bit resolution of the SD ADC 390 is 16 bits and the SD ADC 390 receives an 8-bit enable signal, scaling to 8 bits is performed to output digital sensing data. When the SD ADC 390 receives an enable signal requesting an increase to 10 bits while outputting 8-bit sensing data, 10-bit sensing data can be generated and output by controlling the CDAC.

[0066] Here, the gain control signal received from the CNN control unit 100 is generated by executing the CNN AI algorithm on the sensing data output from the sensor signal processor 300. The CNN control unit 100 uses the sensing data output from the sensor signal processor 300 and the additional workload data as input to generate and output a gain control signal for controlling the gain of the TIA 320, the LNA 330, and / or the VGA 370.

[0067] Figure 42 is a block diagram showing a configuration of the analog PIM module 1000, and FIG. 5 is a diagram for describing the internal structure and operation of the analog PIM.

[0068] According to the present invention, Figure 4 The analog PIM module 1000 is configured to be able to perform convolution operations in the analog signal domain, and is configured to be able to dynamically adjust (enlarge or reduce) the resolution of the convolution operation. By using the analog PIM module 1000 that can adjust the calculation resolution in a scalable manner, the signal processing circuit 10 can dynamically perform optimal convolution calculations and accordingly apply the optimal CNN algorithm regardless of the type or kind of the sensor 30.

[0069] Furthermore, the signal processing circuit 10 can reduce power consumption and increase calculation speed by adjusting the calculation resolution for optimal convolution calculation. Figure 4 The analog PIM module 1000 may be configured as an analog PIM circuit and is preferably incorporated in an integrated circuit.

[0070] according to Figure 4 , the analog PIM module 1000 includes an analog PIM 1100 and an analog PIM controller 1500 . Figure 4 The analog PIM module 1000 is preferably integrated into the signal processing circuit 10 of the sensor device 1 , which is an integrated circuit.

[0071] The analog PIM 1100 is configured to perform convolution operations between a weight matrix (e.g., 8*8, 4*4, 3*3, 2*2, 1*1, etc.) scaled to a variable size in the analog signal domain and an input data matrix (e.g., 8*8, 4*4, 3*3, 2*2, 2*1, etc.).

[0072] The analog PIM 1100 includes a plurality (e.g., 8, 16, 64, etc., hereinafter referred to as a "first plurality") of SRAM operators 1130. Figure 4 The first plurality of SRAM operators 1130 in ①) perform convolution operations in the analog domain using the SRAM 1131 and the capacitor 1133 therein, and generate analog convolution result signals according to the convolution operations.

[0073] Among the first plurality of SRAM operators 1130, an enabled number (e.g., 1, 4, 8, 16, etc., hereinafter referred to as the "second number") of SRAM operators 1130 perform a convolution operation under the control of the analog PIM controller 1500. Among the first plurality of SRAM operators 1130, the remaining disabled SRAM operators 1130 suspend the execution of the analog convolution operation. According to the disable signal, the power supply to the SRAM operator 1130 may be suspended, or the capacitor 1133 of each SRAM operator 1130 may be turned on to suspend the execution of the convolution operation.

[0074] Each of the first plurality of SRAM operators 1130 (temporarily) stores one weight (a weight of 1 or 0) of the weight matrix in the SRAM 1131, and stores one piece of input data of the input data matrix as an analog signal in the capacitor 1133. The SRAM operator 1130 performs a product operation between the weight of the SRAM 1131 and the stored input data of the capacitor 1133 in the analog domain, and the first plurality of SRAM operators 1130 performs an averaging operation on the product operation using the provided capacitor (the total capacity of 1133) to generate an analog convolution result signal.

[0075] The analog PIM 1100 further includes a first plurality of DACs 1150. Each of the first plurality of DACs 1150 is connected to the first plurality of SRAM operators 1130, and the DACs 1150 convert digital input data (eg, workload data or sensing data) into analog input data and output the analog input data to the SRAM operators 1130.

[0076] According to the present invention, the first plurality of SRAM operators 1130 ( Figure 4 Each of the plurality of SRAMs 1131 and one or more (preferably two) capacitors 1133 therein, and one or more jump switches 1135 (see Figure 5a In the structure of the SRAM operator 1130 , the jump switch is configured to control the storage of analog input data in one or more capacitors 1133 through the DAC 1150 .

[0077] The SRAM 1131 including 10 transistors or the like stores the weight of a binary signal 1 or 0, and the capacitor 1133 stores analog input data through the connected DAC 1150. The skip switch 1135 can cause the SRAM operator 1130 to store the analog input data in the capacitor 1133 according to the reception of an on control signal (zero skip (ZS) control signal) from the analog PIM controller 1500, and can cause the SRAM operator 1130 not to store the analog input signal in the capacitor 1133 according to the reception of an off control signal.

[0078] In addition, the analog PIM 1100 further includes one or more ADCs 1110. When the first plurality of SRAM operators 1130 in the analog PIM module 1000 is L SRAM operators 1130 (L is an integer of 1 or greater, for example, Figure 4 As shown in 4), the analog PIM 1100 may have four ADCs 1110.

[0079] The ADC 1110 is connected to the first plurality of SRAM operators 1130 in a specific row, converts analog convolution analog operation signals calculated by the first plurality of DRAM operators 1140 into digital convolution data, and outputs the converted digital convolution data.

[0080] The analog PIM controller 1500 of the analog PIM module 1000 controls the analog PIM 1100 so that the convolution operation is performed in the analog domain. The analog PIM controller 1500 includes a gate through a logic circuit (e.g., a circuit generated according to an RTL code), and controls the execution of the convolution operation by controlling the analog PIM 1100 according to a control signal received from the CNN control unit 100 of the signal processing circuit 10.

[0081] The analog PIM controller 1500 is preferably configured to perform convolution operations in the analog domain by controlling an enabled second number of SRAM operators 1130 of the first plurality of SRAM operators 1130 .

[0082] The analog PIM controller 1500 stores the weight in the SRAM 1131 of the enabled second number of SRAM operators 1130 among the first plurality of SRAM operators 1130, and stores the input data in the capacitor 1133. The analog PIM controller 1500 stores the weight and the input data through the DAC 1150 connected to the second number of SRAM operators 1130. When the weight value stored in the weight storage process is a specified value (e.g., logic (bit) 0), the analog PIM controller 1500 outputs a disconnection control signal for disconnecting the jump switch 1135 to the SRAM operator 1130.

[0083] By outputting the off control signal to the SRAM operator 1130, the analog PIM controller 1500 omits performing the analog operation (product operation) in the corresponding SRAM operator 1130. When the stored weight is a different value (e.g., logic (bit) 1), the analog PIM controller 1500 outputs an on control signal to the SRAM operator 1130 to turn on (connect) the jump switch 1135.

[0084] By further including control of the skip switch 1135 in the SRAM operator 1130, when the weight is 0, the analog product operation in the SRAM operator 1130 may be omitted, thereby reducing power consumption of the analog PIM 1100 and increasing execution speed of the convolution operation.

[0085] Figure 5b An example of a weight matrix of size 4*4 (4*4, the second number is 16) is shown. Figure 5b As in the example, when the weight of (the element of) a particular row and column of the weight matrix is ​​0, the jump switch 1135 of the corresponding SRAM operator 1130 is turned off (disconnected), and the corresponding SRAM operator 1135 omits the product operation.

[0086] Based on the convolution data output from ADC 1110, the analog PIM controller 1500 outputs an enable control signal to the first plurality of SRAM operators 1130 to enable a second number of SRAM operators 1130 (the second number is 1 or more and is less than or equal to the number of the second plurality of SRAM operators 1140) in the entire first plurality of SRAM operators 1131 that is less than or equal to the first plurality of SRAM operators 1130.

[0087] The analog PIM controller 1500 outputs enable control signals to the second number of SRAM operators 1130 to activate the second number of SRAM operators 1130 in the first plurality of SRAM operators 1130 , and outputs disable control signals to the remaining SRAM operators 1130 to deactivate the remaining SRAM operators 1130 in the first plurality of SRAM operators 1130 .

[0088] Each disable signal suspends the execution of the analog operation (product operation) in each of the remaining SRAM operators 1130. For example, the disable signal blocks the power supplied to the SRAM operator 1130, or disconnects the capacitor 1133 connected to the ground through a switch provided separately from the read / write line (see Figure 5a ).

[0089] Figure 5c1 shows an example formula and process for a convolution operation performed by the analog PIM 1100 through conversion from the digital domain to the analog domain. The input data (workload data and / or sensor data) and weights are converted to analog input data and weights through the DAC 1150 and stored in a second number (e.g., 16) of SRAM operators 1130 (see FIG. 11 ). Figure 5c In ①).

[0090] Each of the second number (e.g., 16) of SRAM operators 1130 in the first plurality of SRAM operators 1130 performs a product operation between the SRAM 1131 and the capacitor 1133, and averages (MAV) signals (e.g., voltage signals) according to the capacity of the total capacitors 1133 in the second number (e.g., 16) of SRAM operators 1130 to generate an analog convolution result signal (see Figure 5c ②). The ADC 1110 connected to the enabled second number of SRAM operators 1130 converts the digital convolution data according to the size of the convolution result signal (see Figure 5c ③).

[0091] The converted digital convolution data is output to the CNN controller 2000 of the CNN control unit 100, and the CNN controller 2000 may extract and classify features according to the CNN AI algorithm.

[0092] The analog PIM controller 1500 is preferably configured to dynamically change the resolution of the data and operation used in the convolution operation. The analog PIM controller 1500 is preferably configured to be able to adjust the resolution of the input data of the SD ADC 390 and / or the resolution of the analog convolution operation (the second number of SRAM operators 1130) based on the input data output from the SD ADC 390 connected to the analog PIM module 1000 and / or the convolution data output from the ADC 1110 of the analog PIM 1100.

[0093] For example, when the input data output from the SD ADC 390 of the sensor signal processor 300 connected to the analog PIM module 1000 is the same set number of times or more, the analog PIM controller 1500 increases the output resolution of the SD ADC 390, and after the output resolution of the SD ADC 390 is increased, determines a second number of new SRAM operators 1130 to be enabled among the first plurality of SRAM operators 1130 based on the convolution data output from the ADC 1110.

[0094] In addition, the analog PIM controller 1500 interfaces with the analog PIM 1100 to output the sensing data from the sensor signal processor 300 or the workload data of the power supply to the DAC 1150 connected to the second number of SRAM operators 1130 selected as the input data, and output the weight from the CNN controller 2000 to the DAC 1150 connected to the second number of SRAM operators 1130. In addition, the analog PIM controller 1500 may output the convolution data output by the ADC 1110 to the CNN controller 2000.

[0095] The analog PIM controller 1500 (or CNN controller 2000) periodically samples the input data according to the determined second number of sampling cycles (and events), stores the sampled input data in the analog PIM 1100, and then allows the analog convolution operation to be performed. In this way, the analog PIM controller 1500 outputs the input data adjusted according to the second number changed by the determination to the changed second number of SRAM operators 1130. In addition, the analog PIM controller 1500 receives the weight data adjusted according to the changed second number from the CNN controller 2000, and outputs the adjusted weight to the second number of SRAM operators 1130.

[0096] Figure 6 is a diagram illustrating an example block diagram of an analog PIM controller 1500 .

[0097] according to Figure 6 , the analog PIM controller 1500 includes an ADC output detector 1501 , a comparator 1505 , a memory 1503 , and an ADC / SRAM enable controller 1509 . Figure 6 The analog PIM controller 1500 includes a logic circuit (eg, a circuit generated according to an RTL code) and is implemented as an integrated circuit through a plurality of gates.

[0098] The ADC output detector 1501 detects the convolution data output from the ADC 1110 of the analog PIM 1100, and further detects the input data (sensing data) output from the SD ADC 390 of the sensor signal processor 300 connected to the analog PIM module 1000. The ADC output detector 1501 may detect the data upon receiving a signal representing data output from the CNN controller 2000, other control logic of the analog PIM controller 1500, and / or the sensor signal processor 300. The ADC output detector 1501 outputs the detected data (convolution data or input data).

[0099] The memory 1503 stores previous input data and previous convolution data output to the ADC output detector 1501 .

[0100] The comparator 1505 digitally compares the convolution data output from the ADC output detector 1501 with the convolution signal outputted before (immediately before), and outputs a comparison result signal. In addition, the comparator 1505 also digitally compares the input data outputted from the ADC output detector 1501 with the input signal outputted according to the previous (immediately before) detection, and outputs a comparison result signal.

[0101] The comparator 1505 outputs a comparison result signal indicating whether the data from the memory 1503 and the data output from the ADC output detector 1501 are identical by comparing each of the convolution data and the input data.

[0102] The ADC / SRAM enable controller 1509 determines the number of bits of the SD ADC 390 in the first plurality of SRAM operators 1130 and the second number for the analog convolution operation according to the comparison result in the comparator 1505. The ADC / SRAM enable controller 1509 generates an enable (control) signal and outputs the enable signal to the SD ADC 390 to increase the resolution of the SD ADC 390 according to the comparison between the current data (input data or convolution data) and the previous data, and generates an enable (control) signal and outputs the enable (control) signal to the second number of SRAM operators 1130 of the analog PIM 1100 to increase the resolution of the convolution operation.

[0103] For example, when the ADC / SRAM enable controller 1509 continuously receives the same comparison result signal for a specified number of times (e.g., 4 times, 5 times, etc.) or more set for the convolution data through the comparator 1505, the ADC / SRAM enable controller 1509 determines a second number greater than the currently set second number to increase the resolution of the analog operation of the first plurality of SRAM operators 1130. For example, when there are 64 SRAM operators 1130 in total and four SRAM operators 1130 are currently used, the ADC / SRAM enable controller 1509 increases the second number from 4 to 5 (or 8, 9, etc.). The specified number of times can be set by the control terminal 2.

[0104] The ADC / SRAM enable controller 1509 outputs an enable control signal to the first plurality of SRAM operators 1130 to enable the added second number of SRAM operators 1130. The ADC / SRAM enable controller 1509 outputs an enable signal to the second number of SRAM operators 1130 to enable the added second number of SRAM operators 1130 in the first plurality of SRAM operators 1130, and outputs a disable signal to the remaining SRAM operators 1130 to disable the remaining SRAM operators 1130.

[0105] Through this process, the analog PIM module 1000 can perform a convolution operation that matches the sensor 30 or a convolution operation optimized for the condition of the sensor 30, while reducing power consumption by increasing the analog calculation resolution in real time when the output resolution of the convolution operation on the input data is low.

[0106] In addition, the ADC / SRAM enable controller 1509 increases the output resolution of the SD ADC 390 according to the comparison result of the input data through the comparator 1505. When the ADC / SRAM enable controller 1509 continuously receives the same comparison result signal set for the input data for a specified number of times (e.g., 4 times, 5 times, etc.) or more, the ADC / SRAM enable controller 1509 outputs an enable control signal to the SD ADC 390 to have an output resolution higher than the currently set output resolution. The ADC / SRAM enable controller 1509 outputs an enable control signal to the SD ADC 390 to enable more bits than the currently set bit number of the SD ADC 390 (e.g., increase by 1 bit). The specified number can be set by the control terminal 2, and the specified number for the input data and the specified number for the convolution data can be different from each other.

[0107] The SD ADC 390 controls an internal CDAC, etc. upon receiving an enable control signal, and outputs digital sensing data having a resolution with an increased number of bits.

[0108] After the number of output bits of the SD ADC 390 is increased, the ADC / SRAM enable controller 1509 re-determines the second number enabled in the first plurality of SRAM operators 1130 using the convolution data output by the ADC 1110 of the analog PIM 1100. When the ADC / SRAM enable controller 1509 continuously receives the same comparison result signal set for the convolution data a specified number of times (e.g., 4 times, 5 times, etc.) or more starting from the minimum number (e.g., 1) in the first plurality thereof through the comparator 1505, the ADC / SRAM enable controller 1509 determines the second number greater than the currently set second number to increase the resolution of the analog operation of the first plurality of SRAM operators 1130.

[0109] The ADC / SRAM enabling controller 1509 repeatedly or simultaneously (in parallel) performs a process of determining a second quantity to detect the resolution of the optimal convolution operation while the output resolution of the SD ADC 390 changes, thereby allowing an optimized CNN AI algorithm to be executed based on the connected sensor 30 or the condition of the sensor 30.

[0110] The ADC / SRAM enable controller 1509 outputs an enable control signal to the first plurality of SRAM operators 1130 to enable the second number of SRAM operators 1130, and outputs an enable control signal for activating the second number of DACs 1150 connected to the second number of SRAM operators 1130. In addition, the ADC / SRAM enable controller 1509 may output a disable control signal to the remaining DACs 1150 in the first plurality of DACs 1150 to deactivate the remaining DACs 1150.

[0111] Reference Figure 7 The resolution determination process performed in the ADC / SRAM enable controller 1509 is examined in more detail.

[0112] Figure 7 is a flow chart describing an illustrative control flow for determining an output match with any sensor 30 and calculating a resolution.

[0113] For controlling the analog PIM 1100 Figure 7 The control flow is executed in the signal processing circuit 10, and is preferably executed by the control of the CNN control unit 100 and the analog PIM module 1000 (the analog PIM controller 1500). Figures 1 to 6 A detailed examination was conducted, so the control process will receive more attention and examination below.

[0114] First, the signal processing circuit 10 enables the sensor 30 (S101). The signal processing circuit 10 can start or initialize the operation of the sensor 30 by supplying power to the sensor 30 according to a POR (power on reset) signal applied from the outside or generated by a power supply. The signal processing circuit 10 also initializes the internal circuits 100, 200 and 300.

[0115] Thereafter, the signal processing circuit 10 enables the SD ADC 390 of the sensor signal processor 300 (S103). For example, the signal processing circuit 10 enables the SD ADC 390 so that the SD ADC 390 has an output resolution of all bits.

[0116] The signal processing circuit 10 samples input data (sensing data) output from the SD ADC 390 of the sensor signal processor 300 ( S105 ), and determines a resolution of the SD ADC 390 according to the sampled input data ( S107 ).

[0117] The signal processing circuit 10 samples the input data multiple times, and uses the comparator 1505 to determine the number of bits that changes less than a specified number of times between the minimum number of bits (e.g., 1 bit) and the maximum number of bits (e.g., the number of bits of the maximum resolution that the SD ADC 390 can output, e.g., 16 bits or 32 bits) of the SD ADC 390 as the initial resolution of the SD ADC 390. The signal processing circuit 10 may determine the initial resolution by sequential performance for each bit number, or may determine the initial resolution by parallel determination for all bits.

[0118] The signal processing circuit 10 outputs an enable control signal to the SD ADC 390 to enable the number of bits determined according to the initial resolution, and the SD ADC 390 outputs digital sensing data according to the set resolution.

[0119] After determining the initial (initial) resolution, the signal processing circuit 10 enables the analog PIM 1100 ( S107 ). The CNN controller 2000 may initialize the analog PIM 1100 by controlling the analog PIM controller 1500 .

[0120] After enabling the analog PIM 1100 , the signal processing circuit 10 samples the convolution data output from the ADC 1110 of the analog PIM 1100 ( S109 ), and determines the resolution of the analog PIM based on the sampled convolution data ( S111 ).

[0121] The signal processing circuit 10 can perform comparison tests on the convolution data in sequence from the minimum resolution (the second number is 1) to the maximum resolution (the second number is the first plurality) under the control of the CNN controller 2000, and determine the calculation resolution of the first (initial) analog PIM 1100 based on the comparison test.

[0122] The signal processing circuit 10 may perform a convolution operation on the weight and the input data (sensing data) at a currently set resolution between the minimum resolution and the maximum resolution, increase the currently set resolution when the digital convolution data as a result of the convolution operation is the same for a specified number of times or more, and determine an initial optimal calculation resolution by repeatedly comparing the current convolution data with the previous convolution data at the increased resolution. Through this test process, the signal processing circuit 10 determines the initial second number.

[0123] After determining the number of bits and the second number of SD ADC 390 according to the initialization process (see S101 to S111), the signal processing circuit 10 dynamically adjusts the number of bits and the second number of SD ADC 390 according to the output of SD ADC 390 or ADC 1110 of the analog PIM 1100 to maintain the optimal resolution required for the convolution operation in real time and optimize power consumption.

[0124] More specifically, after determining the number of bits and the second number of the SD ADC 390 (see S101 to S111 ), the signal processing circuit 10 performs the sensor operation under the control of the CNN control unit 100 ( S113 ).

[0125] The signal processing circuit 10 generates an analog convolution result signal according to an analog operation between weights in the second number of SRAM operators 1130 enabled by one or more of the first plurality of SRAM operators 1130 and input data (e.g., sensing data) through the analog PIM 1100, and converts the convolution result signal generated by the ADC 1110 of the analog PIM 1100 into digital convolution data. The signal processing circuit 10 can perform a CNN AI algorithm in real time using the converted convolution data.

[0126] The process of generating the convolution result signal is configured to omit the analog operation (product operation) through the skip switch 1135 according to the non-control signal received from the analog PIM controller 1500 and set according to the weight value in one or more second number SRAM operators 1130.

[0127] After the sensor operation, the signal processing circuit 10 determines whether the number of times the digital input data output from the sensor signal processor 300 is the same reaches or exceeds a specified number of times (N times) (S115). When the number of times the digital input data is the same reaches or exceeds the specified number of times, the signal processing circuit 10 determines that the output resolution of the sensor signal processor 300 is low, and increases the output resolution of the input data (S117).

[0128] The signal processing circuit 10 increases the currently set output bit number of the SD ADC 390 by 1 or more, and outputs an enable control signal indicating the increased bit number to the SD ADC 390 .

[0129] As the output resolution of the input data increases (according to the increase in the number of bits of the SD ADC 390), the signal processing circuit 10 re-executes S109 and S111 to sequentially increase the enabled second number of SRAM operators 1130 between the minimum number and the number of the first plurality of SRAM operators, and re-determines the second number by comparing the convolution data output from the increased second number of SRAM operators 1130 with the previous same second number of convolution data. Through this process, when the number of output bits of the SD ADC 390 changes, the resolution of the optimal analog operation is dynamically calculated to optimize the performance of the CNN AI algorithm.

[0130] In addition, the signal processing circuit 10 dynamically determines a second number of SRAM operators 1130 to be enabled in the first plurality of SRAM operators 1130 based on the convolution data output from the analog PIM 1100 (S119 and S121). When the convolution data output from the analog PIM 1100 is identical to the previous convolution data (continuously) for a specified number of times (N times) or more (S119), the signal processing circuit 10 increases the second number to increase the calculation resolution of the first plurality of SRAM operators 1130, i.e., adds an analog PIM ADC (S121).

[0131] The signal processing circuit 10 determines a second number greater than the currently determined second number, enables the determined second number of SRAM operators 1130 (and the corresponding DAC 1150), and performs subsequent sensor operations accordingly. The signal processing circuit 10 can output an enable control signal for enabling the changed second number of SRAM operators 1130 (and the corresponding second number of DACs 1150) to the first plurality of SRAM operators 1130 (and the corresponding second number of DACs 1150), and outputs a convolution result of the calculation resolution adjusted by the activated SRAM operators 1130 and DACs 1150.

[0132] Through the above control flow, regardless of the type of the sensor 30 , the signal processing circuit 10 can process the sensor 30 by universally combining with the sensor 30 , and has dynamically efficient convolution operation processing and optimized power consumption according to the type or condition of the sensor 30 .

[0133] Hereinafter, as another embodiment of the monitoring system, a system including a plurality of sensors and monitoring an abnormal condition based on the plurality of sensors will be described.

[0134] Figure 8 is a diagram for describing an example of a system for monitoring an abnormal condition based on a plurality of sensors according to an embodiment.

[0135] Reference Figure 8 , a system 800 for monitoring abnormal conditions based on multiple sensors (hereinafter referred to as a monitoring system) may include multiple sensors 811, 812, and 813, a monitoring device 820, a server 830, and one or more user terminals 840 and 850. However, in addition to this, the monitoring system 800 may further include at least one external device. The various components included in the monitoring system 800 may be connected via wired or wireless communications to send and receive data therebetween.

[0136] The plurality of sensors 811, 812, and 813 may be sensors for building an Internet of Things (IoT) system. For example, the plurality of sensors 811, 812, and 813 may detect various objects or phenomena in various environments and situations in an area where the IoT system is built. For example, each of the plurality of sensors 811, 812, and 813 may be a temperature sensor, a humidity sensor, a motion sensor, a gas sensor, a vibration sensor, a lighting sensor, a noise sensor, a distance sensor, a pressure sensor, a proximity sensor, a pH sensor, a UV sensor, a magnetic sensor, an image sensor, a sound sensor, a wind speed sensor, a wind direction sensor, a current sensor, a voltage sensor, and a flow sensor. In addition, the sensors of the monitoring system 800 may include sensors configured to detect objects associated with the IoT system.

[0137] about Figure 8 , the object detected by each sensor may be an object related to an abnormal condition monitored by the monitoring system 800. Here, the abnormal condition monitored by the monitoring system 800 may refer to a condition that exceeds a normal range due to a change in the IoT system or the environment in which the monitoring system 800 operates. For example, the monitoring system 800 may monitor whether an abnormal condition, such as a fire or a gas leak, occurs around an area where the monitoring system 800 is constructed (or an area where the IoT system is constructed) based on changes in temperature, pressure, and gas detected by a temperature sensor, a pressure sensor, and a gas sensor, respectively.

[0138] At the same time, the area where the monitoring system 800 is constructed can be a target area where the monitoring system 800 monitors whether an abnormal condition occurs. That is, a plurality of sensors 811, 812, and 813 can be installed in the target area for monitoring abnormal conditions to respectively detect objects related to the abnormal conditions. At the same time, even if a plurality of sensors 811, 812, and 813 are installed in the target area, the detection range may vary according to the type of the detected object. Therefore, according to the type of object detected by the sensor, the monitoring system 800 may include a plurality of sensors for detecting the same object. For example, the monitoring system 800 may include a first gas sensor, a second gas sensor, and a temperature sensor. However, for ease of description, even when a plurality of sensors for detecting the same object are installed in the target area, the sensor for detecting the object is grouped and described as one sensor.

[0139] In the present disclosure, the plurality of sensors 811, 812, and 813 may not be limited to sensors that detect one object. For example, each of the plurality of sensors 811, 812, and 813 may be a sensor that selectively detects one object or a sensor that simultaneously detects multiple objects. That is, among the sensors included in the monitoring system 800, each sensor that detects a specific object may be implemented as an independent device, or implemented as a device that can perform a function of detecting one object and a function of detecting another object. For example, the monitoring system 800 may include a gas sensor, a pressure sensor, and a temperature sensor that detect gas, pressure, and temperature, respectively. As another example, the monitoring system 800 may include a first sensor configured to detect gas and a second sensor configured to detect pressure and temperature.

[0140] Meanwhile, each of the plurality of sensors 811, 812, and 813 may be Figure 1 For example, the plurality of sensors 811, 812, and 813 may be devices installed in a target area of ​​the monitoring system 800 to sense an external environment. The plurality of sensors 811, 812, and 813 may be operated according to the power provided to detect an object associated with an abnormal condition and output a sensing signal associated with a result of detecting the object.

[0141] The monitoring device 820 may be connected to the plurality of sensors 811, 812, and 813 included in the monitoring system 800 to receive sensing signals from the plurality of sensors 813, 812. That is, the monitoring device 820 may receive sensing signals for detecting different types of objects. For example, in the present disclosure, a first sensing signal associated with a first object detected by a first sensor and a second sensing signal associated with a second object detected by a second sensor may be processed by the monitoring device 820 instead of being received and processed by an independent device.

[0142] The monitoring device 820 may output a monitoring result related to whether an abnormal condition occurs in the target area based on processing the received sensing signal. In more detail, the monitoring device 820 may convert the sensing signal into input data to be input into the neural network model, and perform operations on the input data using the neural network model to obtain a monitoring result related to the classification of the abnormal condition. Figures 9 to 15 An example of specific operation of the monitoring device 0 in the present disclosure is described.

[0143] The server 830 may be a device configured to communicate with external devices including the monitoring device 820 and the user terminals 840 and 850. For example, the server 830 may receive the monitoring result from the monitoring device 820 and store the monitoring result. In addition, the server 830 may transmit the stored monitoring result to the user terminal 840 belonging to the manager 861 or the user terminal 850 belonging to the IoT system user 862.

[0144] User terminals 840 and 850 may be computing devices, each of which has a display device and includes a memory and a processor. For example, the display device may be implemented as a touch screen for receiving user input. Alternatively, each of user terminals 840 and 850 may further include a separate device (e.g., keyboard, mouse, etc.) for receiving user input. For example, user terminals 840 and 850 may include, but are not limited to, smart phones, notebook computers, desktop computers, laptop computers, tablet computers, etc.

[0145] The user terminal 840 of the manager 861 who manages the monitoring system 800 (or IoT system) can output the status information of the IoT system according to the monitoring results in response to receiving the monitoring results from the server 830. For example, through the user terminal 840, the manager 861 can verify whether the environment around the current IoT system is in a normal condition, or whether an abnormal condition has occurred with respect to a predetermined object, or receive an alarm about the occurrence of an abnormal condition. Similarly, the user terminal 850 of the user 862 who uses the IoT system can output the status information in the IoT system according to the monitoring results in response to receiving the monitoring results from the server 830. Similarly, through the user terminal 850, the user 862 can verify whether the environment around the current IoT system is in a normal condition, or whether an abnormal condition has occurred with respect to a predetermined object, or receive an alarm about the occurrence of an abnormal condition. At the same time, referring to Figure 8 , user terminals 840 and 850 are shown as receiving monitoring results from server 830. However, the present invention is not limited thereto. User terminals 840 and 850 may directly receive monitoring results from monitoring device 820, or may receive monitoring results from another device (not shown).

[0146] Fig. 9 is a diagram for describing an example of the internal configuration of a monitoring device according to the embodiment.

[0147] refer to Fig. 9 The monitoring device 900 may include a processor 910, a memory 920, an input / output interface 930, and a communication module 940. For ease of description, Fig. 9 Only components relevant to the present invention are shown. Fig. 9In addition to the components shown, other common components may be further included in the monitoring device 900. In addition, it is obvious to those skilled in the art that the present invention is related to: Fig. 9 The illustrated processor 910 , memory 920 , input / output interface 930 , and communication module 940 may be implemented as independent devices.

[0148] The processor 910 can process computer program instructions by performing basic arithmetic, logic, and input / output operations. Here, instructions can be provided from the memory 920 or an external device (e.g., an external server, etc.). In addition, the processor 910 can generally control the operation of other components included in the monitoring device 900.

[0149] The processor 910 may determine an operation mode based on condition information in a target area for monitoring abnormal conditions.

[0150] For example, the processor 910 may determine any one of a first mode of detecting different objects simultaneously or a second mode of detecting objects related to the situation information in real time as the operation mode.

[0151] (First operation mode)

[0152] The processor 910 may receive sensing signals from a plurality of sensors installed in the target area in response to the first mode of simultaneously detecting different objects being determined as the operation mode.

[0153] Here, the plurality of sensors may include at least one of a gas sensor, a pressure sensor, or a temperature sensor.

[0154] The processor 910 may process the sensing signal and output first sensing data obtained by converting the sensing signal into a digital signal.

[0155] For example, the processor 910 may select one of the sensing signals based on the first selection control signal, amplify the selected sensing signal based on the amplification control signal and the gain control signal, and perform digital conversion on the amplified sensing signal to output sensing data.

[0156] According to an embodiment, the first selection control signal may be a signal configured to perform a control operation to select the sensing signal in time series.

[0157] According to an embodiment, the first selection control signal may be a signal configured to perform a control operation to select one of the sensing signals based on a predetermined cycle.

[0158] The processor 910 may generate first input data to be input to the first neural network model by variably adjusting the size of the first sensing data.

[0159] For example, the processor 910 may accumulate the first sensing data to correspond to the number of bits, and generate the first input data having an output resolution corresponding to the number of bits based on an enable signal designating the number of bits enabled at a maximum resolution.

[0160] The processor 910 may perform an operation on first input data in an analog domain using a first neural network model, and output first operation data obtained by converting the operation result into digital data.

[0161] For example, the processor 910 may convert first input data into an input value in an analog domain, perform a convolution operation in the analog domain on the input value using a plurality of SRAM operators, and output first operation data obtained by converting an analog convolution result as an output value of the operation into data in a digital domain.

[0162] The processor 910 may output a first monitoring result obtained by classifying the abnormal condition based on the first operation data.

[0163] For example, the processor 910 may classify the abnormal condition of each object detected by the plurality of sensors, and output the monitoring result based on the classification result of the abnormal condition of each object.

[0164] According to an embodiment, the processor 910 may determine a category of the abnormal condition based on the abnormal condition classification result of each object, and output a monitoring result by classifying a risk level within the determined abnormal condition category.

[0165] According to an embodiment, the processor 910 may assign a weight to one of the objects detected by the plurality of sensors, and assign the weight to an abnormal condition classification result of the weighted object.

[0166] According to an embodiment, the processor 910 may update the status information in the target area based on the first monitoring result.

[0167] (Second Operation Mode)

[0168] The processor 910 may select a target sensor from among a plurality of sensors installed in the target area in response to the second mode of detecting an object related to the situation information in real time being determined as the operation mode.

[0169] Here, the plurality of sensors may include at least one of a gas sensor, a pressure sensor, or a temperature sensor.

[0170] For example, the processor 910 may select a sensor that detects a main object related to the situation information as a main target sensor among the plurality of sensors, and select a sensor that detects a sub-object related to the main object as a sub-target sensor.

[0171] The processor 910 may process an object signal received from the object sensor and output second sensing data obtained by converting the object signal into a digital signal.

[0172] For example, the processor 910 may receive a target signal from the target sensor based on a second selection control signal in response to the target sensor being selected, amplify the target signal based on an amplification control signal and a gain control signal, and perform digital conversion on the amplified target signal to output second sensing data.

[0173] The processor 910 may generate second input data to be input to the second neural network model by variably adjusting the size of the second sensing data.

[0174] For example, the processor 910 may accumulate the second sensing data to correspond to the number of bits, and generate the second input data having an output resolution corresponding to the number of bits based on an enable signal designating the number of bits enabled at a maximum resolution.

[0175] Here, the second neural network model is a model used in response to the second mode being determined as the operation mode, and can be characterized as a model that performs an operation by considering input data at a time before the operation is performed.

[0176] The processor 910 may perform an operation on the second input data in the analog domain using the second neural network model, and output second operation data obtained by converting the operation result into digital data.

[0177] For example, the processor 910 may convert second input data into a second input value in the analog domain, perform a convolution operation on the second input value in the analog domain using a plurality of SRAM operators, and output second operation data obtained by converting a second analog convolution result as an output value of the operation into data in the digital domain.

[0178] The processor 910 may output a second monitoring result obtained by classifying the abnormal condition based on the second operation data.

[0179] For example, the processor 910 may determine a category of an abnormal condition in response to an object detected by the target sensor, and output a second monitoring result by classifying a risk level within the determined category of the abnormal condition.

[0180] According to an embodiment, the processor 910 may determine the category of the abnormal condition based on the abnormal condition classification results of each of the main object and the sub-object, and output a second monitoring result taking into account the risk level classification of the main object and the risk level classification of the sub-object within the determined abnormal condition category.

[0181] According to an embodiment, the processor 910 may update the status information in the target area based on the second monitoring result.

[0182] The processor 910 may be implemented as an array of multiple logic gates, or may be implemented as a combination of a general-purpose microprocessor and a memory storing a program that can be executed on the microprocessor. For example, the processor 910 may include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some cases, the processor 910 may include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. For example, the processor 910 may refer to a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors and a DSP core, or a combination of processing devices, such as any other such configuration combination.

[0183] The memory 920 may include any non-transitory computer-readable recording medium. For example, the memory 920 may include a random access memory (RAM) and a permanent mass storage device such as a read-only memory (ROM), a disk drive, a solid-state drive (SSD), or a flash memory. As another example, a permanent mass storage device such as a ROM, an SSD, a flash memory, or a disk drive may be a separate permanent storage device different from the memory. In addition, an operating system (OS) and at least one program code (e.g., a program for the processor 910 to execute a program to be referred to later) may be included. Figures 11 to 15 The code for describing the operations) may be stored in the memory 920.

[0184] These software components may be loaded from a computer-readable recording medium that is separate from the memory 920. The independent computer-readable recording medium may be a recording medium that can be directly connected to the monitoring device 900. For example, the recording medium may include a computer-readable recording medium such as a floppy disk drive, a magnetic disk, a magnetic tape, a DVD / CD-ROM drive, or a memory card. Alternatively, the software components may be loaded into the memory 920 via the communication module 940 instead of the computer-readable recording medium. For example, a computer program that can be installed based on a file provided by a developer via the communication module 940 or a file distribution system that distributes application installation files (e.g., a file distribution system for the processor 910 to execute a program referred to later) may be used to distribute the software components. Figures 11 to 15 etc.), loads at least one program into the memory 920.

[0185] The communication module 930 may provide a configuration or function for the plurality of sensors 811, 812, and 813 and the server 830 to communicate with the monitoring device 900 through a network. In addition, the communication module 940 may provide a configuration or function for the monitoring device 900 to communicate with other external devices (e.g., user terminals 840 and 850). For example, control signals, commands, data, etc. provided under the control of the processor 910 may be sent to the server 830 and / or external devices through the communication module 940 and the network.

[0186] Meanwhile, according to the embodiment, although Fig. 9 Although not shown in the figure, the monitoring device 900 may also include a display device. Alternatively, the monitoring device 900 may be connected to an independent display device via wired or wireless communication to send and receive data therebetween. For example, the display device may display status information in the target area according to the monitoring results.

[0187] According to another embodiment, the monitoring device 900 may further include an input / output interface. The input / output interface may be a device for interfacing with a device for input or output (e.g., a keyboard, a mouse, etc.), which may be connected to the monitoring device 900 or included in the monitoring device 900. The input / output interface may be implemented as an element configured separately from the processor 910, or may be configured to be included in the processor 910.

[0188] Fig.10 is a diagram for describing another example of the internal configuration of the monitoring device according to the embodiment.

[0189] refer to Fig.10 , the monitoring device may include a sensor signal processing unit 1020 , a microcontroller unit (MCU) 1030 , and an analog PIM unit 1040 .

[0190] According to an embodiment, the sensor signal processing unit 1020 may correspond to Figures 1 to 7 The sensor signal processor 300 described in Figures 1 to 7 A portion of the operations performed by the sensor signal processor 300 described in can be performed by the sensor signal processing unit 1020.

[0191] For example, the sensor signal processing unit 1020 may include a first MUX 1021 (multiplexing), a TIA 1022, an LNA 1023, a second MUX 1024, a VGA 1025, and a SDADC 1026. These may correspond to the first MUX 310 (multiplexing), the TIA 320, the LNA 330, the second MUX 350, the VGA 370, and the SDADC 390 of the sensor signal processor 300, respectively.

[0192] For example, the first MUX 1021 and the second MUX 1024 may receive a signal and select a signal. The TIA 1022 may convert a sensing signal in the form of a current signal into a voltage signal based on a first amplification control signal. The LNA 1023 may amplify the sensing signal in the form of a voltage signal based on a second amplification control signal to minimize noise. The VGA 1025 may amplify the voltage signal based on a dynamically adjusted gain control signal. The SD ADC 1026 may perform digital conversion on the amplified sensing signal and output sensing data. For example, the SD ADC 1026 may convert an analog sensing signal into a digital signal and output the converted sensing data.

[0193] According to an embodiment, the sensor signal processing unit 1020 may receive sensing signals from a plurality of sensors installed in the target area in response to the first mode of simultaneously detecting different objects being determined as the operation mode.

[0194] According to an embodiment, the second signal processing unit 1020 may select a target sensor from among a plurality of sensors installed in the target area in response to the second mode of detecting an object related to the situation information in real time being determined as the operation mode.

[0195] Here, the plurality of sensors may include at least one of a gas sensor, a pressure sensor, or a temperature sensor.

[0196] According to an embodiment, the second signal processing unit 1020 may select one of the sensing signals based on the first selection control signal, amplify the selected sensing signal based on the amplification control signal and the gain control signal, and perform digital conversion on the amplified sensing signal to output sensing data.

[0197] According to an embodiment, the first selection control signal may be a signal configured to perform a control operation to select the sensing signal in time series.

[0198] According to an embodiment, the first selection control signal may be a signal configured to perform a control operation to select one of the sensing signals based on a predetermined cycle.

[0199] According to an embodiment, the sensor signal processing unit 1020 may select a sensor that detects a main object related to the situation information as a main target sensor among the plurality of sensors, and select a sensor that detects a sub-object related to the main object as a sub-target sensor. The sensor signal processing unit 1020 may process a target signal received from the target sensor and output second sensing data obtained by converting the target signal into a digital signal.

[0200] According to an embodiment, in response to the selected target sensor, the sensor signal processing unit 1020 may receive a target signal from the target sensor based on a second selection control signal, amplify the target signal based on an amplification control signal and a gain control signal, and perform digital conversion on the amplified target signal to output second sensing data.

[0201] According to an embodiment, the sensor signal processing unit 1020 may generate first input data to be input to the first neural network model by variably adjusting the size of the first sensing data. Alternatively, the sensor signal processing unit 1020 may generate second input data to be input to the second neural network model by variably adjusting the size of the second sensing data. For example, the sensor signal processing unit 1020 may accumulate the first sensing data (or the second sensing data) to correspond to the number of bits, and generate the first input data (or the second input data) having an output resolution corresponding to the number of bits based on an enable signal specifying the number of bits enabled at the maximum resolution.

[0202] According to an embodiment, the MCU 1030 may correspond to Figures 1 to 7 CNN controller 2000 described in . Figures 1 to 7 A portion of the operations performed by the CNN controller 2000 described in can be performed by the MCU 1030.

[0203] According to an embodiment, the MCU 1030 may determine the operation mode based on the condition information in the target area for monitoring abnormal conditions. For example, the MCU 1030 may determine any one of the first mode for simultaneously detecting different objects and the second mode for detecting objects related to the condition information in real time as the operation mode.

[0204] According to an embodiment, the MCU 1030 may output a first monitoring result obtained by classifying the abnormal condition based on the first operation data.

[0205] For example, the MCU 1030 may classify the abnormal condition of each object detected by the plurality of sensors, and output the monitoring result based on the abnormal condition classification result of each object.

[0206] According to an embodiment, the MCU 1030 may determine the category of the abnormal condition based on the abnormal condition classification result of each object, and output the monitoring result by classifying the risk level within the determined abnormal condition category.

[0207] According to an embodiment, the MCU 1030 may assign a weight to one of the objects detected by the plurality of sensors, and assign the weight to an abnormal condition classification result of the weighted object.

[0208] According to an embodiment, the MCU 1030 may output a second monitoring result obtained by classifying the abnormal condition based on the second operation data.

[0209] For example, the MCU 1030 may determine a category of an abnormal condition in response to an object detected by the target sensor, and output the second monitoring result by classifying a risk level within the determined category of the abnormal condition.

[0210] According to an embodiment, the MCU 1030 may determine the category of the abnormal condition based on the abnormal condition classification result of each of the main object and the sub-object, and output the second monitoring result taking into account the risk level classification of the main object and the risk level classification of the sub-object within the determined abnormal condition category.

[0211] According to an embodiment, the MCU 1030 may update the status information in the target area based on the first monitoring result or the second monitoring result.

[0212] The analog PIM unit 1040 may include an analog PIM (analog in-memory processor) 1041 and a digital controller 1042. Here, the analog PIM 1041 may correspond to Figures 1 to 7 The analog PIM 1100 described in Figures 1 to 7 A portion of the operations performed by the analog PIM 1100 described in the above may be performed by the analog PIM 1041. In addition, the digital controller 1042 may correspond to Figures 1 to 7 The analog PIM controller 1500 described in Figures 1 to 7 A portion of the operations performed by the analog PIM controller 1500 described in can be performed by the digital controller 1042.

[0213] According to an embodiment, the analog PMIM unit 1040 may perform an operation on first input data in an analog domain using a first neural network model, and output first operation data obtained by converting the operation result into digital data.

[0214] For example, the analog PMIM unit 1040 may convert first input data into an input value in the analog domain, perform a convolution operation in the analog domain on the input value using a plurality of SRAM operators, and output first operation data obtained by converting the analog convolution result as an output value of the operation into data in the digital domain.

[0215] According to an embodiment, the analog PMIM unit 1040 may perform an operation on the second input data in the analog domain using the second neural network model, and output second operation data obtained by converting the operation result into digital data.

[0216] For example, the analog PMIM unit 1040 may convert second input data into an input value in the analog domain, perform a convolution operation in the analog domain on the input value using a plurality of SRAM operators, and output second operation data obtained by converting the analog convolution result as an output value of the operation into data in the digital domain.

[0217] Fig.11 is a flowchart for describing an example of a method for monitoring abnormal conditions based on multiple sensors according to an embodiment.

[0218] In step 111 , the monitoring device may process sensing signals received from a plurality of sensors configured to detect different objects, and output sensing data obtained by converting the sensing signals into digital signals.

[0219] The plurality of sensors may be sensors for detecting objects related to the IoT system, as described above with reference to Figure 8 Furthermore, the objects respectively detected by the plurality of sensors may be objects related to the abnormal conditions monitored by the monitoring system.

[0220] For example, the monitoring device may receive sensing signals from a gas sensor, a pressure sensor, and a temperature sensor. That is, the plurality of sensors may include at least one of a gas sensor, a pressure sensor, or a temperature sensor.

[0221] Meanwhile, even if one sensor is described as detecting a specific object for ease of description, the present invention is not limited thereto. For example, when the monitoring device receives a sensing signal related to a gas from a gas sensor, the monitoring device may receive sensing signals related to the gas from a plurality of gas sensors. That is, the monitoring system may include a plurality of gas sensors, each of which is configured to detect a gas.

[0222] The monitoring device may output sensing data as a result of signal processing on the sensing signals through a process of selecting one of the sensing signals, amplifying the selected sensing signal, and performing digital conversion on the amplified sensing signal.

[0223] In more detail, the monitoring device may select one of the received sensing signals based on the selection control signal. Fig.12 A method of selecting a sensed signal by a monitoring device is described.

[0224] Fig.12 is a diagram for describing an example of sensing data according to an embodiment.

[0225] refer to Fig.12In the first signal processing example 1210, the second signal processing example 1220, and the third signal processing example 1230 shown in FIG. 1 , the monitoring device may receive a sensing signal A, a sensing signal B, and a sensing signal C from a plurality of sensors that detect different objects.

[0226] In the present disclosure, since the monitoring device monitors a plurality of sensors simultaneously, that is, performs signal processing on sensing signals received from the plurality of sensors simultaneously, it may be necessary to assign priority as to which of the received sensing signals will be processed first.

[0227] According to an embodiment, the selection control signal may be a signal configured to perform a control operation to select a sensing signal in time series. That is, the monitoring device may select one of the sensing signals over time. For example, when the monitoring device receives sensing signals from a plurality of sensors during a predetermined time, the monitoring device may divide the predetermined time and select a sensing signal to be received during each time interval.

[0228] According to an embodiment, the selection control signal may be a signal configured to perform a control operation to select one of the sensing signals based on a predetermined period. For example, the predetermined period may be a value preset by an administrator or user of the monitoring system. For example, in the first signal processing example 1210, the monitoring device may select one of the received sensing signals based on a selection control signal, which is configured to perform a control operation to perform a selection based on a period of 10ms. Therefore, the monitoring device may select a sensing signal A 1211 from 0 seconds to 10ms, a sensing signal B 1212 from 10ms to 20ms, and a sensing signal C 1213 from 20ms to 30ms.

[0229] Here, the selection control signal may be a signal that controls the selection of the sensing signal so that the frequency of selecting each of the sensing signals is adjusted. In more detail, the frequency of selecting each sensing signal may be determined based on the weight assigned to each sensing signal. For example, in the second signal processing example 1220, the monitoring device may assign the maximum weight to the sensing signal A1221 among the received sensing signals, and may select the sensing signal based on the selection control signal that controls the selection of the sensing signal, thereby selecting the sensing signal A1221 at the highest frequency. Therefore, in the second signal processing example 1220, the monitoring device may select the sensing signal A1211 from 0 seconds to 10ms, the sensing signal B1212 from 10ms to 20ms, the sensing signal A1211 again from 20ms to 30ms, and the sensing signal C 1213 from 30ms to 40ms.

[0230] Meanwhile, a weight for each of the sensing signals may be assigned based on the importance of the object of the sensing signal (i.e., the object detected by the sensor) relative to the abnormal condition. For example, when the importance of monitoring temperature is high, a high weight may be assigned to the temperature sensor or the sensing signal received from the temperature sensor. As another example, when the importance of monitoring gas is high, a high weight may be assigned to the gas sensor or the sensing signal received from the gas sensor.

[0231] According to another embodiment, the selection control signal may be a signal for controlling signal selection so that the time interval for receiving the signal is adjusted for each sensing signal. In more detail, the time interval for receiving each sensing signal may be determined based on the weight assigned to each sensing signal. For example, in the third signal processing example 1230, the monitoring device may assign the maximum weight to the sensing signal B 1232 among the received sensing signals, and thus may select the sensing signal based on the selection control signal for controlling the sensing signal selection so that the time interval for receiving the sensing signal B 1232 is the longest. Therefore, in the third signal processing example 1230, the monitoring device may select the sensing signal A 1231 from 0s to 5ms, the sensing signal B 1232 from 5ms to 25ms, and the sensing signal C 1233 from 25ms to 30ms.

[0232] at the same time, Fig.12 Each of the sensing signals 1211, 1212, 1213, etc. shown in the first signal processing example 1210, the second signal processing example 1220, and the third signal processing example 1230 may refer to sensing data converted into digital signals. For example, in the first signal processing example 1210, the sensing data A may be sensing data obtained by amplifying the sensing signal A based on the amplification control and gain control signals and performing digital conversion on the amplified sensing signal. Similarly, the sensing data B and the sensing data C may each be sensing data in digital form. That is, in the first signal processing example 1210, the sequence data connected in time series may be data obtained by connecting digital sensing data. The monitoring device may use the corresponding sequence data to generate input data for input to the neural network model. Regarding this, a description will be given again below.

[0233] At the same time, before converting the received sensing signal into a digital signal, the monitoring device may adjust the signal level of the sensing signal, which is an analog signal. For example, since the analog sensing signal received from the gas sensor and the analog sensing signal received from the temperature sensor have different signal levels (or signal amplitudes, etc.), the monitoring device may adjust the signal level to meet the input conditions for converting each sensing signal into a digital signal. To this end, the monitoring device may amplify the selected sensing signal based on the amplification control signal and the gain control signal. For example, the monitoring device may convert the sensing signal in the form of a current signal into a voltage signal based on a first amplification control signal, and amplify the converted voltage signal based on a dynamically adjusted gain control signal. As another example, the monitoring device may amplify the sensing signal in the form of a voltage signal based on a second amplification control signal to minimize noise, and may amplify the sensing signal based on a dynamically adjusted gain control signal.

[0234] The monitoring device may output the sensing data by performing digital conversion on the amplified sensing signal. For example, the monitoring device may convert the analog sensing signal into a digital signal and output the converted sensing data.

[0235] Reference again Fig.11 In step 113, the monitoring device may generate input data to be input into the neural network model by variably adjusting the size of the sensing data.

[0236] Regarding step 113, reference will be made to Fig.13 Give a more detailed description.

[0237] Fig.13 is a diagram for describing an example of input data according to an embodiment.

[0238] The sensing data refers to data obtained by converting the sensing signal into a digital signal as described above. For example, the monitoring device can generate sequence data by connecting the sensing data and adjusting the size of the sequence data to correspond to the size for input to the neural network model, thereby generating input data.

[0239] For example, the monitoring device may accumulate sensing data corresponding to the number of bits based on an enable signal specifying the number of bits enabled within the maximum resolution, and generate input data having an output resolution corresponding to the number of bits. For example, the monitoring device may generate input data using digital sensing data within a maximum resolution of 8 bits, 16 bits, or 32 bits.

[0240] For example, the monitoring device may accumulate (or connect to the sequence data) the sensing data 1321 based on the first enable signal 1311 to generate the first input data 1310. Here, the first enable signal 1311 may be an enable signal that specifies the number of bits (maximum number of bits) corresponding to the maximum resolution. That is, the monitoring device may generate the first input data 1310 by accumulating the sensing data 1321 to the maximum number of bits.

[0241] As another example, the monitoring device may accumulate (or connect to the sequence data) the sensing data 1322 based on the second enable signal 1312 to generate the second input data 1320. Here, the second enable signal 1312 may be an enable signal that specifies the number of bits corresponding to a resolution lower than the maximum resolution. That is, the monitoring device may generate the second input data 1320 by accumulating the sensing data 1322 to a predetermined number of bits.

[0242] Reference again Fig.11 In step 115, the monitoring device may operate the input data in the analog domain using the neural network model, and output operation data obtained by converting the operation result into digital data.

[0243] For example, the monitoring device may convert the input data into an input value in the analog domain. That is, the monitoring device may convert the digital input data into an input value as analog input data and output the converted input value to the SRAM operator.

[0244] In addition, the monitoring device can use multiple SRAM operators to perform convolution operations on input values ​​in the analog domain. Figure 4 The structures and operation methods of multiple SRAM operators are described in detail.

[0245] In addition, the monitoring device may output operation data obtained by converting an analog convolution result, which is an output value of a convolution operation in the analog domain, into data in the digital domain. For example, the monitoring device may convert an analog convolution result signal calculated by an SRAM operator into operation data, which is digital convolution data, and output the converted operation data.

[0246] Meanwhile, according to one embodiment, the monitoring device may include an analog PIM unit 1040. Fig.10 As stated in Figures 1 to 7 At least some of the operations described as being performed by analog PIM module 1000, analog PIM 1100, and analog PIM controller 1500 are performed by a monitoring device.

[0247] Reference again Fig.11In step 117, the monitoring device may output a monitoring result obtained by classifying the abnormal condition based on the calculated data.

[0248] For example, the monitoring device may classify the abnormal condition of each object detected by a plurality of sensors, and output a monitoring result based on the classification result of the abnormal condition of each object.

[0249] An example of monitoring results output based on sensing signals received from the gas sensor, the temperature sensor, and the pressure sensor will be described with reference to Table 1 below.

[0250] [Table 1]

[0251]

[0252] For example, referring to Table 1, the monitoring device can classify the abnormal conditions of each of the gas, temperature and pressure detected by the gas sensor, temperature sensor and pressure sensor as any one of normal, detected abnormality and risk. In addition, the monitoring device can output monitoring results such as fire risk based on the abnormal condition classification results of each object.

[0253] Meanwhile, according to an embodiment, the monitoring device may determine the category of the abnormal condition based on the abnormal condition classification result of each object. For example, the monitoring device may determine the category of the abnormal condition as a category related to fire in response to the abnormal condition classification result of fire being any one of the detected abnormality or risk.

[0254] In addition, the monitoring device may output the monitoring result by classifying the risk level within the category of the determined abnormal condition. For example, the monitoring device may classify the risk level as an incipient fire risk in response to the abnormal state classification result of detecting an abnormal fire, and may classify the risk level as a fire risk in response to the abnormal state classification result of fire being a risk.

[0255] According to another embodiment, the monitoring device may assign a weight to any one of the objects detected by the multiple sensors, and assign a weight to the abnormal condition classification result associated with any one of the weighted objects. For example, the monitoring device may comprehensively consider the abnormal condition classification result of the temperature detected by the temperature sensor and the abnormal condition classification result of the gas detected by the gas sensor to determine the fire as the category of the abnormal condition, or comprehensively consider the abnormal condition classification result of the gas detected by the gas sensor and the abnormal condition classification result of the pressure detected by the pressure sensor to determine the gas leakage as the category of the abnormal condition. However, the abnormal condition related to the fire and the abnormal condition related to the gas leakage are strongly related to the temperature (or the abnormal condition classification result for the temperature) and the gas (or the abnormal condition classification result for the gas), respectively, so the monitoring device may assign a weight to the corresponding object (or the abnormal condition classification result of the object). For example, the monitoring device may determine the category of the abnormal condition or classify the risk level of the abnormal condition by further considering the weight applied to the object (or the abnormal condition classification result of the object).

[0256] Fig.14 is a flowchart for describing another example of a method for monitoring abnormal conditions based on multiple sensors according to an embodiment.

[0257] Reference Fig.14 In step 1410, the monitoring device may determine an operation mode based on the condition information of the target area for monitoring abnormal conditions.

[0258] Here, the condition information in the target area refers to condition information related to the area where abnormal conditions are monitored by multiple sensors. For example, when any one of the objects detected by the multiple sensors deviates from the normal condition, the condition information in the target area may be in an abnormal detection condition. Specifically, the condition information in the target area may be a monitoring result including the category, risk level, etc. of the object deviating from the normal condition.

[0259] Alternatively, when all objects detected by multiple sensors are in normal conditions, the condition information in the target area may be in a normal condition. Here, the normal condition of the object may refer to a normal condition, and the fact that the object is out of the normal condition may mean that the risk level of the object is an abnormal detection condition or a risk condition.

[0260] For example, the monitoring device may determine the first mode of simultaneously detecting different objects as the operating mode in response to the condition information in the target area being a normal condition. As another example, the monitoring device may determine the second mode of real-time detection of objects related to the condition information as the operating mode in response to the condition information in the target area being an abnormal detection condition.

[0261] Hereinafter, steps 1421, 1431, 1441, 1451, 1461 and 1471 show an example of an operation flow of a monitoring device operating in a first mode, and steps 1422, 1432, 1442, 1452, 1462 and 1472 show an example of an operation flow of a monitoring device operating in a second mode. Meanwhile, hereinafter, the operation of the monitoring device operating in the first mode may include Fig.11 part of the monitoring device operation described in. Therefore, Fig.11 The operation of the monitoring device described in Fig.14 The monitoring device is operating in a first mode.

[0262] In step 1421 , in response to the first mode being determined as the operation mode, the monitoring device may receive sensing signals from a plurality of sensors installed in the target area.

[0263] In step 1431 , the monitoring device may process the sensing signal and output first sensing data obtained by converting the sensing signal into a digital signal.

[0264] In step 1441, the monitoring device may generate first input data to be input to the first neural network model by variably adjusting the size of the first sensing data.

[0265] In step 1451, the monitoring device may perform an operation on first input data in an analog domain using a first neural network model, and output first operation data obtained by converting the operation result into digital data.

[0266] In step 1461, the monitoring device may output a first monitoring result obtained by classifying the abnormal condition based on the first operation data.

[0267] In step 1471, the monitoring device processor may update the status information in the target area based on the first monitoring result. For example, when the status information in the target area is updated to (remains as) a normal status, the monitoring device may operate in the first operation mode. Alternatively, when the status information in the target area is updated to (or changed to) an abnormally detected status, the monitoring device may operate in the second operation mode.

[0268] Again, in step 1422, the monitoring device may select a target sensor from a plurality of sensors installed in the target area in response to the second mode of detecting an object related to the condition information in real time being determined as the operation mode.

[0269] That is, the target sensor may be a sensor that detects an object related to the condition information. For example, when the condition information in the target area is an initial fire risk condition, the monitoring device may determine the second mode as the operation mode and select the temperature sensor as the target sensor to monitor the temperature sensor related to the fire in real time.

[0270] Meanwhile, according to an embodiment, the monitoring device may select a sensor that detects a main object related to the condition information as a main target sensor among a plurality of sensors, and select a sensor that detects a sub-object related to the main object as a sub-target sensor.

[0271] For example, when the condition information in the target area is an incipient fire risk condition, the monitoring device may determine temperature as a main object to select a temperature sensor as a main target sensor, and determine a gas related to temperature as a sub-object to determine a gas sensor as a sub-target sensor. That is, according to an embodiment, even when operating in the second mode, the monitoring device may detect an object that finds an abnormal condition in real time, and continuously monitor objects associated with the corresponding object.

[0272] Thereafter, in step 1432 , the monitoring device may process the target signal received from the target sensor and output second sensing data obtained by converting the target signal into a digital signal.

[0273] Meanwhile, the processing process for outputting the second sensing data based on the target signal can be the same as Fig.11 The same processing procedure for outputting sensing data based on the sensing signal described in . Therefore, overlapping descriptions will be omitted. For example, the monitoring device may be configured to output sensing data based on the sensing signal. Fig.11 The target signal is amplified based on the amplification control signal and the gain control signal in the same manner as described in the embodiment, or digital conversion is performed on the amplified target signal to output second sensing data.

[0274] According to one embodiment, the monitoring device may receive a target signal from the target sensor based on the second selection control signal in response to the target sensor being selected. Fig.15 Give a description.

[0275] Fig.15 is a diagram for describing other examples of sensing data according to an embodiment.

[0276] refer to Fig.15 In the first signal processing example 1510 and the second signal processing example 1520 shown in FIG. 1 , the monitoring device may receive a target signal from a target sensor. Here, target signals A 1511 and 1521 may be main target signals received from a main target sensor, and target signal 1522 and target signal C 1523 may be sub-target signals received from a sub-target sensor.

[0277] about Fig.11 The embodiments described above for the selection control signal in the embodiment may be applied unchanged to the second selection control signal. For example, the second selection control signal may be a signal configured to perform a control operation to select a target signal in a time series. As another example, the second selection control signal may be a signal configured to perform a control operation to select a target signal based on a predetermined period.

[0278] According to another embodiment, the second selection control signal may be a signal selected by the control signal so that the time intervals for receiving the main target signal and the sub-target signal are adjusted differently. For example, in the first signal processing example 1510, the monitoring device may select the target signal so that the time interval of the target signal A 1511 selected as the main target signal is longer than the time interval of the target signal B 1512 selected as the sub-target signal based on the second selection control signal. Therefore, the monitoring device may select the target signal A 1511 from 0ms to 15ms and the target signal B 1512 from 15ms to 20ms.

[0279] According to yet another embodiment, the second selection control signal may be a signal for controlling the selection of the target signal, thereby adjusting the frequency of selecting the main target signal and the sub-target signal. For example, in the second signal processing example 1520, the monitoring device may select the target signal so that the frequency of selecting the target signal A 1521 as the main target signal is higher than the frequency of selecting the object signal B 1522 and the object signal C 1523 as the sub-target signal based on the second selection control signal. Therefore, the monitoring device may select the target signal A 1521 from 0ms to 10ms, the target signal B 1522 from 10ms to 20ms, the target signal A 1521 again from 20ms to 30ms, and the target signal C 1523 from 30ms to 40ms.

[0280] Reference again Fig.14 In step 1442, the monitoring device may generate second input data to be input to the second neural network model by variably adjusting the size of the second sensing data.

[0281] At the same time, the process of generating input data based on sensor data can be compared with the reference Fig.11 The described processes are the same, so redundant descriptions will be omitted. For example, based on an enable signal specifying the number of bits enabled within the maximum resolution, the monitoring device can accumulate second sensing data to correspond to the number of bits and generate second data having an output resolution corresponding to the number of bits.

[0282] According to an embodiment, in response to the second mode being determined as the operation mode, the monitoring device may change the neural network model to be used from the first neural network model used when the operation mode is the first mode to the second neural network model. Here, the second neural network model may be characterized as a model that performs an operation by considering input data at a time before the time when the operation is performed. That is, according to this embodiment, the second neural network model is a neural network model with a circular structure, and when the second neural network model is used, continuity or sequence over time may be considered, so the second neural network model may be more suitable for the second operation mode of centrally monitoring the target sensor.

[0283] For example, the first neural network model may be a CNN model, and the second neural network model may also be an RNN model. At the same time, the example of the second neural network model is not limited to the RNN model, and any model with a recursive neural network structure, such as a long short-term memory (LSTM) model or a gated recurrent unit (GRU) model, may correspond to the second neural network model.

[0284] Thereafter, in step 1452, the monitoring device may perform an operation on the second input data in the analog domain using the second neural network model, and output second operation data obtained by converting the operation result into digital data. Similarly, the process of performing an operation on the second input data using the second neural network model may be similar to the process of performing an operation on the second input data using the reference neural network model. Fig.11 The operation procedures described are the same, so redundant descriptions will be omitted.

[0285] For example, the monitoring device may convert second input data into a second input value in the analog domain, perform a convolution operation on the second input value in the analog domain using multiple SRAM operators, and output second operation data obtained by converting the second analog convolution result as the output value of the operation into data in the digital domain.

[0286] Thereafter, in step 1462, the monitoring device may output a second monitoring result obtained by classifying the abnormal condition based on the second operation data. Similarly, the process of outputting the second monitoring result about the target sensor may be similar to the process of referring to Fig.11 The process of outputting the first monitoring result described is the same, so redundant description will be omitted.

[0287] For example, the monitoring device may determine the category of the abnormal condition in response to the object detected by the target sensor, and output the second monitoring result by classifying the risk level within the determined category of the abnormal condition. For example, when the target sensor is a temperature sensor, the monitoring device may determine a fire related to temperature as the category of the abnormal condition. In addition, the monitoring device may classify the risk level regarding the fire into safety, initial risk level, risk level, etc. based on processing the target signal received from the temperature sensor.

[0288] According to an embodiment, the monitoring device can determine the category of the abnormal condition based on the abnormal condition classification result of each of the main object related to the condition information and the sub-object related to the main object. For example, when the main object is temperature and the sub-object is gas, the monitoring device can determine fire, etc. as the category of the abnormal condition.

[0289] In addition, the monitoring device can output the second monitoring result by considering the risk level classification of the main object and the risk level classification of the sub-object within the abnormal condition category. In more detail, the monitoring device can assign a higher weight to the risk level classification of the main object than to the risk level classification of the sub-object. For example, when the risk level of the main object and the risk level of the sub-object are both safety levels, the monitoring device can output information indicating that the target area is in a normal condition as the second monitoring result. As another example, in the case of monitoring an abnormal condition related to a fire, when the risk level related to the temperature (main object) is an initial risk and the risk level related to the gas (sub-object) is a risk, the monitoring device can output information indicating that the target area is at an initial risk of fire as the second monitoring result. As another example, when the risk level related to the temperature (main object) is a risk, and the risk level related to the gas (sub-object) is an initial risk, the monitoring device can output information indicating that the target area is at a risk of fire as the second monitoring result.

[0290] Reference again Fig.14 In step 1472, the monitoring device may update the status information in the target area based on the second monitoring result.

[0291] For example, when the status information in the target area is updated to (changed to) a normal status, the monitoring device can operate in the first operating mode. Alternatively, when the status information in the target area is updated to (or maintained as) an abnormally detected status, the monitoring device can operate in the second operating mode.

[0292] Unless there is a clear order statement or contrary statement for the steps included in the method according to the present invention, these steps can be performed in any suitable order. The present invention is not necessarily limited by the order of the steps. Any examples or illustrative terms (such as, etc.) in the present invention are only used to describe the present invention in detail, and unless limited by the claims, the scope of the present invention is not limited by the examples or illustrative terms. In addition, those skilled in the art will recognize that various modifications, combinations and changes can be made according to design conditions and factors within the scope of the attached claims or their equivalents.

[0293] Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and not only the claims described below but also all scopes equivalent to or changed from the claims equivalently fall within the scope of the spirit of the present invention.

Claims

1. A method for monitoring abnormal conditions based on multiple sensors, the method comprising: determining an operation mode based on condition information in a target area where an abnormal condition is detected; receiving sensing signals from a plurality of sensors installed in the target area in response to a first mode in which different objects are detected simultaneously being determined as the operation mode; processing the sensing signal and outputting first sensing data, wherein the first sensing data is obtained by converting the sensing signal into a digital signal; By variably adjusting the size of the first sensing data, generating first input data to be input to a first neural network model; performing an operation on the first input data in an analog domain using the first neural network model, and outputting first operation data obtained by converting the operation result into digital data; and A first monitoring result obtained by classifying the abnormal condition based on the first operation data is output.

2. The method according to claim 1, wherein the outputting the first sensing data comprises: selecting one of the sensing signals based on a first selection control signal; amplifying the selected sensing signal based on the amplification control signal and the gain control signal; and The amplified sensing signal is digitally converted to output the sensing data. 3 . The method according to claim 1 , further comprising updating the status information in the target area based on the first monitoring result.

4. The method according to claim 1, further comprising: In response to a second mode of detecting an object associated with the condition information in real time being determined as an operation mode, selecting a target sensor from a plurality of sensors installed in the target area; processing a target signal received from the target sensor and outputting second sensing data, the second sensing data being obtained by converting the target signal into a digital signal; By variably adjusting the size of the second sensing data, generating second input data for input to a second neural network model; performing an operation on the second input data in an analog domain using the second neural network model, and outputting second operation data, the second operation data being obtained by converting the operation result into digital data; and A second monitoring result is output, the second monitoring result being obtained by classifying the abnormal condition based on the second operation data.

5. The method according to claim 4, wherein the selecting a target sensor comprises selecting a sensor configured to detect a main object related to the status information as a main target sensor among the multiple sensors, and selecting a sensor configured to detect a sub-object related to the main object as a sub-target sensor.

6. The method according to claim 4, wherein the outputting the second sensing data comprises: receiving the target signal from the target sensor based on a second selection control signal in response to the target sensor being selected; amplifying the target signal based on the amplification control signal and the gain control signal; and The second sensing data is output by performing digital conversion on the amplified target signal.

7. The method according to claim 4, wherein the second neural network model is a model used in response to the second mode being determined as the operation mode, and is a model that performs the operation by considering input data at a time before the operation is performed.

8. The method according to claim 4, wherein the generating the second input data comprises accumulating the second sensing data to correspond to the number of bits enabled within the maximum resolution based on an enable signal specifying the number of bits, and generating the second input data having an output resolution corresponding to the number of bits.

9. The method according to claim 4, wherein the outputting the second operation data comprises: converting the second input data into a second input value in the analog domain; performing a convolution operation on the second input value in the analog domain using a plurality of SRAM operators; and The second operation data obtained by converting the second analog convolution result into data in the digital domain is output, and the second analog convolution result is an output value of the operation.

10. The method according to claim 4, wherein outputting the second monitoring result comprises: determining a category of the abnormal condition in response to an object detected by the target sensor; and The second monitoring result is output by classifying the risk level within the determined category of the abnormal condition.

11. The method according to claim 5, wherein outputting the second monitoring result comprises: determining a category of the abnormal condition based on the abnormal condition classification result of each of the main object and the sub-objects; and The second monitoring result is outputted by considering the risk level classification of the main object and the risk level classification of the sub-object within the determined category of the abnormal condition. 12 . The method according to claim 4 , further comprising updating the status information in the target area based on the second monitoring result.

13. The method of claim 1 or 4, wherein the plurality of sensors comprises at least one of a gas sensor, a pressure sensor, or a temperature sensor.

14. An abnormal condition monitoring device based on multiple sensors, comprising: a memory configured to store at least one program; and A processor configured to execute the at least one program, wherein the processor is configured to: determining an operation mode based on condition information in a target area where an abnormal condition is detected; in response to a first mode in which different objects are simultaneously determined being determined as the operation mode, processing sensing signals received from a plurality of sensors installed in the target area and outputting first sensing data obtained by converting the sensing signals into digital signals, By variably adjusting the size of the first sensing data, generating first input data to be input to a first neural network model; performing an operation on the first input data in an analog domain using the first neural network model, and outputting first operation data obtained by converting the operation result into digital data; and A first monitoring result obtained by classifying the abnormal condition based on the first operation data is output.

15. The abnormal condition monitoring device according to claim 14, wherein the processor is further configured to: In response to a second mode of detecting an object associated with the situation information in real time being determined as the operation mode, selecting a target sensor from among the plurality of sensors installed in the target area; processing a target signal received from the target sensor and outputting second sensing data obtained by converting the target signal into a digital signal; By variably adjusting the size of the second sensing data, generating second input data for input to a second neural network model; performing an operation on the second input data in an analog domain using the second neural network model, and outputting second operation data obtained by converting the operation result into digital data; and A second monitoring result obtained by classifying the abnormal condition based on the second operation data is output.

16. A computer-readable recording medium having stored thereon a program for executing the method according to claim 1 on a computer.

Images