Integrated Sensing System

Abstract

To provide an integrated sensing system that performs environmental multi-modality sensing.SOLUTION: An integrated sensing system includes: a first sensing unit that generates a first modality sensing output on the basis of an environment; a first edge artificial intelligence (AI) engine which controls the first sensing unit, and which generates a first data analysis result on the basis of the first modality sensing output; a second sensing unit that generates a second modality sensing output on the basis of the environment; a second edge AI engine which controls the second sensing unit, and which generates a second data analysis result on the basis of the second modality sensing output; and a computer processor that generates an environmental classification result on the basis of the first data analysis result and of the second data analysis result using a central AI algorithm. The computer processor is directly coupled to the first edge AI engine and to the second edge AI engine.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an integrated sensing system. [Background technology]

[0002] "Modality" is a term that refers to the form of sensation. Modalities include, for example, vision, hearing, touch, taste, smell, temperature, and balance, which correspond to the perception of visible signals, audible signals, vibrations and movement, taste, smell, heat, and balance.

[0003] Cloud computing is a form of computer system resource utilization that makes computer system resources such as data storage and computing power available to many users via the Internet. Computer system resources available via the Internet through cloud computing are called the cloud. Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, most sensing systems only have one modality, which detects specific stimuli within a limited range. As a result, sensing systems are applied in a highly specialized and fragmented manner without a general-purpose platform. When there are multiple systems with very limited sensing modalities, a central computing system in the cloud connected to a network must control the multiple sensing modalities, which increases costs and time delays. [Means for solving the problem]

[0005] In one principal aspect, the present invention relates to an integrated sensing system for multi-modality sensing of an environment. ,beforea first edge artificial intelligence (AI) engine that controls the first sensing unit and generates a first data analysis result based on the first modality sensing output; a second sensing unit that generates a second modality sensing output about the environment; a second edge AI engine that controls the second sensing unit and generates a second data analysis result based on the second modality sensing output; and a computer processor that uses a central AI algorithm to generate a classification result of the environment based on the first data analysis result and the second data analysis result, wherein the computer processor is an integrated sensing system directly connected to the first edge AI engine and the second edge AI engine, the integrated sensing system further comprising a communication interface that communicates with an additional integrated sensing system via a private blockchain network, and the computer processor generates the classification result of the environment by exchanging intermediate classification results between the integrated sensing system and the additional integrated sensing system via the communication interface, and the classification result is generated by cooperation between the integrated sensing system and the additional integrated sensing system based on at least the intermediate classification result.

[0006] In one principal aspect, the present invention relates to a method for multi-modality sensing of an environment, the method comprising: , thgenerating, by a first edge AI engine that controls the second sensing unit, a second data analysis result based on the second modality sensing output; and generating, by a computer processor using a central AI algorithm, a classification result of the environment based on the first data analysis result and the second data analysis result, wherein the computer processor is directly coupled to the first edge AI engine and the second edge AI engine, wherein generating the classification result of the environment by the computer processor includes communicating intermediate classification results between the integrated sensing system and an additional integrated sensing system via a private blockchain network, and the classification result is generated by cooperation between the integrated sensing system and the additional integrated sensing system based on at least the intermediate classification result.

[0007] In one main aspect, the present invention relates to a program for performing multi-modality sensing of an environment. , tha first edge artificial intelligence (AI) engine controlling the first sensing unit generating a first data analysis result based on the first modality sensing output; a second sensing unit generating a second modality sensing output about the environment; a second edge AI engine controlling the second sensing unit generating a second data analysis result based on the second modality sensing output; and a central AI engine using a central AI algorithm realizing functions for generating a classification result of the environment based on the first data analysis result and the second data analysis result, wherein a computer processor is a program directly connected to the first edge AI engine and the second edge AI engine, and generating the classification result of the environment includes exchanging intermediate classification results between the integrated sensing system and an additional integrated sensing system via a private blockchain network, and the classification result is generated by collaboration between the integrated sensing system and the additional integrated sensing system based on at least the intermediate classification result.

[0008] Other aspects of the present invention will become apparent from the following description and the appended claims. [Brief explanation of the drawings]

[0009] [Figure 1] 1 illustrates a system according to one or more embodiments of the present invention. [Figure 2] 1 illustrates a flowchart in accordance with one or more embodiments of the present invention. [Figure 3A] 1 illustrates examples according to one or more embodiments of the present invention. [Figure 3B] 1 illustrates examples according to one or more embodiments of the present invention. [Figure 3C] 1 illustrates examples according to one or more embodiments of the present invention. [Figure 4] 1 illustrates a computing system according to one or more embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, in which like elements in the various figures are designated with the same reference numerals for consistency.

[0011] In the following detailed description of the embodiments of the present invention, numerous specific details are set forth to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without these specific details. Additionally, well-known features have not been described in detail to avoid unnecessarily complicating the description.

[0012] Primarily, an embodiment of the present invention provides an integrated sensing system that comprehensively combines multiple sensing modalities for sensing multiple different stimuli. The integrated sensing system has a sensing unit with a high sensing range and functions as a common sensing platform for many applications. The integrated sensing system independently completes the sensing, control, and decision-making processes using artificial intelligence (AI) without accessing a networked control computing system through the cloud. This realizes a secure sensing method with low cost and fast response time. The sensing of specific stimuli is determined by the decision-making AI within the integrated sensing system, but collaborative intelligence may also be utilized. In collaborative intelligence, multiple integrated sensing systems support each other through a highly secure blockchain.

[0013] FIG. 1 illustrates an integrated sensing system 100 according to one or more embodiments of the present invention. As shown in FIG. 1, the integrated sensing system 100 includes multiple components, including, for example, a buffer 104, a central artificial intelligence (AI) engine 101, and multiple edge AI engines. The edge AI engines are connected to corresponding sensing units. For example, edge AI engine A 110a is connected to sensing unit A 111a, edge AI engine B 110b is connected to sensing unit B 111b, and edge AI engine C 110c is connected to sensing unit C 111c. These sensing units have multiple modalities and are located at various locations within an environment 150. The environment 150 is a physical area within the sensing range of the sensing units of the integrated sensing system 100. The environment 150 may include natural objects (e.g., living and non-living objects) and / or man-made objects (e.g., buildings, machinery, etc.).

[0014] Each component (101, 104, 110a, 110b, 110c) may be located on the same computer device (e.g., a personal computer (PC), laptop, tablet PC, smartphone, multifunction printer, kiosk, server, etc.), or may be located on different computer devices directly connected without a network (e.g., a wide area network or a portion of the Internet of any size having wired and / or wireless segments). Each component will be described below.

[0015] In one or more embodiments of the present invention, the buffer (104) may be implemented in hardware (i.e., circuitry), software, or any combination thereof. The buffer (104) is configured to store data generated and / or used by the integrated sensing system (100). As shown in FIG. 1 , the sensing unit generates sensing outputs (105) for the edge AI engine. The edge AI engine generates data analysis results (106) for the central AI engine (101). The central AI engine (101) generates intermediate classification results (107) and classification results (108). Specifically, the sensing outputs (105) correspond to measurements of changes or events detected in the environment (150) by various modalities at various times. The data analysis results (106) are initial assessments of events occurring in the environment (150) by each edge AI engine based on its respective sensing outputs. Specifically, each individual sensing output is from a single modality. The intermediate classification result (107) and classification result (108) are intermediate and final assessments of events occurring in the environment (150) by the central AI engine (101) based on the aggregate of data analysis results from multiple modalities.

[0016] The sensing output (105), data analysis result (106), intermediate classification result (107), and classification result (108) may be part of the overall intermediate and final data of the integrated sensing system (100). The sensing output (105), data analysis result (106), intermediate classification result (107), and classification result (108) may be of any size and in any suitable format. While the buffer (104) is shown as a single component, in other embodiments of the present invention, it may be divided into multiple separate components. For example, each sensing output (105) and each data analysis result (106) may be stored locally with the corresponding edge AI engine, while the intermediate classification result (107) and classification result (108) may be stored locally with the central AI engine (101).

[0017] In one or more embodiments of the present invention, sensing unit A (111a), sensing unit B (111b), and sensing unit C (111c) are each physical devices configured to detect events or changes in the environment (150). As such, each sensing unit generates a sensing output representative of the detected events or changes. Typically, the sensing units are configured to detect different types of events or changes in the environment (150), with each type of event or change corresponding to a particular modality. For example, the sensing output generated by sensing unit A (111a) is a first modality sensing output, the sensing output generated by sensing unit B (111b) is a second modality sensing output, and the sensing output generated by sensing unit C (111c) is a third modality sensing output. The first, second, and third modality sensing outputs are part of the sensing output (105).

[0018] In one or more embodiments of the present invention, the edge AI engine A (110a), the edge AI engine B (110b), and the edge AI engine C (110c) may each be implemented in hardware (i.e., a circuit), software, or any combination thereof. Typically, the edge AI engines are configured to control corresponding sensing units to analyze corresponding sensing outputs and generate data analysis results. In particular, the edge AI engine A (110a) is configured to control the sensing unit A (111a) to analyze the sensing output of the sensing unit A (111a) and generate corresponding data analysis results (referred to as a first data analysis result). Specifically, the edge AI engine A (110a) generates a first data analysis result based on the sensing output of the sensing unit A (111a). Similarly, the edge AI engine B (110b) is configured to control the sensing unit B (111b) to analyze the sensing output of the sensing unit B (111b) and generate corresponding data analysis results (referred to as a second data analysis result). Specifically, the edge AI engine B (110b) generates a second data analysis result based on the sensing output of the sensing unit B (111b). Furthermore, the edge AI engine C (110c) is configured to control the sensing unit C (111c) to analyze the sensing output of the sensing unit C (111c) and generate a corresponding data analysis result (referred to as a third data analysis result). Specifically, the edge AI engine C (110c) generates a third data analysis result based on the sensing output of the sensing unit C (111c). The first, second, and third data analysis results are part of the data analysis result (106).

[0019] In one or more embodiments of the present invention, the central AI engine (101) may be implemented in hardware (i.e., circuitry), software, or any combination thereof. The central AI engine (101) is configured to generate a classification result (108) of the environment (150) using a central AI algorithm based on the data analysis result (106). The central AI engine (101) is further configured to provide the classification result (108) to an alarm unit (not shown) of the environment (150). The alarm unit is configured to perform an alarm task (e.g., issue a visual or audible warning) for the environment (150) based on the classification result (108). An application example of the central AI engine (101) and the alarm unit of the environment (150) is described with reference to Figures 3A-3C.

[0020] Although not explicitly shown, the central AI engine (101) may include a communication interface configured to communicate with one or more additional integrated sensing systems through a connection network (e.g., a point-to-point connection network or a private blockchain network). Thus, multiple integrated sensing systems, including the integrated sensing system (100), can exchange intermediate classification results via the communication interface and the connection network. As a result, the interconnected integrated sensing systems can collaborate to generate a classification result (108) based on the intermediate classification results.

[0021] In one or more embodiments of the present invention, the central AI engine (101), edge AI engine A (110a), edge AI engine B (110b), and edge AI engine C (110c) perform the above-described functions using the method described below with reference to FIG. 2.

[0022] Although the integrated sensing system (100) is shown as having three edge AI engines (110a, 110b, 110c) with corresponding sensing units and two other components (101, 104), other embodiments of the present invention may have more or fewer edge AI engines and more or fewer other components. Furthermore, the functionality of each of the above-described components may be shared among the components. Furthermore, each of the components (101, 104, 110a, 110b, 110c) may be used multiple times to perform repeated operations.

[0023] FIG. 2 illustrates a flowchart according to one or more embodiments of the present invention. One or more of the steps in FIG. 2 may be performed by components of integrated sensing system (100) (described above with reference to FIG. 1). In one or more embodiments of the present invention, one or more of the steps in FIG. 2 may be omitted, repeated, and / or performed in a different order than that shown in FIG. 2. Thus, the scope of the present invention should not be considered limited to the specific arrangement of steps shown in FIG. 2.

[0024] Referring to FIG. 2, first, in step 201, a sensing unit of a certain modality generates a sensing output. The sensing output may be generated continuously or in response to a command from an edge AI engine that controls the sensing unit. This command may be sent from the edge AI engine to the sensing unit periodically or intermittently, may be sent in response to a certain environmental condition, or may be sent based on other predetermined criteria. In addition, the sensing unit may be activated / deactivated by the edge AI engine that controls the sensing unit. Specifically, a sensing unit that is deactivated does not generate a sensing output.

[0025] In step 202, the sensing output is sent to an edge AI engine, which analyzes the sensing output to generate a data analysis result. Specifically, the edge AI engine generates a data analysis result based on the sensing output. If the sensing output is an analog signal, the sensing output is converted to a digital signal before being analyzed by the edge AI engine. The edge AI engine may analyze the sensing output using a physics-based machine learning algorithm.

[0026] In step 203, it is determined whether any additional sensing units are activated and available for evaluating events occurring in the environment. If the determination is positive, i.e., if at least one additional sensing unit is activated and available for evaluating events occurring in the environment, the process returns to step 201. If the determination is negative, i.e., if all sensing units for evaluating events occurring in the environment are activated and included, the process proceeds to step 204. Steps 201, 202, and 203 may be repeated continuously, periodically, or intermittently, or may be performed in response to certain conditions in the environment or based on other predetermined criteria.

[0027] In step 204, the central AI engine determines whether to include or exclude data analysis results of a particular modality when generating a classification result for the environment based on the data analysis results of all active sensing units. Specifically, the central AI engine determines which modality is suitable for classifying events occurring in the environment based on the initial assessment by the edge AI engines of the active sensing units. The central AI engine can make this determination based, for example, on deep machine learning using a convolutional neural network trained on labeled data from all sensing units in the integrated sensing system.

[0028] In step 205, it is determined whether an inactive sensing unit with an additional modality is required to evaluate events occurring in the environment. If the determination is positive, i.e., if at least one inactive sensing unit with an additional modality is required, proceed to step 207. If the determination is negative, i.e., if no inactive sensing unit with an additional modality is required, proceed to step 208.

[0029] In step 206, it is determined whether any of the active sensing units is unsuitable for assessing the events occurring in the environment. If the determination is positive, i.e., if at least one of the active sensing units is unsuitable, the process proceeds to step 207. If the determination is negative, i.e., if all of the active sensing units are suitable, the process proceeds to step 208.

[0030] In step 207, a command is sent to the edge AI engine to activate or deactivate the sensing unit controlled by the edge AI engine. If it is determined in step 205 that the sensing unit should provide the additional required modality, the command causes the edge AI engine to activate the sensing unit. If it is determined in step 206 that the modality provided by the sensing unit is not suitable, the command causes the edge AI engine to deactivate the sensing unit.

[0031] In step 208, a classification of the environment is generated by the central AI engine. The central AI engine generates the classification using a central AI algorithm based on all available analytical results. In some embodiments of the present invention, multiple integrated sensing systems are used to classify complex events occurring in the environment. In such embodiments, multiple connected integrated sensing systems exchange intermediate classification results and collaborate to generate a final classification result. For example, multiple integrated sensing systems can exchange intermediate classification results via point-to-point connections. As another example, multiple integrated sensing systems can exchange intermediate classification results via a private blockchain network.

[0032] In step 209, the central AI engine provides the environmental classification results to an environmental notification unit. The notification unit may be a system that issues an alarm, for example, an audio or visual alarm. The edge AI engine and the central AI engine exchange data using a direct connection, such as a hardwired electrical connection or a fiber optic connection. However, the central AI engine may provide the classification results to the notification unit via either a direct connection or a network connection.

[0033] 3A-3C illustrate examples according to one or more embodiments of the present invention. The examples of FIGS. 3A-3C are based on the system and method flowcharts described with reference to FIGS. 1 and 2. Specifically, components in FIGS. 1 and 3A-3C that are labeled with the same reference numerals correspond to each other. In one or more embodiments of the present invention, one or more of the components shown in FIGS. 3A-3C may be omitted, repeated, and / or arranged differently. Therefore, the scope of the present invention should not be considered limited to the specific arrangement of components shown in FIGS. 3A-3C.

[0034] In one or more embodiments, the examples in Figures 3A-3C correspond to an integrated sensing system. This integrated sensing system has multiple sensing modalities for sensing many different stimuli across a wide range of sensing applications, serving as a common sensing platform for many applications. Each sensing modality and its range is controlled by artificial intelligence, and the system operates via a networked cloud without the use of computing resources, optimizing operational costs, energy usage, and achieving high performance.

[0035] The table below shows the performance of the integrated sensing system compared to human sensing. [Table 1]

[0036] The integrated sensing system (300) shown in FIG. 3 is an example of the integrated sensing system (100) shown in FIG. 1. The sensing units of the integrated sensing system (300) include a tactile sensor (311a), an audio sensor (311b), a visual sensor (311c), an olfactory sensor (311d), and a temperature sensor (311e). These are examples of the sensing unit A (111a), sensing unit B (111b), and sensing unit C (111c) shown in FIG. 1. Each sensing unit is attached to an edge control unit consisting of an analog-to-digital converter (ADC) and an edge AI engine. The edge control units (310a, 310b, 310c, 310d, and 310e) are examples of the edge AI engine A (110a), edge AI engine B (110b), and edge AI engine C (110c) shown in FIG. 1.

[0037] In the exemplary integrated sensing system (300), signals from each sensing element are analog signals that are converted by an ADC into corresponding digital signals for the edge AI engine. Primary control and data analysis are performed at the edge AI engine level. All sensing elements and the edge AI engine are connected to a central AI engine (101) that performs decision-making. When complex data analysis and decision-making is required, a digital signal is sent from the edge AI engine to the central AI engine (101). The criteria for sending a digital signal from the edge AI engine to the central AI engine (101) are based on the signal-to-noise ratio, signal range, and the number of training datasets for each edge AI engine. For example, if the signal-to-noise ratio is less than 3.0, the signal is above or below the range listed in Table 1 above, and the number of inputs in the training dataset is less than 1,000, the edge AI engine cannot classify the events occurring in the environment (150). Therefore, the information is passed to the central AI engine, which makes the decision.

[0038] The algorithm of the edge AI engine is based on machine learning based on the physical model of each sensing modality. The central AI engine (101) determines which sensing modalities and edge AI engines should be fully activated to obtain the maximum signal for a specific stimulus. Unnecessary sensing modalities are placed in standby mode.

[0039] The central AI engine (101) performs classification and decision-making based on deep learning, which uses a convolutional neural network trained on labeled data from all sensing modalities. The algorithms of the central AI engine (101) also include transfer learning to deal with unlearned situations and reinforcement learning to select the optimal response.

[0040] For example, suppose the environment (150) corresponds to a nursing home. If the edge AI engine (310c) analyzes the sensing output of the visual sensor (311c) and finds that a person may have fallen, the central AI engine (101) activates the audio sensor (311b) to its highest level to detect noise and sound and assess the situation. The other sensors (311a, 311d, 311e) are placed in standby mode to optimize the operation of the integrated sensing system (300). In the optimized integrated sensing system (300), the central AI engine (101) accurately classifies the event occurring in the environment (150) as a person falling in a specific location and appropriately provides the classification result to the notification unit (350). The notification unit (350) is, for example, an alarm system that automatically dispatches nursing home staff to assist the person who has fallen in the specific location.

[0041] As another example, the environment (150) may correspond to a chemical plant. If the edge AI engine (310d) analyzes the sensing output of the olfactory sensor (311d) and finds an abnormal level of a specific odor / chemical, the central AI engine (101) activates the audio sensor (311b) to its highest level to identify the location of the chemical leak. Meanwhile, other unnecessary sensors (311a, 311c, 311e) are placed in standby mode. In an optimized integrated sensing system (300), the central AI engine (101) accurately classifies the event occurring in the environment (150) as a chemical leak at a specific location and appropriately provides the classification result to the alarm unit (350). The alarm unit (350) may be, for example, an automatic shut-off valve that automatically shuts off the piping network to isolate the chemical leak at the specific location.

[0042] As another example, suppose the environment (150) corresponds to an airport. If the edge AI engine (310c) analyzes the sensing output of the visual sensor (311c) and finds abnormal human behavior, the central AI engine (101) activates the olfactory sensor (311d) and audio sensor (311b) to their highest levels to detect volatile organic compounds (VOCs) and abnormal noise and assess the situation. Meanwhile, other unrelated sensors (311a, 311e) are placed in standby mode. In an optimized integrated sensing system (300), the central AI engine (101) accurately classifies an event occurring in the environment (150) as a terrorist act at a specific location and appropriately provides the classification result to the notification unit (350). The notification unit (350) is, for example, a security dispatch system that automatically dispatches security guards to a specific location.

[0043] The edge AI engine and associated sensors may be co-located with the central AI engine (101) or, depending on the specific situation, may be located separately from the central AI engine (101) via a wired or fiber optic connection, in this way ensuring the security of the sensing system and preventing it from being hacked or manipulated by an external source.

[0044] Figure 3B shows an example of a highly complex situation. In this example, multiple integrated sensing systems collaborate to analyze and assess the situation in the environment (150). For example, multiple integrated sensing systems are used when the number of inputs in the labeled training dataset of the central AI engine per integrated sensing system is less than 100,000. Integrated sensing system A (300a), integrated sensing system B (300b), integrated sensing system C (300c), integrated sensing system D (300d), and integrated sensing system E (300e) are connected, for example, by point-to-point connections. These integrated sensing systems A to E are copies or modifications of the integrated sensing system (300) shown in Figure 3A. There is no limit to the number of integrated sensing systems that can collaborate, depending on the complexity of the problem and the desired response time.

[0045] FIG. 3C illustrates an example of a secure collaboration environment (150). In this example, multiple integrated sensing systems are connected as peer nodes through a private blockchain network (310), and all data is exchanged cryptographically. Specifically, the blockchain network (310) is an infrastructure that provides ledger and smart contract services to integrated sensing system A (300a), integrated sensing system B (300b), integrated sensing system C (300c), integrated sensing system D (300d), and integrated sensing system E (300e). Integrated sensing systems A-E are replicas or modifications of the integrated sensing system (300) shown in FIG. 3A. Transactions are primarily performed using smart contracts, which are then transmitted to all peer nodes in the blockchain network (310) and immutably recorded in each peer node's copy ledger.

[0046] Embodiments of the present invention can be implemented in virtually any type of computing system, regardless of the platform used. For example, a computing system may be one or more portable devices (e.g., a laptop computer, a smartphone, a personal digital assistant, a tablet computer, or other portable device), a desktop computer, a server, a blade in a server chassis, or one or more other types of computing devices that include at least the minimum processing power, memory, and input / output devices required to implement one or more embodiments of the present invention. For example, as shown in FIG. 4 , a computing system (400) may include one or more computer processors (402), associated memory (404) (e.g., random access memory (RAM), cache memory, flash memory, etc.), one or more storage devices (406) (e.g., hard disks, optical drives such as compact disc (CD) drives or digital versatile disc (DVD) drives, flash memory sticks, etc.), and many other components and features. The computer processor (402) may be an integrated circuit for processing instructions. For example, the computer processor may be one or more cores or micro-cores of a processor. The computing system (400) may also include one or more input devices (410), such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or other type of input device. The computing system (400) may also include one or more output devices (408), such as a screen (e.g., a liquid crystal display (LCD), plasma display, touchscreen, cathode ray tube (CRT) monitor, projector, or other display device), printer, external storage device, or other output device. One or more of the output devices may be the same as or different from the input devices. The computing system (400) may be connected to a network (412) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or other type of network) via a network interface connection (not shown).Input and output devices may be connected locally or remotely (e.g., via a network 412) to the computer processor 402, memory 404, and storage 406. There are many different types of computing systems, and the input and output devices described above may take other forms.

[0047] Software instructions in the form of computer-readable program code for implementing embodiments of the present invention may be stored, in whole or in part, temporarily or permanently on a non-transitory computer-readable medium such as a CD, DVD, storage device, diskette, tape, flash memory, physical memory, or other computer-readable storage medium. In particular, the software instructions may correspond to computer-readable program code configured to implement embodiments of the present invention when executed by a processor.

[0048] Furthermore, one or more of the components of the computing system 400 may be remotely located and connected to the other components via a network 412. One or more embodiments of the present invention may also be implemented in a distributed system having multiple nodes, with portions of the present invention located on different nodes within the distributed system. In one embodiment of the present invention, a node may represent a separate computing device. Alternatively, a node may represent a computer processor with associated physical memory. Alternatively, a node may represent a computer processor or micro-core of a computer processor with shared memory and / or information sources.

[0049] One or more embodiments of the present invention may have one or more of the following advantages: utilization of an integrated sensing system with multiple highly sensitive modalities; control of the system through the cloud without relying on a large central computer system; completion of all decisions in the central AI engine of the integrated sensing system; utilization of collaborative intelligence for complex decisions; low-cost, fully functional sensing systems and rapid response without communication with the cloud; ability to select the most appropriate sensing modality; and high security through the use of blockchain.

[0050] While the present invention has been described with reference to a limited number of embodiments, those skilled in the art, having the benefit of this disclosure, will recognize that other embodiments may be devised without departing from the scope of the invention disclosed herein. Accordingly, the scope of the present invention is limited only by the appended claims.

Claims

1. 1. An integrated sensing system for multi-modality sensing of an environment, comprising: a first sensing unit that generates a first modality sensing output about the environment; a first edge artificial intelligence (AI) engine that controls the first sensing unit and generates a first data analysis result based on the first modality sensing output; a second sensing unit that generates a second modality sensing output about the environment; a second edge AI engine that controls the second sensing unit and generates a second data analysis result based on the second modality sensing output; a computer processor that uses a central AI algorithm to generate a classification result for the environment based on the first data analysis result and the second data analysis result; The computer processor is directly coupled to the first edge AI engine and the second edge AI engine.

1. An integrated sensing system, comprising: and a communication interface for communicating with the additional integrated sensing system via the private blockchain network. the computer processor generates the classification result of the environment by communicating intermediate classification results between the integrated sensing system and the additional integrated sensing system via the communication interface; The classification result is generated based on at least the intermediate classification result by cooperation of the integrated sensing system and the additional integrated sensing system. Integrated sensing system.

2. The computer processor further provides the classification result of the environment to an annunciator of the environment. The integrated sensing system of claim 1 .

3. a third sensing unit that generates a third modality sensing output about the environment; a third edge AI engine that controls the third sensing unit and generates a third data analysis result based on the third modality sensing output; The computer processor determining, using the central AI algorithm and based on the first data analysis result and the second data analysis result, to include the third data analysis result in generating the classification result for the environment; sending a command to the third edge AI engine to activate the third sensing unit when generating the third modality sensing output about the environment; and generating the classification result of the environment by The integrated sensing system according to claim 1 or 2.

4. The computer processor determining, using the central AI algorithm and based on the first data analysis result and the second data analysis result, not to include the second data analysis result when generating the classification result for the environment; sending a command to the second edge AI engine to deactivate the second sensing unit when generating the second modality sensing output about the environment; and generating the classification result of the environment by The integrated sensing system according to claim 1 or 2.

5. The first edge AI engine generates the first data analysis result using a physics-based machine learning algorithm. The integrated sensing system according to claim 1 or 2.

6. The central AI algorithm is based on deep learning using a convolutional neural network trained with labeled data from at least the first sensing unit and the second sensing unit. The integrated sensing system according to claim 1 or 2.

7. 1. A method for multi-modality sensing of an environment, comprising: generating a first modality sensing output about the environment by a first sensing unit; generating a first data analysis result based on the first modality sensing output by a first edge artificial intelligence (AI) engine controlling the first sensing unit; generating a second modality sensing output about the environment by a second sensing unit; generating a second data analysis result based on the second modality sensing output by a second edge AI engine that controls the second sensing unit; generating, by a computer processor using a central AI algorithm, a classification result for the environment based on the first data analysis result and the second data analysis result; The computer processor is directly coupled to the first edge AI engine and the second edge AI engine, generating the classification result of the environment by the computer processor, communicating the intermediate classification results between the integrated sensing system and the additional integrated sensing system via a private blockchain network; The classification result is generated by cooperation of the integrated sensing system and the additional integrated sensing system based on at least the intermediate classification result.

8. providing, by the computer processor, the classification result of the environment to an annunciator of the environment. The method of claim 7.

9. generating a third modality sensing output about the environment by a third sensing unit; and analyzing the third modality sensing output by a third edge AI engine that controls the third sensing unit to generate a third data analysis result. generating the classification result of the environment by the computer processor, determining, using the central AI algorithm and based on the first data analysis result and the second data analysis result, to include the third data analysis result in generating the classification result for the environment; and sending a command to the third edge AI engine to activate the third sensing unit when generating the third modality sensing output about the environment.

9. The method according to claim 7 or 8.

10. generating the classification result of the environment by the computer processor, determining, using the central AI algorithm and based on the first data analysis result and the second data analysis result, not to include the second data analysis result when generating the classification result for the environment; and sending a command to the second edge AI engine to deactivate the second sensing unit when generating the second modality sensing output about the environment.

9. The method according to claim 7 or 8.

11. The first edge AI engine generates the first data analysis result using a physics-based machine learning algorithm.

9. The method according to claim 7 or 8.

12. The central AI algorithm is based on deep learning using a convolutional neural network trained with labeled data from at least the first sensing unit and the second sensing unit.

9. The method according to claim 7 or 8.

13. A program for performing multi-modality sensing of an environment, the program comprising: generating a first modality sensing output about the environment by a first sensing unit; generating a first data analysis result based on the first modality sensing output by a first edge artificial intelligence (AI) engine that controls the first sensing unit; generating a second modality sensing output about the environment by a second sensing unit; generating a second data analysis result based on the second modality sensing output by a second edge AI engine that controls the second sensing unit; A central AI engine using a central AI algorithm realizes a function for generating a classification result of the environment based on the first data analysis result and the second data analysis result; a computer processor, the computer processor being a program directly coupled to the first edge AI engine and the second edge AI engine; The generation of the classification result of the environment includes: communicating the intermediate classification results between the integrated sensing system and the additional integrated sensing system via a private blockchain network; A program in which the classification result is generated by cooperation between the integrated sensing system and the additional integrated sensing system based on at least the intermediate classification result.

14. The computer, The central AI engine further realizes a function for providing the classification result of the environment to a notification unit of the environment. The program according to claim 13.

15. The computer, generating a third modality sensing output about the environment by a third sensing unit; a third edge AI engine that controls the third sensing unit further realizing a function for analyzing the third modality sensing output and generating a third data analysis result; The generation of the classification result of the environment by the computer processor includes: determining, using the central AI algorithm and based on the first data analysis result and the second data analysis result, to include the third data analysis result in generating the classification result for the environment; and sending a command to the third edge AI engine to activate the third sensing unit when generating the third modality sensing output about the environment.

15. The program according to claim 13 or 14.

16. The generation of the classification result of the environment by the computer processor includes: determining, using the central AI algorithm and based on the first data analysis result and the second data analysis result, not to include the second data analysis result when generating the classification result for the environment; and sending a command to the second edge AI engine to deactivate the second sensing unit when generating the second modality sensing output about the environment.

15. The program according to claim 13 or 14.

17. the first edge AI engine generates the first data analysis result using a physics-model-based machine learning algorithm; The central AI algorithm is based on deep learning using a convolutional neural network trained with labeled data from at least the first sensing unit and the second sensing unit.

15. The program according to claim 13 or 14.

Images