Edge devices, sensor systems, and sensor operation systems for operating multiple edge devices

Abstract

This provides an edge device that can perform artificial intelligence learning processes related to the processing of signals from sensors on the edge side. [Solution] In the edge device 100.j, the first artificial intelligence computing unit 2400 reduces the amount of information contained in the signals from sensors 300.j.1 to 300.j.mj and outputs them as compressed information. The second artificial intelligence computing unit 2600 receives the output from the first artificial intelligence computing unit 2400 and classifies the pattern of change in the observed quantity of the object observed by the sensor. Based on the compressed information from the first artificial intelligence computing unit 2400, the second artificial intelligence computing unit 2600 performs a learning process and updates the parameter group that defines the artificial intelligence model from its initial values.

Description

Technical Field

[0001] The present invention relates to an edge device that executes processing on sensor data on the edge side of a system, a sensor system using the edge device, and a sensor operation system that provides data for operating a plurality of edge devices.

Background Art

[0002] (Expansion of the Application Field of Artificial Intelligence Technology)

[0003] In recent years, the fields of use of artificial intelligence technology have been rapidly expanding. In particular, due to the development of deep learning technology, the computer has become capable of extracting the feature amounts of a target by itself, a large amount of data (big data) necessary for such artificial intelligence learning has become available, and furthermore, there has been development of hardware that can handle learning processes with a large computational processing load. These three factors are said to have led to such an expansion of applications.

[0004] For example, as image recognition using deep learning technology, there are technologies such as a face authentication system and a gait authentication system (see, for example, Patent Document 1 and Patent Document 2). Here, in technologies such as gait authentication, as a pre-authentication step, a learning model (trained model) learned by machine learning such as deep learning is used, and a distance image is input into the trained model, and a "human region" in the distance image is obtained from the trained model. Such a process is executed.

[0005] Furthermore, the free cloud service "Google Colaboratory (Google Colab)" has created an environment where pose estimation technology using artificial intelligence (AI) can be easily implemented. Previously, estimating a person's pose in an image was impossible without using an expensive camera capable of estimating depth, but recent advances in AI technology have made it possible to do so even with an inexpensive monocular camera. Using a system called "Open Pose," it is even possible to perform the advanced task of simultaneously estimating the poses of multiple people by using deep learning.

[0006] Furthermore, with the increasing prevalence of surveillance cameras and visitor monitoring using cameras in public facilities, buildings, and other structures, technologies are being developed that utilize image sensors installed within buildings and sensor systems using image sensors to understand people's movements within buildings through computational processing using AI-based judgment models (see, for example, Patent Document 3).

[0007] Alternatively, Patent Document 4 discloses a technique for collecting multiple data points acquired by sensors installed on monitored equipment such as factories and plants, and using a pre-trained artificial intelligence model to predict the behavior of target data as input, outputting a predicted value of the behavior of the target data as the output value. In this case, the object of observation is time-series data collected from a group of sensors, and the so-called "sparse estimation method" is used as the prediction method.

[0008] On the other hand, there is a demand for processing devices that can perform very large-scale processing tasks, such as image processing, quickly and easily. Therefore, processor devices that meet these needs have been proposed (see, for example, Patent Document 5). In many cases, parallel processing is a promising method for performing large-scale processing tasks at high speed. However, parallel processing requires various highly specialized skills, such as expertise in program construction and communication between processors based on data dependencies.

[0009] Therefore, the processor device disclosed in Patent Document 5 includes a memory access device that reads data to be processed from external memory and writes it to a first group of registers that is not being accessed by any of the processors. The control unit of this processor device sequentially executes the same instruction on each of the multiple processors, changing the address of the group of registers that stores the data to be processed. Based on specified scenario information, the scheduler specifies the instructions to be executed and the group of registers to be processed for the multiple processors, and specifies to the memory access device the group of registers to be written and the data to be processed to be written. A processor device with this configuration can be applied to various arithmetic operations (including artificial intelligence arithmetic operations) while being faster than conventional processor devices. [Prior art documents] [Patent Documents]

[0010] [Patent Document 1] Japanese Patent Publication No. 2019-219721 Specification [Patent Document 2] Japanese Patent Publication No. 2021-124868 Specification [Patent Document 3] Japanese Patent Publication No. 2021-39592 Specification [Patent Document 4] Japanese Patent Publication No. 2021-149590 Specification [Patent Document 5] International Publication 2019 / 208566 Specification [Overview of the project] [Problems that the invention aims to solve]

[0011] However, as mentioned above, the training process in deep learning is computationally intensive. Therefore, the generation and updating of so-called learning models are currently carried out by using on-premise servers with a certain level of computing power (as in Patent Document 3) or servers with large computing power on the cloud, consuming a lot of computing resources.

[0012] However, while aggregating data from multiple sensors onto a single server and performing learning processing may be desirable from a computing resource perspective, as mentioned above, it presents the following problems.

[0013] 1) For example, if each sensor is an image sensor, and the amount of data transmitted from the sensor to the server via communication is large, the network communication load will increase as the number of sensors increases. With the further development of IoT (Internet of Things), as sensors are deployed throughout the physical space, processing delays may occur due to increased communication capacity and increased processing load on the server side.

[0014] 2) System configurations where not only learning processes but also prediction and decision-making processes are performed on the server side are not suitable for applications that require immediate responsiveness.

[0015] 3) If the data acquired by the sensor includes personal information such as images of people (especially facial images), directly transmitting the data from the sensor (edge) to the server may pose security risks.

[0016] 4) Furthermore, when installing sensors (such as image sensors) in multiple locations and areas to collect data for applications such as security or analysis of pedestrian flow and movement patterns, the numerous sensor installation locations place a burden on the system installer in terms of adjustment and calibration during installation.

[0017] The present invention was made to solve the above-mentioned problems, and its objective is to provide an edge device that can perform artificial intelligence learning processing related to the processing of signals from sensors on the edge side.

[0018] Another object of the present invention is to provide a sensor operation system for operating edge devices capable of performing artificial intelligence learning processes at the edge. [Means for solving the problem]

[0019] According to one aspect of this invention, an edge device provided in correspondence with a sensor and for performing processing on signals from the sensor comprises: a sensor interface for receiving signals from the corresponding sensor; a first artificial intelligence module that performs processing by a first artificial intelligence model to reduce the amount of information contained in the signal and output it as compressed information in accordance with the signal from the sensor; and a second artificial intelligence module that receives the output from the first artificial intelligence module and classifies the pattern of change of the observed quantity of an object observed by the sensor by a second artificial intelligence model, wherein the first artificial intelligence model is defined by a first set of parameters, the second artificial intelligence model is defined by a second set of parameters, the second artificial intelligence module updates the second set of parameters by performing a learning process based on the compressed information, and the first artificial intelligence module maintains the first set of parameters in a fixed state at least during the update process of the second set of parameters.

[0020] Preferably, the edge device further comprises a communication interface for communicating with an external network and a non-volatile memory for storing a first set of parameters and a second set of parameters, wherein the initial values ​​of the first set of parameters and the second set of parameters are supplied via the communication interface and stored in the non-volatile memory before the start of the update process by the second artificial intelligence module.

[0021] Preferably, the sensor includes an image sensor, the first artificial intelligence module executes a process of extracting a target from image data from the image sensor and converting it into a feature amount of the target, and the second artificial intelligence module executes a process of classifying a displacement pattern of the target in the time direction, and updates a second parameter group by unsupervised learning.

[0022] Preferably, the second artificial intelligence module updates the second parameter group by learning the occurrence probability of the displacement pattern of the target based on dictionary data created by dictionary learning.

[0023] Preferably, the dictionary learning is sparse dictionary learning.

[0024] Preferably, the second artificial intelligence module updates the second parameter group by learning the occurrence probability of the displacement pattern of the target by reservoir computing.

[0025] Preferably, the first artificial intelligence module and the second artificial intelligence module are executed in the same processor device, and the first parameter group and the second parameter group each include scenario information defining the processing steps of the first artificial intelligence module and the second artificial intelligence module, and the processor device includes a plurality of processors, a plurality of register groups each including a plurality of registers for storing data to be processed, a memory access device that reads data to be processed from non-volatile memory based on an input instruction and writes it to a register group in the plurality of register groups, a control unit that sequentially executes the same instruction in each of the plurality of processors while changing the address of one register group storing the data to be processed corresponding to the number of the plurality of processors, a scheduler that instructs the plurality of processors to execute instructions and register groups to be processed based on the specified scenario information, and instructs the memory access device to write the register group to be written and the data to be processed to be written among the plurality of register groups.

[0026] Preferably, the sensor acquires time-series data, the first artificial intelligence module performs a process to convert the time-series data from the sensor into features, and the second artificial intelligence module performs a process to classify the displacement patterns of the time-series data, updating the second set of parameters by unsupervised learning.

[0027] In another aspect of this invention, a sensor operation system comprising a management server, a plurality of sensors, and a plurality of edge devices provided corresponding to each of the plurality of sensors, wherein each edge device includes a sensor interface for receiving signals from the corresponding sensor, a communication interface for communicating with an external network, a non-volatile memory for storing a first set of parameters and a second set of parameters, a first artificial intelligence module that performs processing by a first artificial intelligence model to reduce the amount of information contained in the signal and output it as compressed information in response to the signal from the sensor, and a second artificial intelligence model that receives the output from the first artificial intelligence device and processes the patterns of movement and morphological changes of the object observed by the sensor. The system comprises a second artificial intelligence module that classifies by a first set of parameters, the first artificial intelligence model being defined by a first set of parameters, the second artificial intelligence model being defined by a second set of parameters, the second artificial intelligence module updating the second set of parameters by performing a learning process based on compressed information, the first artificial intelligence module keeping the first set of parameters fixed for at least the duration of the update process of the second set of parameters, the management server distributing the initial values ​​of the first set of parameters and the second set of parameters to multiple edge devices via a communication interface before the update process by the second artificial intelligence module begins, and each edge device storing the distributed initial values ​​in non-volatile memory.

[0028] Preferably, the first set of parameters includes scenario information defining a first structure of the first artificial intelligence model and parameter values ​​defining the computational processing that the first artificial intelligence model performs based on the first structure, and the second set of parameters includes scenario information defining a second structure of the second artificial intelligence model and parameter values ​​defining the computational processing that the second artificial intelligence model performs based on the second structure.

[0029] Preferably, the multiple sensors each include an image sensor, the management server includes edge servers provided for each of the multiple edge devices to perform data distribution to the multiple edge devices and receive requests from the multiple edge devices, the edge servers include a storage device that stores image data from the image sensors and compressed information generated by processing the image data by the first artificial intelligence module in association with each other, and the management server performs a search for information already stored in the storage device in response to a query request to the edge server that includes specific compressed information.

[0030] Preferably, the image data stored in the storage device is associated with annotation information about the objects contained in the image data, and the edge server, in response to a query request, returns at least one of the image data stored in the storage device that matches specific compression information or annotation information associated with the matching image data.

[0031] Preferably, the first artificial intelligence module and the second artificial intelligence module are executed in the same processor device, the processor device including a plurality of processors, a plurality of register groups each containing a plurality of registers for storing data to be processed, a memory access device that reads data to be processed from non-volatile memory and writes it to a group of registers in the plurality of register groups based on input instructions, a control unit that sequentially executes the same instruction in each of the plurality of processors, changing the address of one group of registers storing data to be processed in accordance with the number of processors, and a scheduler that instructs the plurality of processors on the instructions to be executed and the group of registers to be processed, and instructs the memory access device on the group of registers to be written and the data to be processed to be written from among the plurality of register groups.

[0032] Preferably, the management server includes a service provider server that provides a tool environment for creating scenario information and parameter information to define the processing of the first artificial intelligence module and the second artificial intelligence module through user operation.

[0033] Preferably, the tool environment provides an existing first artificial intelligence module registered with the service provider server, according to the user's selection.

[0034] According to yet another aspect of this invention, a sensor operation system for processing data from multiple sensors comprises multiple edge devices, each corresponding to one of the multiple sensors; a central server for distributing artificial intelligence models executed on the edge devices to the multiple edge devices; and a service provision server for supporting the creation of artificial intelligence models, wherein each edge device includes a sensor interface for receiving signals from the corresponding sensor, a communication interface for communicating with an external network, a non-volatile memory for storing a first set of parameters and a second set of parameters, a first artificial intelligence module that performs processing by a first artificial intelligence model to reduce the amount of information contained in the signal and output it as compressed information in response to the signal from the sensor, and a second artificial intelligence model that receives the output from the first artificial intelligence device and processes the patterns of movement and morphological changes of the object observed by the sensor. The system comprises a first artificial intelligence model defined by a first set of parameters, a second artificial intelligence model defined by a second set of parameters, the second artificial intelligence module updates the second set of parameters by performing a learning process based on compressed information, the first artificial intelligence module maintains the first set of parameters in a fixed state at least during the update process of the second set of parameters, a service provider server provides a tool environment for creating initial models of the first and second artificial intelligence modules by user operation, the initial models include initial values ​​for the first and second set of parameters, a central server distributes the initial values ​​to multiple edge devices via a communication interface before the update process by the second artificial intelligence module begins, and each edge device stores the distributed initial values ​​in non-volatile memory. [Effects of the Invention]

[0035] According to the edge device of the present invention, it is possible to perform artificial intelligence learning processing related to the processing of signals from sensors on the edge side, so even when performing artificial intelligence learning, it is possible to suppress the increase in communication volume.

[0036] According to the edge device of the present invention, the adjustment and calibration of artificial intelligence parameters during installation can be performed on the edge device, thus reducing the man-hours required for installation.

[0037] More specifically, even when requesting information from images measured by sensors, it becomes possible to receive a response to the request by encrypting the communication and suppressing personally identifiable information during transmission. [Brief explanation of the drawing]

[0038] [Figure 1] This is a conceptual diagram illustrating the configuration of the sensor system 1000 of Embodiment 1. [Figure 2] This is a functional block diagram showing the configuration of the sensor device 400.1j. [Figure 3] This is a functional block diagram illustrating an example configuration of edge device 100.j. [Figure 4] This is a conceptual diagram illustrating the processing of time-series data judgment and classification using sparse dictionary learning. [Figure 5] This is a conceptual diagram illustrating the case where the processor device 100' executes the calculations of the first artificial intelligence calculation unit 2400 and the second artificial intelligence calculation unit 2600. [Figure 6] This is a block diagram showing the configuration of processor unit 100'. [Figure 7] This is an illustrative diagram showing the basic processing configuration flow of processor device 100'. [Figure 8] This diagram illustrates the configuration of data (parameters) necessary for the processor device 100' to execute the inference and training processes of deep learning. [Figure 9] This timing chart shows an example of the processing timing of the calculations performed by the first artificial intelligence calculation unit 2400 and the second artificial intelligence calculation unit 2600. [Figure 10] This is a functional block diagram illustrating an example of the configuration of edge device 100.j in a modified example 1 of Embodiment 1. [Figure 11] This is a functional block diagram illustrating an example of the configuration of edge device 100.j in a modified example 2 of Embodiment 1. [Figure 12] This figure shows a map of the store where the sensor system 1000 is applied in Embodiment 2. [Figure 13] This is a conceptual diagram showing the placement of the sensor system 1000 in a store. [Figure 14] This is a conceptual diagram illustrating the distribution and training of artificial intelligence models in a surveillance camera system, as well as the provision of services in camera system 1000. [Figure 15] This is a conceptual diagram illustrating an example of a customer behavior response service. [Figure 16] This is a conceptual diagram illustrating the operation of edge device 100.j for performing a modified version of Embodiment 2. [Figure 17] This is a conceptual diagram illustrating the process by which store staff can retrieve annotation information about a specific person from a terminal. [Figure 18] This is a block diagram illustrating the hardware configuration of Central Server 3000. [Figure 19] This is a functional block diagram illustrating the functional configuration of central server 3000. [Figure 20] This is a conceptual diagram showing the sensor system 1000 deployed in a theme park. [Figure 21] This is a conceptual diagram illustrating the process that takes place when a child gets lost. [Figure 22] This is a conceptual diagram illustrating the process by which the central server 300 functions as an artificial intelligence platform. [Modes for carrying out the invention]

[0039] The configuration of the edge device and sensor system according to the embodiments of the present invention will be described below. In the following embodiments, components and processing steps denoted by the same reference numerals are identical or equivalent, and their descriptions will not be repeated unless necessary.

[0040] In the following description, the server-side processing of the present invention will be described as a computer program installed on a single computer device that executes the corresponding processing. However, the server-side processing may be distributed across multiple computer devices, and the number of arithmetic units that execute the computer processing may be one or more.

[0041] Furthermore, edge device processing is not limited to execution as a program installed on a computer device; it may also be implemented as a dedicated semiconductor integrated circuit combining an arithmetic unit and a memory device, or as a single-board computer, FPGA (Field-Programmable Gate Array), or other electronic circuit. [Embodiment 1] (Sensor system configuration)

[0042] Figure 1 is a conceptual diagram illustrating the configuration of the sensor system 1000 of Embodiment 1.

[0043] Referring to Figure 1, sensor devices 400.11 to 400.1N (N: natural number) are installed in region 1, and sensor devices 400.21 to 400.1M (M: natural number) are installed in region 2.

[0044] While not particularly limited, "domain" refers to an area where data on the environment, fixed equipment, and operating devices within that domain, as well as data on moving objects (cars, bicycles, mobile robots, drones, etc.) and people within that domain, are collected by a group of sensors installed within that domain, such as a floor within a store, a specific building, a specific region (shopping street, shopping mall, etc.), or a factory or plant, as described later.

[0045] Furthermore, while Figure 1 uses two regions as an example, the number of regions is not limited to this.

[0046] For example, the sensor device 400.11 is equipped with sensors 301.1 to 300.1.m1 (m1: natural number), and has an edge device 100.1 for performing artificial intelligence processing on data from these sensors.

[0047] A sensor is an element or device that collects information about a target and converts it into a signal that can be handled by a machine. For example, it includes devices that perform detection to convert the state of an object or environment into numerical data, such as surveillance cameras and infrared cameras that acquire image information as still images or videos, microphones that acquire sound information, infrared thermometers and contact thermometers that acquire temperature information, pressure gauges that acquire pressure information, stress meters that acquire stress information, and magnetic field sensors that acquire magnetic field information.

[0048] Therefore, the signals (physical quantities) that sensors collect and replace can be a variety of types, including temperature, light, color, pressure, magnetism, velocity, and acceleration.

[0049] Furthermore, the other sensor devices 400.12~400.1N and 400.21~400.1M also similarly have sensors and corresponding edge devices.

[0050] In Figure 1, an edge server 500.1 is provided to aggregate data from sensor devices 400.11 to 400.1N installed in region 1, perform predetermined processing on the data collected by the sensors, and communicate data with the central server 3000 via network 2. Edge servers 500.2 are also provided in region 2 and for sensor devices 400.21 to 400.1M.

[0051] As described later, edge servers 500.1 and 500.2 aggregate data from sensor devices and store it in memory, and send requests to the central server (service provider server) 3000 in response to instructions and operations from users in the corresponding area. The central server 3000 replies with response information to requests from edge servers 500.1 and 500.2, and distributes data necessary for edge devices to operate, as described later.

[0052] Figure 2 is a functional block diagram showing the configuration of the sensor device 400.1j (j: natural number).

[0053] Figure 2 is an example of the configuration of a sensor device, and other sensor devices have a similar configuration.

[0054] Referring to Figure 2, the sensor device 400.1j includes sensors 300.j.1 to 300.j.mj and the edge device 100.j.

[0055] The edge device 100.j includes a sensor interface 2100 for receiving signals from sensors 300.j.1 to 300.j.mj, a computing module 2200 for receiving digitized signals detected by sensors from the sensor interface 2100 and executing artificial intelligence processing as described later, a non-volatile memory 2900 for temporarily storing parameter data and data in progress for the computing module 2200 to execute processing, and a network interface unit 2800 for exchanging data between the computing module 2200 and the edge server 500.1.

[0056] The arithmetic module 2200 includes a first artificial intelligence arithmetic unit 2400 that performs processing by a first artificial intelligence model to reduce the amount of information contained in a signal and output it as compressed information in response to a signal from sensor 300.j.1 to 300.j.mj, a second artificial intelligence arithmetic unit 2600 that receives the output from the first artificial intelligence arithmetic unit 2400 and classifies the pattern of change in the observed quantity of the object observed by the sensor using a second artificial intelligence model, and a controller 105 that controls the operation of the arithmetic module 2200.

[0057] As will be described later, the processing performed by the first artificial intelligence calculation unit 2400 and the processing performed by the second artificial intelligence calculation unit 2600 can be switched by the control of the controller 105. In general, the first artificial intelligence calculation unit 2400 may be configured to divide its output information into multiple related groups and output them to the second artificial intelligence calculation unit 2600 in group units. For example, when image data is input from a sensor, the first artificial intelligence calculation unit 2400 extracts skeletal information of multiple people in the image, and the second artificial intelligence calculation unit 2600 performs processing based on such skeletal information, such as the posture and changes of each person in the image, and the position and changes of each person. In such cases, the second artificial intelligence calculation unit 2600 receives the skeletal information extracted by the first artificial intelligence calculation unit 2400 for the image as a "group of related data" for each person.

[0058] The first artificial intelligence model of the first artificial intelligence computing unit 2400 is defined by a first set of parameters, and the second artificial intelligence model is defined by a second set of parameters.

[0059] As described later, the initial values ​​of the first and second parameter groups can be distributed from the central server 3000 via the edge server 500.1. The second artificial intelligence computing unit 2600 updates the second parameter group from its initial values ​​by performing a learning process based on the compressed information from the first artificial intelligence computing unit 2400.

[0060] As described later, the first artificial intelligence computing unit 2400 performs processing with fixed values ​​for at least the first set of parameters, without changing them from their initial values, during the update processing period for the second set of parameters.

[0061] Here, each edge device can send the updated second set of parameters to an edge server or central server and share them with other edge devices. For example, an edge device that is added later can receive the updated second set of parameters from an edge device that was installed earlier in the same area, thereby reducing the training time.

[0062] Although not particularly limited, the configuration may also involve a switch by the controller 105 described above, which allows data to be directly passed from the sensor interface unit 2100 to the second artificial intelligence calculation unit 2600 for processing.

[0063] Figure 3 is a functional block diagram illustrating an example of the configuration of edge device 100.j shown in Figure 2.

[0064] Here, sensor 300.j.1 is assumed to be a camera that captures optical images as a video.

[0065] Furthermore, the first artificial intelligence model used in the processing of the first artificial intelligence computing unit 2400 is a model that has been pre-trained using deep learning to detect people in an image and / or to detect a person's posture.

[0066] The parameters for defining such a first artificial intelligence model (for example, parameters that define the structure of the neural network and parameters that represent the weights of the nodes in the neural network, thus identifying the "trained model") are initially distributed from the central server 3000 via the edge server 500.1 and stored in non-volatile memory 2900 as target detection model parameters 2920.

[0067] The first artificial intelligence computing unit 2400 includes an object detection unit 2410 that performs processing to detect a person in an image and / or detect their posture (e.g., skeletal information) using a first artificial intelligence model defined by object detection model parameters 2920.

[0068] Although not particularly limited, the non-volatile memory 2900 may be configured to function as a working memory for the target detection unit 2410 to execute processing and to hold the target image temporary storage data 2910.

[0069] Furthermore, the second artificial intelligence model used in the processing of the second artificial intelligence computing unit 2600 is a model that classifies patterns of human movement detected in video footage through dictionary learning.

[0070] The dictionary data obtained as a result of dictionary learning includes, for example, time-series pattern dictionary data 2950 that shows movement patterns, and classification dictionary data 2960 that indicates which behavioral classification the extracted time-series patterns belong to.

[0071] While not particularly limited, for example, the classification processing unit 2610 classifies human behavior patterns in image data sent from sensor 300.j.1 based on time-series pattern dictionary data 2950 and classification dictionary data 2960. The classification processing unit 2610 may learn the probability of generating behavior patterns as a probabilistic model.

[0072] This, too, is not particularly limited, but for example, this probability model can be used to determine whether a person's actions in an image deviate from normal behavior (hereinafter referred to as "abnormal behavior").

[0073] The initial values ​​of the time-series pattern dictionary data 2950 and classification dictionary data 2960, as well as the initial values ​​of the probabilistic model of the classification processing unit 2610 for performing classification processing, can be configured to be distributed as pre-trained data from the central server 3000 via the edge server 500.1 and stored in the non-volatile memory 2900. The parameters of the probabilistic model of the classification processing unit 2610 are called edge model parameters 2980, and in this example, they correspond to the second set of parameters.

[0074] Furthermore, the pattern probability distribution learning processing unit 2620 updates the edge model parameters 2980 from their initial values ​​based on data from the sensor 300.j.1 at the installation site.

[0075] Compared to deep learning, learning a probabilistic model using a dictionary generally does not involve a large computational load, even if the learning process is performed on the edge side. (Sparse dictionary learning: for time-series data)

[0076] Figure 3 illustrates an example of learning a probabilistic model based on initially provided dictionary data.

[0077] However, when the features extracted by the first artificial intelligence computing unit 2400 are time-series data, a more general approach to compressing and processing time-series data has been proposed: sparse dictionary learning (sparse coding), which decomposes the data into a small number of dictionary bases.

[0078] For example, the following document discloses a time series decomposition method that uses a skip representation based on time skip operations to obtain a more interpretable time series decomposition, and a program that identifies time series using the decomposed and compressed representation of time series data. Publication 1: Genta Yoshimura, Atsunori Kanemura, Hideki Asoh, "Reconstructable and Interpretable Representations for Time Series with Time-Skip Sparse Dictionary Learning", Proceedings of the on Thematic Workshops of ACM Multimedia 2017, doi: 10.1145 / 3126686.3126724

[0079] Furthermore, the following literature also discloses a technique for representing time-series data using a sparse representation base called shapelets. Publicly known document 2: https: / / www.ismll.uni-hildesheim.de / pub / pdfs / grabocka2014e-kdd.pdf

[0080] Furthermore, the following literature also discloses techniques for classifying time-series data using sparse representation bases. Publicly known document 3: https: / / proceedings.neurips.cc / paper / 2006 / file / c922de9e01cba8a4684f6c3471130e4c-Paper.pdf

[0081] Therefore, the "dictionary data" shown in Figure 3 may be a "sparse dictionary" using such a "sparse representation base".

[0082] Figure 4 is a conceptual diagram illustrating an example of processing time-series data for judgment and classification using sparse dictionary learning.

[0083] As shown in Figure 4, in the learning process, time-series data from multidimensional sensors acquired in parallel by multiple sensors is first converted into a dimensionality-reduced representation through a learning process. Note that "time-series data from multidimensional sensors" also includes cases where data is acquired in time series for multiple objects within an image acquired by an image sensor.

[0084] However, such dimensionality reduction processing could, for example, involve using a one-dimensional convolutional neural network and applying a sliding convolutional filter to a one-dimensional input to learn features, or it could involve compressing the amount of information using a time skip operation, as described in the aforementioned publicly available document 1.

[0085] In this way, after reducing the dimensionality of the multidimensional sensor time series data, the feature representations are registered as dictionary data in a database, and a probabilistic model of the patterns in the time series data is generated by machine learning. In this case, for example, supervised learning may be used, but clustering may also be performed using unsupervised learning, and the classification significance of each cluster may be assigned to create a classification dictionary.

[0086] On the other hand, using the trained model generated in this way, classification and judgment results are output for data that has been dimensionally reduced from multidimensional sensor time series data.

[0087] While not particularly limited, when the second artificial intelligence computing unit 2600 learns the second artificial intelligence model on the edge side, it is generally difficult to obtain training data. Therefore, one example is to perform clustering as a probabilistic model for classification, as an "unsupervised learning" method.

[0088] However, if training data can be obtained on the edge device, it is of course possible to use "supervised learning." (Configuration of the arithmetic module 2200 implemented using semiconductor integrated circuits)

[0089] In the above explanation, the first artificial intelligence computing unit 2400 and the second artificial intelligence computing unit 2600 were described as being implemented using separate hardware, such as a board computer equipped with a GPU (Graphical Processing Unit).

[0090] However, in a single semiconductor integrated circuit as described in Patent Document 5 (International Publication 2019 / 208566), it is also possible to configure the system so that, based on scenario information and parameter data, the controller 105 controls the switching of processing so that the processing of the first artificial intelligence calculation unit 2400 and the second artificial intelligence calculation unit 2600 is executed by the same processor device 100'. (The concept of switching artificial intelligence processing via a controller)

[0091] Figure 5 is a conceptual diagram illustrating the case in which the first artificial intelligence calculation unit 2400 and the second artificial intelligence calculation unit 2600 are executed by the processor device 100'.

[0092] Based on the parameters and scenario information stored in the non-volatile memory 2900, the scheduler 110 switches the processing in the processor unit 100', and the arithmetic unit 10 executes the corresponding arithmetic processing.

[0093] The computational processes that are switched and executed include those of a Convolutional Neural Network (CNN). Here, CNNs can include both one-dimensional (time-series processing) and two-dimensional (image recognition processing).

[0094] Furthermore, the computational processing includes not only classification and decision processing based on dictionary learning, but also processes that are not necessarily performed by neural networks, such as coordinate transformation processing, frequency transformation processing, transcendental function calculation processing, and encryption calculation processing. (Conceptual concept of the configuration of arithmetic processing using semiconductor integrated circuits)

[0095] The configuration of such a processor device is disclosed in detail in Patent Document 5, so its configuration and operation will be described by extracting it to the extent necessary for the following explanation. (Processor unit configuration)

[0096] Therefore, the following describes a configuration in which the edge device 100.j in Figure 2 is realized by such a processor device 100'.

[0097] The processor unit 100' comprises external memory, multiple processors, multiple register groups, a memory access device, a control unit, and a scheduler. Each register group includes multiple registers for storing data to be processed. The memory access device reads the data to be processed from the external memory and writes it to a first register group among the multiple register groups that is not being accessed by any of the processors. The control unit causes each of the multiple processors to execute the same instruction sequentially, changing the address of the single register group storing the data to be processed. Based on specified scenario information, the scheduler specifies the instructions to be executed and the register groups to be processed for the multiple processors, and specifies to the memory access device the register group to be written and the data to be processed to be written.

[0098] Here, "external memory" refers to a large-capacity recording medium that has the function of storing the data to be processed.

[0099] Furthermore, "register memory" is a recording medium that has the function of temporarily storing data to be processed and has a smaller capacity than external memory.

[0100] A "processor" is the hardware component in a computer system that executes instructions.

[0101] "Access" refers to reading data from or writing data to memory.

[0102] A "bank" refers to a region of register memory that is accessible to the processor. By setting banks for register memory, different processors can simultaneously access different banks, as long as their accesses do not conflict.

[0103] Furthermore, "defining a bank for a processor" means specifying the address range that the processor can access within the register memory.

[0104] Figure 6 is a block diagram showing the configuration of the processor device 100'.

[0105] As shown in Figure 6, the processor unit 100' includes a DMA (Dynamic Memory Access controller) 101, memory 102, register memory 104, scheduler 110, and multiple arithmetic units 10. Each arithmetic unit 10 includes a multiplexer 103, a controller 107, and multiple processors 106. The scheduler 110 is connected to the DMA 101, the register memory 104, and the controller 107 of each arithmetic unit 10. The DMA 101 is connected to the scheduler 110, memory 102, register memory 104, and each of the processors 106 of each arithmetic unit 10. The register memory 104 is connected to the DMA 101, the scheduler 110, and the multiplexer 103 of each arithmetic unit 10.

[0106] The scheduler 110 reads data from memory 102 and instructs DMA 101 to write the data necessary for processing to register memory 104, according to the scenario information and program stored in memory 102. The scheduler 110 also identifies the calculation to be performed according to the scenario information and designates one arithmetic unit 10 from among multiple arithmetic units 10 to perform the processing corresponding to the calculation. The scheduler 110 instructs the arithmetic unit 10 on the calculation to be performed and the area (address) of register memory 104 to access.

[0107] In this explanation, we have used the example of specifying one arithmetic unit 10 as the arithmetic unit 10 that performs processing according to the calculation content, but the number of arithmetic units 10 that perform processing according to the calculation content is not limited to one, and there may be multiple units.

[0108] The DMA101 functions as a memory access device that reads data from memory 102 and stores it in register memory 104 according to instructions from the scheduler 110. The DMA101 also reads the results of calculations performed by the processor 106 stored in register memory 104 and writes them to memory 102.

[0109] Memory 102 is a large-capacity storage device that has the function of storing data to be processed by the processor device 100', as well as scenario information and programs that indicate the content of the processing.

[0110] Here, "large capacity" means a capacity greater than that of register memory 104. Memory 102 can be implemented, for example, by SDRAM (Synchronous Dynamic Random Access Memory). In the example shown in Figure 2, memory 102 corresponds to non-volatile memory 2900.

[0111] The multiplexer 103 accesses the register memory 104 according to the instructions given to the processor 106 for register access.

[0112] The register memory 104 consists of multiple registers. The register memory 104 is a memory that temporarily stores data to be processed by the processor 106. The register memory 104 can be implemented, for example, by SRAM (Static Random Access Memory). The number of registers provided corresponds to the number of logical processors. For example, if each processor has 16 registers, and there are 8 physical processors, and the logical number of processors is 1024, then the configuration can have 16 × 1024 = 16384 registers. In other words, the number of registers logically held is equivalent to that of 1024 processors, and the number that can be physically processed per unit time (1 cycle) is a maximum of 8.

[0113] The controller 107 within the arithmetic unit 10 specifies the calculation content to be processed (the instruction to be executed by the processor) and the address of the register memory 104 to each processor within the arithmetic unit 10, according to the processing instructed by the scheduler 110 (the calculation content that the arithmetic unit 10 should execute) and the address of the register memory 104. In other words, the controller 107 within the arithmetic unit 10 functions as a control device that sequentially executes instructions for each processor 106 while changing the address of the register memory 104 to be accessed. The controller 107 sequentially executes the same instruction (the calculation content that the arithmetic unit 10 should execute) for each processor 106 within the arithmetic unit 10, while changing the address of the register memory 104 to be processed. The controller 107 manages each processor 106 within the arithmetic unit 10, and when it receives a completion signal for the specified processing from the processor 106, it specifies the next instruction.

[0114] Therefore, the functions of controller 105 in the configuration of Figure 2 correspond to the functions realized by controller 107 and scheduler 110 in the configuration of Figure 6.

[0115] In this embodiment, there are eight physical processors 106 within the arithmetic unit 10. Here, the logical number of SIMD (single instruction, multiple data) operations that repeat the same calculation at once in the processor 106 is set to 1024. The processors 106 within the arithmetic unit 10 read and process data stored at a specified address in the register memory 104 via the multiplexer 103, in accordance with instructions from the controller 107 within the arithmetic unit 10, and store the calculation result at a specified address in the register memory 104. Each of the processors 106 within the arithmetic unit 10 executes the same instruction (the calculation that the arithmetic unit 10 should execute), while changing the data to be processed, that is, changing the address of the register memory 104 from which the data was read.

[0116] Figure 7 is an illustrative diagram showing the flow of the basic processing configuration of the processor device 100'.

[0117] The processor device 100' according to this embodiment executes the following processes according to the input scenario information (scenario information stored in memory 102) 200.

[0118] (1) The scheduler 110 receives the scenario information 200 as input and identifies the arithmetic unit 10 specified in the scenario information 200 from among the multiple arithmetic units 10. The scheduler 110 also communicates to the identified arithmetic unit 10 the processing content that the arithmetic unit 10 should execute and the address of the register memory 104 that it should access. Furthermore, the scheduler 110 specifies to the DMA 101 the data to be read from memory 102 and written to register memory 104.

[0119] (2) The DMA 101 reads the data to be processed from memory 102 and stores it in the second register group 104b. Here, the second register group 104b is a storage area composed of multiple registers belonging to register memory 104.

[0120] (3) Each processor 106 in the arithmetic unit 10 executes the same instruction (the operation that the arithmetic unit 10 should perform) specified by the controller 107 on the data stored in the second register group 104b, changing the address of the second register group 104b it references. In Figure 7, the passage of time is shown by depicting multiple processors 106 diagonally. That is, it shows that multiple processors 106 perform some operation on the data read from the register memory 104, and then perform another operation on the resulting value.

[0121] (4) While each processor 106 in the arithmetic unit 10 is executing the same instruction (the calculation that the arithmetic unit 10 should perform) using the data stored in the second register group 104b, the DMA 101 reads the next data to be processed from the memory 102 and stores it in the first register group 104a. Here, the first register group 104a is a storage area in the register memory 104 that is composed of multiple registers that do not belong to the second register group 104b.

[0122] (5) Each processor 106 in the arithmetic unit 10 stores the processing result (the result of the calculation that the arithmetic unit 10 should perform) in the second register group 104b. The storage location is predetermined and may be any register group other than the second register group 104b, as long as it is an area not accessed by the DMA 101.

[0123] (6) DMA101 reads the data stored in the second register group 104b and stores it in memory 102.

[0124] (7) Meanwhile, while the DMA 101 reads the calculation result stored in the second register group 104b and stores it in the memory 102, each processor 106 in the arithmetic unit 10 executes the same instruction (the calculation that the arithmetic unit 10 should execute) specified by the controller 107, changing the address of the first register group 104a that it references. As mentioned above, in Figure 7, the passage of time is shown by depicting the multiple processors 106 diagonally.

[0125] By repeating the processes described in (1) to (7) above, the arithmetic unit 10 does not need to wait for the data to be processed to be stored in the register memory 104. Therefore, it is possible to create a state in which each arithmetic unit 10 is always operating and executing processing, thereby achieving faster processing.

[0126] In this manner, each arithmetic unit 10 of the processor device 100' accesses the first register group 104a and the second register group 104b at timings specified by the scheduler 110 to read data and execute the same instruction (the processing instructed to the arithmetic unit 10). Meanwhile, the DMA 101 reads processing results from the register groups that are not being processed by the arithmetic unit 10 and writes them to memory 102, or writes data to be processed next. This allows the processor device 100' to process faster than when all the data to be processed is stored in register memory 104. Furthermore, in this configuration, the processor device 100' does not need to use an expensive register memory 104 that allows for high-speed access, thus providing an inexpensive processor device 100'.

[0127] Note that in Figure 7, the processing within the arithmetic unit 10 is shown, and therefore, to make the processing easier to understand, the multiplexer 103 is not shown.

[0128] The processor device 100' can be used for inference and training processes in deep learning. Inference and training processes in deep learning mostly involve multiply-accumulate operations and are well-suited for parallel processing.

[0129] Figure 8 illustrates the configuration of data (parameters) necessary for the processor device 100' to execute such deep learning inference and training processes.

[0130] Figure 8(a) shows an example of a learning model related to inference and learning processes in deep learning.

[0131] Generally, it is known that the learning model is represented by links connecting nodes. Figure 8(a) shows a simplified example, but the learning model consists of an input layer, a convolutional layer, a pooling layer, an output layer, etc. At each node, the processing content (integration, addition, etc.) is defined, and the dependencies are indicated by links. For example, when performing an inference process to estimate what kind of image an input image is, the register memory 104 stores the pixel values ​​for one line of the image, and each processor 106 performs the same processing (each processing indicated by the nodes f1, f2, ...) on these values. Then, once the processing for one line is completed, each processor 106 processes the next line, and so on, so that the inference processing for one image can be completed. DMA 101 stores the data for the next line in a group of registers other than the group of registers that store the data for the line being processed by multiple processors 106, thereby enabling uninterrupted processing and execution without logical inconsistencies in the processing.

[0132] As shown in Figure 8(a), the parameters used in the processing of each node, such as weighting coefficients, will be referred to below as "parameters of the trained model."

[0133] Figure 8(b) is a data conceptual diagram showing a detailed example of scenario information 200 provided to the processor device 100'. The scenario information 200 is stored in memory 102 for each task to be processed. The scenario information 200 may be configured to be set and stored in advance by a program, or it may be configured as shown in Figure 8(a), where the scheduler 110 accepts input from a learning model, subdivides it, and defines the scenario information 200.

[0134] The scenario information 200 shown in Figure 8(b) is a meta-description of the AI ​​structure (learning model), which subdivides the processing at one node of the learning model shown in Figure 8(a).

[0135] In other words, "scenario information" indicates the "structure of the learning model (processing content and its order)."

[0136] In Figure 8(b), the unit number is an identifier that makes each node in Figure 8(a) identifiable.

[0137] The following list is information indicating the next process (unit number) to be executed after the process shown in scenario information 200 in Figure 8(b), and it is information indicating dependencies with other scenario information 200.

[0138] The processing unit number indicates the processing content to be executed in scenario information 200, and specifies which process (addition, multiplication, etc.) to execute. For example, if processing unit number 1 is an adder, the processing in this scenario information 200 will be addition. The processing unit number here is information to specify one or more of the arithmetic units 10. The image size is information that defines the size of the data to be processed. Although it is called image size here, this is because image processing is assumed; if processing other than image processing is performed, it will be processing size. The parameter address is information that specifies the address of the register memory 104 that stores the data to be processed. The buffer information is information that specifies the unit in which processing will be performed (for example, in the case of an image, it may be in frames or lines). The transfer amount is information that indicates the number of dimensions of processing to be performed; for example, in the case of image processing, it will be 2-dimensional processing.

[0139] The scheduler 110 refers to the scenario information 200 shown in Figure 8(b) and instructs the DMA 101 to read the data to be processed from memory 102 and store the data in the register of register memory 104 that has the address indicated by the parameter address. The scheduler 110 also checks in other scenario information 200 to confirm that all processes having the unit number indicated in the scenario information 200 to be processed have been completed, and then specifies the process indicated by the processing unit number to the arithmetic unit 10 and instructs it to start processing. As a result, multiple processors 106 within the arithmetic unit 10 execute the same specified instruction (the calculation content that the arithmetic unit 10 should execute) for the specified number of processes, changing the address of the register memory 104.

[0140] In the processor device 100' described above, a configuration in which banks are set for the register memory 104 may also be used. A bank is a unit of area that each processor 106 can access. A bank can also be described as an area where access requests occur. That is, in the processor device 100', a bank can be described as an area where data that is the target of processing for the same instruction (the calculation content that the arithmetic unit 10 should perform) executed by each processor 106 in the arithmetic unit 10 is stored. In this embodiment, a bank is a group of registers consisting of multiple registers belonging to the register memory 104, similar to the first register group 104a and the second register group 104b described above. Note that a bank is defined for each of the multiple processors 106, and each of the multiple processors 106 reads data from the group of registers defined as a bank for itself and executes an instruction. Note that registers belonging to a bank defined for one processor 106 do not belong to other banks defined for other processors 106.

[0141] The bank may be configured to pre-set a specific range (group of registers) of the register memory 104, or the controller 107 in the arithmetic unit 10 may set that range for the register memory 104 according to the processing content. When the controller 107 in the arithmetic unit 10 sets it, it determines, based on the buffer information of the scenario information 200, the area in the register memory 104 that is not being accessed to store the data for the processing to be executed in the scenario information 200.

[0142] Figure 9 is a timing chart showing an example of the processing timing of the calculations performed by the first artificial intelligence calculation unit 2400 and the second artificial intelligence calculation unit 2600.

[0143] During the period from time t1 to t2, the central server 3000 distributes the initial values ​​of the first parameter group and the second parameter group to the sensor device 400 (hereinafter, when referring to sensor devices collectively, they will be called sensor device 400) via the edge server 500.1.

[0144] Between times t2 and t3, the edge device 100 receives a first set of parameters and a second set of parameters for initially defining the artificial intelligence model, and stores them in the non-volatile memory 2900 between times t3 and t4.

[0145] During the period from time t4 to t5, the second artificial intelligence computing unit 2600 performs a learning process based on the compressed information from the first artificial intelligence computing unit 2400 to update the second set of parameters from their initial values. For example, this corresponds to a process that adapts the output of the second artificial intelligence computing unit 2600 to the environment of the installation site after installation.

[0146] Subsequently, during the period from time t5 to t6, the second artificial intelligence calculation unit 2600 performs classification and decision-making processing on the data from the sensor 300.

[0147] On the other hand, during the period from time t4 to t6 (the period for updating the second set of parameters), the first artificial intelligence calculation unit 2400 performs processing with fixed values ​​for at least the first set of parameters, without changing them from their initial values. [Modification 1 of Embodiment 1]

[0148] Figure 10 is a functional block diagram illustrating an example of the configuration of edge device 100.j in Modification 1 of Embodiment 1.

[0149] In the description of Embodiment 1 above, the first artificial intelligence calculation unit 2400 and the second artificial intelligence calculation unit 2600 were described as having a one-to-one relationship and exchanging data.

[0150] However, for example, the configuration may be such that multiple second artificial intelligence calculation units 2600.1 to 2600.n (n: a natural number of 2 or more) each perform different types of artificial intelligence processing on the output information from the first artificial intelligence calculation unit 2400. In this case, as shown in Figure 10, the output information from the first artificial intelligence calculation unit 2400 can be sequentially switched by the data switching unit 2602 and supplied to the second artificial intelligence calculation units 2600.1 to 2600.n. Note that Figure 10 illustrates the case where n is 2.

[0151] Alternatively, although functionally similar, the configuration may be such that the data switching unit 2602 is replaced with a controller 105 that controls the processing of multiple second artificial intelligence calculation units 2600.1 to 2600.n.

[0152] Conversely, it is also possible to configure the system so that a single second artificial intelligence unit 2600 uses the output information from the first artificial intelligence computing units 2400.1 to 2400.m (m: a natural number greater than or equal to 2), each performing different processing, as input information for a predetermined artificial intelligence processing. In this case, it corresponds to a "multimodal artificial intelligence" that performs so-called "multimodal learning." "Multimodal artificial intelligence" refers to a single artificial intelligence model (for example, a deep neural network model) that can process multiple types of data (modalities) such as numerical data, images, text, and audio, by combining or associating them.

[0153] While not particularly limited, one possible example is that the first artificial intelligence calculation unit 2400.1 extracts "people" and "their postures" from an image, the first artificial intelligence calculation unit 2400.2 converts audio data acquired in parallel with the image into text data, and the second artificial intelligence calculation unit 2600 takes "data on the position (changes) of people," "data on the posture (changes) of people," and "speech data" as input to learn "multimodal artificial intelligence" so that it can determine "the relationships between multiple people" contained in the image (for example, whether they are unrelated people or companions).

[0154] Furthermore, more generally, it is also possible to configure the system so that output information from multiple first artificial intelligence computing units 2400.1 to 2400.m is learned by multiple second artificial intelligence computing units 2600.1 to 2600.n. [Modification 2 of Embodiment 1]

[0155] Figure 11 is a functional block diagram illustrating an example of the configuration of edge device 100.j in a modified example 2 of Embodiment 1.

[0156] The configuration in Figure 11 corresponds to the configuration in Figure 3.

[0157] The difference between the configuration in Figure 11 and the configuration in Figure 3 is that, while the second artificial intelligence computing unit 2600 performed dictionary learning and classification / decision processing based on dictionary data, the second artificial intelligence computing unit 2600 is configured such that, after the first artificial intelligence computing unit 2400 extracts features from time-series data from sensor 300.j.1, the second artificial intelligence computing unit 2600 performs learning and classification processing using so-called "reservoir computing".

[0158] Reservoir computing is a generalized concept of a specific model of recurrent neural network, and is one of the machine learning frameworks suitable for time-series information processing. The greatest feature of reservoir computing is that it learns extremely quickly compared to other recurrent neural network models.

[0159] Reservoir computing typically consists of an input layer, a reservoir, and an output layer. By fixing the connection weights between the input layer and the reservoir, and the feedback weights within the reservoir, and optimizing only the connection weights between the reservoir and the output layer using a computationally intensive linear learner, faster learning becomes possible compared to general recurrent neural networks that learn all connection weights.

[0160] It is known that reservoir computing can be applied to clustering in unsupervised learning by utilizing it to generate so-called "self-organizing maps." Publicly available document 4: Ryuhei Matsuo and Hiroshi Dozono, "Time series analysis using Reservoir self-organizing maps," Information Processing Society of Japan Research Report, https: / / www.ipsj-kyushu.jp / page / ronbun / hinokuni / 1010 / Papers / B5-3.pdf

[0161] Furthermore, a configuration called the "chaos Boltzmann machine" has also been proposed for reservoir computing. Publicly available document 5: Ichiro Kawashima, "FPGA Implementation of Chaos Boltzmann Machines and Their Application to Reservoir Computing," March 3, 2021, URL http: / / doi.org / 10.18997 / 00008357 As described above, with the edge device of this embodiment, it is possible to perform artificial intelligence learning processing related to the processing of signals from sensors on the edge side, so even when performing artificial intelligence learning, it is possible to suppress the increase in the amount of communication.

[0162] [Embodiment 2] Embodiment 2 describes use cases for the sensor system 1000 of Embodiment 1.

[0163] Figure 12 shows a map of the store where the sensor system 1000 is applied in Embodiment 2.

[0164] For example, area 1, as explained in Figure 1, corresponds to the store shown in Figure 12. The store is envisioned to be a small supermarket or convenience store.

[0165] Therefore, there are other stores with store layouts similar to those shown in Figure 12, which correspond to, for example, area 2 in Figure 1.

[0166] As explained in Figure 1, the number of regions may be greater than 2. Let's assume that multiple such stores exist, for example, as franchises or chain stores.

[0167] As shown in Figure 12, customers enter the store from outside, pushing a shopping cart, and then shop by browsing the shelves and display cases inside the store, which are filled with items such as vegetables, processed foods, condiments, fresh fish, meats, sweets, luxury goods, frozen foods, meat products, alcoholic beverages, bread, drinks, and dairy products.

[0168] Once you have finished shopping, you will pay at either register 1 or register 2 and then leave the store.

[0169] Figure 13 is a conceptual diagram showing the placement of the sensor system 1000 in the store described in Figure 12.

[0170] Figure 13 shows two stores, Store A and Store B, as examples.

[0171] Furthermore, in comparison with Figure 1, sensor devices 400.11 to 400.1N and sensor devices 400.21 to 400.1M are equipped with surveillance cameras 300.1.1 to 300.N.1 and surveillance cameras 300.1.2 to 300.M.2 (not shown) as sensors, respectively. In other words, sensor system 1000 is a surveillance camera system deployed across multiple stores.

[0172] Figure 14 is a conceptual diagram illustrating the distribution and training of an artificial intelligence model in a surveillance camera system, as well as the provision of services in camera system 1000.

[0173] As described in Embodiment 1, the central server 300 distributes to each edge device 100 (when referring to edge devices collectively, they are referred to as "edge device 100") via the edge server 500.1 the initial values ​​of the first parameter group that defines the first artificial intelligence model of the first artificial intelligence calculation unit 2400, which is executed in the calculation module 2200 (first initial model), and the initial values ​​of the second parameter group that defines the second artificial intelligence model of the second artificial intelligence calculation unit 2600, which are referred to as "second initial model".

[0174] In this case, the calculation module 2200 will, for example, use the configuration described in Figure 3 to perform customer movement analysis and customer behavior pattern analysis based on the movement analysis, using video images from the camera.

[0175] Here, the following are some examples of customer behavior patterns that can be anticipated.

[0176] 1) Abnormal behavior: Behavior that is considered an outlier in the normal shopping flow. For example, this includes behavioral patterns such as walking back and forth in front of a specific product more than a predetermined number of times, or staring at the location of a store's security camera more than a predetermined number of times.

[0177] 2) Interest in a specific product 1: This corresponds to a behavioral pattern where a user stays in front of a product shelf for a predetermined amount of time or longer, but does not purchase the product. The edge device sends information about the time spent along the user's movement path to the central server, which then compares this information with the product information on the shelf and the purchase record at the register to determine whether or not this is an interest in a specific product.

[0178] 3) Interest in a specific product 2: This corresponds to a behavioral pattern in which, compared to the customer's past purchase history, the customer is moving towards a location containing a product that has been purchased a certain number of times or more in the past. The edge device sends a prediction of the movement path to the central server, which then compares it with past purchase history to determine whether or not it is an interest behavior.

[0179] On the other hand, as will be described later, the central server 300 provides customer behavior response services to terminals (not shown) installed at the cash registers of each store or terminals (not shown) held by the store staff, in accordance with the classification results of customer movement analysis and customer information transmitted from the sensor device 400.

[0180] Here, customer behavior response services are not particularly limited, but could include "digital signage," "personalized recommendations," and "suspicious person detection."

[0181] In Figures 13 and 14, data from surveillance cameras is aggregated in the central server 3000, and the central server provides the customer behavior response services described above. However, the configuration is not necessarily limited to this.

[0182] By exchanging data between a group of surveillance cameras and the edge servers under their control, it is also possible to configure the edge servers to provide customer behavior response services.

[0183] Therefore, in the following, when referring to both the central server and the edge servers collectively, we will use the term "management server," and the term "management server" may refer to either the edge server or a configuration that includes both the central server and the edge servers.

[0184] Figure 15 is a conceptual diagram illustrating an example of a customer behavior response service.

[0185] As explained in Figure 14, first, when the "identifiable behavior of a suspicious person" is extracted as in P1, the central server 3000 notifies the store staff of the presence of a suspicious person via a terminal installed at the cash register of the corresponding store, or sends the camera footage of the suspicious person to the terminal via the edge server 500.1 and sends and displays a warning pop-up on the terminal.

[0186] Furthermore, as in P2, when a customer is identified who spends a long time in a particular sales area (in front of a shelf) but does not purchase any items from that shelf, the central server 3000 compares this information with the member information obtained from the member card, etc., at the time of payment at the register, and sends follow-up information via direct mail after the customer returns home.

[0187] Alternatively, as in P3, it is possible to obtain each customer's level of interest in products based on their visits to the sales area and their actions in front of product shelves, and then individually recommend products to that customer through digital signage within their line of sight. Such displays may also be installed in the customer's shopping cart and configured to move with the customer.

[0188] [Modified version of Embodiment 2] Figure 16 is a conceptual diagram illustrating the operation of edge device 100.j for performing a modified version of Embodiment 2.

[0189] The configuration shown in Figure 16, for example, performs movement analysis and behavior pattern classification using the configuration shown in Figure 3, but the second artificial intelligence processing unit 2600 also includes an encryption processing unit 2630 that performs encryption processing of information to be output externally.

[0190] Furthermore, it is assumed that image data from camera 300.j.1 is also transmitted to edge server 500.1 in synchronization with the processing of the first artificial intelligence processing unit 2400 and the second artificial intelligence processing unit 2600.

[0191] The compressed information (features) extracted by the first artificial intelligence processing unit 2400 is then encrypted by the encryption processing unit 2630 and sent to the edge server 500.1. It is then associated with the image data from the camera 300.j.1 and temporarily stored in the memory device 502.1 of the edge server 500.1. At this time, the edge server 500.1 performs a process to assign a person ID to the image data and the encrypted features.

[0192] Here, "feature quantities" can refer to a configuration that includes facial features for face recognition and gait features for gait recognition.

[0193] As explained in Embodiment 2, once the "identifiable behavior of a suspicious person" is extracted, the central server 3000 notifies the store staff of the presence of a suspicious person via a terminal installed at the cash register of the corresponding store, or transmits the camera footage of the suspicious person to the terminal via the edge server 500.1, and sends a pop-up to the terminal to alert the staff. At this time, if the store staff inputs annotation information about the person from the terminal, this annotation information is transmitted to the edge server 500.1, associated with the person ID, and stored in the storage device 502.1.

[0194] While not particularly limited, for example, if annotation information is not sent to the edge server 500.1, the person is considered to have no abnormal behavior, and after a certain period of time, the person ID, image data, feature quantities, and annotation information of the person can be deleted from the storage device 502.1.

[0195] On the other hand, if annotation information is sent to the edge server 500.1, it is determined that the person has exhibited abnormal behavior, and the person ID, image data, feature quantities, and annotation information of the person are stored and retained in the storage device 502.1 for a period longer than the aforementioned fixed time.

[0196] Furthermore, "long period" can refer to the maximum period permitted by law, for example, if the image data or feature data constitutes personal information.

[0197] Furthermore, since the data is stored and retained in the storage device 502.1 for a "long period" in this manner, for system security reasons, it is also possible to configure the system so that not only feature quantities but also image data and person annotation information are encrypted and stored in the edge server 500.1.

[0198] Furthermore, when encrypting features on the edge device 100.j, it is possible to configure it to encrypt using the public key set on the edge server 500.1, although this is not particularly limited. On the edge server 500.1, as described later, it is possible to configure it to decrypt the encrypted data using the private key when a query is received from the central server 3000.

[0199] Figure 17 is a conceptual diagram illustrating the process by which store staff can retrieve annotation information about a specific person from a terminal.

[0200] Referring to Figure 17, for example, the storage devices 502.1 of edge server 500.1 and 502.2 of edge server 500.2 store person IDs, image data, feature quantities, and person annotation information, as described in Figure 16.

[0201] A person lookup request is sent from a terminal at store A via edge server 500.1 (S01). At this time, the person lookup request includes encrypted feature information. In this case, for example, the person lookup request is encrypted using the public key of central server 3000 at edge server 500.1 and sent to central server 3000 via network 2 (S02).

[0202] The central server 3000 decrypts the feature quantities included in the person lookup request, encrypts them with the public key of the edge server 500.2, and then sends them together with the person lookup query to the edge server 500.2 at store B (S03). If edge servers are also provided at other stores, the central server 3000 also sends the person lookup query along with the encrypted feature quantities to these other edge servers.

[0203] The edge server 500.2, upon receiving the person lookup query along with encrypted features, decrypts the features using its own secret key and then performs a matching process with the features stored in the memory device 502.2.

[0204] If matching information exists, the edge server 500.2 encrypts at least one of the annotation information and image data about the matched person and uploads it to the central server 3000 (S04).

[0205] The central server 3000 encrypts at least one of the uploaded annotation information and image data with the public key of the edge server 500.1 that sent the person lookup request, and sends it to the edge server 500.1. After decryption at the edge server 500.1, at least one of the image data and annotation information of the person looked up and the person that was found is displayed on the terminal at store A.

[0206] With this configuration, the staff at store A can obtain information about any person currently visiting store A who is exhibiting any abnormal behavior, including whether similar abnormal behavior occurred at other stores and what that behavior was.

[0207] As a result, the store staff at store A will be able to take necessary security measures (for example, alerting security guards).

[0208] Furthermore, all information exchange for such personal inquiries is conducted using encrypted data, and the source data for inquiries is distributed across each edge server and stored encrypted as needed. This ensures that when using personal information for security purposes, the responsibility for protecting personal information is fulfilled to the fullest extent.

[0209] In addition, while Figures 16 and 17 describe how data from surveillance cameras is aggregated in the central server 3000 and how the central server provides the customer behavior response services described above, the configuration is not necessarily limited to this.

[0210] By exchanging data between a group of surveillance cameras and the edge servers under their control, it is also possible to configure the edge servers to provide customer behavior response services.

[0211] Furthermore, the services offered are not limited to customer behavior tracking services that span multiple stores (or multiple customer service locations) as described above. They can also be provided as security services at the same customer service location, allowing for retrieval of at least one of previously stored image data and annotation information of a specific individual.

[0212] Figure 18 is a block diagram illustrating the hardware configuration of the central server 3000.

[0213] As mentioned above, the central server 3000 may be configured so that its own CPU performs the calculations, or it may be configured so that part of the program's processing is executed on other servers. In the following explanation, we will assume that the CPU within its own chassis performs the calculations.

[0214] Referring to Figure 18, the central server 3000 comprises a computer device 3010, a network communication unit 3300 for communicating with the network, and a recording medium (e.g., a memory card) 3210 for recording external data and providing it to the computer device 3010.

[0215] For example, the recording medium 3210 can be a USB memory stick, memory card, or external storage device. The network communication unit 3300 can utilize, for example, wired LAN or wireless LAN communication functions.

[0216] As shown in Figure 18, the computer unit comprising this computer device 3010 includes, in addition to the disk drive 3030 and memory drive 3020, a CPU (Central Processing Unit) 3040 connected to the bus 3050, memory including ROM (Read Only Memory) 3060 and RAM (Random Access Memory) 3070, a non-volatile rewritable non-volatile storage device 3080, and an input / output interface 3090 for communication over a network and data exchange with the outside. An optical disc can be installed in the disk drive 3030. A memory card 3210 can be installed in the memory drive 3020. For the non-volatile storage device 3080, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive) can be used, and in the following description, an SSD will be used.

[0217] For the operation of the computer device 3010, the data and programs that store the information fundamental to the computer's operation will be described as being stored in the SSD 3080.

[0218] In Figure 18, the medium on which information such as programs to be installed on the computer can be recorded may be, for example, a DVD-ROM (Digital Versatile Disc), a memory card, or a USB memory stick. To accommodate such cases, the computer is equipped with drive devices (memory drive 3020, disk drive 3030) that can read these media.

[0219] The main components of the computer device 3010 consist of computer hardware and software executed by the CPU 3040. Generally, such software is stored in a storage medium and distributed, or distributed via a network, and retrieved via the disk drive 3030 or network communication unit 3300, and temporarily stored in the SSD 3080. Then, it is read from the SSD 3080 into the RAM 3070 in memory and executed by the CPU 3040. In the case of a network connection, the software may be loaded directly into RAM and executed without being stored in the SSD 3080.

[0220] The program for functioning as a computer device 3010 does not necessarily need to include an operating system (OS) that causes the computer body 3010 to execute functions such as those of an information processing device, for distribution purposes. The program only needs to include the instruction portion that calls appropriate functions (modules) in a controlled manner and obtains the desired result. How the computer system 3010 operates is well known, so a detailed explanation is omitted.

[0221] Figure 19 is a functional block diagram illustrating the functional configuration of the central server 3000.

[0222] Referring to Figure 19, the central server 3000 includes a network communication unit 3300 for exchanging data with edge servers 500.1 and 500.2 via network 2, an input / output interface (hereinafter referred to as input / output I / F) 3090 for taking data into the server or outputting data from the server between the network communication unit 3300 and the other, a computing unit 3040, and a non-volatile storage device 3080.

[0223] The computing unit 3040 includes, as functional blocks, a model distribution processing module 3041 for distributing the data 3082 of the first artificial intelligence initial model and the data 3083 of the second artificial intelligence initial model, stored in the non-volatile memory device 3080, to the sensor device 400 (referred to collectively as sensor device 400) via edge servers 500.1 and 500.2, and a region information distribution processing module 3042 for distributing the edge individual region information 3084, stored in the non-volatile memory device 3080, to the sensor device 400 via edge servers 500.1 and 500.2.

[0224] Here, "edge individual area information 3084" includes, for example, store product arrangement and map information. In Figure 2, this information is included in the classification dictionary data 2960 and is used for classifying customer behavior patterns, and includes information about the products displayed on the shelves.

[0225] As explained in Figure 17, the computing unit 3040 includes, as functional blocks, a query information retrieval processing module 3043 for sending a person inquiry to the edge server when a person inquiry request is sent from the edge server, a response information acquisition module 3046 for receiving annotation information uploaded from the edge server, and a query information response transmission processing module 3045 for sending a person information response to the edge server that initiated the person inquiry.

[0226] Furthermore, the non-volatile memory device 3080 also stores information such as membership card details, as well as member purchase history and email recipient information, as history information 3085.

[0227] As explained above, when looking up individuals, it is possible to search past image data using encrypted feature quantities as keys, making it possible to search for security-related information while protecting personal information.

[0228] [Embodiment 3] Figure 20 is a conceptual diagram showing the sensor system 1000 deployed in a theme park.

[0229] Theme park is merely an example; any other facility or area that is under the jurisdiction of a specific manager and where it is necessary to search for a person within that jurisdiction, such as a shopping mall, may also be used.

[0230] Referring to Figure 20, a surveillance camera 100.1.1 is installed at the entrance gate of the theme park, and images of visitors are associated with, for example, the ID information of the visitors' admission tickets and stored on the edge server 500.1.

[0231] Alternatively, at the entrance gate, park staff could offer a service of taking commemorative photos of visitors with theme park characters using their smartphones, and the photos of visitors on that day could be stored on the visitors' smartphones.

[0232] In Figure 20, data from surveillance cameras is aggregated at the central server 3000, and the central server is shown to provide a lost child search service as described later. However, the configuration is not necessarily limited to this.

[0233] By exchanging data between a group of surveillance cameras and the edge servers under their control, it is also possible to configure the edge servers to provide a lost child search service.

[0234] Figure 21 is a conceptual diagram illustrating the process when a child gets lost.

[0235] Referring to Figure 21, when a child gets lost, with the parent's consent, feature data is extracted from the image of the lost child stored on the parent's smartphone or from the image of the child at the time of entry stored on the edge server 500.1. Based on these feature data, inquiry information is sent to each sensor device 400. Here, the sensor device 400 is assumed to have surveillance cameras as sensors.

[0236] On the edge device 100 of each sensor device 400, a matching process is performed between the features extracted from the images captured by the surveillance camera and the features of the lost child's image.

[0237] Each sensor device 400 is configured to discard image data acquired after a certain period following the execution of such matching processing. When a match is found, each sensor device 400 responds via the edge server 500.1 that a lost child has been found in the captured image.

[0238] With this configuration, each sensor device 400 only performs feature matching, and the captured image data is discarded within a predetermined period each time. Therefore, there is no problem with acquiring personal information, and it becomes possible to search for lost children.

[0239] [Embodiment 4] Figure 22 is a conceptual diagram illustrating the process by which the central server 3000 functions as an artificial intelligence platform.

[0240] In Embodiment 4, multiple edge servers are described as being under the control of the central server 3000.

[0241] Here, we will explain the surveillance camera system within a theme park, as described in Embodiment 3, as an example.

[0242] Sensor manufacturers that produce and sell surveillance cameras purchase edge devices like those described in Figures 6 to 8 and either integrate them into their own surveillance cameras and sell them, or sell them as kits that can be retrofitted to their surveillance cameras.

[0243] Theme parks will deploy surveillance cameras and edge devices as assets within their premises.

[0244] The central server 3000, which functions as an artificial intelligence platform, acts as an app store, processing a tool environment that provides libraries of artificial intelligence models, such as a face recognition engine executed on the first artificial intelligence computing unit 2400, to third parties creating applications such as lost child search apps. Here, the face recognition engine is assumed to be written in a high-level language that describes machine learning processing.

[0245] Furthermore, the central server 3000 processes the tools and environment to provide third parties developing applications with a Software Development Kit (SDK), which includes conversion kits for converting models written in high-level languages ​​for machine learning into scenario information and parameters, enabling them to run on edge devices.

[0246] The central server 3000 also provides a test environment for testing whether third-party lost child search apps, and in particular facial recognition engines and movement prediction models (models that predict the direction of movement from images in the camera) that operate when searching for lost children, function correctly on edge devices.

[0247] Furthermore, the central server 3000 receives lost child search apps from third parties, performs authentication processing, and for authenticated apps and models, for example, the artificial intelligence model (initial model) is distributed to edge devices via the edge server, while the lost child search app is also distributed to visitors' smartphones and devices held by park staff.

[0248] When a request to search for a lost child is sent via a visitor's smartphone or a terminal held by park staff, as explained in Figure 21, the search request is sent to each sensor device (surveillance camera) via the edge server. The sensor device that detects the presence of the person being searched for in the captured image sends the information back to the smartphone or terminal as the lost child's location information. At this time, a movement prediction model may be used to predict the next location the person will appear in based on their movement in the image, and this information may also be sent back to the smartphone or terminal.

[0249] As explained above, if an artificial intelligence platform like the one in this embodiment is realized, an environment will be created in which third parties can participate in the creation of applications and software that utilize edge devices.

[0250] On the other hand, if the manufacturer and seller of edge devices operates this artificial intelligence platform, they will not need to disclose to third parties the scenario information and methods for converting it into parameters necessary for the AI ​​model to run on the edge device.

[0251] In Figure 22, the central server 3000 is described as performing the following: processing of a tool environment (first tool environment) that provides a library of artificial intelligence models to third parties; processing of a tool environment (second tool environment) that provides third parties with a conversion kit as an SDK to convert models written in a high-level language for machine learning into scenario information and parameters so that they can run on edge devices; processing of a test environment; processing of artificial intelligence models; and distribution processing that distributes the authenticated artificial intelligence models (initial models) to edge devices via the edge server. However, the configuration is not necessarily limited to this, and for example, the central server 3000 is responsible for the distribution processing of artificial intelligence models (initial models), and the provision of the first tool environment, the provision of the second tool environment, the provision of the test environment, and the processing of models are performed by service provision servers different from the central server 3000. Therefore, more generally, each of these processes can be shared and distributed among multiple servers. In this embodiment, the term "management server" is used to refer collectively to the central server and the service provision servers. More generally, since some of the functions performed by the "management server," such as the distribution process of artificial intelligence models (initial models), may be carried out jointly by the central server and edge servers, the term "management server" collectively refers to the central server, service provision servers, and edge servers.

[0252] The embodiments disclosed herein are illustrative of configurations for specifically carrying out the present invention and do not limit the technical scope of the present invention. The technical scope of the present invention is indicated by the claims rather than by the description of the embodiments, and modifications within the literal scope and equivalent meaning of the claims are intended. [Explanation of Symbols]

[0253] 100.j Edge device, 300.j.1~300.j.mj Sensor, 400.1j Sensor device, 2100 Sensor interface, 2200 Computation module, 2400 First artificial intelligence computing unit, 2600 Second artificial intelligence computing unit, 2800 Network interface unit, 2900 Non-volatile memory, 3000 Central server.

Claims

1. An edge device provided in correspondence with a sensor and for performing processing on signals from the sensor, A sensor interface for receiving signals from the corresponding sensor, A first artificial intelligence module that performs processing by a first artificial intelligence model to reduce the amount of information contained in the signal and output it as compressed information in response to the signal from the sensor, The system includes a second artificial intelligence module that receives the output from the first artificial intelligence module and classifies the pattern of change in the observed quantity of the object observed by the sensor using a second artificial intelligence model, The first artificial intelligence model is defined by a first set of parameters, and the second artificial intelligence model is defined by a second set of parameters. The second artificial intelligence module updates the second set of parameters by performing the learning process based on the compressed information. The first artificial intelligence module is an edge device that keeps the first set of parameters fixed at least during the update process of the second set of parameters.

2. A communication interface for communicating with an external network, The system further comprises a non-volatile memory for storing the first set of parameters and the second set of parameters, The edge device according to claim 1, wherein the initial values ​​of the first parameter group and the second parameter group are supplied via the communication interface and stored in the non-volatile memory before the start of the update process by the second artificial intelligence module.

3. The aforementioned sensor includes an image sensor, The first artificial intelligence module extracts an object from the image data from the image sensor and performs a process to convert it into a feature quantity of the object. The edge device according to claim 1 or 2, wherein the second artificial intelligence module performs a process to classify the temporal displacement pattern of the target, and updates the second set of parameters by unsupervised learning.

4. The edge device according to claim 3, wherein the second artificial intelligence module updates the second set of parameters by learning the probability of occurrence of the target displacement pattern based on dictionary data created by dictionary learning.

5. The edge device according to claim 4, wherein the dictionary learning is sparse dictionary learning.

6. The edge device according to claim 3, wherein the second artificial intelligence module updates the second set of parameters by learning the probability of occurrence of the target displacement pattern through reservoir computing.

7. The first artificial intelligence module and the second artificial intelligence module are executed on the same processor device. The first set of parameters and the second set of parameters each include scenario information that defines the processing steps of the first artificial intelligence module and the second artificial intelligence module, The aforementioned processor device, Multiple processors, A group of registers, each containing multiple registers for storing the data to be processed, A memory access device that reads data to be processed from the non-volatile memory based on input instructions and writes it to a group of registers in the group of registers, Each of the aforementioned multiple processors is configured to sequentially execute the same instruction, while changing the address of a group of registers storing the data to be processed in accordance with the number of the multiple processors. The edge device according to claim 2, comprising a scheduler that, based on specified scenario information, instructs the plurality of processors on the instructions to be executed and the group of registers to be processed, and instructs the memory access device on the group of registers to be written and the data to be written from among the plurality of registers.

8. The aforementioned sensor acquires time-series data, The first artificial intelligence module performs a process to convert time-series data from the sensor into features, The edge device according to claim 1 or 2, wherein the second artificial intelligence module performs a process of classifying the displacement patterns of the time series data and updates the second set of parameters by unsupervised learning.

9. Management server and Multiple sensors, The system comprises a plurality of edge devices, each corresponding to one of the plurality of sensors, Each of the edge devices is A sensor interface for receiving signals from the corresponding sensor, A communication interface for communicating with an external network, A non-volatile memory for storing the first set of parameters and the second set of parameters, A first artificial intelligence module that performs processing by a first artificial intelligence model to reduce the amount of information contained in the signal and output it as compressed information in response to the signal from the sensor, The system includes a second artificial intelligence module that receives output from the first artificial intelligence device and classifies the movement and morphological changes of the object observed by the sensor using a second artificial intelligence model, The first artificial intelligence model is defined by the first set of parameters, and the second artificial intelligence model is defined by the second set of parameters. The second artificial intelligence module updates the second set of parameters by performing the learning process based on the compressed information. The first artificial intelligence module maintains the first set of parameters in a fixed state, at least during the update process of the second set of parameters. The management server distributes the initial values ​​of the first parameter group and the second parameter group to the multiple edge devices via the communication interface before the update process by the second artificial intelligence module begins. Each edge device stores the distributed initial values ​​in the non-volatile memory, comprising a sensor operation system.

10. The first set of parameters is, Scenario information defining the first structure of the first artificial intelligence model, The first artificial intelligence model includes parameter values ​​that define the computational processing performed based on the first structure, The second set of parameters is, Scenario information defining the second structure of the second artificial intelligence model, The sensor operation system according to claim 9, comprising parameter values ​​that define the computational processing performed by the second artificial intelligence model based on the second structure.

11. Each of the aforementioned sensors includes an image sensor. The management server includes edge servers provided for each of the multiple edge devices in order to distribute data to the multiple edge devices and receive requests from the multiple edge devices. The edge server includes a storage device that stores the image data from the image sensor and the compressed information generated by processing the image data by the first artificial intelligence module in association with each other. The sensor operation system according to claim 9, wherein the management server performs a search for information already stored in the storage device in response to a query request to the edge server that includes specific compressed information.

12. The image data stored in the storage device is associated with and stored annotation information for the object contained in the image data. The sensor operation system according to claim 11, wherein the edge server, in response to the query request, returns at least one of the image data stored in the storage device that matches the specific compressed information or annotation information related to the matching image data.

13. The first artificial intelligence module and the second artificial intelligence module are executed on the same processor device. The aforementioned processor device, Multiple processors, A group of registers, each containing multiple registers for storing the data to be processed, A memory access device that reads data to be processed from the non-volatile memory based on input instructions and writes it to a group of registers in the group of registers, Each of the aforementioned multiple processors is configured to sequentially execute the same instruction, while changing the address of a group of registers storing the data to be processed in accordance with the number of the multiple processors. The sensor operation system according to claim 10, comprising a scheduler that, based on specified scenario information, instructs the plurality of processors on the instructions to be executed and the group of registers to be processed, and instructs the memory access device on the group of registers to be written and the data to be written from among the plurality of registers.

14. The sensor operation system according to claim 10 or 11, wherein the management server includes a service provider server that provides a tool environment for creating scenario information and parameter information for defining the processing of the first artificial intelligence module and the second artificial intelligence module through user operation.

15. The sensor operation system according to claim 14, wherein the tool environment provides the existing first artificial intelligence module registered on the service provision server according to the user's selection.

16. A sensor operation system for processing data from multiple sensors, Multiple edge devices are provided, each corresponding to a different sensor, A central server for distributing the artificial intelligence model executed on the edge device to the multiple edge devices, The service includes a service server to support the creation of the aforementioned artificial intelligence model, Each of the edge devices is A sensor interface for receiving signals from the corresponding sensor, A communication interface for communicating with an external network, A non-volatile memory for storing the first set of parameters and the second set of parameters, A first artificial intelligence module that performs processing by a first artificial intelligence model to reduce the amount of information contained in the signal and output it as compressed information in response to the signal from the sensor, The system includes a second artificial intelligence module that receives output from the first artificial intelligence device and classifies the movement and morphological changes of the object observed by the sensor using a second artificial intelligence model, The first artificial intelligence model is defined by the first set of parameters, and the second artificial intelligence model is defined by the second set of parameters. The second artificial intelligence module updates the second set of parameters by performing the learning process based on the compressed information. The first artificial intelligence module maintains the first set of parameters in a fixed state, at least during the update process of the second set of parameters. The service provider server provides a tool environment for creating initial models of the first artificial intelligence module and the second artificial intelligence module through user operation, and the initial models include initial values ​​for the first parameter group and the second parameter group. The central server distributes the initial values ​​to the multiple edge devices via the communication interface before the update process by the second artificial intelligence module begins. Each edge device stores the distributed initial values ​​in the non-volatile memory, comprising a sensor operation system.

Images