Diversified edge computer and edge computer control system

Abstract

To provide an edge computer and control system thereof that can bring out the diversity of data processing.SOLUTION: In this embodiment, an edge computer that can communicate with multiple devices including portable terminals and IoT devices and a cloud server is realized. The edge computer is equipped with an application engine for executing multiple application rules previously imported from the cloud server, each of the multiple devices, and multiple microservices provided for applying the individual application rules corresponding to the cloud server, and can operate the internal application rules, the external multiple devices, and the cloud server in cooperation with each other.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] An embodiment of the present invention relates to a versatile edge computer and edge computer control system. [Background technology]

[0002] A computer system can be equipped with various programs and have diversity, but the various programs are used by switching between them individually, and after switching, data processing is completed within the processing range of the switched program. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-185797 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present embodiment aims to provide an edge computer and an edge computer control system that are versatile enough to enable processing in which various unique functions are linked to each other. [Means for solving the problem]

[0005] In this embodiment, an edge computer capable of communicating with a plurality of devices including IoT devices and a cloud server, an application engine for executing a plurality of application rules pre-fetched from the cloud server; a plurality of microservices configured to apply the application rules to each of the plurality of devices and the cloud server; The application rule has a basic rule of execution (THEN) for a condition (IF), The application engine includes: An edge computer is provided, which is characterized in that, when a first application rule is used, a first API command as the condition is given to a first microservice to obtain a return value, and a second API command as the execution is given to a second microservice different from the first microservice based on the return value. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is an explanatory diagram showing the basic concept of this embodiment. [Diagram 2] FIG. 2 is an explanatory diagram showing a basic system in this embodiment. [Diagram 3] FIG. 3 is an explanatory diagram showing problems that arise when using conventional technology without combining it with image recognition technology. [Figure 4] FIG. 4 is a diagram illustrating the effect when only the head temperature of a person is used for judgment. [Diagram 5] FIG. 5 is an explanatory diagram showing the operation flow executed in the basic system shown in FIG. [Figure 6] FIG. 6 is an explanatory diagram illustrating an example of a scene in which this embodiment is used. [Figure 7] FIG. 7 is an explanatory diagram showing another example of this embodiment. [Figure 8] FIG. 8 is an explanatory diagram showing a program configuration used in an Android-based smartphone. [Figure 9] FIG. 9 is an explanatory diagram showing the configuration of an edge computer and its relationship with a cloud server. [Figure 10] FIG. 10 is a table showing a list of the application (IF-THEN) engine functions. [Figure 11] FIG. 11 is a block diagram expressing the concept of an application example in this embodiment. [Figure 12] FIG. 12 is an explanatory diagram illustrating the effect of writing a microservice in an OS-independent programming language. [Figure 13] FIG. 13 is an explanatory diagram showing the class configuration of a microservice. [Figure 14] FIG. 14 is an explanatory diagram of the functions of the microservices set in the individual device control package. [Figure 15] FIG. 15 is an explanatory diagram showing a list of APIs provided by the microservice control template class (BaseIms Class). [Figure 16] FIG. 16 is an explanatory diagram showing a list of APIs provided by the device control template class (Base Device Class). [Figure 17] FIG. 17 is a state transition diagram of a device. [Figure 18] FIG. 18 is an explanatory diagram showing the relationship between the device state and the microservice state. [Figure 19] FIG. 19 is an explanatory diagram showing the relationship between the blackbody radiation source temperature and the spectral radiance characteristics. [Figure 20] FIG. 20 shows an explanatory diagram of the internal structure of a triple-lens infrared camera. [Figure 21] FIG. 21 explains the inter-class cooperative operation within the class configuration shown in FIG. [Figure 22] FIG. 22 is an explanatory diagram showing a setting method for notifying multiple media when a person with a high fever is detected. [Figure 23] FIG. 23 is an explanatory diagram showing the class configuration of information provision-related microservices. [Figure 24] FIG. 24 is an explanatory diagram showing the processing steps up to notifying the user of information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The basic concept of this embodiment will be described with reference to FIG. 1. As shown in FIG. 1(a), a place where multiple people have gathered is assumed. If there is a person among them whose body temperature is above a predetermined value (e.g., 37.5 degrees Celsius), that person is automatically detected individually. In the display example of FIG. 1(b), the person indicated by the diagonal lines corresponds to a person with a high fever of 37.5 degrees Celsius or higher. This information is then automatically notified (FIG. 1(c)). There are multiple options for this notification method, and in this embodiment, the option can be selected in a very simple manner. Here, multiple notification methods may be selected simultaneously.

[0008] In Figure 1(a), the number of people gathered at the gathering place is relatively small, so each person is shown separately. However, as the population density of the people gathering increases, partial overlap occurs. Conventional thermography technology, which measures the surface temperature distribution of the object being measured, cannot identify individuals in places where multiple people overlap. Therefore, when using only conventional thermography technology, it is not possible to individually detect only those with high fevers in crowded places (individual extraction from crowded places).

[0009] In this embodiment, image recognition technology is combined with thermography technology using an infrared camera to enable individual detection (individual extraction) of only those with high fevers. As a specific combination method, in this embodiment, a program (image recognition class AI Analyze Class) 2018 having an image recognition function using AI technology is embedded 76 in a part (or inside) of the microservice 80C serving as the control means 4 for the infrared camera 806. Details will be described later with reference to Figs. 11 and 13.

[0010] In order to reduce the risk of virus infection in places where people gather and are active, a technology is required that can easily detect and notify people with high fevers who are suspected of infection. In this embodiment, it can be used simply by attaching a commercially available infrared camera 806 to an edge computer 6 such as an existing smartphone or tablet, and installing software (service provision application program) 92 called ifLink (registered trademark) app, the details of which will be described later.

[0011] As the commercially available infrared camera 806 described above, an infrared camera 806 that can be connected to an edge computer 6 such as a commercially available smartphone or tablet may be used. If this infrared camera 806 is used, it can be very easily connected to an edge computer 6 such as a commercially available smartphone or tablet, so that the high fever person detection system of this embodiment can be easily constructed.

[0012] In addition, the above-mentioned ifLink app 92 works as an app on smartphones and tablets that many people have. As a result, it is effective in reducing the risk of infection at each activity base by using the existing infrastructure (usage environment) of smartphones, tablets, and infrared cameras 806, and by simply installing the ifLink app 92, which is a very simple method.

[0013] In this embodiment, image recognition is combined with video or still images captured by the infrared camera 806 to individually detect (individually extract) only those with high fevers from a group, and automatically notify that information.

[0014] There have been methods of detecting the surface temperature of an object to be measured using thermography technology with an infrared camera 806. However, as mentioned above, when people are crowded together and multiple people overlap, it has been necessary for an observer of the video / still image to visually check and separate (individually extract) each person. Furthermore, since conventional thermography alone cannot distinguish whether the object to be measured is a human or another object, a person who can identify the object to be measured is required. For this reason, conventional thermography technology could only be used at the reception desk of a company or facility that can deploy a dedicated monitor. In this embodiment, the incorporation 76 of an image recognition class (AI Analyze Class) 2018 using AI technology allows a computer (edge ​​computer 6) to recognize people with high fevers from images (video or still images) obtained from an infrared camera. Since the recognition is performed automatically within the edge computer 6 in this way, people with high fevers can be detected and notified even without a monitor. Therefore, this embodiment has the effect of significantly reducing the cost of monitoring (automatic individual detection of people with high fevers). In this embodiment, the method of recognizing a person with a high fever is not limited to the use of the AI ​​Analyze Class 2018 image recognition method, but any image recognition method may be adopted. For example, a unique hardware with a dedicated image recognition function may be prepared in advance. In this case, an image obtained from an infrared camera may be sent to this hardware for image recognition, and the result may be returned.

[0015] In Fig. 1(c), when a person with a high fever whose body temperature exceeds a predetermined value (e.g., 37.5 degrees Celsius) is detected, the result is automatically notified. As an example of an option for this automatic notification method, an image (video or still image) clearly indicating the presence of a person with a high fever may be displayed 1210 on the display screen 12 of the edge computer 6 such as a smartphone or tablet. In addition, without being limited to this, a warning may be displayed 1220 on a signage (electronic bulletin board) 804 installed near the edge computer 6 that detected the person with a high fever.

[0016] The display form is not limited to a specific image, and a warning screen in which a "warning message is written in text format" may be displayed 1230 on the display screen 12 of the edge computer 6 or on the signage 804. Furthermore, in addition to visually appealing with a specific image or text warning message, notification may be given by a method that appeals to the sense of hearing, touch, or smell, such as an audio warning 1240.

[0017] Furthermore, remote notification may be performed by using network communication etc. to notify a location away from where a person with a high fever was detected. This remote notification method may be email communication (230) 1250 or posting of information on the Web screen (210) 1260. The information posted on the Web screen 1260 is not limited to individual information on people with a high fever, and may include the results of summarizing information obtained from multiple different locations.

[0018] Furthermore, the form of notification is not limited to the "information transmission" described above, but may be a physical operation involving device operation 1290, such as a blocking process 1270 of the passage gate (820).

[0019] The notification method is not limited to the above-described options, and the presence of a person with a high fever may be notified by any method. In addition, the notification method may be a method of notifying a plurality of options at the same time.

[0020] Conventional monitoring systems have a fixed mechanism for notifying after detection, and therefore there is no flexibility in the method of use. In contrast, the notification method after detecting a person with a high fever in this embodiment can be flexibly set according to the devices, software, etc. owned by the user. As options that can be flexibly set, a Patlite (registered trademark) may be turned on, an alarm sound 1240 may be sounded, a voice guide 1280 (voice warning 1240) using voice synthesis, notification by email transmission 1250 or messaging software such as SNS (Social Networking Service), screen display 1210, 1230, etc. can be arbitrarily set. As the notification method, a method set by the user in advance is used. As a technique for enabling flexible settings in accordance with the devices, software, etc. owned by the user, this embodiment involves setting application (IF-THEN) rules 70 and setting an application (IF-THEN) engine 90 that utilizes the rules, as will be described later with reference to Figures 7, 11, and 22. In other words, the user can flexibly set one or more notification methods from among the multiple notification method mechanisms described above, for example, on the screen of a smartphone or the like.

[0021] The basic system (high fever detection system) in this embodiment will be described with reference to Fig. 2. This basic system (high fever detection system) uses an infrared camera 806, which corresponds to a first optical sensor having infrared light, and a normal camera 816, which is a second optical sensor including an image sensor having wavelengths other than infrared light.

[0022] Generally, the wavelength range of 380 nm to 810 nm is called "visible light", and the wavelength range of 810 nm to 1 mm is generally called "infrared light" or "infrared light". However, the wavelength range of "infrared light" suitable for "human body temperature measurement" using blackbody radiation is within the range of 1 μm to 100 μm, as shown in FIG. 19, and preferably within the range of 2.5 μm to 20 μm. Therefore, the wavelength range of "infrared light" or "infrared light" used in the infrared camera 806 as the first optical sensor here means the range of 1 μm to 100 μm (preferably within the range of 2.5 μm to 20 μm).

[0023] Therefore, wavelength light outside the above range is referred to here as "wavelength light other than infrared light". In this embodiment, a second signal obtained from a second optical sensor (normal camera 816) including an imaging element having the above-mentioned "wavelength light other than infrared light" is also used to detect a person with a high fever whose body temperature exceeds a predetermined value. Suitable light for this purpose includes ultraviolet light (wavelength of 10 nm or more), visible light, near infrared light (wavelength of 2.5 μm or less), and microwaves (wavelength of 10 cm or less). Taking the above information into consideration, the wavelength range of "wavelength light other than infrared light" is in the range of 10 nm to 2.5 μm or the range of 20 μm to 10 cm (preferably in the range of 380 nm to 1 μm or the range of 100 μm to 10 cm).

[0024] The term "normal camera 816" generally refers to a second optical sensor including an imaging element having a "visible light" wavelength range of 380 nm to 810 nm. However, the "normal camera 816" depicted in FIG. 2 refers to an optical sensor including an imaging element having a wavelength range of 10 nm to 2.5 μm or 20 μm to 10 cm (preferably 380 nm to 1 μm or 100 μm to 10 cm).

[0025] Many edge computers 6 such as smartphones and tablets are equipped with a standard built-in normal camera 816. Therefore, the camera that is standardly built into the edge computer 6 such as a smartphone or tablet may be used as the normal camera 816 shown in FIG.

[0026] On the other hand, when a commercially available infrared camera 806 is used, the first signal (temperature characteristic signal) obtained from the infrared camera 806 (first optical sensor having infrared light) may be communicated to the edge computer 6 using wireless communication means or wired communication means. Alternatively, the signal may be directly transferred via the connection unit 10 as described later with reference to FIG. 3.

[0027] As shown in Fig. 2, the application 100 (software program) of this system is composed of a function of a recognition setting unit 110 capable of setting a recognition logic according to a usage scene, a function of a judgment temperature setting unit 120, a function of a detection rule setting unit 130 that sets an application (IF-THEN) rule 70 (described later with reference to Figs. 7, 11, and 22), a function of a detection time operation setting unit 140, and a function of a detection information transmission unit 150. In the embodiment of Fig. 2, the application 100 of this system takes the form of software installed in the edge computer 6. However, the present invention is not limited to this, and some of the functions of the components 110 to 150 may be configured with dedicated hardware (electronic circuits).

[0028] Therefore, the edge computer 6 on which the application 100 (software program) of the above-mentioned system is installed, or the edge computer 6 on which dedicated hardware (electronic circuits) for executing part of each function 110 to 150 is incorporated corresponds to the high fever detection device in this embodiment.

[0029] A first signal (temperature characteristic signal) obtained from the infrared camera 806 (first optical sensor having infrared light) is transmitted to an infrared image recognition unit 116 (first image recognition unit) of the recognition setting unit (function) 110. Then, in this infrared image recognition unit 116, the spatial temperature distribution characteristic is image-recognized (first image recognition).

[0030] An image signal (corresponding to a video signal or a still image signal, corresponding to the second signal) obtained from a normal camera 816 (a second optical sensor including an image pickup element having wavelength light other than infrared) is transmitted to a normal camera image recognition unit 112 (a second image recognition unit) of a recognition setting unit (function) 110. Then, as the second image recognition performed by this normal camera image recognition unit 112, 1) image recognition (image analysis) of the image signal is performed, 2) determination of whether or not there is a person in the screen (person detection), 3) if there is a person in the screen, calculation of the number of people (number detection), and 4) only if there is a person in the screen, recognition of the position of the head of each person (head detection). At the same time, 5) calculation (detection) of the relationship between the person and the infrared camera 806 may also be performed for each person identified in the screen.

[0031] The judgment temperature setting unit (function) 120 also has a person's head temperature judgment unit 125, in which the temperature of the person's head facing the direction of the infrared camera 806 is measured as the body temperature. Then, a person with a high fever whose body temperature exceeds a predetermined value (for example, 37.5 degrees Celsius) is detected (determined whether or not the person has a high fever). If a person with a high fever is detected as a result of the temperature judgment, a notification is automatically issued. Specifically, the detection information is transmitted to the detection operation instruction unit (application (IF-THEN) engine) 90 of the detection rule setting unit (function) 120. Then, the detection unit operation setting unit (function) 140 operates to execute the notification process.

[0032] 3 and 4, the "2) Person detection" performed by the normal camera image recognition unit 112 will be described. The type of heat source of blackbody radiation (what is radiating heat from?) cannot be identified from only the first signal (temperature characteristic signal) obtained from the infrared camera 806. Therefore, if the image recognition technology performed by the normal camera image recognition unit 112 is not combined, the temperature of objects other than people will also be determined.

[0033] As an example shown in FIG. 3(a), consider a person drinking hot coffee from a coffee cup held in their hand. Let us also assume that at this time the surface temperature of the coffee cup is 50°C. As shown in FIG. 3(b), the thermography screen (output of the infrared image recognition unit 116) obtained from this scene detects a surface temperature of 50°C of the coffee cup. If the judgment temperature for whether or not a person has a high fever is set to "37.5°C", the surface temperature of the coffee cup, 50°C, exceeds this judgment temperature. If the type of heat source of blackbody radiation (what is the heat radiating from?) cannot be identified, it will be erroneously judged that "there is a person with a high fever" (FIG. 3(c)).

[0034] Figure 4 shows the effect of "using only the temperature of a person's head for judgment." Here, Figure 4(a) shows the same diagram as Figure 3(a). And when "using only the temperature of a person's head for judgment," the surface temperature of a coffee cup, 50°C, is excluded from the detection target (Figure 4(b)).

[0035] The result of "2) Person detection" performed by the normal camera image recognition unit 112 and the spatial temperature distribution characteristics within the screen output from the infrared image identification unit 116 are integrated and processed by the person's head temperature determination unit 125 (Fig. 2). Then, as shown in Fig. 4(c), the person's head temperature determination unit 125 uses "only the temperature of the person's head" to "determine whether or not a person has a high fever." Here, the temperature of the person's head is used to measure the body temperature, but any part of the person may be used to measure the body temperature. When measuring the body temperature of a part other than the head, the name of the "person's head temperature determination unit 125" may be changed to a "person's body temperature determination unit" that has the same function. In this case, "the person's body temperature determination unit detects whether or not a person has a high fever whose body temperature exceeds a predetermined value based on the results of the first image recognition and the second image recognition."

[0036] In this manner, in this embodiment, a first signal obtained from a first optical sensor (infrared camera 806) having infrared light and a second signal obtained from a second optical sensor (normal camera 816) including an image sensor having wavelengths other than infrared light are used in combination to detect a person with a high fever whose body temperature exceeds a predetermined value. As a result, the accuracy of the normal judgment (FIG. 4(d)) is significantly improved.

[0037] Next, we will explain "4) Head detection" which is performed by the normal camera image recognition unit 112 (Fig. 2) mentioned above. As described above with reference to Figs. 3 and 4, acquiring only the temperature of the head has the effect of eliminating false detections caused by other heat sources. Furthermore, A) the torso and limbs are covered by clothing and not exposed, so there is a large error in the body temperature measurement results from these, and B) the head is relatively exposed, so it is relatively easy to measure the body temperature. Therefore, acquiring only the temperature of the head has the effect of improving the accuracy of the body temperature measurement.

[0038] Furthermore, the relationship with "3) Number of people detection" normally performed by the camera image recognition unit 112 will also be described. In images (video or still images) obtained from a place where multiple people are crowded together, C) parts of the body (torso and limbs) tend to overlap with adjacent people, but the head positions are relatively separate. Therefore, in this embodiment, by using the head position in particular, not only is it possible to simplify the image recognition process of "3) Number of people detection", but the accuracy of "3) Number of people detection" is also improved.

[0039] A specific method for "2) Person detection" and "3) Number of people detection" using "4) Head detection" performed by the normal camera image recognition unit 112 will be described. A number of standard samples relating to people's head shapes and hairstyles are stored in advance in the normal camera image recognition unit 112. Then, the degree of pattern matching between the image (second signal) obtained from the normal camera 816 (a second optical sensor including an imaging element having light of wavelengths other than infrared) and the above-mentioned multiple standard samples is calculated. Then, an area where the degree of pattern matching exceeds a predetermined value is estimated to be a "candidate for a human head."

[0040] In many cases, there is a torso under the "human head," and limbs (hands and feet) are arranged around the torso. "2) Person detection" is performed only when a torso or limb is detected directly under the estimated "human head candidate" in this way. The number of areas on the screen where "2) people are detected" is then the result of "3) number of people detection."

[0041] However, if the area "4) detected as the head" is the "back of the head", accurate body temperature measurement becomes difficult. Consider the case where the person to be measured for body temperature is facing away. In this case, the blackbody radiation emitted from the epidermis of the head is scattered by hair on the optical path toward the infrared camera 806. This significantly reduces the amount of blackbody radiation detected by the infrared camera 806, significantly reducing the accuracy of body temperature measurement (the detection principle of thermography will be described later using Figure 19). For this reason, it is desirable to detect only "the head facing forward toward the infrared camera 806".

[0042] As a method of "4) Head detection" of only the "head facing the infrared camera 806", the positions of the "eyes", "nose" and "mouth" arranged in front of the head may be used. As a specific method, a plurality of standard samples relating to the shapes of the eyes, nose and mouth are stored in advance in the normal camera image recognition unit 112. Then, the image obtained from the normal camera 816 is divided into small areas, and the degree of pattern matching between the standard samples of the eyes, nose and mouth for each divided area is calculated. Then, the "head facing the infrared camera 806" is detected by using the positional relationship between the obtained "position of the area that can be the eyes", "position of the area that can be the nose" and "position of the area that can be the mouth".

[0043] In the above explanation, "4) Head detection" is performed for "a head facing the direction of the infrared camera 806", and the temperature of the head is measured by the person's head temperature determination unit 125. However, without being limited to this, the temperature may be measured using an area other than the head.

[0044] Furthermore, the method of "2) Person detection" does not necessarily have to use the results of "4) Head detection". In this embodiment, "2) Person detection" can be applied flexibly according to the usage scene. For example, the number of visitors visiting the reception counter is often less than a few at a time, so it is relatively easy to identify individual people. However, the normal camera 816 installed behind the reception counter can only capture the upper body of the visitor.

[0045] In addition, while it is possible to capture the whole body when photographing a passageway with a normal camera 816, it is necessary to recognize moving people. In other words, high speed is required to detect people with high fevers in passageways. On the other hand, when detecting people with high fevers in a classroom with about 40 students, it is necessary to measure the body temperatures of many people at the same time.

[0046] It is preferable to set an optimal person detection logic for each different usage scenario in advance and select the optimal logic for each usage scenario to enable flexible response. As a method for enabling flexible response according to this usage scenario, for example, a selection menu that allows the user to select the usage scenario may be provided so that the optimal person detection logic is executed. Also, as will be described later with reference to FIG. 13, multiple methods (optimal person detection logic for each usage scenario) are placed in an image recognition class (AI Analyze Class) that uses AI. Then, the optimal method for each usage scenario is selectively called 76 using an API command.

[0047] Next, "5) Detection of each person identified on the screen" performed by the normal camera image recognition unit 112 in Fig. 2 will be described. In the infrared camera 806, if the person is far from the heat source (person's head) that emits black body radiation, the temperature tends to be read lower than the actual temperature (this will also be described later with reference to Fig. 19). Therefore, by using the results of "5) detection", α) body temperature correction (temperature correction) is performed in the person's head temperature determination unit 125 for the spatial temperature distribution characteristics obtained by the infrared image recognition unit 116, or β) for people who are farther away from the infrared camera 806 than a certain distance, even if the temperature is lower than the judgment temperature, the detection time operation instruction unit (application (IF-THEN) engine) 90 can perform processing such as issuing a warning or issuing an instruction.

[0048] Therefore, by using “5) Detection”, the accuracy of body temperature detection is improved, and the accuracy of instructions from the detection operation instruction unit 90 is improved.

[0049] As a method of the above-mentioned "5) detection", individual images obtained from multiple normal cameras 816 may be used to calculate using "trigonometry" (a method used in surveying, etc., to calculate using the relationship between the sides and angles of a triangle). For example, when a smartphone or tablet is used as the edge computer 6, a normal camera 816 is built into the edge computer 6 as standard. Therefore, if individual images obtained from multiple edge computers 6 are shared by communication means, it is possible to calculate using "trigonometry". In addition, if a triple-lens infrared camera with two built-in visible light sensors 46 and 48 described later in FIG. 20 is used, "5) detection" can be performed by itself.

[0050] Alternatively, the maximum and minimum temperature positions may be pinpointed and used from within the image analyzed by the normal camera image recognition unit 112 in Fig. 2. In other words, the maximum and minimum temperature positions may be calculated using "trigonometry" by using information on the maximum and minimum temperature positions estimated by the normal camera image recognition unit 112 and information on the maximum and minimum temperature positions extracted by the infrared image recognition unit 116.

[0051] As another embodiment, location information using a global positioning system (GPS) or a beacon may be used. For example, consider a case where it is possible to identify each person detected as a result of image recognition by the normal camera image recognition unit 112. In that case, communication can be performed with a mobile terminal (smartphone) owned by the identified person to collect the other person's location information.

[0052] The detection operation setting unit 140 has a function of setting the control method of various notification means for notifying the presence of a person with a high fever whose body temperature exceeds a predetermined value. Specific notification means include options such as generation of an alarm sound 1240, voice guidance 1280, screen display 1230, sending of email 1250, and device operation 1290 (1270). Among these, the options used for automatic notification (multiple options can be selected at the same time) are specified according to application (IF-THEN) rules 70 (described later using FIG. 7 and FIG. 11) predefined by the detection operation instruction unit (application (IF-THEN) engine) 90. Information required for the notification means (options) specified according to the application (IF-THEN) rules 70 is communicated (sent) from the detection operation instruction unit (application (IF-THEN) engine) 90.

[0053] The application 100 of this system has a built-in function of a detection information transmission unit 150. Necessary information is appropriately transmitted from the detection information transmission unit (function) 150 to the cloud server 2 which manages, monitors, and configures the execution of the application 100 of this system.

[0054] The application (IF-THEN) rules 70 used in the detection rule setting unit (function) 140 may be set from the cloud server 2. In this case, the execution status of the application (IF-THEN) rules 70 needs to be monitored at multiple locations, and therefore the cloud server 2 is required.

[0055] Alternatively, the application (IF-THEN) rules 70 may be set locally from the display screen of the application 100 of this system. In that case, the application 100 of this system can be executed even without the cloud server 2.

[0056] Figure 5 shows the flow of operations executed within the basic system shown in Figure 2. The flow of operations shown in Figure 5 will now be described with reference to Figure 2.

[0057] During the operation of the application 100 of this system, periodic processing S01 is continued, which repeats the operations shown in Fig. 5. First, in step 02 (S02), an image signal (video or still image) captured by the normal camera 816 (second optical sensor) is transmitted as a second signal to the normal camera image recognition unit 112 (second image recognition unit). Then, this normal camera image recognition unit 112 performs "1) image recognition", and executes "3) number of people detection" and "head position calculation (4) part of head detection)" using the result of "4) head detection", and "5) detection" (up to this point is S02). Here, if the number of people is not 1 or more as a result of "3) number of people detection" (i.e., no people are included in the image obtained by the normal camera 816) (No in S03), the process proceeds to the end of processing in step 04 (S04).

[0058] On the other hand, if the number of people is one or more (a person is included in the image obtained by the normal camera 816) (Yes in S03), the person head temperature determination unit (person body temperature determination unit) 125 acquires temperature information at the head positions of each person corresponding to the number of people (S05). Prior to this, the infrared image recognition unit 116 (first image recognition unit) calculates the spatial temperature distribution characteristics within the screen using a temperature characteristic signal (first signal) acquired from the infrared camera 860 (first optical sensor having infrared light). The person head temperature determination unit (person body temperature determination unit) 125 superimposes the head position information of each person acquired from the normal camera image recognition unit 112 (second image recognition unit) on the spatial temperature distribution characteristics within the screen to acquire the head temperature (body temperature) of each person.

[0059] Next, as shown in step 06 (S06), a "set temperature determination" is performed to check whether the head temperature (body temperature) of each person exceeds a predetermined value (e.g., 37.5 degrees Celsius) in person head temperature determination unit (person body temperature determination unit) 125. If the result of this set temperature determination is that no person's body temperature exceeds the predetermined value, the process proceeds to the end of step 07 (S07).

[0060] 5, two types of judgment values ​​(predetermined values) are set for "set temperature judgment", namely, an "alarm value" (e.g., 37.5 degrees Celsius) and a "caution value" (e.g., 37 degrees Celsius). Here, the "alarm value" is set to a value higher than the "caution value".

[0061] If the determination result in step 06 (S06) is equal to or greater than the "alarm value", an alarm is issued as the notification process shown in step 10 (S10). This specific notification method is performed by selecting an option 1240 to 1290 suitable for "alarm notification" in detection operation setting unit 140 according to the provisions of application (IF-THEN) rules 70 set in detection operation instruction unit (application (IF-THEN) engine) 90.

[0062] On the other hand, if the judgment result in step 06 (S06) is below the "alarm value" and above the "caution value," the judgment is made taking into account the result of "5) detection" described above. As an example of how to use the result of "5) detection," it was explained that "β) for a person who is farther away from the infrared camera 806 than a certain (set value), even if the temperature is lower than the judgment temperature, the detection time operation instruction unit (application (IF-THEN) engine) 90 issues an instruction to warn the person."

[0063] If the distance between the infrared camera 806 and the person being measured increases, the measurement result (body temperature) will be lower than the actual temperature (details will be described later using FIG. 19). Therefore, as shown in step 08 (S08), it is determined whether the distance between the head position of the person being measured and the infrared camera 806 is farther than a predetermined set value. If this distance is closer than the set value (No in S08), it is deemed that the "measurement accuracy of the body temperature of the person being measured is high." In this case, it is determined that "there is no person with a high fever," and the process ends (S09).

[0064] On the other hand, if the corresponding value is far from the set value (Yes in S08), it is assumed that the measurement accuracy of the subject's body temperature is low. In other words, it is presumed that, although the subject has a high fever and the body temperature exceeds the "alarm value" (37.5 degrees Celsius), the correct body temperature could not be measured due to poor measurement accuracy of the thermography. Then, as shown in step 11 (S11), a "warning" notification process is performed.

[0065] In this case as well, in accordance with the provisions of the application (IF-THEN) rules 70 set in the detection operation instruction section (application (IF-THEN) engine) 90, the detection operation setting section 140 operates the options 1240 to 1290 suitable for a "warning notification".

[0066] 6 shows an example (use case) of the edge computer 6 in FIG. 2 corresponding to the high fever person detection device or a high fever person detection system including it. The high fever person detection system shown in FIG. 6 is composed of a triple-lens infrared camera 806 having a structure described later in FIG. 20 and an edge computer 6 (high fever person detection device) such as a smartphone. The two are connected via a connection unit 10.

[0067] Many people operate their smartphones while on a train or in a place where people gather (such as a restaurant). The upper part of FIG. 6 shows a situation where a group of people is nearby inside a train or in a place where people gather. The lower part of FIG. 6 shows a situation where another application program 22 is being used on the smartphone (edge ​​computer 6). If there is a person with a high fever in the nearby group, the detection result is displayed on the display screen 12 as a screen display 1230.

[0068] As an example of a warning screen display 1230 controlled by the detection operation setting unit 140, the detected person with a high fever is clearly indicated by diagonal lines. At the same time, a warning message saying "There is a person with a high fever nearby! Please be careful!" is also displayed.

[0069] In the method shown in FIG. 6, the user is automatically notified when another application program 22 is used, so that the immediacy of notification and convenience for the user are greatly improved.

[0070] In the use case of the high fever detection system shown in Fig. 6, a warning screen display 1230 is performed on the display screen 12 of the edge computer 6. However, the warning screen display 1230 may be performed on a head mounted display (HMD) 810, which will be described later with reference to Fig. 17.

[0071] Another example of this embodiment will be described with reference to FIG. 7. In the system configuration shown in FIG. 7, the cloud server 2 and the edge computer 6 are configured to be able to communicate with each other 18. The edge computer 6 can provide various services to users by controlling various devices 8 such as a sensor 802, a signage (electronic bulletin board) 804, an infrared camera 806, a mobile terminal 808 such as a smartphone or tablet, an HMD (Head Mounted Display) 810, and an IoT device (gate, etc.) 820. When these various devices are various sensors such as a temperature sensor, a pressure sensor, a gyro, or a normal camera 816, they may be built into the edge computer 6. However, some of these various devices may be arranged outside the edge computer 6 and controlled by communication.

[0072] This edge computer 6 has an application (IF-THEN) engine 90 inside, which performs processing (controls various devices 8) according to pre-recorded application (IF-THEN) rules 70 (pre-installed from the cloud server 2) to provide services to the user.

[0073] 7, microservices 80A-80F that individually control various devices 8 such as a sensor 802, a signage (electronic bulletin board) 804, an infrared camera 806, a mobile terminal 808 such as a smartphone or tablet, an HMD (head mounted display) 810, and an IoT device (gate, etc.) 820 are built into the edge computer 6 in advance (installed in advance from the cloud server 2). An application (IF-THEN) engine 90 operates these microservices 80A-80F to control cooperation between the various devices 8. At this time, an application program interface (API) command is issued from the application (IF-THEN) engine 90 to the microservices 80A-80F to be operated.

[0074] In addition, in this embodiment, the edge computer 6 may have (pre-install) an IoT server microservice 80G that cooperates with the IoT service server 220, a mailer microservice 80H that cooperates with the mail server 230, a Web microservice 80I that accesses the Web service server 210 to obtain a Web service, and the like. Specific control examples of the Web microservice 80I may include control to automatically write necessary information in a form element frame specified in HTML (Hyper Text Mark-Up Language) and transition control between Web pages. In this way, by setting up an area for the Web microservice 80I in the software architecture of the edge computer 6, the user can enjoy advanced services such as Web services. Not limited to the IoT server microservice 80G and the mailer microservice 80H shown in FIG. 7, a microservice 80 that can individually access any service available in a cyberspace (not shown) may be arranged in the same order as the Web microservice 80I. Examples of cyberspace services not shown in Figure 7 include user services such as automatic buying and selling of securities (specific brands of stocks), ordering of specific products and automatic transfer of the received product price (billing processing), application for purchasing movie or music tickets, and application for reservations for specified trips.

[0075] In this way, by making it possible to place (pre-install) microservices 80 that can individually access each service in cyberspace within the software architecture of the edge computer 6, users can enjoy collaborative services that connect different services in cyberspace.

[0076] Conventionally, the cooperation between services using various devices 8 in the real world and services in the cyber world was relatively weak. By making it possible to arrange microservices 80A-80F that control various devices 8 on the same level as the Web microservice 80I as shown in Figure 7, users can enjoy advanced cooperation services that seamlessly and seamlessly straddle the services in the real or physical world using various devices 8 and the services in the cyber world.

[0077] When the technology to handle video and audio, which were previously handled as analog data, as digital data was born, the word "multimedia" became popular, indicating the function of "integrating and digitally handling all media." Today, this digital world has become the norm.

[0078] In this embodiment, a dedicated control software (microservice 80) that individually controls various devices 8 that detect and operate objects and / or instances that exist in the real space and their states is newly defined, creating a "multi-service" world that allows the provision of more advanced services that combine (mix) services that can be provided in the real / physical space with services that can be provided in the cyberspace. The "method of combining various services" to provide advanced services to users is specified in the application (IF-THEN) rules 70.

[0079] The relationship between this FIG. 7 and the above-mentioned FIG. 2 will be described. The normal camera 816 in FIG. 2 corresponds to a type of sensor 802, and the infrared camera 806 in FIG. 2 and FIG. 7 means the same device 8. Therefore, the normal camera image recognition unit 112 in FIG. 2 corresponds to the sensor microservice 80A. And the infrared recognition unit 116 and the person's head temperature determination unit 125 in FIG. 2 correspond to the infrared camera microservice 80C. In FIG. 2, the result of the image recognition performed by the normal camera image recognition unit 112 is transferred to the person's head temperature determination unit 125. In FIG. 7, information is exchanged between the sensor microservice 80A and the infrared camera microservice 80C via the application (IF-THEN) engine 90.

[0080] 7. Moreover, the detection time operation instruction unit (application (IF-THEN) engine) 90 of the detection rule setting unit 130 corresponds to the application (IF-THEN) engine 90 shown in FIG.

[0081] 2 is processed by the mailer microservice 80H. When the screen display 1230 is displayed on the display unit of the HMD 810, the function of the detection time operation setting unit 140 that controls the screen display 1230 is processed by the HMD microservice 80E.

[0082] For example, when performing an equipment operation 1290 (1270) such as "closing the passage gate to restrict the passage of a detected person with a high fever," the function of the corresponding detection time operation setting unit 140 is processed by the individual equipment microservice 80F.

[0083] With regard to the application (IF-THEN) rules 70 and various devices 8 used by each edge computer 6, the cloud server 2 has a rule setting and distribution unit (function) 202 and a device management unit (function) 206 as functions to be executed. In addition, the cloud server 2 may have a function as a microservice registration unit (function or module) 204 (the term "module" will be defined later).

[0084] Various data obtained when the edge computer 6 provides a service to a user according to the application (IF-THEN) rules 70 is managed / accumulated 208 by the cloud server 2 and appropriately stored in the data / information storage device 200.

[0085] 8 shows an example of a program configuration used in an Android based smartphone. An ifLink widget 1100 displays the status of ifLink.

[0086] Microservices 80A to 80F written in Java (registered trademark) are installed in a lower area of ​​an ifLink application 1000. An application (IF-THEN) engine 90 runs as part of this ifLink application 1000. In addition, application (IF-THEN) rules 70 are also used in this ifLink application 1000. In other words, this ifLink application 1000 manages the microservices 80A to 80F and application (IF-THEN) rules 70 connected to ifLink.

[0087] The relationship between the inside of the edge computer 6 and the cloud server 2 shown in FIG. 7 will be described in detail with reference to FIG. 9. The edge computer 6 has an ifLink application 92, which is a service providing application program, pre-installed. The software architecture of the ifLink application 92 includes an application (IF-THEN) engine unit 90, a setting and management screen configuration unit 98, a data transmission unit 96, and a rule reception unit 94 as components. The edge computer 6 also has a memory area for storing application (IF-THEN) rules 70 written in, for example, an XML (Extensible Markup Language) format. The application (IF-THEN) engine unit 90 operates the corresponding microservice 80 according to the application (IF-THEN) rules 70.

[0088] Here, the ifLink application 92 has been described as a "software program," but it is not limited to this, and the ifLink application 92 may be configured as "hardware." In this case, the memory area for storing the application (IF-THEN) rules 70, the application (IF-THEN) engine unit 90, the setting and management screen configuration unit 98, the data transmission unit 96, and the rule reception unit 94 may each be configured as physically separate electronic circuits.

[0089] As shown in Figure 9, the software architecture of the cloud server 2 is composed of a microservice registration unit 204, a rule setting and distribution unit 202, a data collection service unit (DDS: Data Destination Service) 216, an endpoint management unit (EPM: End Point Manager) 214, a Web-API control unit 218, and a data / information storage device 200.

[0090] This rule setting and distribution unit 202 manages the setting of application (IF-THEN) rules 70 in accordance with user requests and the distribution of the application (IF-THEN) rules 70 to the user.

[0091] The data collected by the ifLink application is transferred from the data transmission unit 96 of the edge computer 60 to the data collection service unit (DDS) 216 of the cloud server 2 via the communication path 18. The data organized by the data collection service unit (DDS) 216 is then stored in the data / information storage device 200 one by one.

[0092] In addition, application (IF-THEN) rules set according to user requests are also stored in the data / information storage device 200. Then, at the required timing, the application (IF-THEN) rules 70 are read out from the data / information storage device 200 and distributed from the endpoint management unit (EPM) 214 to the rule receiving unit 94 of the ifLink application via the communication path 18. Then, the application (IF-THEN) rules 70 received by the rule receiving unit 94 are stored in a storage area of ​​the edge computer 6 (not shown). In addition, the microservice 80 authenticated and registered by a specified institution is also installed in the edge computer 6 via the endpoint management unit (EPM) and the rule receiving unit 94.

[0093] On the other hand, various data collected from the ifLink app and stored in the data / information storage device 200 are processed by various application services 1800 via Web-API. Specific service contents of this application service 1800 include visualization 1802 of data collected from the ifLink app, management 1804 of data collected from the ifLink app, status monitoring and abnormality response 1806 regarding the status of the edge computer 6 and the status of various modules 9 using data collected from the ifLink app (the module 9 will be explained later), analysis 1808 of data collected from the ifLink app, service optimization 1810 for users using the ifLink app and optimization 1810 of module 9 setting conditions, and information service 1812 using information obtained as a result of the data analysis 1808.

[0094] Fig. 10 shows functions implemented in the application (IF-THEN) engine 90 shown in Fig. 7 and Fig. 9. Specific functions include activation 902, rule management control 904, rule execution control 906, sensor data control 910, JOB control 912, log output function 914, identical event ignoring time setting function 916, IF-THEN function 918 across other edge computers 6, rule enabling / disabling function 920, execution time control function 922, intermediate data holding object function 924, etc.

[0095] In activation 902 , when the application (IF-THEN) engine 90 is started up from the ifLink application 92 , a connection with the ifLink application 92 is established.

[0096] The rule management control 904 reads the application (IF-THEN) rules 70 stored in advance when the application (IF-THEN) engine 90 is started, and constructs an internal state. That is, for example, it becomes a waiting state for detection output or a command from the actual sensor 802 (a preparation state with several options).

[0097] The rule execution control 906 extracts application (IF-THEN) rules 70 that match the conditions based on the internal state, based on the sensor data information notified from the ifLink application 92. In other words, a rule is selected according to the contents of the detection output (command contents).

[0098] The sensor data control 910 receives the device information 68 from the ifLink application 92 and starts execution control corresponding to the application (IF-THEN) rule 70 .

[0099] In the JOB control 912, necessary JOB information is notified to the ifLink application 92.

[0100] A log output function 914 outputs a log (for example, a service history) according to a log level (importance or detail of the log contents).

[0101] The same event ignoring time setting function 916 refers to a function for suppressing occurrence of the same sensor data for a specified period of time even if the same sensor data is input.

[0102] In the IF-THEN function 918 across other edge computers 6, the JOB is notified to another specific edge computer 6, and the JOB is executed there.

[0103] The rule enable / disable function 920 refers to a function for switching ON / OFF the rule specified in the execution (THEN) 78 (described later with reference to FIG. 11) of the application (IF-THEN) rule 70.

[0104] The execution time control function 922 refers to a function that can shift the timing of a particular execution (THEN) 78 so that a subsequent execution (THEN) 78 (described later with reference to FIG. 11) can be performed after a certain time.

[0105] The intermediate data holding object function 924 means a function that holds intermediate data generated during processing in a storage device and enables input to a condition (IF) 72 of another rule (described later with reference to FIG. 11).

[0106] 11 is a block diagram expressing the concept of an application example in this embodiment. In the world of "multi-service" proposed by this embodiment, for example, the result of change detection of an entity or state (or information related to them) 7 occurring in the physical real world directly cooperates with a service in cyberspace (Web service 210), making it possible to provide a service to a user.

[0107] That is, in FIG. 7, the Web service server 210 and the mail server 230 are explicitly shown as service forms for users that utilize devices other than the device 8. In contrast, here, various devices 8 (corresponding to 802-820 in FIG. 7) as means for providing services to users in the physical real space and means for providing services to users in the cyberspace (e.g., a Web screen provided by the Web service server 210) are collectively referred to as "instance, status, and / or information 7 that can be recognized by humans (corresponding to 7A-7F)". In addition, the above-mentioned device 8 corresponds to a type of "instance / status / information 7 that can be recognized by humans". In addition, a general-purpose means for individually controlling these is defined as "control means 4 (4A-4F)".

[0108] As mentioned above, in Fig. 7, the Web service server 210 and the mail server 230 are explicitly shown as service forms for users that utilize devices other than the device 8. In contrast to this, here, "instance, status, and / or information 7 (7A-7F) that can be recognized by humans" is newly proposed. The device 8 mentioned above corresponds to one type of "instance / status / information 7 that can be recognized by humans". In addition, the general-purpose means for individually controlling these are defined as "control means 4 (4A-4F)".

[0109] For example, consider a case where an AI (Artificial Intelligence) program is used to control a web screen (such as automatically filling in required information within a form element frame specified in HTML, or controlling transitions between web pages), or a case where a web screen is used to process an order, contract, or application. In this case, the information on the web screen (as a specific example, a "format" on a web screen that should be filled in during order processing, contract processing, or application processing) corresponds to a type of "entity, state, or information 7 that can be recognized by humans," and the AI ​​program that controls that information (as a specific example, automatically filling in required items in a "format" on a web screen, and performing processing such as pressing an "Execute" button on the web screen after user confirmation) corresponds to control means 4.

[0110] As another example, consider a case where big data collected using a sensor 802 is stored in the data / information storage device 200 and then analyzed using statistical analysis software. In this case, the sensor 802 alone is classified as a device 8, and thus corresponds to a type of "human-recognizable entity / state / information 7." The sensor microservice 80A that controls the operation of this sensor 802 is a type of control means (microservice) 4.

[0111] Furthermore, the data / information storage device 200 used to store big data is classified as a device 8, and therefore corresponds to a type of "human-recognizable entity / state / information 7." The big data stored in this data / information storage device 200 and the information obtained after data analysis can also be classified as a type of "human-recognizable entity / state / information 7." In comparison, the data analysis process performed by statistical analysis software falls under the broad definition of "data control." Therefore, this statistical analysis software corresponds to a type of control means (microservice) 4.

[0112] Here, the function of the control means 4 shown in Fig. 11 may be realized by a hardware configuration (for example, a combination of logic circuits) or by a software program. For example, the control means 4 in the form of a program using an object-oriented programming language is particularly called a microservice 80. Here, it goes without saying that a microservice created in a non-object-oriented programming language (for example, assembler) may also be used.

[0113] The devices 8 and the means for providing services in cyberspace are collectively (generalized) named as “recognizable entities / states / information 7”, and the generalized name including microservices 80 is called “control means 4”, and from now on, we will give a generic explanation for all types of services provided to users.

[0114] For example, the sensing function can only be realized when the "control means 4" controls the sensor 802 alone. The means for realizing a specific function related to the service provided to the user in this way are called "modules 9 (9A-9F)." Module 9 is basically composed of "recognizable entity / state / information 7" and "control means 4." As an installation form of this module 9, the entire module γ9C may be built into the edge computer 6. Alternatively, only the recognizable entities / states / information 7D, 7E of the modules 9D, 9E may be arranged outside the edge computer 6. In this case, the control means (microservices) 4D, 4E arranged (pre-installed) in the edge computer 6 and the recognizable entities / states / information 7D, 7E are signal-connected via communication means 18.

[0115] Furthermore, the control means (microservices) 4A and 4B may exist in the cloud server α2A, not limited to the edge computer 6.

[0116] Application rule β70B, which specifies the method of cooperation between module γ9C and module δ9D to provide a specified service to the user, is basically composed of basic logic that transitions over time from condition (IF) β72B to execution (THEN) β78B.

[0117] Using an example logic (application rule β70B) for "turn on the lights when it gets dark," this "condition (IF) β72B ⇒ execution (THEN) β78B" logic and its specific processing content will be explained. In this case, an API command 75 is issued from the application (IF-THEN) engine 90 to the control means (microservice) γ4C of the illuminance sensor as processing corresponding to the condition (IF) β72B. In response to this, the control means (microservice) γ4C of the illuminance sensor controls the illuminance sensor γ7C of the recognizable entity / state / information 7 to measure the illuminance of the surrounding environment. If the obtained illuminance is lower than a preset value, the control means (microservice) γ4C of the illuminance sensor returns the return value of the API command 75 to the application (IF-THEN) engine 90.

[0118] Then, the application (IF-THEN) engine 90 starts the operation of the execution (THEN) β78B according to the application rule β70B. At this time, the application (IF-THEN) engine 90 issues instructions to the necessary microservices based on the interpretation of the IF-THEN rule for the signal received from the microservice. In other words, the application (IF-THEN) engine 90 does not execute processing autonomously, but rather compares the data received from the microservice specified in IF with the conditions of the rule, and when the conditions are met, issues an execution instruction to the microservice specified in THEN.

[0119] Then, the application (IF-THEN) engine 90 issues the next API command 75 to the light switch control means (microservice) δ4D. Then, the light switch control means (microservice) δ4D controls the light switch δ7D, which is a recognizable entity / state / information 7, to turn on the light switch. This executes the function of the module δ9D, which has the function of controlling the light switch.

[0120] Next, a case where a user purchases new modules ε9E and ζ9F to expand the system will be described. The control means (microservices) ε4E and ζ4F of these modules ε9E and ζ9F and the application rule γ70C that combines them are generated in the cloud server β2B of the Web server. After that, they are installed 180 via the communication line 18 that connects the cloud server β2B and the edge computer 6.

[0121] The installed 180 control means (microservices) ε4E, ζ4F and application rule γ70C are then stored in a predetermined recording area of ​​the edge computer 6 in the form of a file (for example, as a microservice file 2602 or an application rule file).

[0122] In parallel with this, the cloud server β2B generates an integrated rule 700 that integrates the existing application rule β70B and the newly created application rule γ70C. Then, the cloud server β2B verifies the consistency between the existing application rule β70B and the newly created application rule γ70C. If any defect is found as a result of this verification, the user is notified at the time of the above-mentioned installation 180 or when the service is provided to the user.

[0123] Cloud server β2B can verify the application rules of a large number of edge computers. Edge computers include those in factories, hospitals, schools, government offices, farms, homes, etc., and each facility adopts its own application rules.

[0124] Terms used within this patent specification, including those used in the above description, are defined below.

[0125] ●[Module 9]--means a means for realizing a predetermined function, and is composed of a combination of a [recognizable entity or state, information (instance, status, and / or information) 7] and its [controller 4] that humans can recognize in the real space where the predetermined function is realized. In this embodiment, a service is provided to a user by a combination of operations (motions) of multiple [modules 9] controlled by individual [controllers 5]. [Module 9] described in this embodiment merely indicates the "concept" of a "combination for realizing a predetermined function". Therefore, the [recognizable entity or state, information 7] and its [controller 4] that constitute [Module 9] do not need to be "physically integrated". For example, the [recognizable entity or state, information 7] and its [controller 4] may be installed in physically separate locations, and the two may cooperate with each other using the communication means 18. As will be described later, neither of them needs to form a "physically real object".

[0126] ●[Human recognizable entity / status / information7] -- Refers to an object that realizes a specific function in real space. The object here also includes measurable [status] such as temperature, humidity, and pulse rate. In this patent specification, the content, data / information, processing, and types of devices described below are included in the above [entity, status, and / or information7] that can be recognized in real space.

[0127] ● [Data and / or Information] -- Includes both acquired data / information and provided data / information. Here, data collected by a [device] such as a sensor corresponds to a type of acquired data, and information obtained as a result of analyzing this acquired data corresponds to a type of acquired information. Naturally, this [data] also includes the measurement results of [conditions] such as temperature, humidity, and pulse rate. In addition, data and information provided to users as a [service] is classified as provided data / information. As a specific example, educational materials and educational teaching materials correspond to this provided data / information.

[0128] ● [Processing] -- Refers to the process of realizing a given function, and requires the intervention of electronic devices. In other words, it includes all specific actions using electronic devices. For example, specific actions such as ordering using a smartphone or web screen, remittance / bank transfer processing, currency exchange processing, contract processing (such as insurance), travel and business scheduling, and travel arrangements are included. However, manual processing such as "putting handwritten documents on an application form into a mailbox" does not involve electronic devices, so it does not fall under the category of [processing] described in this specification.

[0129] ●[Device 8]--Includes any electronic device that exists in the real world and is used to realize a specific function. Therefore, it also includes electronic devices that execute the above [processing], electronic devices that display the above [content], and storage devices that store the above [information / data]. This [device 8] may be a standalone device with a built-in communication function for communicating with external electronic devices. This [device 8] may also have a function that acts on physical (or chemical) states and phenomena in the natural world. Devices that act passively on the natural world include sensors, etc. Devices that act actively on the natural world include robots, drive mechanisms, display devices (display elements), etc. The physical form of this [device 8] is not limited to a stationary form, but may also be a mobile, portable form. For example, it may be a band-type sensor fixed to the user's arm or leg, a belt-type sensor fixed to the waist, or a VR (virtual reality) type or AR (augmented reality) type HMD 810 worn on the head, which can be worn by the user (wearable).

[0130] ● [Control means (controller) 4] -- means for controlling [human-recognizable entities / states / information 7] to realize a specified function. A communication function may be built in this, and the [human-recognizable entities / states / information 7] may be controlled using the communication means. Examples of using this [control means 4] include transition control between web pages, automatic input control of necessary parts in a web page, data analysis control of accumulated data, control of the buying and selling of goods, automatic travel arrangement processing control, and robot movement control. In addition, in the control means for sensors such as temperature and humidity, the [human-recognizable entities / states / information 7] detects the physical (or chemical) [states] of the natural world and controls them to be quantified (digitized). Here, in this control means, detailed procedures for realizing individual detailed functions are specified. For example, the detailed procedures may be specified in a "hardware form" in which control logic is formed by a combination of logic circuits. In addition, the detailed procedures may be specified in a "software form" such as a program that can be installed on an [edge computer 6] or [cloud server 2].

[0131] ● [Micro-service 80 / IMS]--means [control means 4] formed by a program using an object-oriented programming language. As mentioned above, the control target (object) of this microservice is not limited to devices, but also includes processes, contents, and information / data. Therefore, for example, program software using AI technology to control transitions between web pages or screen operations corresponding to a specific web page is also included as a type of microservice or as a part of it. Similarly, program software that uses AI technology to automatically extract the number of people appearing in a specific video content, for example, and program software that analyzes acquired data (for example, using statistical analysis, etc.) are also included as a type of microservice or as a part of it. The information obtained as a result of the analysis here and the recording device that stores that information are treated as [human-recognizable entity / state / information 7]. In addition, since [microservice 80] is a form of [control means 4], as mentioned above, by installing [microservice 4], it becomes possible to control the control target (human-recognizable entity / state / information 7) by [edge computer 6] or [cloud server 2]. When specifying detailed procedures in this "software format," forming the [control means 4] with a program using an object-oriented programming language such as Java (registered trademark) or Objective-C allows for effective use of existing program assets (such as embedding (import) in other programs or calling (issuing API command 75)) (details will be described later in Chapter 2). Furthermore, writing a program in Java, which is independent of the OS (operating system), improves the versatility of the [control means 4]. In the following explanation, it will also be called IMS (ifLink Micro-service) instead of microservice.

[0132] ● [Control Method] -- Refers to a program that constitutes part of the [microservice 80] and specifies the detailed procedures for realizing a specific individual detailed function. Each control method is treated as the smallest functional unit (i.e., subprogram) that realizes an individual detailed function. A control method name corresponding to a specific individual detailed function can be called as a corresponding function (command) as an API command 75 from the [application engine 90]. This allows the [application engine 90] to execute the individual detailed function corresponding to the control method name. In this embodiment, by writing the [control method] program (specifying the program contents) using an object-oriented programming language, the [control method] can be linked to the [application rule 70] as described above. (A detailed explanation with specific examples will be given later in Chapter 2.) ● [Control Class] -- Refers to a collection of individual [Control Methods] grouped by common similar functions. A control method that corresponds to the initial setting function of the instance (entity) to be created is called a constructor. The constructor name (control method name corresponding to the constructor) included in the same collection may be made to match the control class name. In the aforementioned [Microservice 80], the management unit that groups control methods by common similar functions corresponds to a [Control Class]. Therefore, multiple different [Control Classes] may exist in a [Microservice 80] with the same purpose of use.

[0133] ●[Control Package]--Indicates a set of classes with similar functions. In other words, it is defined according to a mechanism for managing a set of classes with similar functions by dividing them into folders. In particular, in this embodiment, a control package is divided for each entity / state / information 7 included in the same module 9, and a different control package name (identification information for each control package) is set for each. Therefore, the contents of each entity / state / information can be identified by using the individually set control package name (identification information for each control package). Here, when setting the name of a [control package] for controlling a [device] belonging to an entity / state / information, identification information of the manufacturer or distributor of the module (or device), type information of the module (or device), individual serial numbers, etc. may be used (set) as identification information for each control package.

[0134] ● [Microservice file 2602]--Indicates the storage unit (storage form / storage format) for saving [microservice 80] in the storage area of ​​the cloud server 2 or edge computer 6. If [microservice 80] is written in Java, for example, the contents of [microservice 80] are a series of descriptive statements in Java. To save this information, a file is constructed for each control class (in which the processing content for each control class is also described). The extension of this control class file is ".class". Also, a "folder" for each control package is placed in the storage area of ​​the cloud server 2 or edge computer 6, and the control class file is stored in it. The name of this folder is made to match the name of the control package. Then, the manufacturer / vendor identification information of the module (or device) and the type information of the module (or device) can be known from this folder name, making it easy to search for the required control class file.

[0135] ●[API (API command) 75]--Indicates a means of communication (communication tool) between the [application (IF-THEN) engine 90] and the [microservice 80] for operating each microservice 80 from the [application (IF-THEN) engine 90]. Each [microservice 80] has a hierarchical structure of a control package / control class / control method. In the [API (API command) 75], a specific control method name indicating a detailed procedure (program) for realizing an individual detailed function is specified, and individual detailed function operations are often performed on a control method basis. For example, an example of a method for executing an individual detailed function of this specific control method in an application program 22 (including the [ifLink application (service providing application program)] described later) written in Java language will be described. First, in the class of this application program 22, a control class of a control package containing a corresponding control method is incorporated (import processing is performed). Then, by specifying the corresponding control method name in the above class or method, this individual detailed function can be executed. ● [Application (IF-THEN) rule 70]--Indicates information on a combination method (rule) of a plurality of different control means 4 (or modules 9) that indicates a method of providing a predetermined service to a user. When the [Application (IF-THEN) rule 70] is configured, for example, by hardware configured by a combination of logic circuits, the output terminal of this combination logic circuit is electrically connected directly to the input terminal side of the control logic configured by the combination of logic circuits of the [Control means 4]. On the other hand, when this [Application (IF-THEN) rule 70] is specified in a description format conforming to a predetermined description method (including HTML that configures a Web screen), the application (IF-THEN) engine 90 of the cloud server 2 or edge computer 6 deciphers the contents of the above [Application (IF-THEN) rule 70]. Then, based on the result of the interpretation, the application (IF-THEN) engine 90 issues an API command 75 to the corresponding microservice 80, and control of the recognizable entity / state / information 7 from the microservice 80 is started.

[0136] ● [Integrated rule 700]--This rule is generated when multiple different [Application (IF-THEN) rules 70] are specified in the same edge computer 6 or the same ifLink application. The purpose of generating this [Integrated rule 700] is to discover in advance contradictions and malfunctions that may occur when multiple different [Application (IF-THEN) rules 70] are combined, and to avoid trouble when providing services to users. When a user wishes to specify multiple different [Application (IF-THEN) rules 70], the [Integrated rule 700] is first generated in the cloud server 2, and the cloud server 2 simulates operations according to this [Integrated rule 700]. If a contradiction or malfunction occurs in the simulation results, the user is notified and a remedy is proposed. After it is confirmed that no contradiction or malfunction occurs in the simulation results, this integrated rule 700 is installed 180 from the cloud server 2 to the edge computer 6. After this installation 180, the edge computer 6 operates various microservices 80 (control means 4) according to this [Integrated rule 700].

[0137] ● [Edge computer 6]--Having an [application (IF-THEN) engine 90], the [application (IF-THEN) rule 70] and various [microservices 80] necessary for providing the corresponding service to the user are stored in advance. When a user wants to receive a specific service from [Place 1], the [application (IF-THEN) rule 70] and the corresponding [microservices 80] are installed in advance from the cloud server β2B based on a request from the user to [Place 1]. The [edge computer 6] is composed of a processor, a memory unit, and a communication unit. The [application (IF-THEN) rule 70] and the corresponding [microservices 80] installed here are stored in the memory unit. In addition, the above processor performs the function of the [application (IF-THEN) engine 90] according to this [application (IF-THEN) rule 70]. As long as it is composed of a processor, a memory unit, and a communication unit, the [edge computer 6] can take any form. As a specific example of a form, a personal computer, a smartphone, a tablet, a signage, a gateway, a router, etc. may function as the [edge computer 6].

[0138] ● [ifLink application (service providing application program) 92]--means a program processed by the cloud server α2A or the edge computer 6 to provide a specific service to the user 1700. This is composed of an [application (IF-THEN) engine 90] that executes processing according to the contents of the [application (IF-THEN) rule 70], a setting and management screen unit 98, a data transmission 96 involved in communication between the cloud server 2, and a rule receiving unit 94. This [ifLink application (service providing application program) 92] is also programmed to first refer to the contents of a specific [application (IF-THEN) rule 70] that has been saved in advance. Then, the [application (IF-THEN) engine 90] of the cloud server α2A or the edge computer 6 executes processing according to the contents programmed in this [ifLink application (service providing application program) 92], and provides the specific service to the user 1700.

[0139] ● [Application (IF-THEN) engine 90] -- It is built into the cloud server α2A or the edge computer 6, and means a place (function) where processing / execution is performed within the cloud server α2A or the edge computer 6. In terms of hardware, a calculation processor (or its processing state) may be used. This [Application (IF-THEN) engine 90] reads the [Application (IF-THEN) rule 70] and deciphers its contents. Then, referring to the deciphered contents, it executes processing according to the contents programmed in the [ifLink app (service provision application program) 92]. During the processing / execution, it issues the necessary [API command 75] to the microservice 80. Inside this [Application (IF-THEN) engine 90], a programming language interpretation engine corresponding to the description format (corresponding programming language) in which the [Application (IF-THEN) rule 70] is written, and a control engine that executes processing according to the program specified in the [ifLink app (service provision application program)] while referring to the interpretation result. A Web browser function may be built in in response to the case where the [application (IF-THEN) rule 70] is described (expressed) in HTML.

[0140] ● [Cloud server 2]--Has an [application (IF-THEN) engine 90] and stores in advance [application (IF-THEN) rules 70] and various [microservices 80] necessary for providing services to users corresponding to the rules. When a user wishes to receive a specific service from [Place 1], the [application (IF-THEN) rules 70] and the corresponding [microservices 80] are transmitted to the edge computer 6 based on a request from the user to [Place 1]. When a link function (Anchor Element) that specifies a URL in HTML is used as an example of a method of transmission, this [cloud server 2] may have a web server function.

[0141] ● [Service]--means an active action that combines the control of a given [entity / state / information 7] to provide a user with a given [processing] or [state] change, or [content] or [data / information], either for a fee or free of charge. In this embodiment, a [service] is provided to the user according to an [application (IF-THEN) rule 70]. In this embodiment, as a form of providing this [service], a combination (composite) service that crosses (spans) the boundaries between a service in the real or physical space using various devices 8 and a service in the cyberspace using a Web service or the like may be provided.

[0142] The application concept of this embodiment described in Fig. 11 is summarized below. That is, the module 9 is composed of a human-recognizable entity or state / information 7 and a control means 4 that controls it. And a plurality of modules 9 can be defined. That is, a first control means γ4C for a first module γ9C and a second control means δ4D for a second module δ9D are defined individually.

[0143] Here, application rules 70 are set for providing services to users by combining this first module γ9C and the second module δ9D. Then, according to the contents of this application rule 70, API commands 75 are issued from the application (IF-THEN) engine 90 to the first control means γ4C and the second control means δ4D individually to operate the first or second control means γ4C, δ4D. The module control method in this embodiment is used.

[0144] The control means 4 includes a microservice 80. The microservice 80 may be formed (written) in a program using an object-oriented programming language. (However, the microservice 80 may be written in any programming language.) Here, a first microservice 80 (control means γ4C) that controls a first entity or state, information γ7C that can be recognized by humans, and a second microservice 80 (control means δ4D) that controls a second entity or state, information δ7D that can be recognized by humans are defined. Then, an application rule β70B is set for providing a service to a user by combining the first microservice 80 and the second microservice 80. Then, in response to an API command 75 issued from an application (IF-THEN) engine 90 according to the application rule β70B, the first and second microservices 80 individually control the first entity or state, information γ7C, δ7D that can be recognized by the first and second microservices 80.

[0145] In addition, in the edge computer 6 in this embodiment (application example), the application rule γ70C (and the integrated rule 700) and the microservice 80 (control means 4F) are received from the cloud server β2B and pre-installed 180. As a result, the first microservice 80 (control means ε4E) that controls the first entity or state, information ε7E that can be recognized by humans, the second microservice 80 (control means ζ4F) that controls the second entity or state, information ζ7F that can be recognized by humans, and the application rule γ70C (and the integrated rule 700) related thereto are pre-installed in the edge computer 6. Here, the application rule γ70C (and the integrated rule 700) specifies a rule for providing a service to a user by a combination of the first microservice 80 (control means ε4E) and the second microservice 80 (control means ζ4F). The edge computer 6 then operates the first microservice 80 (control means ε4E) and the second microservice 80 (control means ζ4F) to individually control the second entity or state, information ζ7F, which is recognizable by a human, or the second entity or state, information ζ7F, which is recognizable by a human, to provide a service to the user.

[0146] In particular, the edge computer 6 in this embodiment (application example) has an application (IF-THEN) engine 90. Then, according to the rules defined in the above application rule γ70C (and the integration rule 700), the application (IF-THEN) engine 90 issues an API command 75 to the first microservice 80 (control means ε4E) and the second microservice 80 (control means ζ4F). As a result, the first or second microservice 80 (control means ε4E, ζ4F) individually controls recognizable entities or states, information ε7E, ζ7F to provide a service to the user. And in the service providing method in this embodiment (application example), the above-mentioned method is used to provide a service to the user.

[0147] As described above, in this embodiment (application example), the contents of the microservice 80 can be written in an OS-independent programming language such as the Java language.

[0148] The effect of this will be explained with reference to Figure 12. In the case of Java, an OS-independent general-purpose programming language, translation areas α1300A and β1300B (Virtual Machines) corresponding to the individual OS layers α30A and β30B are prepared in advance. The contents of the microservice 80 written in Java are translated via the translation areas α1300A and β1300B (Virtual Machines) and passed to the individual OSs α30A and β30B.

[0149] In particular, the commands of the microservice 80 in this embodiment are written as a combination of basic API commands 75 at the OS layer α 30A and β 30B levels. As a result, the effect of being able to perform basic control of each device in detail is achieved. 13 shows the class configuration (software architecture) of the microservice 80. Note that, although this chapter will mainly explain the microservice 80, the following description is also applicable to the control means 4 in a broad sense.

[0150] A basic function required for the microservice 80 is a "function to control each corresponding device 8". However, in addition to that, "peripheral functions related to the control of each device 8" are also required as functions of the microservice 80. As a specific example, it is necessary to perform interface processing with an application (IF-THEN) engine and to check whether the corresponding device 8 is in an operable state.

[0151] In this embodiment, the microservice 80 is separated into different classes according to different functions, improving the functional extensibility of the microservice 80. That is, to execute the "control function of each corresponding device 8", an individual device control class (Custom Device Class) 2102 is set. Then, multiple methods in which detailed procedure program contents for each individual detailed function of the corresponding device 8 are written are placed (written) in this.

[0152] On the other hand, a control class (Customims Class) 2110 within the microservice corresponding to an individual device is set up as a program for executing "peripheral functions related to the control of each individual device 8".

[0153] Here, the individual control methods of the individual device control class (Custom Device Class) 2102 are called (embedded) 76 from the individual device-compatible microservice control class (Customims Class) 2110, making them executable. Specifically, when import [individual device control package name].[individual device control class name (Custom Device)]; is specified using an import statement before the individual device-compatible microservice control class (Customims Class) 2110, the individual control class in the individual device control class 2102 can be embedded 76 in the individual device-compatible microservice control class (Customims Class) 2110. Then, the individual control method name, arguments, and return value format (type) in the individual device control class 2102 are described in the individual device-compatible microservice control class (Customims Class) 2110 or a specific control class therein. This makes it possible to call / use 76 the individual control method in the individual device control class 2102 in the individual device-compatible microservice control class (Customims Class) 2110 or a specific control class therein.

[0154] In this embodiment, it is possible to transmit data E60 that is optimal for the characteristics of each device 8 to be controlled to the device 8. To make this possible, in this embodiment, it is possible to set different individual device control packages 2100 for different manufacturing / sales makers and device models.

[0155] On the other hand, it is necessary to guarantee compatibility between systems delivered to users and future expandability. In order to ensure such compatibility between systems and future expandability, a template class that serves as a standard for the description content (program) of each class may be provided. That is, for the individual device control class (Custom Device Class) 2102, a device control template class (Base Device Class) 2002 is provided to any programmer as a template class. Also, for the individual device corresponding microservice control class (Customims Class) 2110, a microservice control template class (BaseIms Class) 2010 is provided to any programmer as a template class.

[0156] A programmer who wishes to program a new microservice 80 can create an individual device control class (Custom Device Class) 2102 and an individual device-compatible microservice control class (Customims Class) 2110 by only making minor modifications (customizations) 74 to the template contents, which has the effect that the programmer who creates a new microservice 80 can create the microservice 80 efficiently in a short period of time.

[0157] Then, to show that each class inherits (Extends) 74 the functionality of the original template class, if you write, for example, public class Custom Device extends Base Device { in the place where you define the name of each individually created class, the relationship of functional inheritance (Extends) 74 between classes will be clearly stated. Using this method of description not only makes it easier to manage individual microservices 80, but also makes it easier to ensure compatibility and scalability between systems.

[0158] Incidentally, the "public" above means that "use (repurposing) of the Custom Device Class in other systems is permitted." If "private" were written here instead of "public," use by any class other than your own (in the above example, the Custom Device Class) would be prohibited. Writing "protected" would mean that it would only be available to the package's classes and classes that inherit functionality74. When many programmers are involved in creating (modifying) microservices80, problems such as unintentional tampering are likely to occur. Making rules like the above has the effect of reducing the frequency of problems occurring between multiple programmers.

[0159] As shown in FIG. 13, an API command (linkage specification) 75 from an application (IF-THEN) engine 90 is processed by a control template class (BaseIms Class) 2010 within the microservice.

[0160] Also, while a specific service is being provided to the user 1700, an abnormality such as the battery running out of the device 8 may occur. An abnormality detection class (Health Check Task Class) 2016 having a function for dealing with such an emergency abnormality is provided, making it possible to quickly deal with the abnormality. This abnormality detection class (Health Check Task Class) 2016 is called / embedded from the microservice control template class (BaseIms Class) 2010 as a call / embedded process 76. The call / embedded process 76 here is processed in the same manner as described above.

[0161] In particular, in this embodiment, AI processing such as image recognition is required. For this reason, the basic control package 2000 includes an AI-based image recognition class (AI Analyze Class) 2018 that specializes in image recognition processing using AI technology.

[0162] In the normal camera image recognition unit 112 in Fig. 2 (corresponding to the sensor microservice 80A in Fig. 7), 1) image recognition (image analysis) of the image signal is performed, 2) whether or not there is a person on the screen (person detection), 3) if there is a person on the screen, the number of people is calculated (number of people detection), 4) only if there is a person on the screen, the position of the head of each person is recognized (head detection), and 5) for each person identified on the screen, the position of the person and the infrared camera 806 is calculated (detected). Correspondingly, in the image recognition class using AI (AI Analyze Class) 2018, individual methods that perform the individual processing of the above [1] to [5] are arranged (programmed).

[0163] Then, using an API command corresponding to each method, each corresponding method is called 76 from the microservice control template class (BaseIms Class) 2010 or the individual device corresponding microservice control class (Customins Class) 2110. Here, the corresponding methods are called 76 in sequence according to the flow procedure shown in FIG.

[0164] As already explained, the image recognition process for "2) Person detection" performed by the normal camera image recognition unit 112 in Fig. 2 requires flexible processing according to the usage scene. For example, the number of visitors visiting the reception counter is often less than a few at a time, so it is relatively easy to identify individual people. However, the normal camera 816 installed behind the reception counter can only capture the upper half of the visitor's body.

[0165] In addition, while it is possible to capture the whole body when photographing a passageway with a normal camera 816, it is necessary to recognize moving people. In other words, high speed is required to detect people with high fevers in passageways. On the other hand, when detecting people with high fevers in a classroom with about 40 students, it is necessary to measure the body temperatures of many people at the same time.

[0166] In this way, image recognition classes (AI Analyze Class) 2018 using multiple AIs each corresponding to the optimal person detection logic (image recognition algorithm) according to the usage scene such as "reception counter", "in the corridor", "in the classroom", etc. may be simultaneously arranged in the same basic control package 2000. (However, in this case, the class name changes for each class.) Then, from the microservice control template class (BaseIms Class) 2010 or the individual device compatible microservice control class (Customims Class) 2110, the corresponding class name is designated and each method individually arranged (written) therein is called and processed 76.

[0167] In addition, in the AI-based image recognition class (AI Analyze Class) 2018, all of the individual methods that perform the individual processing of [1] to [5] above according to the usage scenario, such as "at the reception counter," "in the hallway," or "in the classroom," may be placed (programmed) in the class.

[0168] In this way, by arranging an image recognition class (AI Analyze Class) 2018 using AI in the same basic control package 2000, it becomes possible to call 76 / utilize each corresponding class in the microservice 80. This makes it possible to select the optimal person detection logic for each different usage scenario, enabling flexible and highly accurate image recognition and AI processing.

[0169] A method of data movement used for image recognition (or AI processing) in the same microservice 80 will be described. The movement of data between blocks shown in Fig. 2 and the movement of data between methods in the same microservice 80 are essentially synonymous. For example, in Fig. 2, the image recognition result in the block of the normal camera image identification unit 112 is moved to the block of the human head temperature determination unit 125.

[0170] 13 is used in the corresponding method of an image recognition class using AI (AI Analyze Class) 2018. The data movement at this time is controlled and managed by a predetermined program of a microservice control template class (BaseIms Class) 2010 or a microservice control class (Customins Class) 2110 for individual devices.

[0171] Furthermore, without being limited to this, detection data of the sensor 802 acquired by the device control template class (Base Device Class) 2002 or the individual device control class (Custom Device Class) 2102 may be temporarily stored in a buffer memory 2008 (for temporarily storing stream data). Then, a corresponding method of an image recognition class using AI (AI Analyze Class) 2018 may read out necessary data from there and use it.

[0172] 13, an image recognition class (AI Analyze Class) 2018 (data analysis processing program) that performs specialized image recognition is placed as an example of AI processing. However, without being limited to this, any AI class (AI program) specialized for a specific function using any artificial intelligence technology, such as a data control function, a data manipulation function, or an automatic data collection function (using the Internet), may be placed in the basic control package 2000.

[0173] By providing an AI class that specializes in data analysis in the basic control package 2000 in this way, advanced data analysis processing can be executed within the microservice 80. This allows only the data analysis results processed within the microservice 80 to be passed to the application (IF-THEN) engine 90, which has the effect of significantly reducing the processing load on the application (IF-THEN) engine 90. Furthermore, since an AI class (AI program) specialized for a specific function using artificial intelligence technology can be arbitrarily placed in the basic control package 2000, advanced processing using artificial intelligence technology can be arbitrarily performed within the microservice 80.

[0174] When a device 8 is installed outside the cloud server 2 or the edge computer 6, the microservice 80 controls the device 8 via a communication line 18. Here, for example, if the communication line 18 becomes congested or is disconnected, there is a risk that the system will freeze until a response is returned from the device 8.

[0175] On the other hand, by installing a response time monitoring class (Timeout Check Task Class) 2006, it is possible to monitor the communication status between the microservice 80 and the device 8. Then, by detecting a communication problem and taking a quick response, it is possible to prevent the system from freezing.

[0176] Depending on the type of sensor device 802, various data ranging from binary data (two-value data of "1" or "0") to video streams must be handled. By providing a stream control engine class (Stm Engine Class) 2004, it is possible to efficiently handle various data in an integrated manner. A stream analysis class / method using AI technology may be placed inside this stream control engine class (Stm Engine Class) 2004 or at the same level as it. That is, as described above, an import statement is used to incorporate a stream analysis class using AI technology into the stream control engine class (Stm Engine Class) 2004 or the device control template class (Base Device Class) 2002 (when placed at the same level as the stream control engine class (Stm Engine Class) 2004). This makes it possible to use (call) the stream analysis class / method using AI technology in the stream control engine class (Stm Engine Class) 2004 or the device control template class (Base Device Class) 2002.

[0177] This makes it possible to pass not only raw stream data obtained from the sensor device 802 to the application (IF-THEN) engine 90, but also only analysis result information obtained as a result of automatically analyzing the stream data to the application (IF-THEN) engine 90, resulting in an effect of significantly reducing the processing load of the application (IF-THEN) engine 90. In this case, the raw stream data obtained from the sensor device 802 may be sequentially stored as a time-series file in the data / information storage device 2 in the edge computer 6 or the cloud server 2. Then, a method of the stream analysis class using AI technology plays the time-series file (raw stream data file) at appropriate times. Then, a specific condition (IF) β may be determined using the analysis information obtained by the data analysis using AI technology.

[0178] 13, the above-mentioned response time monitoring class (TimeoutCheckTask Class) 2006 and stream control engine class (StmEngine Class) 2004 can be called / embedded 76 from the device control template class (Base Device Class). The specific method of this call / embedding 76 process is the same as that described above.

[0179] In this embodiment, when the microservice 80 is written using an object-oriented programming language (such as the Java language), the existing program assets of the basic control package 2000 can be effectively used in the individual device control package 2100. This significantly improves the efficiency of the programmer in developing the microservice 80.

[0180] For example, by writing import statements such as import [name of basic control package 2000].[abnormality detection class name (HealthCheckTask)];, import [name of basic control package 2000].[response time monitoring class name (TimeoutCheckTask)];, and import [name of basic control package 2000].[stream control engine class name (StmEngine)]; before the individual device-compatible microservice control class (Customins Class) 2110 and the individual device control class (Custom Device Class), it is possible to use the programs of the methods defined in each class.

[0181] Furthermore, the destination of the API command (collaboration specification) 75 from the application (IF-THEN) engine 90 is changed from the microservice control template class (BaseIms Class) 2010 to the individual device-compatible microservice control class (CustomIms Class) 2110.

[0182] As a result of this series of processes (program changes), the device control template class (Base Device Class) 2002 and the control template in microservice (BaseIms Class) 2010, which are the corresponding classes in the base control package 2000, are essentially replaced 71 with the individual device control class (Custom Device Class) 2102 and the control class in microservice corresponding to the individual device (Customims Class) 2110, which are the corresponding classes in the individual device control package 2100. Writing the microservice 80 using an object-oriented programming language in this way has the effect of allowing easy and accurate program editing with only a very slight change in the written content.

[0183] The functions of the microservice 80 set in the individual device control package will be described with reference to Figure 14. The three items circled in the required column 2204 (initialization 2210, termination 2212, and start / stop 2214) are the minimum required functions. The remaining two items (sensor data transmission 2218 and job reception 2220) may be unnecessary functions depending on the characteristics of the corresponding device 8.

[0184] Among the microservice functions, Initialization 2210 indicates the function of connecting to the ifLink app and registering the microservice 80 and the device 8 to be controlled when launched from the ifLink app (see the definitions of terms above).

[0185] The function of termination 2212 is a function that, when terminated from the ifLink app, disconnects and terminates the connection with the ifLink app, the microservice 80, and the device 8 that it controls.

[0186] The start / stop 2214 function is a function for starting or stopping the device 8 when the rule of the device 8 registered as the application (IF-THEN) rule 70 becomes valid or invalid.

[0187] Next, the function of the status notification 2216 will be described. The ifLink application manages the status of registered devices. This status notification 2216 indicates a function of notifying the ifLink application in order to update the status of the device 8 managed by the ifLink application.

[0188] The function of sensor data transmission 2218 is a function of transmitting sensor data sent from the controlled device 8 to the ifLink application.

[0189] Finally, we will explain the function of JOB reception 2220. JOB contents (operation instructions / API) are notified from the cloud server 2 or application (IF-THEN) engine 90 via the IfLink application. This JOB reception 2220 is a function that processes JOB information received from the IfLink application.

[0190] Fig. 15 shows a list of APIs provided by the microservice control template class (BaseIms Class) 2010 or the individual device compatible microservice control class (Customims Class). That is, each control method shown in Fig. 15 is described (placed) in the microservice control template class (BaseIms Class) 2010. That is, each control method described in the method / function column 2302 in Fig. 15 must be provided as standard in the microservice 80.

[0191] The "void" or "int" at the beginning of each control method listed in the method / function column 2302 indicates the type (format / type) of the return value of the control method. For example, when this control method is used by calling 76 from a specific class or a specific method, it indicates "what type of data will be returned to the class or method being used" after the processing of this control method is completed.

[0192] For example, a control method that begins with the word "void" indicates a state of "no return value" (i.e., after the control method has finished processing, no specific data is returned to the class or method that is using it).

[0193] Similarly, in a control method that starts with the letters "int" or "long", an integer in the 32-bit or 64-bit range is returned after the control method has finished processing.

[0194] In a control method where the word “boolean” is written first, a truth value of “true” or “false” is returned after the control method has finished processing.

[0195] Additionally, within the parentheses of the control method, the "argument type (format / type)" and "argument" that will be passed on to the control method are listed as a pair separated by a "space." If multiple "arguments" are passed on here, a "," (comma) is placed between the pairs of arguments.

[0196] As for the type (format / type) of this argument, "String" represents a "character string." Additionally, "Map" represents a "key / value database" in which keys and values ​​are stored in pairs. Additionally, "HashMap" refers to the above-mentioned "key / value database" or a class (HashMap Class) that uses this mechanism. In this way, using a "key / value database" to control device 8 has the effect of enabling data / information to be shared between methods across different control packages. Furthermore, using a "key / value database" as the storage format for this data / information improves the convenience of data searches.

[0197] Moreover, "Message" means a "message" that can be sent directly to the application (IF-THEN) engine 90 or to the ifLink application. The "constructor" shown in Fig. 15 has already been explained in the definition of terms related to the [control class].

[0198] "EPA", "epa" and "Epa" in FIG. 15 are abbreviations for "End Point Access" and indicate the application (IF-THEN) engine 90 and the ifLink application. In relation to this, "void onActivationResult(boolean result, EPADevice device)" is described at the 28th position in FIG. 15. This process outline 2304 is described as "notification of service registration result from ifLink application". Here, as arguments, "EPADevice" is specified as "argument type (format / type)" and "device" is specified as "argument". "Service registration" here means "registration of device 8 operated by application (IF-THEN) rule 70". The result is entered in an argument called "result". If "registration was successful", "true" is entered in "result" and the process returns. On the other hand, if "registration was not successful", "false" is entered in "result" and the process returns. The above argument "device" means a specific device 8 to which identification information called "device" is set. And the "EPADevice" that specifies the "argument type (format / type)" means "a device 8 that the application (IF-THEN) engine 90 or the ifLink application can identify."

[0199] The device 8 used in this embodiment also includes a portable device 8. Therefore, when an attempt is made to operate the portable device 8 according to the application (IF-THEN) rule 70, there are many cases in which the target portable device 8 is taken out of the operation area. Therefore, before providing a service to a user, it is necessary to confirm in advance whether the target portable device 8 is present in the operation area. For this purpose, it is necessary to pre-register the portable device 8 whose presence in the operation area should be confirmed in advance.

[0200] For this advance registration, execute the callback registration "void registerCallback()" or "void registerCallback(String cookie)" described in item 11 or 12. For the portable device 8 for which advance confirmation of whether it is in the operation area is no longer necessary, execute the callback cancellation "void unregisterCallback()" described in item 13.

[0201] If the target device 8 can be confirmed in advance, the 14th "boolean createDevice()" is used to perform "device registration." Conversely, for devices 8 that could not be confirmed in advance, the 15th "boolean deleteDevice()" is used to perform "device deletion."

[0202] Also, while providing a service to a user, for example, a specific device 8 may run out of battery, making it difficult to continue the service. To deal with this situation, the 21st “void startHealthCheck(long interval)” can be used to “start a health check” for each device 8. In this method, instead of “continuous health checks”, checks may be performed at regular intervals specified by “interval”.

[0203] When a problem occurs, a "warning notification" must be sent from the microservice 80 to the application (IF-THEN) engine 90 or the ifLink app. In this case, the 17th method "int send_alert(-)" can be executed to send an "alert."

[0204] In the detection of a person with a high fever performed in this embodiment, there are two types of functions required of the sensor microservice 80A and the infrared camera microservice 80C by the application (IF-THEN) engine 90: a) detection (discovery) of a person with a high fever, and b) provision of information necessary to notify the user.

[0205] For example, in the use case shown in Figure 6, when "a) a person with a high fever is detected (discovered)," the detected person with a high fever is crossed out on the display screen 12 to warn the user. In this way, "b) information provision" is required to "visualize the detected person with a high fever." Incidentally, this "image showing the detected person with a high fever crossed out" is generated by the AI-based image recognition class (AI Analyze Class) 2018.

[0206] In such a case, the format of data provided to the application (IF-THEN) engine 90 (the type (format / type) of the return value of the control method) is different between (a) and (b) above.

[0207] In order to flexibly handle such situations, it is necessary to specify a "data format to be sent as a response" from the application (IF-THEN) engine 90 to the corresponding microservices 80A and 80C.

[0208] Therefore, as an API provided by the microservice control template class (BaseIms Class) 2010 or the individual device corresponding microservice control class (Customims Class), void setSendDataFormat(String format) is required to specify the data format to be sent from the corresponding method of the microservice control template class (BaseIms Class) 2010 or the individual device corresponding microservice control class (Customims Class).

[0209] The data type of the argument format of this API is set to String. In other words, the data format to be sent in text format is specified by the application (IF-THEN) engine 90.

[0210] Here, in the above "a) Report on the detection (discovery) of people with high fever", a large amount of data is not required. In this case, long send_data(Map map) is used for the data sending process, and information on "how many people with high fever were detected (discovered)" is returned to the application (IF-THEN) engine 90. Even if no people with high fever were detected, it is possible to return "0 people with high fever were detected (discovered)". The Map specified in this API refers to a format of "data that is a collection of key-value pairs".

[0211] That is, when the application (IF-THEN) engine 90 requests the corresponding microservices 80A and 80C to "a) report the results of detecting (finding) people with high fever", it first issues the API command void setSendDataFormat(String format) to execute "image recognition processing to calculate the number of people with high fever". Then it issues the API command long send_data(Map map) to "a) report the results of detecting (finding) people with high fever".

[0212] If a person with a high fever is detected (discovered), the application (IF-THEN) engine 90 requests the corresponding microservices 80A and 80C to "b) provide information necessary for notifying the user." In this case, the API command void setSendDataFormat(String format) is also issued, but the difference in the request content is specified in text format in the argument format.

[0213] As can be easily seen from the use case example in Figure 6, the format of "b) information necessary for notifying the user" may include stream data such as video and still images. Also, there are many different data compression methods for stream data such as video, still images, and audio information. Therefore, the data type, data compression method, necessary stream data content, and its display format (for example, only those with high fevers are clearly indicated by slashing them) are specified in the argument format of void setSendDataFormat(String format) issued from the application (IF-THEN) engine 90.

[0214] For example, when requesting video information, the application (IF-THEN) engine 90 issues an API command Stream send_data(Stream movie). Here, Stream means "stream format data," and video information is entered in movie.

[0215] On the other hand, if you want to request still image information, issue the API command Stream send_data(Stream image, long time). In this case, the still image information is stored in image. Here, the time information can be used to set the "time to capture (import) the still image information."

[0216] As can be easily seen from the use case in Fig. 6, an image (video or still image) may be used to notify the user. If the image to notify the user is generated on the microservices 80H and 80F related to the application (IF-THEN) engine 90 or the detection time operation setting unit 140, a large load is placed on the generation process.

[0217] As in this embodiment, by having the microservices 80A and 80C related to the condition (IF) 72 (Figure 11) perform "b) providing information necessary for notifying the user," the processing load on the high fever detection device (or the entire high fever detection system) can be reduced.

[0218] 16 shows a list of APIs (commands issued from the application (IF-THEN) engine 90) provided by the device control template class (Base Device Class) or the individual device control class (Custom Device Class). The description rules are the same as those in FIG.

[0219] An application rule 70 is basically composed of a combination of predetermined actions (THEN) 78 corresponding to predetermined conditions (IF) 72. For example, the recognizable entity / state / information 7 that is the subject of the condition (IF) 72 judgment changes from moment to moment, just as the temperature, humidity, and illuminance of the user's environment change from moment to moment. In order to accommodate this change over time, in this embodiment, the recognizable entity / state / information 7 can be recorded in chronological order on a "HashMap" that corresponds to a "key / value type database."

[0220] For example, when it is desired to obtain the current recognizable entity / state / information 7 (i.e., to obtain sensor data obtained from the sensor device 802, for example), the "HashMap createSensorData()" is called 76 from the microservice in-control template class (BaseIms Class) 2010 to "generate sensor data". On the other hand, when it is desired to "generate sensor data" at a specified time, the specified time for obtaining data is set in the argument "time" of "HashMap createSensorData(long time)". Also, when it is desired to obtain data at a specific time interval (= "generate sensor data"), the program of the microservice in-control template class (BaseIms Class) 2010 automatically calculates the data acquisition time according to the specific time interval. Then, the automatically calculated times are sequentially specified in the argument "time" of "HashMap createSensorData(long time)".

[0221] The sensor data (contents of recognizable entities / states / information 7) accumulated in a chronological order is then "sent" from the device control template class (Base Device Class) 2002 to the microservice internal control template class (BaseIms Class) 2010 using "long sendSensor(Map map)".

[0222] As an example, Fig. 16 shows a list of control classes of the device control template class (Base Device Class) 2002 corresponding to a sensor device 802. However, in addition to this, as shown in Fig. 7, it is necessary to control various robots (or drive mechanisms) included in a mobile terminal device 808, an HMD device 810, and an IoT device 820. Therefore, in addition to the contents shown in Fig. 16, the process overview 2304 may include, for example, "execute drive" and a previously specified "display screen / image".

[0223] Furthermore, when various devices 8 are placed outside the edge computer 6 or the cloud server 2, communication control between the devices 8 and the corresponding microservices 80 is required. Although the explanation of the source code of the control method shown in FIG. 16 is omitted here, the control required for communication control may also be described in the source code. For example, by incorporating 76 the "Socketlmpl Class" of the "java (registered trademark).net" package, communication control for the device 8 can be performed at the IP address level. Specifically, import java.net.Socketlmpl; is written before the device control template class (Base Device Class). This makes it possible to use various methods of the Socketlmpl Class that perform basic communication control.

[0224] Using HashMap setDeviceData(HashMap map, long time), which is a function corresponding to void setSendDataFormat(String format) in Fig. 15, sensor data is set from the microservice control template class (BaseIms Class) 2010 or the individual device-compatible microservice control class (Customims Class) 2110. The argument map used at this time sets the type of sensor data, such as "Should video be collected?" or "Should still images be collected?". The argument time indicates the time to capture (obtain) a still image.

[0225] If you want to collect video, first issue the Stream createSensorData() command to have the corresponding sensor 802 or infrared camera 806 collect video. If you want to stop collection by the sensor 802 or infrared camera 806 midway through, issue the void removeCreateSensorData() API command to stop (cancel) sensor data generation. If you then issue Stream sendSensor(Stream movie), the collected video will be available for use in other programs (methods) of the microservice control template class (BaseIms Class) 2010 or the individual device compatible microservice control class (Customims Class) 2110.

[0226] Also, if you want to collect still images, first issue the command Stream createSensorData(long time) to have the corresponding sensor 802 or infrared camera 806 collect still images. If you then issue Stream sendSensor(Stream image, long time), the collected still images can be used in other programs (methods) of the microservice control template class (BaseIms Class) 2010 or the individual device compatible microservice control class (Customims Class) 2110.

[0227] The state transition of the device 8 in this embodiment will be described with reference to Fig. 17. Five states of the device 8 are defined: a stopped state 2400, an operating state 2402, an executing state 2408, a ready state 2410, and an error state 2414.

[0228] Immediately after control of the device 8 starts 2300, the device is in a stopped state 2400. Then, when "createDevice()" (see number 14 in FIG. 15), which means starting device control by the microservice 80, is executed 2502, the device 8 moves to an operating state 2402. If the response 2506 of "createDevice()" fails 2510, the device enters an error occurrence state 2414, and returns to the stopped state 2400 when the ifLink application is disconnected 2500. Even when the device 8 is in an operating state 2402, if the ifLink application is disconnected 2500, the device 8 returns to the stopped state 2400.

[0229] On the other hand, if the response 2506 to "createDevice()" is successful 2512, the device 8 enters a ready state 2410. Then, when an operation instruction 2508 arrives from the application (IF-THEN) engine 90, the device starts operating and enters an executing state 2408. If a stop instruction 2504 arrives from the application (IF-THEN) engine 90, the device operation stops and the device 8 returns to the ready state 2410. Furthermore, regardless of whether the device 8 is in the ready state 2410 or the executing state 2408, if the ifLink application is disconnected, the device 8 enters a stopped state 2400.

[0230] If the device operation fails 2530 while the device 8 is running 2408, a retry 2530 to start the device operation is required, and the device 8 enters an error occurrence state 2414. Similarly, if the time period during which there is no response from the device 8 while the device 8 is running 2408 becomes long, a timeout 2520 occurs and the device 8 enters an error occurrence state 2414.

[0231] The module 9 that realizes a specified function is separated into a device 8 (or a human-recognizable entity / state / information 7) and a microservice 80 (or control means 4) that controls it. This allows the state of the device 8 and the state of the microservice 80 to be managed separately. Meanwhile, while a service is being performed for the user, the user may "change the contents of the application (IF-THEN) rules 70" (i.e., the device 8 operated from the application (IF-THEN) rules 70 may suddenly change). Managing the state of the device 8 and the state of the microservice 80 separately in this way has the effect of making it easier to make real-time changes to the "changes to the application (IF-THEN) rules 70" made by the user.

[0232] As shown in Fig. 18, six states can be defined as the state of the microservice 80 in this embodiment: a quiescent state 2400, connecting state 2412, operating state 2402, device control 2404, starting 2300, and error occurred 2414. Then, as shown in Fig. 18, a relationship is established between the state of the microservice 80 and the state of the device 8. Figure 19 shows the relationship between the source temperature and the spectral radiance of the radiation emitted from a blackbody radiation source. It is known that any object with a certain temperature radiates electromagnetic waves (blackbody radiation) according to its temperature. Therefore, by measuring the spectral characteristics of the blackbody radiation emitted from it, the temperature of the blackbody radiation source can be determined.

[0233] It can be seen from the spectroscopic characteristics diagram in Figure 19 that when only electromagnetic waves of a specific wavelength 14 are extracted, the radiance changes with the temperature of the blackbody radiation light source. In thermography technology, only electromagnetic waves of a specific wavelength 14 are extracted from the radiant light emitted from the object to be measured, and the temperature of the object to be measured is measured from the intensity of the extracted electromagnetic waves.

[0234] When using this thermography technology to measure a person's scalp surface temperature, the blackbody radiation light is scattered by hair along the optical path. Therefore, the accuracy of measuring scalp surface temperature in areas of the head where there is a lot of hair (such as the back of the head) is significantly reduced (a lower temperature is measured than the actual temperature due to the light scattering loss caused by hair). Therefore, when using thermography technology to measure a person's scalp surface temperature, it is desirable to measure on the face where there is less hair.

[0235] Furthermore, the intensity of the electromagnetic waves of the predetermined wavelength 14 detected by the infrared camera 806 is inversely proportional to the square of the distance from the object to be measured to the infrared camera 806. Therefore, when measuring the surface temperature of the object to be measured using thermography technology, the detected temperature decreases as the distance from the object to be measured to the infrared camera 806 increases.

[0236] When using thermography technology to accurately measure the body temperature of a person's head, it is important to take into consideration the above-mentioned 1) measurement of the face (the front of the head, where the scalp is exposed) and 2) the relationship between the object to be measured and the infrared camera 806, as well as 3) the relationship between water absorption (the effects of sweat and humidity on the face surface) and the wavelength used for temperature measurement, and 4) removing the effects of dark current, which has a temperature dependency of the infrared camera 806 itself.

[0237] As already explained, it is desirable to use light in the wavelength range of 2.5 μm to 20 μm for measuring human body temperature, based on the characteristics shown in Figure 19. However, with light in this range, large water absorption peaks are observed near central wavelengths of 2.955 μm and 6.100 μm. Furthermore, in the wavelength range longer than 6.9 μm, the amount of water absorption tends to increase as the wavelength increases.

[0238] Here is a concrete example of the amount of light absorbed by actual water molecules. When light with a wavelength of 2.955 μm passes through a water film that is 0.3 μm thick, 80% of the amount of light is attenuated (i.e. only 20% of the incident light is transmitted). Many people have probably experienced sweat breaking into droplets and dripping down when they sweat profusely. If we consider that the diameter of such droplets is about 1 to 2 mm, we can see how large the amount of light absorbed mentioned above is.

[0239] On the other hand, the wavelength-dependent characteristic of the amount of light absorbed by water molecules reaches its minimum value near wavelengths of 3.75 μm and 5.30 μm. Therefore, the value of the predetermined wavelength 14 used in thermography is preferably near 3.75 μm or near 5.30 μm. Taking into account the light absorption characteristics of water molecules, it is desirable to use light with a wavelength range of 3.3 μm or more and 5.7 μm or less for measuring human body temperature. If light with wavelengths in this range is used for the predetermined wavelength 14 in thermography, body temperature can be measured with relatively little effect from sweat and humidity in the air.

[0240] If more accurate body temperature measurement is required, the detected temperature may be measured using two different wavelengths of light. By combining the detected temperature characteristics using these two wavelengths of light with the wavelength-dependent characteristics of water molecule light absorption described above, the molecular weight of water absorbed during the infrared light path can be estimated. By correcting the detected temperature using this estimated value of the molecular weight of water, an accurate body temperature can be calculated.

[0241] It turns out that the dark current (the pseudo detection current that flows when the blackbody radiation is blocked) of the infrared camera 806 is unexpectedly large. Therefore, to accurately measure a person's body temperature, it is necessary to subtract the value of the dark current from the detection current value. Here, the value of this dark current changes depending on the temperature inside the infrared camera 806.

[0242] When measuring the dark current value of the infrared camera 806, it is not preferable to block the entrance of the infrared camera 806. Blackbody radiation from a light-shielding member at room temperature enters the infrared camera 806, making it impossible to measure the dark current value. It is preferable to point the infrared camera 806 at a sufficiently distant object and measure the dark current value frequently between measuring the human body temperature. Since the brightness of blackbody radiation is inversely proportional to the square of the distance between the infrared camera 806 and a sufficiently distant object, almost no blackbody radiation from a sufficiently distant object enters the infrared camera 806. In this state, the amount of signal (current value) obtained from the infrared camera 806 is approximately equal to the value of the dark current.

[0243] 20 shows the internal structure of a triple-lens infrared camera, which is an application example of this embodiment. Here, "triple" refers to a total of "three lenses," consisting of one lens 32 (single lens) that focuses infrared light 38 on infrared light sensor 34, and two lenses 42, 44 (two lenses) that focus visible light 40 on visible light sensors 46, 48.

[0244] In an infrared camera 806 incorporating an infrared light sensor 34 having infrared light 38, a bandpass optical filter 24 is disposed immediately before a lens 32 that collects the infrared light 38. As described above, this bandpass optical filter 24 is designed to pass only light in the wavelength range of 3.3 μm or more and 5.7 μm or less (or light with a wavelength near 3.75 μm or light with a wavelength near 5.30 μm).

[0245] A detection signal (first signal) obtained from the infrared light sensor 34 is transferred to a temperature distribution video / image generating unit 36. Then, the temperature distribution video / image generating unit 36 ​​generates a spatial detected temperature distribution characteristic.

[0246] 20 has a built-in function for correcting the drop in detected temperature caused by the above-mentioned reason (change between the measurement object 20 and the infrared camera 806). That is, the L to the measurement object (person being measured) is measured using two of the three lenses (lenses 42, 44) that correspond to visible light 40, and the detected temperature is corrected according to the measured L.

[0247] That is, in the triple-lens infrared camera 806, the two lenses 42, 44 that transmit visible light 40 are arranged at positions separated from each other. Therefore, the imaging positions for the measurement object 20 are shifted between the visible light sensors (imaging elements) 46 and 48. By detecting the amount of this shift, the distance L from the measurement object 20 to the triple-lens infrared camera 806 can be calculated based on "trigonometry."

[0248] Specifically, detection signals (second signals) obtained from the visible light sensors (imaging elements) 46, 48 are used to generate imaging patterns for the measurement target 20 in the visible video / visible image generating units 52, 54. Then, the distribution calculating unit 62 for each measurement target examines the amount of deviation between the imaging patterns of both, and calculates L from the measurement target 20 to the triple-lens infrared camera 806.

[0249] As described above, the luminance of the blackbody radiation emitted from the measurement object 20 is inversely proportional to the square of L. Therefore, the temperature distribution correction unit 60 uses the value of L obtained here to correct the spatial distribution characteristics of the detected temperature and calculates the actual body temperature.

[0250] Only the "corrected temperature distribution" can be obtained from the temperature distribution correction unit 60. It is difficult to identify the measurement object 20 from the temperature distribution characteristics alone. For this reason, the synthesis unit 68 superimposes an image of the contour, eyes, nose, mouth, and other shapes of the measurement object 20 obtained by capturing an image using visible light onto the "corrected temperature distribution" described above. This synthesized image is then output to the temperature distribution output terminal 58.

[0251] Here, the shape images of the contour, eyes, nose, mouth, etc., which are superimposed on the "corrected temperature distribution", are obtained from the visible video / visible image contour shape extraction unit (including eye contour) 66. In particular, the center position of this shape image of the contour, eyes, nose, mouth, etc. must match the center position of the "corrected temperature distribution". As can be seen from FIG. 20, there is a misalignment between the position of the lens 32 of the infrared camera 806 and the positions of the lenses 42, 44 of the normal cameras 816-1, 2. Therefore, it is necessary to correct the above-mentioned center position misalignment.

[0252] After synthesis, a visible light planar video / image generator 64 generates a visible image in which the central position shift has been corrected. To do this, the images obtained by the visible video / visible image generators 52, 54 and information L obtained from the distribution calculator 62 for each measurement object are used. The visible image obtained here is then output from a visible video / visible image output terminal 56.

[0253] The correspondence between Fig. 20 and Fig. 2 described above will be described below. The normal camera 816 in Fig. 2 corresponds to the normal cameras 816-1 and 816-2 in Fig. 20, and Fig. 2 and Fig. 20 have the same infrared camera 806. Also, the visible video / visible image generating units 52 and 54 to the distribution calculating unit 62 for each measurement object and the synthesized visible light planar video / image generating unit 64 in Fig. 20 correspond to a part of the normal camera image recognizing unit 112 in Fig. 2. And the temperature distribution video / image generating unit 36 ​​to the temperature distribution correcting unit 60 and synthesizing unit (temperature distribution also displaying the contour of visible light) 68 in Fig. 20 correspond to the infrared image recognizing unit 116 in Fig. 2.

[0254] Also, the triple-lens infrared camera 806 itself shown in Fig. 20 corresponds to the infrared camera 806 in Fig. 7. And the person's head temperature determination unit 125 in Fig. 2 corresponds to a part of the infrared camera microservice 80C in Fig. 7. Furthermore, the detection time operation setting unit 140 in Fig. 2 corresponds to the mailer microservice 80H, HMD microservice 80E, individual device microservice 80F, etc. in Fig. 7.

[0255] In the normal camera image recognition unit 112 in Fig. 2, it has been explained that advanced "image recognition" is performed, from "1) image analysis" to "2) person detection", "3) number of people detection", "4) head detection", and "5) detection. And it has been explained that the advanced "image recognition" is executed by the image recognition class (AI Analyze Class) 2018 (Fig. 13) using AI of the sensor microservice 80A.

[0256] Figure 21 shows an example of cooperation between classes in the class structure (software architecture) of the sensor microservice 80A described in Figure 13. First, import (embedding) processes 74, 76 of the Custom Device Class and the AI ​​Analyze Class are performed in the Customims Class. This makes it possible to use the methods defined (described) in the Custom Device Class and the AI ​​Analyze Class in the Customims Class.

[0257] Then, in step 44 (S44), a send data format is specified by the API command void setSendDataFormat (String format) issued from the application (IF-THEN) engine 90. If the above argument format includes a "request for the number of people obtained after recognizing (analyzing) the image signal from the normal camera 816", image recognition processing using a method of the AI ​​Analyze Class is started.

[0258] Before starting the image recognition process, it is necessary to obtain an image signal (corresponding to a video signal or a still image signal, which corresponds to the second signal) from the normal camera 816. For this purpose, the HashMap SetDeviceData() is called in the CustomDevice Class. Then, the corresponding method previously written in the Custom Device Class runs (S45), and the acquisition of the image signal using the normal camera 816 begins.

[0259] In step 46 (S46), an image signal is obtained using the normal camera 816. Next, when Stream sendSensor() is called in the Customims Class, the image signal obtained by the normal camera 816 is read into the Customims Class. The image signal read here may be stored in the buffer memory 2008 (FIG. 13) (S48).

[0260] A corresponding method previously written in the AI ​​Analyze Class is called by the Customims Class (S49), and the image signal stored in the buffer memory 2008 is read by the image recognition processing side (S50). Similarly, a corresponding method previously written in the AI ​​Analyze Class is called by the Customims Class to issue an instruction to calculate the person's head position and an instruction to calculate the number of people obtained as a result (S51). In response to this, the corresponding method previously written in the AI ​​Analyze Class operates and data analysis (image recognition) S52 is performed. Then, when the execution of the corresponding method is completed, the number of people obtained as a result of the data analysis (image recognition) is known (S53).

[0261] In step S54, a process is performed to determine whether or not it is necessary to obtain the head temperature depending on the calculated number of people. Note that up to this point, the process flow of the sensor microservice 80A that controls the normal camera 816 alone has been described.

[0262] The head temperature extraction process in step 55 (S55) and step 56 (S56) is performed as a collaborative process between the sensor microservice 80A and the infrared camera microservice 80C, or as a process in the infrared camera microservice 80C using the image recognition results in the sensor microservice 80A.

[0263] Fig. 22 shows an example of a method for setting application (IF-THEN) rules when notifying multiple media simultaneously when a person with a high fever is detected. In Fig. 22, not only various videos and various still images 128 obtained from a triple-lens infrared camera 806, but also location information 138 obtained from a GPS location sensor 830 is used as a detection signal.

[0264] Here, the setting of the notification destination (medium to be used for notification) when a person with a high fever is detected is specified (described) in the application (IF-THEN) rules 70 stored in the memory area 71 of the application (IF-THEN) engine 90.

[0265] In the example of Fig. 22, notification is sent via both email communication / email transmission 1250 and warning screen display 1230. What is important here is not only that "a person with a high fever has been detected," but also that the user who receives the notification must be able to "identify the detected person with a high fever." In the example of warning screen display 1230 in Fig. 6, "the person on the far right is displayed with a diagonal line," making him or her identifiable. In this way, "visualization using an image" is performed when notifying the user, which has the effect of allowing the user who receives the notification to easily identify the person with a high fever (the person who should be noted).

[0266] For "visualization" at the time of this notification, a high fever specific image 136 may be sent to a screen display microservice 80K that controls a warning screen display 1230. Also, a high fever face image 132 may be sent to a mailer microservice 80H that controls email communication / email transmission 1250. Here, the user who receives the email is likely to be located away from the site. Therefore, high fever person location information 134 obtained from location information 138 acquired from a GPS location sensor 830 may be sent to the mailer microservice 80H.

[0267] What is particularly important here is that "information 132-136 to be sent to microservices 80H and 80K on the notification side (i.e., associated with execution (THEN) 78 in Fig. 11) is created and sent by microservices 80C and 80J on the sensor side (i.e., associated with condition (IF) 72 in Fig. 11)." If the information required for user notification (i.e., execution (THEN) 78 in Fig. 11) is created by microservices 80C and 80J on the sensor side (i.e., associated with condition (IF) 72 in Fig. 11), this has the effect of greatly improving the processing efficiency inside the high fever detection device (or the entire high fever detection system).

[0268] An example of notification information when a person with a high fever is detected is composed of a "warning message" and a "display image that makes it easy to identify a person with a high fever," as shown on display screen 12 in Figure 6. Furthermore, the content of the above-mentioned "warning message" can often be formatted into a fixed form. Therefore, standard templates (fixed formats) can be used frequently as a method of "providing information to users" in the processing performed by the execution part (THEN) 78 (Figure 11) of the application (IF-THEN) rule 70. Therefore, in the microservice 80 related to "providing information to users," the processing of "creating a standard format using an existing template" can be implemented.

[0269] 23 shows the class configuration (software architecture) of information provision-related microservices. A basic control package 2000 related to information provision includes a standard format generation class (FillFormat Class) 2019 that uses an existing template. Here, a set of a "standard text" (such as the above-mentioned "warning text") related to the information to be notified to the user and a "display image" (such as the above-mentioned "display image for easily identifying people with a high fever") is generated.

[0270] The creation format 77 created (by a specified method) in the standard format generation class (FillFormat Class) 2019 using this existing template is temporarily stored in a buffer memory for storing created formats. Then, this temporarily stored creation format 77 is displayed on the screen 1230 to the user via the device control template class (Base Device Class) 2002 or the individual device control class (Custom Device Class) 2101.

[0271] Fig. 24 explains a method of communicating information to a user using the microservices related to information provision (automatic notification) shown in Fig. 23. Using Fig. 22, it was explained that "information 132-136 to be sent to microservices 80H and 80K on the notification side (i.e., related to execution (THEN) 78 in Fig. 11) is created and sent by microservices 80C and 80J on the sensor side (i.e., related to condition (IF) 72 in Fig. 11)." Here, the specific processing flow is explained.

[0272] Steps 24 (S24) to 29 (S29) in Fig. 24 show the process of having the sensor-side microservice 80C (i.e., associated with condition (IF) 72 in Fig. 11) create and receive the necessary information. Steps 30 (S30) to 31 (S31) send information using the information created by the sensor-side microservice 80C to the notifying-side microservices 4D, 80H, and 80K (i.e., associated with execution (THEN) 78 in Fig. 11) to perform the notification process.

[0273] In this embodiment, notification to the user is required only after a person with a high fever is detected. Therefore, first, the microservice 80C on the sensor side (i.e., associated with the condition (IF) 72 in FIG. 11) is made to respond to whether the condition (IF) 72 is met, i.e., whether a person with a high fever has been detected (S27). Then, if a person with a high fever is detected, the microservice 80C on the sensor side (i.e., associated with the condition (IF) 72 in FIG. 11) is made to create / send (S29) information 128 required to be sent to the microservices 4D, 80H, and 80K on the notification side (i.e., associated with the execution (THEN) 78 in FIG. 11).

[0274] Here, in step 24 (S24) and step 28 (S28), the application (IF-THEN) engine 90 issues an API command of void setSendDataFormat (String format) (Fig. 15) to the sensor-side microservice 80C. This instructs the sensor-side microservice 80C to switch the information content to be created / sent.

[0275] Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment, its modifications, and combinations of the embodiments are also included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0276] 2···Cloud server, 6···Edge computer, 70···Application (IF-THEN) rules, 80···Microservices, 90···Application (IF-THEN) engine, 92···ifLink app.

Claims

1. An edge computer capable of communicating with a plurality of devices including IoT devices and a cloud server, an application engine for executing a plurality of application rules pre-fetched from the cloud server; a plurality of microservices configured to apply the application rules to each of the plurality of devices and the cloud server; The application rule has a basic rule of execution (THEN) for a condition (IF), The application engine includes: When a first application rule is used, a first API command as the condition is provided to a first microservice to obtain a return value thereof, and a second API command as the execution is provided to a second microservice different from the first microservice based on the return value. An edge computer characterized by:

2. 2. The edge computer according to claim 1, wherein the first microservice is connected to a buffer memory for temporarily storing data, and further includes an AI class that performs analysis processing of video data connected to the buffer memory, and sends a result of the data analysis processing to the application engine to reduce the processing load of the application rule.

3. 2. The edge computer according to claim 1, wherein the first application rule is a rule that has been verified by the cloud server and that the application engine has received in advance via a third microservice.

4. The edge computer according to claim 1, characterized in that it is a portable computer having a telephone function.

5. An edge computer control system capable of communicating with a plurality of devices including IoT devices and a cloud server, an application engine for executing a plurality of application rules pre-fetched from the cloud server; a plurality of microservices configured to apply the application rules to each of the plurality of devices and the cloud server; The application rule has a basic rule of execution (THEN) for a condition (IF), The application engine includes: An edge computer control system configured to, when using a first application rule, provide a first API command as the condition to a first microservice to obtain its return value, and provide a second API command as the execution to a second microservice different from the first microservice based on the return value, and characterized in that it links multiple microservices.

Images