Versatile edge computers and edge computer control systems

Abstract

To provide an edge computer and control system thereof that can bring out the diversity of data processing.SOLUTION: In this embodiment, an edge computer that can communicate with multiple devices including portable terminals and IoT devices and a cloud server is realized. The edge computer is equipped with an application engine for executing multiple application rules previously imported from the cloud server, each of the multiple devices, and multiple microservices provided for applying the individual application rules corresponding to the cloud server, and can operate the internal application rules, the external multiple devices, and the cloud server in cooperation with each other.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] Embodiments of the present invention relate to an edge computer with diversity and an edge computer control system.

Background Art

[0002] A computer system can be equipped with various programs and have diversity. However, the various programs are individually switched and used, and after the switch, data processing ends within the processing range of the switched program.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] According to one embodiment, an edge computer capable of communicating with a plurality of devices including IoT devices and a cloud server, an application engine for executing a plurality of application rules pre-imported from the cloud server, and a plurality of microservices provided for applying the application rules to each of the plurality of devices and the cloud server, where the application rule has an execution (THEN) as a basic rule for a condition (IF), and the application engine, when using a first application rule, gives a first API command as the condition to a first microservice to obtain its return value return value, and based on the return value, gives a second API command as the execution to a second microservice different from the first microservice picture, Furthermore, the application engine implements multiple functions, The aforementioned execution (THEN) can be toggled on or off with a rule enable / disable function, An execution time control function that can shift the execution timing of a specific execution (THEN), Prepare Ruko Edge computing characterized by the following will be provided. [Means for solving the problem]

[0005] According to one embodiment, an edge computer capable of communicating with multiple devices, including IoT devices, and a cloud server, An application engine for executing multiple application rules that have been pre-loaded from the aforementioned cloud server, To apply the application rules to each of the aforementioned multiple devices and the aforementioned cloud server They are individually installed and operate independently of each other. It has multiple microservices, The aforementioned application rule is based on the principle of executing (THEN) in response to a condition (IF). The aforementioned application engine is When using the first application rule, Corresponding to the first device among the plurality of devices The first microservice is given a first API command as the condition and its return value is obtained, and based on the return value, the second API command as the execution is executed on a second device different from the first device. Alternatively, by providing it to a second microservice corresponding to the cloud server, the first microservice and the second microservice are made to work together. Furthermore, the application engine implements multiple functions, The aforementioned execution (THEN) can be toggled on or off with a rule enable / disable function, An edge computer is provided that is equipped with an execution time control function that can shift the execution timing of a specific execution (THEN). [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 is an explanatory diagram illustrating the basic concept of this embodiment. [Figure 2] Figure 2 is an explanatory diagram showing the basic system in this embodiment. [Figure 3] Figure 3 is an explanatory diagram showing the problems when using the prior art without combining image recognition technology. [Figure 4] Figure 4 shows an explanatory diagram of the effect when only the head temperature of a person is used for determination. [Figure 5] Figure 8 is an explanatory diagram showing the operation flow executed in the basic system shown in Figure 2. [Figure 6] Figure 6 is an explanatory diagram explaining an example usage scenario of this embodiment. [Figure 7] Figure 7 is an explanatory diagram showing another example of this embodiment [Figure 8] Figure 8 is an explanatory diagram showing the program configuration used in an Android-based smartphone. [Figure 9] Figure 9 is an explanatory diagram showing the configuration of an edge computer and its relationship with a cloud server. [Figure 10] Figure 10 is a table showing the function list of an application (IF-THEN) engine. [Figure 11] Figure 11 is a block diagram representing the application example concept in this embodiment. [Figure 12] Figure 12 is an explanatory diagram explaining the effect when a microservice is described in an OS-independent programming language. [Figure 13] Figure 13 is an explanatory diagram showing the class configuration of a microservice. [Figure 14] Figure 14 is an explanatory diagram regarding the functions of a microservice set in an individual device control package. [Figure 15] Figure 15 is an explanatory diagram showing the list of APIs provided by a microservice internal control template class (BaseIms Class). [Figure 16] Figure 16 is an explanatory diagram showing the list of APIs provided by a device control template class (Base Device Class). [Figure 17] Figure 17 is a state transition diagram of a device. [Figure 18]FIG. 18 is an explanatory diagram showing the relationship between the device state and the state of the microservices. [Figure 19] FIG. 19 is an explanatory diagram showing the relationship between the temperature of the blackbody radiation source and the spectral radiance characteristics. [Figure 20] FIG. 20 shows an explanatory diagram of the internal structure of a three-eye infrared camera. [Figure 21] FIG. 21 explains the inter-class cooperation operation within the class configuration shown in FIG. 13. [Figure 22] FIG. 22 is an explanatory diagram showing a setting method for notifying a plurality of media when detecting a person with a high fever. [Figure 23] FIG. 23 is an explanatory diagram showing the class configuration of microservices related to information provision. [Figure 24] FIG. 24 is an explanatory diagram showing the processing steps until information is notified to the user.

MODE FOR CARRYING OUT THE INVENTION

[0007] Using FIG. 1, the basic concept of this embodiment will be described. As shown in FIG. 1(a), assume a place where a plurality of people gather. If there is a person with a high fever whose body temperature is above a predetermined value (for example, 37.5 degrees Celsius) among them, that person with a high fever is automatically detected individually. In the display example of FIG. 1(b), the person indicated by the slanted lines corresponds to a person with a high fever of 37.5 degrees Celsius or more. And that information is automatically notified (FIG. 1(c)). There are a plurality of options as the notification method, and in this embodiment, the options can be selected in a very simple way. Here, regarding the notification method, a plurality of methods may be selected simultaneously.

[0008] In FIG. 1(a), since the number of people gathered at the gathering place is relatively small, each person is displayed separately. However, when the population density of the people gathering increases, partial overlaps occur. In the conventional thermography technology for measuring the surface temperature distribution of the measurement object, individual identification at a place where a plurality of people overlap each other cannot be performed. Therefore, when only the conventional thermography technology is used, only individual detection (individual extraction from a crowded place) of a person with a high fever in a crowded place cannot be performed.

[0009] In this embodiment, thermography technology using an infrared camera is combined with image recognition technology to enable individual detection (individual extraction) of only those with high fever. Specifically, in this embodiment, a program (image recognition class AI Analyze Class) 2018 with image recognition functionality using AI technology is embedded 76 into a part (or inside) of the microservice 80C which serves as the control means 4 for the infrared camera 806. Details will be described later using Figures 11 and 13.

[0010] To mitigate the risk of viral infection in places where people gather and engage in activities, there is a need for technology that can easily detect and notify individuals with high fevers who are suspected of being infected. In this embodiment, a commercially available infrared camera 806 is attached to an existing edge computer 6 such as a smartphone or tablet, and the system can be used simply by installing software (service provision application program) 92 called the ifLink® app, which will be described in detail later.

[0011] As described above, a commercially available infrared camera 806 that can be connected to an edge computer 6 such as a commercially available smartphone or tablet may be used as the infrared camera 806. Using this infrared camera 806 makes it very easy to connect to an edge computer 6 such as a commercially available smartphone or tablet, so the high fever detection system in this embodiment can be easily constructed.

[0012] Furthermore, the ifLink app 92 mentioned above operates as an application on smartphones and tablets, which many people already own. As a result, it can effectively reduce the risk of infection at each activity base by utilizing existing smartphone and tablet infrastructure (usage environment) and simply by installing the ifLink app 92.

[0013] In this embodiment, image recognition is combined with video or still images captured by the infrared camera 806 to individually detect (individually extract) only those with high fevers from a group, and this information is automatically notified.

[0014] Methods for detecting the surface temperature of an object using thermographic technology with an infrared camera 806 have existed for some time. However, as mentioned above, when people are densely packed together and multiple people overlap, it was previously necessary for the observer of the video / still image to visually confirm and separate (individually extract) each individual. Furthermore, conventional thermographic technology alone cannot distinguish between a person and other objects, requiring an object identifyer. For this reason, conventional thermographic technology could only be used in places such as reception areas of companies or facilities that could deploy dedicated monitoring personnel. In this embodiment, by incorporating an AI-based image recognition class (AI Analyze Class) 2018 76, the computer (edge ​​computer 6) recognizes individuals with high fevers from images (video or still images) obtained from an infrared camera. Since this recognition is performed automatically within the edge computer 6, individuals with high fevers can be detected and notified even without a monitoring officer present. Therefore, this embodiment has the effect of significantly reducing the cost of monitoring (automatic individual detection of individuals with high fevers). In this embodiment, the method for recognizing individuals with high fever is not limited to using the AI-based image recognition class (AI Analyze Class) 2018; any image recognition method may be employed. For example, proprietary hardware with dedicated image recognition capabilities may be prepared in advance. In this case, images obtained from an infrared camera may be sent to this hardware for image recognition, and the results may be returned.

[0015] In Figure 1(c), when a person with a high fever whose body temperature exceeds a predetermined value (for example, 37.5 degrees Celsius) is detected, the result is automatically notified. As an example of the options for this automatic notification method, an image (video or still image) clearly indicating the presence of a person with a high fever may be displayed on the display screen 12 of an edge computer 6 such as a smartphone or tablet 1210. Alternatively, a warning may be displayed 1220 on a signage (electronic bulletin board) 804 installed near the edge computer 6 that detected the person with a high fever.

[0016] Furthermore, the display format is not limited to specific images; a warning screen with a warning message written in text format may also be displayed on the display screen 12 of the edge computer 6 or on the signage 804 1230. In addition to appealing to the visual sense with predetermined images and text warning messages, notification may also be given by methods that appeal to the auditory sense, such as audio warnings 1240, or to the senses of touch and smell.

[0017] Furthermore, remote notification may be performed using network communication or other means to notify locations far from where high fever individuals are detected. This remote notification method may include email communication (230)1250 or posting information on a web screen (210)1260. The information posted on this web screen 1260 is not limited to individual high fever individuals; aggregated results of information obtained from multiple different locations may also be posted.

[0018] Furthermore, the notification method is not limited to the "information transmission" described above; physical operations involving equipment operation 1290, such as the blocking process 1270 of a passage gate (820), may also be performed.

[0019] You may notify the presence of a person with a high fever using any method other than those described above. Furthermore, you may notify multiple individuals simultaneously using multiple notification methods.

[0020] Conventional monitoring systems have a fixed notification mechanism after detection, lacking flexibility in how they can be used. In contrast, the notification method after detecting a person with a high fever in this embodiment can be flexibly configured to suit the user's equipment, software, etc. These flexible configuration options include illuminating a patrol light (registered trademark), sounding an alarm 1240, using voice synthesis for voice guidance 1280 (voice warning 1240), sending an email 1250, notifying via messaging software such as SNS (Social Networking Service), or displaying a screen 1210, 1230, etc., which can be arbitrarily configured. The notification method used is one that has been set in advance by the user. In this embodiment, as a technology that allows for flexible configuration according to the user's equipment, software, etc., the application (IF-THEN) rule 70 and the application (IF-THEN) engine 90 that utilize it are configured as described later using Figures 7, 11, and 22. In other words, the user can flexibly configure one or more notification methods from the above-described notification method mechanisms on the screen of, for example, a smartphone.

[0021] The basic system (high fever detection system) in this embodiment will be explained using Figure 2. In this basic system (high fever detection system), an infrared camera 806, which corresponds to a first optical sensor having infrared light, and a normal camera 816, which is a second optical sensor including an image sensor having wavelengths other than infrared light, are used in combination.

[0022] Generally, wavelengths ranging from 380 nm to 810 nm are called "visible light," and wavelengths ranging from 810 nm to 1 mm are generally called "infrared light" or "infrared ray." However, as shown in Figure 19, the wavelength range of "infrared ray" suitable for "measuring human body temperature" using blackbody radiation is within the range of 1 μm to 100 μm, and preferably within the range of 2.5 μm to 20 μm. Therefore, the wavelength range of "infrared light" or "infrared ray" used in the infrared camera 806 as the first light sensor here means within the range of 1 μm to 100 μm (preferably within the range of 2.5 μm to 20 μm).

[0023] Therefore, light with wavelengths outside the above range is referred to here as "light with wavelengths other than infrared." In this embodiment, a second signal obtained from a second optical sensor (a normal camera 816) that includes an image sensor having the aforementioned "light with wavelengths other than infrared" is also used to detect a person with a high fever whose body temperature exceeds a predetermined value. Suitable light for this purpose includes ultraviolet light (wavelength 10 nm or more), visible light, near-infrared light (wavelength 2.5 μm or less), and microwaves (wavelength 10 cm or less). Taking the above information into account, the wavelength range of "light with wavelengths other than infrared" is in the range of 10 nm to 2.5 μm or in the range of 20 μm to 10 cm (preferably in the range of 380 nm to 1 μm or in the range of 100 μm to 10 cm).

[0024] The term "typical camera 816" generally refers to a second optical sensor that includes an image sensor having a wavelength range of 380 nm to 810 nm in "visible light". However, the "typical camera 816" shown in Figure 2 refers to an optical sensor that includes an image sensor having a wavelength range of 10 nm to 2.5 μm or 20 μm to 10 cm (preferably 380 nm to 1 μm or 100 μm to 10 cm).

[0025] Many edge computers 6, such as smartphones and tablets, have a standard built-in camera 816. Therefore, the camera that is standardly built into the edge computer 6, such as a smartphone or tablet, can be used as the standard camera 816 shown in Figure 2.

[0026] On the other hand, when using a commercially available infrared camera 806, the first signal (temperature characteristic signal) obtained from the infrared camera 806 (first photosensor having infrared light) may be communicated to the edge computer 6 using wireless communication means or wired communication means. However, as will be described later using Figure 3, the signal may also be directly transferred via the connection unit 10.

[0027] As shown in Figure 2, the application 100 (software program) of this system consists of the functions of a recognition setting unit 110 that allows setting of recognition logic according to the usage scenario, a judgment temperature setting unit 120, a detection rule setting unit 130 that sets application (IF-THEN) rules 70 (described later using Figures 7, 11, and 22), a detection operation setting unit 140, and a detection information transmission unit 150. In the embodiment shown in Figure 2, the application 100 of this system takes the form of software installed on the edge computer 6. However, it is not limited to this, and some of the functions of each component 110 to 150 may be configured as dedicated hardware (electronic circuits).

[0028] Therefore, the edge computer 6 on which the application 100 (software program) of the system described above is installed, or the edge computer 6 incorporating dedicated hardware (electronic circuitry) that executes some of the functions 110 to 150, corresponds to the high-temperature person detection device in this embodiment.

[0029] The first signal (temperature characteristic signal) obtained from the infrared camera 806 (first photosensor having infrared light) is transmitted to the infrared image recognition unit 116 (first image recognition unit) of the recognition setting unit (function) 110. The spatial temperature distribution characteristics are then recognized as an image (first image recognition) by this infrared image recognition unit 116.

[0030] The image signal (corresponding to a video signal or still image signal, and corresponding to the second signal) obtained from the normal camera 816 (a second optical sensor including an image sensor having wavelengths of light other than infrared) is transmitted to the normal camera image recognition unit 112 (second image recognition unit) of the recognition setting unit (function) 110. The second image recognition performed by this normal camera image recognition unit 112 is as follows: 1) Image recognition (image analysis) of the image signal; 2) Determining whether there are people in the screen (person detection); 3) If there are people in the screen, calculating the number of people (people detection); and 4) Only if there are people in the screen, recognizing the position of each person's head (head detection). At the same time, 5) For each person identified in the screen, calculation (detection) between that person and the infrared camera 806 may also be performed.

[0031] Furthermore, the temperature determination setting unit (function) 120 includes a human head temperature determination unit 125, which measures the temperature of a person's head facing the infrared camera 806 as their body temperature. It then detects individuals with high fever whose body temperature exceeds a predetermined value (for example, 37.5 degrees Celsius) (determining the presence or absence of individuals with high fever). If an individual with high fever is detected as a result of this temperature determination, a notification is automatically sent. Specifically, detection information is transmitted to the detection operation instruction unit (application (IF-THEN) engine) 90 of the detection rule setting unit (function) 120. The detection unit operation setting unit (function) 140 then operates and the notification process is executed.

[0032] Figures 3 and 4 will be used to explain "2) Person detection" performed by the standard camera image recognition unit 112. The first signal (temperature characteristic signal) obtained from the infrared camera 806 alone cannot identify the type of heat source radiating from the blackbody (what is radiating the heat from?). Therefore, if the image recognition technology performed by the standard camera image recognition unit 112 is not combined, the temperature of objects other than people will also be determined.

[0033] As an example shown in Figure 3(a), consider a person drinking hot coffee from a coffee cup they are holding. Suppose the surface temperature of the coffee cup is 50°C. The thermographic image (output of the infrared image recognition unit 116) obtained from this scene, as shown in Figure 3(b), detects a surface temperature of 50°C for the coffee cup. If the threshold temperature for determining whether a person has a high fever is 37.5°C, then the coffee cup's surface temperature of 50°C exceeds this threshold. If the type of blackbody radiating heat source (what is the heat being radiated from?) cannot be identified, a false determination that "a person with a high fever is present" will be made (Figure 3(c)).

[0034] Figure 4 shows the effect of using only the temperature of a person's head for determination. Here, Figure 4(a) is the same as Figure 3(a). When using only the temperature of a person's head for determination, the surface temperature of the coffee cup (50°C) is excluded from detection (Figure 4(b)).

[0035] The results of "2) Person detection" performed by the standard camera image recognition unit 112 and the spatial temperature distribution characteristics within the screen output from the infrared image recognition unit 116 are integrated and processed by the person head temperature determination unit 125 (Figure 2). In the person head temperature determination unit 125, as shown in Figure 4(c), "only the temperature of the person's head" is used to "determine the presence or absence of a person with a high fever." Here, the temperature of the person's head is used for measuring body temperature, but it is not limited to this, and any part of the person may be used for measuring body temperature. If body temperature is measured from a place other than the person's head, the name of the "person head temperature determination unit 125" may be changed to "person body temperature determination unit," which has the same function. In this case, "the person body temperature determination unit will detect the presence or absence of a person with a high fever whose body temperature exceeds a predetermined value based on the results of the first image recognition and the results of the second image recognition."

[0036] In this embodiment, a first signal obtained from a first optical sensor (infrared camera 806) that emits infrared light and a second signal obtained from a second optical sensor (conventional camera 816) that includes an image sensor that emits light of wavelengths other than infrared light are used in combination to detect a person with a fever exceeding a predetermined body temperature. As a result, the accuracy of the normal judgment (Figure 4(d)) is greatly improved.

[0037] Next, we will explain "4) Head detection," which is performed by the conventional camera image recognition unit 112 (Figure 2) mentioned above. As described above using Figures 3 and 4, acquiring only the temperature of the head has the effect of eliminating false detections caused by other heat sources. Furthermore, A) the torso and limbs are covered by clothing and not exposed, so the error in the temperature measurement results from there is large, while B) the head is relatively exposed and therefore relatively easy to measure the temperature of. Therefore, acquiring only the temperature of the head also has the effect of improving the accuracy of temperature measurement.

[0038] Furthermore, we will explain the relationship with "3) Person detection," which is normally performed by the camera image recognition unit 112. Images (video or still images) obtained from a place where multiple people are densely packed together have the characteristic that C) parts of the body (torso or limbs) tend to overlap with adjacent people, but the heads are positioned relatively separately. Therefore, in this embodiment, by utilizing the head position in particular, not only can the image recognition processing of "3) Person detection" be simplified, but the accuracy of "3) Person detection" can also be improved.

[0039] This section explains the specific methods for "2) Person Detection" and "3) Number of People Detection" using "4) Head Detection," which is normally performed in the camera image recognition unit 112. Multiple standard samples related to the shape of a person's head and hairstyle are pre-stored in the camera image recognition unit 112. The degree of pattern matching between the image (second signal) obtained from the camera 816 (a second optical sensor including an image sensor having wavelengths of light other than infrared) and the above-mentioned multiple standard samples is calculated. The region where the degree of pattern matching exceeds a predetermined value is estimated to be a "candidate human head."

[0040] In most cases, the torso is located below the "human head," with the limbs (hands and feet) positioned around the torso. "2) Person detection" is performed only when a torso or limbs are detected directly below the estimated "candidate human head." The number of areas on the screen where "2) person detection" is performed becomes the result of "3) person detection."

[0041] Incidentally, if the area detected in "4) Head" is the "back of the head," accurate temperature measurement becomes difficult. Consider the case where the person being measured turns their back. In this case, the blackbody radiation emitted from the epidermis of the head is scattered by the hair in the optical path toward the infrared camera 806. As a result, the amount of blackbody radiation detected by the infrared camera 806 decreases significantly, and the accuracy of temperature measurement decreases significantly (the detection principle of thermography will be explained later using Figure 19). Therefore, it is desirable to detect only the "head facing forward toward the infrared camera 806."

[0042] For the "4) Head Detection" method, only "heads facing the direction of the infrared camera 806" may be used, utilizing the positions of "eyes," "nose," and "mouth" located in front of the head. Specifically, multiple standard samples related to the shapes of eyes, noses, and mouths are pre-stored in the normal camera image recognition unit 112. The image obtained from the normal camera 816 is then divided into fine regions, and the degree of pattern matching between each divided region and the standard samples of eyes, noses, and mouths is calculated. The positional relationships between the obtained "positions of candidate regions for eyes," "positions of candidate regions for noses," and "positions of candidate regions for mouths" are then used to detect "heads facing the direction of the infrared camera 806."

[0043] In the above explanation, "4) Head detection" was performed on a "head facing the direction of the infrared camera 806," and the body temperature of the head was measured by the human head temperature determination unit 125. However, the body temperature may be measured using areas other than the head.

[0044] Furthermore, the method for "2) Person Detection" does not necessarily require the use of the results of "4) Head Detection." In this embodiment, "2) Person Detection" can be performed using a flexible method depending on the usage scenario. For example, since the number of visitors to the reception counter is often only a few people at a time, individual identification of people is relatively easy. However, a standard camera 816 installed behind the reception counter can only capture the upper body of visitors.

[0045] Furthermore, while the standard camera 816 can capture full-body images when photographing a corridor, it also needs to recognize moving individuals. In other words, high speed is required for detecting people with high fevers in corridors. On the other hand, when detecting people with high fevers in a classroom with around 40 students, it is necessary to measure the body temperature of many people simultaneously.

[0046] It is preferable to pre-configure the optimal person detection logic for each different usage scenario, allowing for flexible responses by selecting the optimal logic for each application. As a way to enable this flexible response according to the usage scenario, for example, a selection menu may be provided that allows the user to select the usage scenario, so that the optimal person detection logic is executed. Alternatively, as will be described later using Figure 13, multiple methods (optimal person detection logic for each usage scenario) can be placed within the AI-based image recognition class (AI Analyze Class). Then, the most suitable method for each usage scenario can be selectively called using API commands.

[0047] Next, we will explain "5) Detection of each person identified within the screen," which is performed by the normal camera image recognition unit 112 in Figure 2. With the infrared camera 806, if the distance from the heat source radiating blackbody (a person's head) is great, the temperature tends to be read lower than the actual temperature (this will also be explained later using Figure 19). Therefore, using the result of "5) Detection," the person head temperature determination unit 125 can perform body temperature correction (temperature correction) on the spatial temperature distribution characteristics obtained by the α) infrared image recognition unit 116, or, for people who are a certain distance further from the infrared camera 806, even if the temperature is lower than the determined temperature, the detection operation instruction unit (application (IF-THEN) engine) 90 can perform actions such as issuing a warning or instructions.

[0048] Therefore, using "5) Detection" improves the accuracy of body temperature detection, resulting in improved accuracy of instructions from the detection operation instruction unit 90.

[0049] As a method for "5" detection described above, the individual images obtained from multiple standard cameras 816 may be used to calculate the value using "trigonometry" (a method used in surveying, etc., that calculates values ​​such as sides and angles using the relationship between the sides and angles of a triangle). For example, when using a smartphone or tablet as an edge computer 6, a standard camera 816 is built into the edge computer 6 as standard. Therefore, if individual images obtained from multiple edge computers 6 are shared via communication, the value can be calculated using "trigonometry". Furthermore, if a three-lens infrared camera with two visible light sensors 46 and 48, which will be described later in Figure 20, is used, "5" detection can be performed using this camera alone.

[0050] Alternatively, the maximum and minimum temperature locations can be pinpointed and estimated from the image analyzed by the normal camera image recognition unit 112 in Figure 2, and these can be used. That is, the maximum / minimum temperature location information estimated by the normal camera image recognition unit 112 and the maximum / minimum temperature location information extracted by the infrared image recognition unit 116 can be used to calculate the maximum temperature using "trigonometry".

[0051] As another embodiment, location information using GPS (Global Positioning System) or beacons may be used. For example, consider a case where individual person identification is possible as a result of image recognition by the camera image recognition unit 112. In that case, communication can be performed with the mobile device (smartphone) owned by the identified person, and the location information of the other party can be collected.

[0052] The detection operation setting unit 140 has the function of setting control methods for various notification means that notify of the presence of a person with a high fever whose body temperature exceeds a predetermined value. Specific notification means include options such as alarm sound generation 1240, voice guidance 1280, screen display 1230, email transmission 1250, and device operation 1290 (1270). Among these, the options used during automatic notification (multiple selections are possible simultaneously) are specified in accordance with the application (IF-THEN) rule 70 (described later using Figures 7 and 11) predetermined by the detection operation instruction unit (application (IF-THEN) engine) 90. Then, the necessary information for the notification means (options) specified according to this application (IF-THEN) rule 70 is communicated (transmitted) from the detection operation instruction unit (application (IF-THEN) engine) 90.

[0053] The application 100 of this system has a built-in detection information transmission unit 150. The necessary information is sent from the detection information transmission unit (function) 150 as needed to the cloud server 2, which manages, monitors, and configures the execution of the application 100 of this system.

[0054] The application (IF-THEN) rule 70 used in the detection rule setting unit (function) 140 may be configured from the cloud server 2. In this case, the cloud server 2 is necessary because the execution status of the application (IF-THEN) rule 70 needs to be monitored in multiple locations.

[0055] Furthermore, the application (IF-THEN) rule 70 can also be set locally from the display screen of application 100 of this system. In that case, application 100 of this system can be executed even without cloud server 2.

[0056] Figure 5 shows the operation flow performed within the basic system shown in Figure 2. Here, we will explain the operation flow shown in Figure 5 while referring to Figure 2.

[0057] While the application 100 of this system is running, the periodic processing S01, which repeats the operation shown in Figure 5, continues. In step 02 (S02), first, the image signal (video or still image) captured by the normal camera 816 (second optical sensor) is transmitted as a second signal to the normal camera image recognition unit 112 (second image recognition unit). Then, the normal camera image recognition unit 112 performs "1) image recognition," and then performs "3) person detection" and "head position calculation (part of 4) head detection" and "5) detection" using the result of "4) head detection" (this completes S02). If the result of "3) person detection" is that the number of people is not 1 or more (i.e., no people are included in the image obtained by the normal camera 816) (No. in S03), the process proceeds to the end of step 04 (S04).

[0058] On the other hand, if there is one or more people (i.e., people are included in the image obtained by the normal camera 816) (Yes in S03), the person head temperature determination unit (person body temperature determination unit) 125 acquires temperature information at the head positions of each individual corresponding to the number of people (S05). Prior to this, the infrared image recognition unit 116 (first image recognition unit) calculates the spatial temperature distribution characteristics within the screen using the temperature characteristic signal (first signal) obtained from the infrared camera 860 (first light sensor having infrared light). The person head temperature determination unit (person body temperature determination unit) 125 overlays the head position information of each individual obtained from the normal camera image recognition unit 112 (second image recognition unit) onto these spatial temperature distribution characteristics within the screen to acquire the head temperature (body temperature) of each individual.

[0059] Next, as shown in step 06 (S06), the human head temperature determination unit (human body temperature determination unit) 125 performs a "set temperature determination" to determine whether the head temperature (body temperature) of each individual exceeds a predetermined value (for example, 37.5 degrees Celsius). If, as a result of this set temperature determination, no one's body temperature exceeds the predetermined value, the process proceeds to the end of step 07 (S07).

[0060] In the embodiment shown in Figure 5, two types of judgment values ​​(predetermined values) are set for the "set temperature determination": an "alarm value" (for example, 37.5 degrees Celsius) and a "caution value" (for example, 37 degrees Celsius). Here, the value of the "alarm value" is set to a higher value than the "caution value".

[0061] If the result of the judgment in step 06 (S06) is equal to or greater than the “alarm value”, an alarm is notified as a notification process as shown in step 10 (S10). The specific notification method is determined by the detection operation setting unit 140 selecting an option 1240 to 1290 suitable for “alarm notification” in accordance with the provisions of the application (IF-THEN) rule 70 set in the detection operation instruction unit (application (IF-THEN) engine) 90.

[0062] On the other hand, if the judgment result in step 06 (S06) is below the "alarm value" but above the "caution value", the judgment is made by taking into account the result of "5) detection" mentioned above. As an example of how to use the result of "5) detection", it was explained that "for people who are further away from the β) infrared camera 806 than a certain (set value), even if the temperature is lower than the judgment temperature, the detection operation instruction unit (application (IF-THEN) engine) 90 will issue an instruction to warn them."

[0063] If the distance between the infrared camera 806 and the person being measured increases, the measurement result (body temperature) will be lower than the actual temperature (details will be described later using Figure 19). Therefore, as shown in step 08 (S08), it is determined whether the distance between the head position of the person being measured and the infrared camera 806 is greater than or equal to a predetermined set value. If this distance is closer than the set value (No in S08), it is considered that "the accuracy of the measured body temperature of the person being measured is high." In this case, it is determined that "there are no people with high fever," and the process ends (S09).

[0064] On the other hand, if the response is far from the set value (Yes in S08), it is considered that "the accuracy of the measured body temperature of the subject is low." In other words, it is assumed that "although the subject actually had a high fever and their body temperature exceeded the "alarm value" (37.5 degrees Celsius), the correct body temperature could not be measured due to the poor accuracy of the thermography measurement." Then, as shown in step 11 (S11), a "caution" notification process is performed.

[0065] In this case as well, the detection operation setting unit 140 operates the options 1240 to 1290 suitable for “warning notification” in accordance with the provisions of the application (IF-THEN) rule 70 set by the detection operation instruction unit (application (IF-THEN) engine) 90.

[0066] Figure 6 shows an example of a use case for the edge computer 6 shown in Figure 2, or a high fever detection system including it, which is compatible with the high fever detection device. The high fever detection system shown in Figure 6 consists of a three-lens infrared camera 806 with a structure described later in Figure 20, and an edge computer 6 (high fever detection device) such as a smartphone. The two are connected via a connection part 10.

[0067] Here, many people use their smartphones while riding in vehicles (such as trains) or in places where people gather (such as restaurants). The upper part of Figure 6 shows a situation where there is a group of people nearby inside a vehicle or in a place where people gather. The lower part of Figure 6 shows a situation where other application programs 22 are being used on a smartphone (edge ​​computer 6). If a person with a high fever is detected in the nearby group, the detection result is displayed on the display screen 12 1230.

[0068] As an example of the warning screen display 1230 controlled by the detection operation setting unit 140, the detected person with a high fever is clearly indicated with diagonal lines. At the same time, the warning message, "Warning: There is a person with a high fever nearby!! Please be careful!" is also displayed.

[0069] The method shown in Figure 6 automatically notifies the user when using other application programs 22, significantly improving the immediacy of notifications and user convenience.

[0070] In the example use case of the high fever detection system shown in Figure 6, the warning screen 1230 is displayed on the display screen 12 of the edge computer 6. However, it is not limited to this; for example, the warning screen 1230 may also be displayed on the HMD (Head Mounted Display) 810, which will be described later in Figure 17.

[0071] Another embodiment of this design will be described using Figure 7. In the system configuration shown in Figure 7, the cloud server 2 and the edge computer 6 are configured to communicate with each other. The edge computer 6 can also control various devices 8 such as sensors 802, signage (electronic bulletin boards) 804, infrared cameras 806, mobile terminals 808 such as smartphones and tablets, HMDs (Head Mounted Displays) 810, and IoT devices (gates, etc.) 820 to provide various services to users. If these various devices are, for example, temperature sensors, pressure sensors, gyroscopes, or conventional cameras 816, they may be built into the edge computer 6. However, some of these various devices may be located outside the edge computer 6 and controlled by communication.

[0072] The edge computer 6 has an application (IF-THEN) engine 90 that performs processing (controls various devices 8) according to pre-recorded (pre-installed from the cloud server 2) application (IF-THEN) rules 70, and provides services to the user.

[0073] As shown in Figure 7, microservices 80A to 80F that individually control various devices 8 such as sensors 802, signage (electronic bulletin boards) 804, infrared cameras 806, mobile terminals 808 such as smartphones and tablets, HMDs (head-mounted displays) 810, and IoT devices (gates, etc.) 820 are pre-installed on the edge computer 6 (pre-installed from the cloud server 2). The application (IF-THEN) engine 90 then operates these microservices 80A to 80F to control the coordination between the various devices 8. At this time, API (application program interface) commands are issued from the application (IF-THEN) engine 90 to the microservices 80A to 80F that it wants to operate.

[0074] Furthermore, in this embodiment, the edge computer 6 may also possess (pre-installed on the edge computer 6) an IoT server microservice 80G that cooperates with and controls the IoT service server 220, a mailer microservice 80H that cooperates with the mail server 230, and a Web microservice 80I for accessing the Web service server 210 to obtain Web services. As a concrete example of control by this Web microservice 80I, it may perform actions such as automatically writing necessary information into the frame of a form element specified in HTML (Hyper Text Mark-Up Language) or controlling transitions between Web pages. By placing the Web microservice 80I area within the software architecture of the edge computer 6 in this way, users can enjoy advanced services such as Web services. Not limited to the IoT server microservice 80G and mailer microservice 80H shown in Figure 7, any microservice 80 that can individually access any service available in the cyber space (not shown) may be placed on the same level as the Web microservice 80I. Examples of services in cyberspace not shown in Figure 7 include automated trading of securities (shares of specific stocks), ordering of specific products and automatic transfer of payment for received products (billing process), and user services such as purchasing movie or music tickets or making reservations for specific trips.

[0075] By enabling the placement (pre-installation) of microservices 80, which allow individual access to each service in cyberspace, within the software architecture of the edge computer 6, users can enjoy collaborative services that connect different services in cyberspace.

[0076] Previously, the coordination between services utilizing various devices 8 in the physical space and services in the cyber space described above was relatively weak. As shown in Figure 7, by making it possible to place the microservices 80A to 80F that control the various devices 8 on the same level as the Web microservice 80I, users can enjoy advanced coordinated services that seamlessly and without any sense of incongruity, spanning services in the physical space or in the physical world that utilize the various devices 8 and services in the cyber space described above.

[0077] When technology emerged that allowed for the digital handling of video and audio, which were previously treated as analog data, the term "multimedia" became popular, signifying the ability to integrate and handle all media digitally. Today, this digital world is commonplace.

[0078] In this embodiment, by newly defining dedicated control software (microservices 80) for each of the various devices 8 responsible for detecting and manipulating objects and / or instances and their states in real space, it becomes possible to construct a "multi-service" world that enables the provision of more advanced services by combining (mixing) services that can be provided in real space / physical space with services that can be provided in cyber space. The "method of combining various services" to provide advanced services to the user is defined by the application (IF-THEN) rule 70.

[0079] Let's explain the relationship between Figure 7 and the aforementioned Figure 2. The normal camera 816 in Figure 2 corresponds to a type of sensor 802, and the infrared camera 806 in Figure 2 and Figure 7 represent the same device 8. Therefore, the normal camera image recognition unit 112 in Figure 2 corresponds to the sensor microservice 80A. The infrared recognition unit 116 and the human head temperature determination unit 125 in Figure 2 correspond to the infrared camera microservice 80C. In Figure 2, the result of image recognition performed by the normal camera image recognition unit 112 is transferred to the human head temperature determination unit 125. In Figure 7, information exchange between the sensor microservice 80A and the infrared camera microservice 80C takes place via the application (IF-THEN) engine 90.

[0080] Furthermore, the detection operation instruction unit (application (IF-THEN) engine) 90 of the detection rule setting unit 130 corresponds to the application (IF-THEN) engine 90 shown in Figure 7.

[0081] On the other hand, the function of the detection operation setting unit 140 that controls the email sending 1250 in Figure 2 is processed by the mailer microservice 80H. Also, when displaying a screen 1230 on the display unit of the HMD 810, the function of the detection operation setting unit 140 that controls the screen display 1230 is processed by the HMD microservice 80E.

[0082] For example, when performing equipment operation 1290 (1270) such as "blocking a passage gate to restrict the passage of a person with a high fever," the corresponding function of the detection operation setting unit 140 is processed by the individual equipment microservice 80F.

[0083] Regarding the applications (IF-THEN) rules 70 and various devices 8 used by individual edge computers 6, the cloud server 2 has functions such as a rule setting / distribution unit (function) 202 and a device management unit (function) 206. However, the cloud server 2 may also have functions as a microservice registration unit (function or module) 204 (the definition of the term [module] will be described later).

[0084] When the edge computer 6 provides services to the user in accordance with the application (IF-THEN) rule 70, various data obtained are managed and stored 208 on the cloud server 2 and saved appropriately in the data / information storage device 200.

[0085] Figure 8 shows an example of a program configuration used in an Android®-based smartphone. The ifLink® widget 1100 displays the status of ifLink.

[0086] Microservices 80A-80F, written in Java®, are located in a sub-area of ​​the ifLink application 1000. The application (IF-THEN) engine 90 operates as part of this ifLink application 1000. Application (IF-THEN) rules 70 are also used within this ifLink application 1000. In other words, this ifLink application 1000 manages the microservices 80A-80F and application (IF-THEN) rules 70 that are connected to ifLink.

[0087] Using Figure 9, the relationship between the internal workings of the edge computer 6 shown in Figure 7 and the cloud server 2 will be explained in detail. The edge computer 6 has the service provision application program ifLink app 92 pre-installed. The software architecture of this ifLink app 92 consists of an application (IF-THEN) engine unit 90, a settings / management screen configuration unit 98, a data transmission unit 96, and a rule reception unit 94. The edge computer 6 also has a memory area for storing application (IF-THEN) rules 70, which are written in, for example, XML (Extensible Markup Language) format. The application (IF-THEN) engine unit 90 then operates the corresponding microservices 80 according to the application (IF-THEN) rules 70.

[0088] Here, the ifLink app 92 is described as a "software program," but it is not limited to that; the ifLink app 92 may also be configured as "hardware." In this case, the memory area for storing the application (IF-THEN) rules 70, the application (IF-THEN) engine unit 90, the settings / management screen configuration unit 98, the data transmission unit 96, and the rule receiving unit 94 may each be composed of physically separate electronic circuits.

[0089] As shown in Figure 9, the software architecture of Cloud Server 2 consists of a microservice registration unit 204, a rule setting and distribution unit 202, a data acquisition service unit (DDS: Data Destination Service) 216, an endpoint management unit (EPM: End Point Manager) 214, a Web-API control unit 218, and a data / information storage device 200.

[0090] The rule setting and distribution unit 202 then manages the setting of application (IF-THEN) rules 70 according to user requirements and the distribution of application (IF-THEN) rules 70 to users.

[0091] The data collected by the ifLink app is transferred from the data transmission unit 96 of the edge computer 60 to the data collection service unit (DDS) 216 of the cloud server 2 via the communication path 18. The data processed by the data collection service unit (DDS) 216 is then sequentially stored in the data / information storage device 200.

[0092] Application (IF-THEN) rules configured according to user requests are also stored in the data / information storage device 200. Then, at the required time, the application (IF-THEN) rules 70 are read from the data / information storage device 200 and distributed from the endpoint management unit (EPM) 214 to the rule receiving unit 94 of the ifLink application via the communication path 18. The application (IF-THEN) rules 70 received by the rule receiving unit 94 are then stored in the memory area of ​​the edge computer 6 (not shown in the diagram). Microservices 80 that have been certified and registered by a designated organization are also installed on the edge computer 6 via the endpoint management unit (EPM) and the rule receiving unit 94.

[0093] On the other hand, various data collected from the ifLink app and stored in the data / information storage device 200 are processed by various application services 1800 via Web API. The specific services provided by this application service 1800 include: visualization of data collected from the ifLink app 1802; management of data collected from the ifLink app 1804; status monitoring and error response regarding the status of the edge computer 6 and various modules 9 using data collected from the ifLink app 1806 (module 9 will be explained later); analysis of data collected from the ifLink app 1808; optimization of services for users using the ifLink app 1810 and optimization of module 9 setting conditions 1810; and information services 1812 using the information obtained as a result of the data analysis 1808.

[0094] Figure 10 shows the functions implemented in the application (IF-THEN) engine 90 shown in Figures 7 and 9. Specific functions include activation 902, rule management control 904, rule execution control 906, sensor data control 910, job control 912, log output function 914, same event ignore time setting function 916, IF-THEN function spanning other edge computers 6 918, rule enable / disable function 920, execution time control function 922, and intermediate data holding object function 924.

[0095] In activation 902, when the application (IF-THEN) engine 90 is launched from the ifLink app 92, it establishes a connection with the ifLink app 92.

[0096] In the rule management control 904, when the application (IF-THEN) engine 90 is started, the application (IF-THEN) rule 70 that has been held in advance is read and the internal state is constructed. In other words, for example, it becomes a waiting state for detection output or command from the actual sensor 802 (a ready state with several options).

[0097] In the rule execution control 906, based on the sensor data information notified from the ifLink application 92, an application (IF-THEN) rule 70 that matches the conditions is extracted based on the internal state. In other words, a rule is selected according to the content of the detection output (command content).

[0098] The sensor data control 910 receives device information 68 from the ifLink application 92 and initiates execution control corresponding to the application (IF-THEN) rule 70.

[0099] The JOB control unit 912 notifies the ifLink application 92 of the necessary JOB information.

[0100] The log output function 914 outputs logs (e.g., service history) according to the log level (the importance or detail of the log content).

[0101] The same event ignore time setting function 916 means that even if similar sensor data is input, the occurrence will be suppressed for a specified period of time.

[0102] The IF-THEN function 918, which spans multiple edge computers 6, notifies another specific edge computer 6 of the JOB and has it executed there.

[0103] The rule enable / disable function 920 refers to the function that switches the ON / OFF state of the rule specified in the application (IF-THEN) rule 70 and the execution (THEN) 78 (described later using Figure 11).

[0104] The execution time control function 922 refers to a function that allows a specific execution (THEN) 78 timing to be shifted, enabling a subsequent execution (THEN) 78 (described later using Figure 11) to occur after a certain period of time.

[0105] The intermediate data retention object function 924 refers to a function that retains intermediate data generated during processing in a memory device and allows it to be input to the condition (IF) 72 of another rule (described later using Figure 11).

[0106] Figure 11 is a block diagram representing the application concept in this embodiment. In the world of "multi-service" proposed by this embodiment, for example, the detection results of changes in entities or states (or related information) 7 occurring in the physical real world and services (Web services 210) in cyberspace can be directly linked to provide services to users.

[0107] In other words, Figure 7 explicitly shows the Web service server 210 and the mail server 230 as service forms for users that utilize devices other than device 8. In contrast, here, various devices 8 (corresponding to 802-820 in Figure 7) as means of providing services to users in physical space, and means of providing services to users in cyberspace (for example, the Web screen provided by the Web service server 210) are collectively referred to as "human-recognizable entities or states, information (instance, status, and / or information) 7 (corresponding to 7A-7F)." Furthermore, the aforementioned devices 8 are a type of "human-recognizable entity / state / information 7." General-purpose means for individually controlling these are defined as "control means 4 (4A-4F)."

[0108] As mentioned earlier, Figure 7 explicitly shows the Web service server 210 and the mail server 230 as service models for users that utilize devices other than device 8. In contrast, we propose a new concept here: “human-recognizable entities or states, information (instance, status, and / or information) 7 (7A-7F).” Device 8, mentioned above, falls under the category of “human-recognizable entities / states / information 7.” Furthermore, we define the general means for individually controlling these as “control means 4 (4A-4F).”

[0109] For example, consider the use of an AI (Artificial Intelligence) program to control a web screen (such as automatically filling in necessary information within the frame of a form element specified in HTML, or controlling transitions between web pages), or the use of a web screen for order processing, contract processing, or application processing. In this case, the information on the web screen (specifically, the "format" on the web screen that should be filled in during order processing, contract processing, or application processing) falls under the category of "human-recognizable entities or states, information 7", and the AI ​​program that controls this information (specifically, the process of automatically filling in the necessary items in the "format" on the web screen and pressing the "execute" button on the web screen after user confirmation) falls under the category of control means 4.

[0110] As another example, consider the case where big data collected using sensor 802 is stored in data / information storage device 200 and then analyzed using statistical analysis software. In this case, sensor 802 alone is classified as device 8 and corresponds to a type of "human-recognizable entity / state / information 7". The sensor microservice 80A that controls the operation of this sensor 802 becomes a type of control means (microservice) 4.

[0111] Furthermore, the data / information storage device 200 used for storing big data is classified as a device 8 and therefore falls under the category of "human-recognizable entities / states / information 7". The big data stored in this data / information storage device 200 and the information obtained after data analysis can also be classified as a type of "human-recognizable entities / states / information 7". In contrast, the data analysis processing performed by statistical analysis software falls under "data control" in a broad sense. Therefore, this statistical analysis software falls under the category of control means (microservices) 4.

[0112] Here, the functional implementation form of the control means 4 shown in Figure 11 may be implemented in hardware (e.g., a combination of logic circuits) or in software. For example, a control means 4 that takes the form of a program using an object-oriented programming language is specifically called a microservice 80. Of course, a microservice created in a non-object-oriented programming language (e.g., assembler) is also acceptable.

[0113] The device 8 and the means of providing services in cyberspace are collectively (generalized) referred to as "recognizable entity / state / information 7," and the generalized collective name including the microservices 80 is called "control means 4," and this will be used to explain all forms of services provided to users in general from now on.

[0114] For example, the sensing function can only be realized when the sensor 802 itself is controlled by the "control means 4". The means for realizing predetermined functions in relation to the services provided to the user are called "modules 9 (9A-9F)". Module 9 basically consists of "recognizable entities / states / information 7" and "control means 4". As for the installation configuration of module 9, the entire module γ9C may be built into the edge computer 6. However, only the recognizable entities / states / information 7D and 7E of modules 9D and 9E may be located outside the edge computer 6. In this case, the control means (microservices) 4D and 4E (which are installed in advance) on the edge computer 6 and the recognizable entities / states / information 7D and 7E are signal-connected by the communication means 18.

[0115] Furthermore, control means (microservices) 4A and 4B may exist not only on the edge computer 6, but also on the cloud server α2A.

[0116] Application rule β70B, which defines the method of cooperation between module γ9C and module δ9D in order to provide a specified service to the user, basically consists of a basic logic that progresses over time from condition (IF) β72B to execution (THEN) β78B.

[0117] This section explains the "condition (IF) β72B ⇒ execution (THEN) β78B" logic and its specific processing using an example logic (application rule β70B) that "turns on the lights when it gets dark." In this case, as processing corresponding to condition (IF) β72B, the application (IF-THEN) engine 90 issues an API command 75 to the illuminance sensor control means (microservice) γ4C. In response, the illuminance sensor control means (microservice) γ4C controls the illuminance sensor γ7C of the recognizable entity / state / information 7 to measure the illuminance of the surrounding environment. If the obtained illuminance is lower than the preset value, the illuminance sensor control means (microservice) γ4C returns the return value of API command 75 to the application (IF-THEN) engine 90.

[0118] Then, the application (IF-THEN) engine 90 starts the execution (THEN) β78B operation in accordance with the application rule β70B described above. At this time, the application (IF-THEN) engine 90 receives signals from the microservices and, based on the interpretation of the IF-THEN rule, issues instructions to the necessary microservices. In other words, the application (IF-THEN) engine 90 does not autonomously execute processing, but rather, it receives data from the microservices specified in IF, compares it with the conditions of the rule, and when the conditions are met, issues an execution instruction to the microservices specified in THEN.

[0119] Then, the application (IF-THEN) engine 90 issues the following API command 75 to the lighting switch control means (microservice) δ4D. The lighting switch control means (microservice) δ4D then controls the lighting switch δ7D, which is a recognizable entity / state / information 7, to turn on the lighting switch. As a result, the function of module δ9D, which has the function of controlling the lighting switch, is executed.

[0120] Next, we will explain the case where a user purchases new modules ε9E and ζ9F to expand the system. The control means (microservices) ε4E and ζ4F for these modules ε9E and ζ9F, and the application rules γ70C that combine them, are generated within the web server's cloud server β2B. Subsequently, they are installed 180 via the communication line 18 that connects the cloud server β2B and the edge computer 6.

[0121] The installed control means (microservices) ε4E, ζ4F and application rule γ70C are then saved in the form of files (for example, as microservice file 2602 and application rule file) in a predetermined recording area of ​​the edge computer 6.

[0122] Simultaneously, Cloud Server β2B generates Integration Rule 700, which integrates the existing application rule β70B with the newly created application rule γ70C. Then, Cloud Server β2B verifies the consistency between the existing application rule β70B and the newly created application rule γ70C. If any problems are found as a result of this verification, the user will be notified during the installation 180 mentioned above or when the service is provided to the user.

[0123] Cloud Server β2B can test application rules for numerous edge computers. These edge computers can be found in factories, hospitals, schools, government offices, farms, and homes, each employing application rules tailored to its specific location.

[0124] The terms used in this patent specification, including those used in the above explanation, are defined below.

[0125] ●[Module 9]--This refers to the means for realizing a predetermined function, and is composed of a combination of a [recognizable entity or state, information (instance, status, and / or information) 7] and its [controller 4] in the physical space where this predetermined function is realized. In this embodiment, a service is basically provided to the user by a combination of operations (movements) of multiple [Modules 9] controlled by individual [controllers 5]. The [Module 9] described in this embodiment merely represents the "concept" of a "combination for realizing a predetermined function". Therefore, the [recognizable entity or state, information 7] and its [controller 4] that constitute the [Module 9] do not need to be "physically integrated". For example, the [recognizable entity or state, information 7] and its [controller 4] may be installed in physically separate locations and cooperate with each other using communication means 18. Furthermore, as will be described later, neither of them needs to form a "physically real object".

[0126] ●[Human-recognizable entity / state / information (instance, status, and / or information)7]--This refers to the object of realization of a predetermined function in real space. Measurable [states] such as temperature, humidity, and pulse rate are also included in the real-space recognizable [entity or state, information (instance, status, and / or information)7] described below.

[0127] ●[Data / Information]--This includes both acquired data / information and provided data / information. Here, data collected by [devices] such as sensors is a type of acquired data, and the information obtained as a result of analyzing this acquired data is a type of acquired information. Naturally, this [data] also includes measurement results of [conditions] such as temperature, humidity, and pulse rate. Furthermore, data and information provided to users as a [service] are classified as provided data / information. A concrete example of this provided data / information is educational materials and teaching materials.

[0128] ●[Processing] -- This refers to the process of realizing a predetermined function, and the intervention of electronic devices is a prerequisite. In other words, it includes all specific actions using electronic devices. For example, this includes specific actions such as ordering, money transfer, foreign exchange, contract processing (such as insurance), scheduling of travel and business, and travel arrangements using smartphones or web screens. However, manual processing such as "putting a handwritten application form into a mailbox" does not involve electronic devices and therefore does not fall under the definition of [processing] as described in this specification.

[0129] ●[Device 8]--Includes all electronic devices that exist in real space and are used to realize a predetermined function. Therefore, it also includes electronic devices that perform the above [processing], electronic devices that display the above [content], and memory devices that store the above [information / data]. This [Device 8] may have a built-in communication function for communicating with external electronic devices. This [Device 8] may also have a function that acts on the physical (or chemical) state or phenomenon of the natural world. Devices that act passively on the natural world include sensors, etc. Devices that act actively on the natural world include robots, drive mechanisms, display devices (display elements), etc. The physical form of this [Device 8] is not limited to a stationary type, but may also take the form of a portable type that can be moved. For example, it may take the form of a band-type sensor that is fixed to the user's arm or leg, a belt-type sensor that is fixed to the waist, or a wearable form such as a VR (virtual reality) type or AR (augmented reality) type HMD810 that is worn on the head.

[0130] ●[Controller 4]--This refers to a means for controlling [human-recognizable entities / states / information 7] in order to realize a predetermined function. A communication function may be built into this, and the [human-recognizable entities / states / information 7] may be controlled using the communication means. Examples of using this [Controller 4] include control of transitions between web pages, automatic input control of necessary parts within web pages, data analysis control of accumulated data, control of buying and selling processing of goods, automatic travel arrangement processing control, and robot movement control. Furthermore, if the [human-recognizable entities / states / information 7] is a control means for sensors such as temperature and humidity, it performs control that detects and quantifies (converts into data) the physical (or chemical) [state] of the natural world. In this control means, detailed procedures for realizing individual detailed functions are defined. For example, the detailed procedures may be defined in a "hardware form" in which the control logic is formed by a combination of logic circuits. However, it is not limited to this, and the detailed procedures may also be defined in a "software form" such as a program that can be installed on an [edge computer 6] or a [cloud server 2].

[0131] ●[Micro-service 80 / IMS]--This refers to a [control means 4] formed by a program using an object-oriented programming language. As mentioned above, the object controlled by this micro-service is not limited to devices, but also includes processes, content, and information / data. Therefore, for example, a program software using AI technology to control transitions between web pages or screen operations corresponding to a specific web page is included as a type of micro-service or a part of it. Similarly, a program software that uses AI technology to automatically extract the number of people appearing in a specific video content, or a program software that analyzes acquired data (for example, using statistical analysis), is also included as a type of micro-service or a part of it. The information obtained as a result of the analysis and the recording device that stores that information are treated as [human-recognizable entities / states / information 7]. Also, since [micro-service 80] is a form of [control means 4], as mentioned above, by installing [micro-service 4], it becomes possible to control the object controlled by the [edge computer 6] or [cloud server 2] (human-recognizable entities / states / information 7). When defining the detailed procedures in this "software form," forming the [control means 4] with a program using an object-oriented programming language such as Java® or Objective-C allows for effective utilization of existing program assets (such as embedding (import) or calling (issuing API command 75) in other programs) (details will be described later in Chapter 2). Furthermore, writing the program in the OS-independent Java language improves the versatility of the [control means 4]. In the following explanation, we will also refer to it as IMS (ifLink Micro-service) instead of microservice.

[0132] ●[Control Method]--This refers to a program that constitutes part of the [Microservice 80] and defines the detailed procedures for realizing specific individual detailed functions. Each control method is treated as the smallest functional unit (i.e., a subprogram) that realizes the individual detailed function. The control method name corresponding to a specific individual detailed function can be called as a corresponding function (command) as an API command 75 from the [Application Engine 90]. This allows the [Application Engine 90] to execute the individual detailed function corresponding to the control method name. In this embodiment, by writing the program for the [Control Method] using an object-oriented programming language (defining the program content), the [Control Method] can be linked to the [Application Rule 70] as described above. (A detailed explanation with specific examples will be given later in Chapter 2.) ●[Control Class]--This refers to a collection of individual [Control Methods] grouped together according to common similar functions. A control method corresponding to the initial setup function of the instance (entity) to be created is called a constructor. The name of the constructor (the name of the control method corresponding to the constructor) included in the same collection may match the name of the control class. In the aforementioned [Microservice 80], a management unit that groups control methods according to common similar functions corresponds to a [Control Class]. Therefore, multiple different [Control Classes] may exist in a [Microservice 80] for the same purpose.

[0133] ●[Control Package]-- Indicates a collection of classes with similar functions. That is, it is defined according to a mechanism for managing collections of classes with similar functions by dividing them into folders. In this embodiment in particular, a control package is separated for each entity / state / information 7 included in the same module 9, and a different control package name (identification information for each control package) is set individually for each. Therefore, it is possible to identify the contents of each entity / state / information using the individually set control package name (identification information for each control package). When setting the name of the [control package] for controlling a [device] belonging to an entity / state / information, the identification information of the manufacturer or distributor of the module (or device), the type of module (or device), or the individual manufacturing number may be used (set) as the identification information for each control package.

[0134] ●[Microservice File 2602]-- Indicates the storage unit (storage form / storage format) for saving [Microservice 80] in the storage area of ​​Cloud Server 2 or Edge Computer 6. If [Microservice 80] is written in Java, for example, the contents of [Microservice 80] are a sequence of Java code. To save this information, files are constructed in units of control classes (which also contain descriptions of the processing content for each control class). The extension of these control class files is ".class". In addition, a "folder" for each control package is placed in the storage area of ​​Cloud Server 2 or Edge Computer 6, and the control class files are stored in it. The folder name should match the control package name. This makes it easy to find the necessary control class files because the manufacturer / distributor identification information and module (or device) type information can be found from the folder name.

[0135] ●[API (API command) 75]--This indicates a means of communication (communication tool) between the [Application (IF-THEN) Engine 90] and the [Microservice 80] for manipulating individual microservices 80 from the [Application (IF-THEN) Engine 90]. Each [Microservice 80] has a hierarchical structure of control packages / control classes / control methods. [API (API command) 75] often specifies a particular control method name that indicates the detailed procedure (program) for realizing individual detailed functions, and individual detailed function operations are performed on a control method basis. For example, an example of how to execute the individual detailed function of this particular control method in an application program 22 written in the Java language (including the [ifLink app (service provision application program)] described later) is explained below. First, within the class of this application program 22, the control class of the control package containing the corresponding control method is incorporated (imported). Then, by specifying the corresponding control method name within the above class or method, this individual detailed function can be executed. ●[Application (IF-THEN) Rule 70]--This rule specifies information for a method (rule) that combines multiple different control means 4 (or modules 9) to indicate how to provide a predetermined service to a user.If the [Application (IF-THEN) Rule 70] is composed of hardware consisting of a combination of logic circuits, for example, the output terminals of these combined logic circuits are electrically directly connected to the input terminals of the control logic composed of the combination of logic circuits of the [Control Means 4].On the other hand, if the [Application (IF-THEN) Rule 70] is defined in a description format (including HTML that constitutes a web screen) according to a predetermined description method, the application (IF-THEN) engine 90 of the cloud server 2 or edge computer 6 deciphers the contents of the above [Application (IF-THEN) Rule 70].Based on the decipherment result, the application (IF-THEN) engine 90 issues an API command 75 to the corresponding microservice 80, and control of the recognizable entity / state / information 7 from the microservice 80 is initiated.

[0136] ●[Integrated Rule 700]--This rule is generated when multiple different [Application (IF-THEN) Rules 70] are defined for the same edge computer 6 or the same ifLink application. The purpose of generating this [Integrated Rule 700] is to detect in advance any inconsistencies or problems that may occur when multiple different [Application (IF-THEN) Rules 70] are combined, thereby avoiding trouble when providing services to users. If a user requests to define multiple different [Application (IF-THEN) Rules 70], the [Integrated Rule 700] is first generated on the cloud server 2, and the operation in accordance with this [Integrated Rule 700] is simulated on the cloud server 2. If inconsistencies or problems occur in the simulation results, the user is notified and a solution is proposed. After it is confirmed that no inconsistencies or problems occur in the simulation results, this Integrated Rule 700 is installed 180 from the cloud server 2 to the edge computer 6. After this installation 180, the edge computer 6 operates various microservices 80 (control means 4) in accordance with this [Integrated Rule 700].

[0137] ●[Edge Computer 6]--It has an [Application (IF-THEN) Engine 90] and pre-stores [Application (IF-THEN) Rules 70] and various [Microservices 80] necessary for providing services to users corresponding to those rules. When a user wants to receive a specific service from [Place 1], the [Application (IF-THEN) Rules 70] and the corresponding [Microservices 80] are pre-installed from the cloud server β2B based on the user's request to [Place 1]. [Edge Computer 6] consists of a processor, a memory unit, and a communication unit. The [Application (IF-THEN) Rules 70] and the corresponding [Microservices 80] installed here are stored in the memory unit. The processor also performs the functions of the [Application (IF-THEN) Engine 90] according to these [Application (IF-THEN) Rules 70]. As long as it consists of a processor, a memory unit, and a communication unit in this way, [Edge Computer 6] can take any form. As a concrete example of form, personal computers, smartphones, tablets, signage, gateways, routers, etc., may function as [Edge Computer 6].

[0138] ●[ifLink app (service provision application program) 92]--This refers to a program processed by the cloud server α2A or edge computer 6 in order to provide a predetermined service to user 1700. This consists of an [application (IF-THEN) engine 90] that executes processing in accordance with the contents of the [application (IF-THEN) rule 70], a settings / management screen section 98, and data transmission 96 and rule reception section 94 involved in communication between the cloud server 2. Furthermore, this [ifLink app (service provision application program) 92] is programmed to first refer to the contents of a predetermined [application (IF-THEN) rule 70] that has been saved in advance. The [application (IF-THEN) engine 90] on the cloud server α2A or edge computer 6 then executes processing in accordance with the contents programmed within this [ifLink app (service provision application program) 92] and provides the predetermined service to user 1700.

[0139] ●[Application (IF-THEN) Engine 90]--This refers to the location (function) built into the cloud server α2A or edge computer 6 that performs processing / execution within the cloud server α2A or edge computer 6. Hardware-wise, it may correspond to an arithmetic processing processor (or its processing state). This [Application (IF-THEN) Engine 90] reads the [Application (IF-THEN) Rule 70] and deciphers its contents. Then, referring to the deciphered contents, it executes processing in accordance with the content programmed in the [ifLink App (Service Provisioning Application Program) 92]. During this processing / execution, it issues the necessary [API Command 75] to the microservice 80. The inside of this [Application (IF-THEN) Engine 90] consists of a programming language interpretation engine corresponding to the description format (corresponding programming language) in which the [Application (IF-THEN) Rule 70] is written, and a control engine that executes processing in accordance with the program defined in the [ifLink App (Service Provisioning Application Program)] while referring to the interpretation result. [Application (IF-THEN) Rule 70] may include built-in web browser functionality to handle cases where the application is written (expressed) in HTML.

[0140] ●[Cloud Server 2]--It has an [Application (IF-THEN) Engine 90] and has pre-stored [Application (IF-THEN) Rules 70] and various [Microservices 80] necessary for providing services to users corresponding to those rules. When a user wants to receive a specified service from [Place 1], the [Application (IF-THEN) Rules 70] and the corresponding [Microservices 80] are sent to the Edge Computer 6 based on the user's request to [Place 1]. As an example of this transmission method, if a link function (Anchor Element) that specifies a URL in HTML is used, this [Cloud Server 2] may also have a Web server function.

[0141] ●[Service]--This refers to an active operation that provides a user with a predetermined [process], a change in [state], or [content] or [data / information], either for a fee or free of charge, by combining the control of a predetermined [entity / state / information 7]. In this embodiment, the [service] is provided to the user in accordance with the [application (IF-THEN) rule 70]. In this embodiment, as a form of [service] provision, a combined (complex) service may be provided that transcends (spans) the boundaries between services in the real or physical space using various devices 8 and services in cyberspace using Web services, etc.

[0142] The application concept of this embodiment, as explained in Figure 11, is summarized below. Specifically, module 9 consists of a human-recognizable entity or state / information 7 and a control means 4 for controlling it. Multiple modules 9 can be defined. That is, a first control means γ4C for the first module γ9C and a second control means δ4D for the second module δ9D are defined separately.

[0143] Here, an application rule 70 is set for providing services to the user using the combination of the first module γ9C and the second module δ9D. Then, according to the contents of this application rule 70, API commands 75 are issued individually from the application (IF-THEN) engine 90 to the first control means γ4C and the second control means δ4D to operate the first or second control means γ4C and δ4D. This utilizes the module control method of this embodiment.

[0144] The control means 4 described above includes a microservice 80. This microservice 80 may be formed (described) as a program using an object-oriented programming language. (However, it is not limited to this, and the microservice 80 may be described in any programming language.) Here, a first microservice 80 (control means γ4C) that controls a first entity or state, information γ7C that is recognizable by humans, and a second microservice 80 (control means δ4D) that controls a second entity or state, information δ7D that is recognizable by humans are defined. An application rule β70B is set to provide a service to the user using this combination of the first microservice 80 and the second microservice 80. Then, in response to the API command 75 issued from the application (IF-THEN) engine 90 according to the application rule β70B, these first / second microservices 80 individually control the first entity or state, information γ7C and δ7D that are recognizable by humans.

[0145] Furthermore, in this embodiment (and its application example), the edge computer 6 receives application rule γ70C (and integration rule 700) and microservice 80 (control means 4F) from the cloud server β2B and pre-installs them 180. As a result, a first microservice 80 (control means ε4E) that controls a first human-recognizable entity or state, information ε7E, a second microservice 80 (control means ζ4F) that controls a second human-recognizable entity or state, information ζ7F, and related application rule γ70C (and integration rule 700) are pre-built into the edge computer 6. Here, the application rule γ70C (and integration rule 700) defines rules for providing services to the user through a combination of the first microservice 80 (control means ε4E) and the second microservice 80 (control means ζ4F). Then, the edge computer 6 operates the first microservice 80 (control means ε4E) and the second microservice 80 (control means ζ4F) to individually control a second entity or state, information ζ7F that is recognizable to humans, and to provide services to the user.

[0146] In particular, the edge computer 6 in this embodiment (and its application example) has an application (IF-THEN) engine 90. The application (IF-THEN) engine 90 issues API commands 75 to the first microservice 80 (control means ε4E) and the second microservice 80 (control means ζ4F) according to the rules defined in the application rule γ70C (and integration rule 700). As a result, the first or second microservice 80 (control means ε4E, ζ4F) individually control recognizable entities or states, information ε7E, ζ7F, and provide services to the user. In the service provision method in this embodiment (and its application example), services are provided to the user using the method described above.

[0147] In this embodiment (and its application example), as described above, the contents of the microservice 80 can be described using an OS-independent programming language such as Java.

[0148] The effect will be explained using Figure 12. In the case of the Java language, an OS-independent general-purpose programming language, translation areas α1300A and β1300B (Virtual Machines) corresponding to individual OS layers α30A and β30B are prepared in advance. The contents of the microservice 80 written in Java are translated via translation areas α1300A and β1300B (Virtual Machines) and passed to the individual OS α30A and β30B.

[0149] In particular, the commands of the microservice 80 in this embodiment are described by a combination of basic API commands 75 at the OS layer α30A and β30B levels. As a result, the basic control of each device can be performed in detail. Figure 13 shows the class structure (software architecture) of the microservice 80. While this chapter focuses on the microservice 80, the content described below can also be applied to the broader control means 4.

[0150] A fundamental requirement for microservices 80 is the ability to control individual devices 8. However, beyond that, peripheral functions related to the control of individual devices 8 are also required as part of the microservices 80's functionality. Specific examples include interface processing with the application (IF-THEN) engine and checking whether the corresponding device 8 is operational.

[0151] In this embodiment, the microservice 80 is separated into different classes based on functions different from those described above, thereby improving the extensibility of the microservice 80. Specifically, a Custom Device Class 2102 is set up to execute the control functions of the individual devices 8. Multiple methods describing the detailed procedure program content for each individual detailed function of the corresponding device 8 are then placed (written) within this class.

[0152] On the other hand, a Customims Class 2110 is set up as a program to execute peripheral functions related to the control of individual devices 8.

[0153] Here, the individual control methods of the Custom Device Class 2102 are called (integrated) from within the Customims Class 2110, making them executable. Specifically, by specifying `import [Package Name for Custom Device Control].[Custom Device Class Name];` using an import statement before the Customims Class 2110, the Customims Class 2110 can integrate the individual control classes within the Custom Device Class 2102. Then, within the Customims Class 2110 or a specific control class within it, the individual control method name, arguments, and return value format (type) within the Custom Device Class 2102 are described. This makes it possible to call / use the individual control methods within the Custom Device Class 2102 within the Customims Class 2110 or a specific control class within it.

[0154] In this embodiment, optimal data E60 tailored to the individual characteristics of each device 8 to be controlled can be transmitted to the device 8. To enable this, this embodiment allows for the configuration of different individual device control packages 2100 for each manufacturer / distributor and device model.

[0155] On the other hand, it is necessary to guarantee compatibility between systems delivered to users and future expandability. To ensure such compatibility between systems and future expandability, template classes that serve as the basis for the description content (program) of each class may be provided. Specifically, for the Custom Device Class 2102, the Base Device Class 2002 is provided as a template class to any programmer. Similarly, for the CustomIms Class 2110, the BaseIms Class 2010 is provided as a template class to any programmer.

[0156] Programmers attempting to program a new microservice 80 can create a Custom Device Class 2102 and a Customims Class 2110 within a microservice with individual device support by making only minor modifications (customizations) 74 to the template content described above. This allows programmers creating a new microservice 80 to create it efficiently and in a short amount of time.

[0157] To indicate that each class extends the functionality of the original template class, you can specify the extension relationship between classes by writing, for example, `public class Custom Device extends Base Device {` in the place where you define each individually created class name. This method of description not only makes it easier to manage individual microservices, but also makes it easier to ensure compatibility and extensibility between systems.

[0158] By the way, the "public" mentioned above means "permission to use (adapt) the Custom Device Class in other systems." If you write "private" instead of "public" here, use by any class other than its own (Custom Device Class in the example above) will be prohibited. Also, if you write "protected," it will only be usable by classes in the package and classes that have inherited its functionality. When many programmers are involved in creating (modifying) microservices, problems such as unintended tampering are likely to occur. By establishing rules as described above, the frequency of problems occurring among multiple programmers can be reduced.

[0159] As shown in Figure 13, the API command (coordination specification) 75 from the application (IF-THEN) engine 90 is processed by the microservice control template class (BaseIms Class) 2010.

[0160] Furthermore, while providing a predetermined service to user 1700, an abnormal situation may occur, such as the battery of device 8 running out. An abnormality detection class (Health Check Task Class) 2016 is provided to handle such urgent abnormal situations, enabling rapid response to these situations. This abnormality detection class (Health Check Task Class) 2016 is called / embedded by the microservice control template class (BaseIms Class) 2010. The call / embedded process 76 here is processed in the same way as described above.

[0161] In this embodiment in particular, AI processing such as image recognition is required. Therefore, the basic control package 2000 includes an AI-based image recognition class (AI Analyze Class) 2018 that specializes in image recognition processing using AI technology.

[0162] Figure 2 shows the normal camera image recognition unit 112 (corresponding to the sensor microservice 80A in Figure 7), which performs a series of image recognition processes including: 1) image recognition (image analysis) of the image signal; 2) determining whether there is a person in the screen (person detection); 3) calculating the number of people if there is a person in the screen (people detection); 4) recognizing the position of each person's head only if there is a person in the screen (head detection); and 5) calculating (detecting) the relationship between each person and the infrared camera 806 for each person identified in the screen. Accordingly, the AI-based image recognition class (AI Analyze Class) 2018 has individual methods for performing the above individual processes [1] to [5], respectively, which are described in the program.

[0163] Then, using API commands corresponding to each method, the corresponding methods are called individually from the microservice control template class (BaseIms Class) 2010 or the individual device-specific microservice control class (Customins Class) 2110, and the processing is carried out 76. Here, the corresponding methods are called sequentially according to the flow procedure shown in Figure 5 76.

[0164] As previously explained, the image recognition processing related to "2) Person Detection" performed by the standard camera image recognition unit 112 in Figure 2 requires flexible processing depending on the usage scenario. For example, since the number of visitors to a reception counter is often only a few people at a time, individual identification of people is relatively easy. However, the standard camera 816 installed behind the reception counter can only capture the upper body of visitors.

[0165] Furthermore, while the standard camera 816 can capture full-body images when photographing a corridor, it also needs to recognize moving individuals. In other words, high speed is required for detecting people with high fevers in corridors. On the other hand, when detecting people with high fevers in a classroom with around 40 students, it is necessary to measure the body temperature of many people simultaneously.

[0166] In this way, multiple AI image recognition classes (AI Analyze Class) 2018, each using an optimal person detection logic (image recognition algorithm) tailored to a specific usage scenario such as a "reception counter," "corridor," or "classroom," may be simultaneously placed within the same basic control package 2000. (However, in this case, the class name will change for each class.) Then, from the microservice control template class (BaseIms Class) 2010 or the individual device-compatible microservice control class (CustomIms Class) 2110, the corresponding class name is specified, and the methods individually placed (described) within it are called to perform the processing.

[0167] Furthermore, the AI-based image recognition class (AI Analyze Class) 2018 may also contain (program) all of the individual methods that perform the specific processing described in [1] to [5] above, depending on the usage scenario, such as a "reception counter," "corridor," or "classroom."

[0168] By placing the AI-based image recognition class (AI Analyze Class) 2018 in the same basic control package 2000, it becomes possible to call and utilize the corresponding classes in the microservice 80. This allows for the selection of the optimal person detection logic for each different usage scenario, enabling flexible and highly accurate image recognition and AI processing.

[0169] This section describes the method of data movement used for image recognition (or AI processing) within the same microservice 80. The data movement between blocks shown in Figure 2 and the data movement between methods within the same microservice 80 are essentially synonymous. For example, in Figure 2, the image recognition results from the camera image identification unit 112 block are moved into the human head temperature determination unit 125 block.

[0170] The detection data from sensor 802 acquired by the device control template class (Base Device Class) 2002 or the custom device control class (Custom Device Class) 2102 in Figure 13 is used within the corresponding method of the AI-based image recognition class (AI Analyze Class) 2018. The data transfer at this time is controlled and managed by a predetermined program in the microservice control template class (BaseIms Class) 2010 or the custom device-compatible microservice control class (Customins Class) 2110.

[0171] Furthermore, detection data from sensor 802 acquired by the device control template class (Base Device Class) 2002 or the individual device control class (Custom Device Class) 2102 may be temporarily stored in buffer memory 2008 (for temporary storage of stream data). Then, the corresponding method of the AI-based image recognition class (AI Analyze Class) 2018 may read the necessary data from there and use it.

[0172] Figure 13 shows an example of AI processing, where an image recognition class (AI Analyze Class) 2018 (data analysis processing program) is placed to perform specialized image recognition. However, it is not limited to this; any AI class (AI program) specialized in a specific function using any artificial intelligence technology, such as data control functions, data manipulation functions, or automatic data collection functions (using the internet), may be placed in the basic control package 2000.

[0173] By including an AI class specializing in data analysis within the basic control package 2000, advanced data analysis processing can be performed within the microservice 80. This allows only the data analysis results processed within the microservice 80 to be passed to the application (IF-THEN) engine 90, significantly reducing the processing load on the application (IF-THEN) engine 90. Furthermore, since AI classes (AI programs) specialized for specific functions utilizing artificial intelligence technology can be arbitrarily placed in the basic control package 2000, advanced processing using artificial intelligence technology can be arbitrarily performed within the microservice 80.

[0174] If device 8 is located outside of cloud server 2 or edge computer 6, microservices 80 control device 8 via communication line 18. However, if, for example, communication line 18 becomes congested or disconnected, there is a risk that the system will freeze until a response is received from device 8.

[0175] By installing a Timeout Check Task Class 2006, the communication status between microservice 80 and device 8 can be monitored. This allows for the detection of communication problems and prompt action, preventing system freezes.

[0176] Furthermore, depending on the type of sensor device 802, it may be necessary to handle a variety of data, from binary data (binary data of "1" or "0") to video streams. By providing a Stream Control Engine Class (Stm Engine Class) 2004, the effect of efficiently and integrally handling diverse data is achieved. Note that a stream analysis class / method using AI technology may be placed within or at the same level as this Stream Control Engine Class (Stm Engine Class) 2004. That is, as described above, an import statement is used to incorporate the stream analysis class using AI technology into the Stream Control Engine Class (Stm Engine Class) 2004 or the Base Device Class (Base Device Class) 2002 (when placed at the same level as the Stream Control Engine Class (Stm Engine Class) 2004). As a result, the Stream Control Engine Class (Stm Engine Class) 2004 or the Base Device Class (Base Device Class) 2002 can use (call) the stream analysis class / methods within the Stream Analysis Class using AI technology.

[0177] This makes it possible to pass not only the raw stream data obtained from the sensor device 802 to the application (IF-THEN) engine 90, but also only the analysis result information obtained from the automatic analysis of the stream data to the application (IF-THEN) engine 90, resulting in a significant improvement in the processing load of the application (IF-THEN) engine 90. In this case, the raw stream data obtained from the sensor device 802 may be sequentially saved as a time-series file in the data / information storage device 2 on the edge computer 6 or the cloud server 2. The method of the stream analysis class using AI technology then plays back the time-series file (raw stream data file) as needed. The analysis information obtained from the data analysis using AI technology may then be used to perform a specific condition (IF) β judgment.

[0178] As shown in Figure 13, the device control template class (Base Device Class) can call / embed the response time monitoring class (TimeoutCheckTask Class) 2006 and the stream control engine class (StmEngine Class) 2004 as described above. The specific method for this call / embed process is the same as described above.

[0179] In this embodiment, writing the microservice 80 using an object-oriented programming language (such as Java) has the effect of effectively utilizing the existing program assets of the basic control package 2000 in the individual device control package 2100. This significantly improves the programmer's efficiency in developing the microservice 80.

[0180] For example, by using import statements before the individual device-compatible microservice control class (Customins Class) 2110 and the individual device control class (Custom Device Class), such as import [Name of basic control package 2000].[Name of anomaly detection class (HealthCheckTask)]; import [Name of basic control package 2000].[Name of response time monitoring class (TimeoutCheckTask)]; and import [Name of basic control package 2000].[Name of stream control engine class (StmEngine)];, you can utilize the programs of the methods defined in each class.

[0181] Furthermore, the destination of the API command (coordination specification) 75 from the application (IF-THEN) engine 90 is changed from the microservice control template class (BaseIms Class) 2010 to the individual device-compatible microservice control class (CustomIms Class) 2110.

[0182] As a result of this series of processes (program changes), the Base Device Class 2002 and BaseIms Class 2010, which are corresponding classes within the Basic Control Package 2000, are effectively replaced with the Custom Device Class 2102 and CustomIms Class 2110, which are corresponding classes within the Individual Device Control Package 2100.71 By writing microservices 80 using an object-oriented programming language in this way, program editing can be done easily and accurately with only very minor changes to the code.

[0183] Figure 14 illustrates the functions of the microservice 80 configured in the individual device control package. The three items circled in the required field 2204 (initialization 2210, termination 2212, start / stop 2214) are the minimum required functions. The remaining two items (sensor data transmission 2218 and JOB reception 2220) may be unnecessary depending on the characteristics of the corresponding device 8.

[0184] Within the microservices functionality, initialization 2210 refers to the function of establishing a connection with the ifLink application (see the definition of terms above) and registering the microservice 80 and the device 8 to be controlled.

[0185] The termination function 2212 disconnects and terminates the connection to the ifLink application, as well as the microservices 80 and the controlling device 8, when the application is terminated from the ifLink application.

[0186] The Start / Stop function 2214 is a function that starts or stops device 8 when the rule for device 8, which is registered as application (IF-THEN) rule 70, becomes enabled or disabled.

[0187] Next, we will explain the function of status notification 2216. The ifLink app manages the status of registered devices. This status notification 2216 is a function that notifies the ifLink app to update the status of device 8, which is managed by the ifLink app.

[0188] The function of sensor data transmission 2218 is to send sensor data sent from the controlled device 8 to the ifLink app.

[0189] Finally, let's explain the function of JOB receiver 2220. The content of the job (instructions for operations / API) is notified from the cloud server 2 or the application (IF-THEN) engine 90 via the IfLink app. This JOB receiver 2220 is a function that processes the job information seen from the IfLink app.

[0190] Figure 15 shows a list of APIs provided by the microservice control template class (BaseIms Class) 2010 or the individual device-specific microservice control class (CustomIms Class). In other words, the control methods shown in Figure 15 are written (placed) in the microservice control template class (BaseIms Class) 2010. That is, each control method listed in the Methods and Functions column 2302 of Figure 15 must be standard equipment in the microservice 80.

[0191] The “void” or “int” at the beginning of each control method listed in Methods and Functions section 2302 indicates the type (format / type) of the control method's return value. For example, if this control method is called from a given class or method and used in 76, it indicates "what type of data will be returned to the class or method using this control method" after the processing of this control method is complete.

[0192] For example, a control method that begins with the word "void" indicates a state of "no return value" (i.e., after the control method finishes processing, no specific data is returned to the class or method using it).

[0193] Similarly, in control methods where the characters "int" or "long" are listed first, an integer with a size in the range of 32 bits or 64 bits will be returned after the control method finishes processing.

[0194] And in control methods where the word "boolean" is written first, a boolean value of "true" or "false" is returned after the control method has finished processing.

[0195] Furthermore, within the parentheses of a control method, the "type (format / type) of the argument" and the "argument" itself, which are passed on to the control method, are listed as a pair separated by a "space". If multiple "arguments" are passed on, a comma (,") is placed between the pairs of arguments.

[0196] The argument type "String" represents a "string". "Map" represents a "key / value database" where keys and values ​​are stored as pairs. "HashMap" refers to the above "key / value database" or a class that uses this mechanism (HashMap Class). By using a "key / value database" to control device 8 in this way, data / information can be shared between methods across different control packages, as shown in Figure AA. Furthermore, using a "key / value database" as the storage format for this data / information improves the convenience of data retrieval.

[0197] Furthermore, "Message" refers to a "message" that can be sent directly to the application (IF-THEN) engine 90 or to the ifLink app. The "constructor" shown in Figure 15 has already been explained in the definition of terms related to [control classes].

[0198] In Figure 15, “EPA,” “epa,” and “Epa” are abbreviations for “End Point Access,” referring to the application (IF-THEN) engine 90 and the ifLink app. Related to this, the 28th item in Figure 15 is “void onActivationResult(boolean result, EPADevice device).” The processing overview 2304 for this is described as “Notification of service registration result from the ifLink app.” Here, “EPADevice” is specified as the “argument type (format / type)” and “device” is specified as the “argument.” “Service registration” here means “registration of device 8 operated by application (IF-THEN) rule 70.” The result is then entered into the “result” argument. If “registration was successful,” “true” is entered into “result” and the result is returned. On the other hand, if “registration failed,” “false” is entered into “result” and the result is returned. The “device” argument above refers to a specific device 8 to which the identification information “device” has been set. The "EPADevice" that specifies the "type (format / type) of the argument" means "a device 8 that the application (IF-THEN) engine 90 or the ifLink app can identify."

[0199] The device 8 used in this embodiment also includes portable devices 8. Therefore, when attempting to operate a portable device 8 according to the application (IF-THEN) rule 70, there are many cases where the target portable device 8 is taken outside the operating area. Consequently, prior to providing services to the user, it is necessary to confirm in advance whether or not the target portable device 8 is present in the operating area. This requires prior registration of the portable devices 8 whose presence in the operating area should be confirmed in advance.

[0200] For this pre-registration, the callback registration methods “void registerCallback()” or “void registerCallback(String cookie)” described as the 11th or 12th item are performed. Additionally, for portable devices 8, for which prior confirmation of whether or not they exist in the operation area is no longer required, the callback cancellation method “void unregisterCallback()” described as the 13th item is performed.

[0201] If the target device 8 can be pre-verified, the 14th function, “boolean createDevice()”, is used to “register the device”. Conversely, for device 8 that could not be pre-verified, the 15th function, “boolean deleteDevice()”, is used to “deactivate the device”.

[0202] Furthermore, during the provision of services to users, there may be situations where it becomes difficult to continue the service, for example, due to the battery running out on a specific device 8. To address this situation, the 21st method, “void startHealthCheck(long interval),” can be used to “start a health check” for each device 8. In this method, instead of “continuous health checks,” checks may be performed at regular intervals specified by “interval.”

[0203] When a problem occurs, the microservice 80 needs to send a "warning notification" to the application (IF-THEN) engine 90 or the ifLink app. In this case, the 17th method, "int send_alert(-)", can be executed to "send an alert".

[0204] In the high fever detection performed in this embodiment, there are two types of functions required from the application (IF-THEN) engine 90 to the sensor microservice 80A and the infrared camera microservice 80C: a) detection (discovery) of the person with a high fever and b) provision of information necessary for notifying the user.

[0205] For example, in the use case shown in Figure 6, when “a) a person with a high fever is detected (discovered),” the user is warned by “displaying the detected person with a high fever with diagonal lines” on the display screen 12. In this way, “b) providing information” is necessary to “visualize the detected person with a high fever.” Incidentally, this “image with the detected person with a high fever displayed with diagonal lines” is generated by the AI-based image recognition class (AI Analyze Class) 2018.

[0206] In such cases, the format of the data provided to the application (IF-THEN) engine 90 (the type (format / type) of the return value of the control method) differs between (a) and (b) above.

[0207] To flexibly respond to such situations, it is necessary to specify the "data format to send in response" to the corresponding microservices 80A and 80C from the application (IF-THEN) engine 90.

[0208] Therefore, the void setSendDataFormat(String format) method is required as an API provided by the microservice control template class (BaseIms Class) 2010 or the individual device-compatible microservice control class (Customims Class) to specify the data format to be sent from the corresponding method of the microservice control template class (BaseIms Class) 2010 or the individual device-compatible microservice control class (Customims Class).

[0209] The data type (format / type) of the `format` argument of this API is set to String. In other words, the data format to be sent in text format is specified by the application (IF-THEN) engine 90.

[0210] In this case, the above "a) Reporting the results of detecting (finding) high-fever individuals" does not require a large amount of data. In this case, the data transmission process uses long send_data(Map map), and the information that "how many high-fever individuals were detected (found)" is returned to the application (IF-THEN) engine 90. Even if no high-fever individuals were detected, it is acceptable to return "0 high-fever individuals were detected (found)". The Map specified in this API refers to a data format in which "key-value pairs are collected".

[0211] In other words, when the application (IF-THEN) engine 90 requests the corresponding microservices 80A and 80C to "a) report the results of detecting (finding) people with high fever," it first issues the API command void setSendDataFormat(String format) to execute the "image recognition process to calculate the number of people with high fever." Then it issues the API command long send_data(Map map) to send "a) report the results of detecting (finding) people with high fever."

[0212] If a person with a high fever is detected (discovered) here, the application (IF-THEN) engine 90 requests the corresponding microservices 80A and 80C to "b) provide the information necessary to notify the user". At this time, the API command void setSendDataFormat(String format) is also issued, but the difference in the request content is specified in text format in the argument format.

[0213] As can be easily seen from the example use case in Figure 6, the format of “b) Information necessary for notifying the user” may include stream data such as video and still images. Furthermore, there are many different data compression methods for stream data such as video, still images, and audio information. Therefore, the data type, data compression method, the content of the required stream data, and its display format (for example, highlighting only those with high fevers with diagonal lines) are specified in the `format` argument of `void setSendDataFormat(String format)` issued by the application (IF-THEN) engine 90.

[0214] For example, when requesting video information, the application (IF-THEN) engine 90 issues the API command Stream send_data(Stream movie). Here, Stream means "data in stream format," and the video information is stored in movie.

[0215] On the other hand, to request information about a still image, you issue the API command `Stream send_data(Stream image, long time)`. In this case, the still image information is stored in `image`. Here, the `time` parameter allows you to set the time at which the still image information is captured (imported).

[0216] As can be easily seen from the use case example in Figure 6, images (video or still images) may be used to notify users. If these images for user notification are generated on the application (IF-THEN) engine 90 or the microservices 80H and 80F related to the detection operation setting unit 140, a significant load will be placed on the generation process.

[0217] As in this embodiment, by having the microservices 80A and 80C related to condition (IF) 72 (Figure 11) perform "b) providing information necessary for notifying the user," the processing load on the high fever detection device (or the entire high fever detection system) can be reduced.

[0218] Figure 16 shows a list of APIs provided by the device control template class (Base Device Class) or the custom device control class (Custom Device Class) (commands are issued from the application (IF-THEN) engine 90). The notation rules are the same as in Figure 15.

[0219] Application rules 70 basically consist of a combination of predetermined executions (THEN) 78 that correspond to predetermined conditions (IF) 72. For example, just as the temperature, humidity, and illuminance of the user environment change moment by moment, the recognizable entities / states / information 7 that are the subject of this condition (IF) 72 judgment change moment by moment. In order to accommodate this time change, in this embodiment, the recognizable entities / states / information 7 can be recorded chronologically on a "HashMap" corresponding to a "key / value database".

[0220] For example, if you want to acquire the currently recognizable entity / state / information 7 (i.e., acquire sensor data obtained from, for example, a sensor device 802), you call “HashMap createSensorData()” from the microservice control template class (BaseIms Class) 2010 to “generate sensor data”. On the other hand, if you want to “generate sensor data” at a specified time, you set the specified time for obtaining the data in the “time” argument of “HashMap createSensorData(long time)”. Also, if you want to acquire data (= “generate sensor data”) at specific time intervals, the program in the microservice control template class (BaseIms Class) 2010 automatically calculates the data acquisition time corresponding to the specific time interval. Then, you sequentially specify the automatically calculated time in the “time” argument of “HashMap createSensorData(long time)”.

[0221] The sensor data (contents of recognizable entities / states / information 7) accumulated in chronological order is then sent from the device control template class (Base Device Class) 2002 to the microservice control template class (BaseIms Class) 2010 using "long sendSensor(Map map)".

[0222] Figure 16 shows, as an example, a list of control classes for the base device class 2002 corresponding to the sensor device 802. However, as shown in Figure 7, control of various robots (or drive mechanisms) included in the mobile terminal device 808, HMD device 810, and IoT device 820 is also required. Therefore, the processing overview 2304 is not limited to what is described in Figure 16 and may include, for example, "execute drive" or a pre-specified "display screen / video".

[0223] Furthermore, when various devices 8 are located outside of the edge computer 6 or cloud server 2, communication control between the device 8 and the corresponding microservice 80 becomes necessary. Although the explanation of the source code of the control method shown in Figure 16 is omitted here, the control necessary for communication control may also be written in the source code. For example, by incorporating the "Socketlmpl Class" from the "java(registered trademark).net" package 76, communication control to device 8 becomes possible at the IP address level. Specifically, import java.net.Socketlmpl; is written before the device control template class (Base Device Class). This makes it possible to use various methods of the Socketlmpl Class that perform basic communication control.

[0224] The HashMap setDeviceData(HashMap map, long time) function, which corresponds to the void setSendDataFormat(String format) function in Figure 15, is used to configure sensor data from the microservice control template class (BaseIms Class) 2010 or the individual device-specific microservice control class (CustomIms Class) 2110. The argument map used at this time specifies the type of sensor data, such as "collect video?" or "collect still images?". The argument time indicates the time at which the still image is captured (acquired).

[0225] If you want to collect video, first issue the command Stream createSensorData() to have the corresponding sensor 802 or infrared camera 806 collect video. If you want to stop the collection of data from sensor 802 or infrared camera 806 midway through, issue the API command void removeCreateSensorData() to stop (cancel) the generation of sensor data. After that, issuing Stream sendSensor(Stream movie) will make the collected video available for use in other programs (methods) of the microservice control template class (BaseIms Class) 2010 or the individual device-specific microservice control class (CustomIms Class) 2110.

[0226] To collect still images, first issue the command Stream createSensorData(long time) to have the corresponding sensor 802 or infrared camera 806 collect still images. After that, issuing Stream sendSensor(Stream image, long time) will make the collected still images available for use in other programs (methods) of the microservice control template class (BaseIms Class) 2010 or the individual device-specific microservice control class (CustomIms Class) 2110.

[0227] The state transitions of device 8 in this embodiment will be explained using Figure 17. Five states are defined for device 8: stopped state 2400, operating state 2402, running state 2408, ready state 2410, and error state 2414.

[0228] Immediately after control of device 8 begins at 2300, it is in a stopped state at 2400. Then, when "createDevice()" (see the 14th item in Figure 15), which signifies the activation of device control by microservice 80, is executed at 2502, device 8 moves to an operational state at 2402. If the response to "createDevice()" at 2506 fails at 2510, it enters an error state at 2414, and returns to the stopped state at 2400 when the ifLink application is disconnected at 2500. Also, even when device 8 is in the operational state at 2402, if the ifLink application is disconnected at 2500, device 8 returns to the stopped state at 2400.

[0229] On the other hand, if the response 2506 to “createDevice()” is successful 2512, device 8 enters the ready state 2410. Then, when an operation instruction 2508 arrives from the application (IF-THEN) engine 90, the device starts operating and enters the running state 2408. If a stop instruction 2504 arrives from the application (IF-THEN) engine 90, the device stops operating and device 8 returns to the ready state 2410. Also, regardless of whether device 8 is in the ready state 2410 or the running state 2408, if the ifLink application is disconnected, device 8 enters the stopped state 2400.

[0230] If device 8 fails to operate during execution (2408), a retry of the device operation (2530) is required, and device 8 enters an error state (2414). Similarly, if there is no response from device 8 for an extended period during execution (2408), a timeout (2520) occurs, and device 8 enters an error state (2414).

[0231] Module 9, which implements a predetermined function, separates the device 8 (or a human-recognizable entity / state / information 7) from the microservice 80 (or control means 4) that controls it. This allows for the separate management of the state of device 8 and the state of microservice 80. On the other hand, during the execution of a service to a user, the user may change the content of the application (IF-THEN) rule 70 (i.e., the device 8 operated by the application (IF-THEN) rule 70 may suddenly change). By separating and managing the state of device 8 and the state of microservice 80 in this way, it becomes easier to make real-time changes in response to user changes to the application (IF-THEN) rule 70.

[0232] As shown in Figure 18, in this embodiment, six states can be defined for the microservice 80: quiescent state 2400, connected state 2412, operational state 2402, device control state 2404, start state 2300, and error occurrence state 2414. Then, as shown in Figure 18, a relationship is established between the state of the microservice 80 and the state of the device 8. Figure 19 shows the relationship between the source temperature and spectral radiance of the radiation emitted from a blackbody radiation source. It is known that all objects with a given temperature emit electromagnetic waves (blackbody radiation) corresponding to that temperature. Therefore, the temperature of a blackbody radiation source can be determined by measuring the spectral characteristics of the blackbody radiation emitted from it.

[0233] Figure 19 shows that when only electromagnetic waves of a predetermined wavelength 14 are extracted, the radiance changes with the temperature of the blackbody radiation source. Thermography technology extracts only electromagnetic waves of a predetermined wavelength 14 from the radiation emitted from the object being measured, and measures the temperature of the object from the intensity of these extracted electromagnetic waves.

[0234] When measuring the scalp surface temperature of a person using this thermographic technique, the blackbody radiation is scattered by hair in the optical path. Therefore, measuring the scalp surface temperature in areas of the head where hair is densely distributed (such as the back of the head) significantly reduces the accuracy of the measurement (the temperature measured is lower than the actual temperature due to the effect of light scattering loss by hair). For this reason, when measuring the scalp surface temperature of a person using thermographic technique, it is desirable to measure on the face where hair distribution is sparse.

[0235] Furthermore, the electromagnetic wave intensity of a predetermined wavelength 14 detected by the infrared camera 806 is inversely proportional to the square of the distance from the object being measured to the infrared camera 806. Therefore, in surface temperature measurement of an object being measured using thermography technology, the detected temperature decreases as the distance from the object to the infrared camera 806 increases.

[0236] When accurately measuring the body temperature of a person's head using thermographic technology, the following points should be considered: 1) measurement of the face (the front of the head, where the scalp is exposed), 2) consideration of the relationship between the object being measured and the infrared camera 806, 3) the relationship between moisture absorption (the effect of sweat and humidity on the facial surface) and the wavelength used for temperature measurement, and 4) the removal of the effect of the temperature-dependent dark current of the infrared camera 806 itself.

[0237] As previously explained, based on the characteristics shown in Figure 19, it is desirable to use light in the wavelength range of 2.5 μm to 20 μm for measuring human body temperature. Within this range, large water absorption peaks are observed near central wavelengths of 2.955 μm and 6.100 μm. Furthermore, in the wavelength range longer than 6.9 μm, the amount of water absorption tends to increase as the wavelength increases.

[0238] Here's a concrete example of light absorption by actual water molecules. When light with a wavelength of 2.955 μm passes through a water film 0.3 μm thick, 80% of the light is attenuated (i.e., only 20% of the incident light is transmitted). Many people have likely experienced sweating profusely, with the sweat turning into water droplets and dripping. Considering that the diameter of these droplets is about 1-2 mm, you can see how large the amount of light absorbed is.

[0239] On the other hand, the wavelength-dependent characteristics of light absorption by water molecules are minimized around wavelengths of 3.75 μm and 5.30 μm. Therefore, the desired value for the predetermined wavelength 14 used in thermography is around 3.75 μm or 5.30 μm. Furthermore, considering the light absorption characteristics of water molecules as described above, it is desirable to use light in the wavelength range of 3.3 μm to 5.7 μm for measuring human body temperature. If light in this wavelength range is used as the predetermined wavelength 14 for thermography, body temperature can be measured with relatively little influence from sweat or humidity in the air.

[0240] For even more precise body temperature measurement, the detection temperature may be measured using two different wavelengths of light. By combining the detection temperature characteristics at these two wavelengths with the wavelength-dependent characteristics of water molecule light absorption described above, the water molecular weight absorbed along the infrared light path can be estimated. By correcting the detection temperature using this estimated water molecular weight value, an accurate body temperature can be calculated.

[0241] It turns out that the dark current (a pseudo-detection current that flows when blackbody radiation is blocked) of the infrared camera 806 is surprisingly large. Therefore, to accurately measure a person's body temperature, it is necessary to subtract the dark current value from the detection current value. Here, the value of this dark current changes with the temperature inside the infrared camera 806.

[0242] For measuring the dark current value of the infrared camera 806, it is undesirable to block light from the entrance of the infrared camera 806. Blackbody radiation from a light-shielding material at room temperature enters the infrared camera 806, making it impossible to measure the dark current value. It is desirable to frequently measure the dark current value by pointing the infrared camera 806 at a sufficiently distant object in between measuring a person's body temperature. Since the brightness of blackbody radiation is inversely proportional to the square of the distance between the infrared camera 806 and the sufficiently distant object, almost no blackbody radiation from the sufficiently distant object enters the infrared camera 806. In this state, the amount of signal (current value) obtained from the infrared camera 806 is approximately equal to the value of the dark current.

[0243] Figure 20 shows the internal structure of a three-lens infrared camera, which is an application example of this embodiment. Here, "three lenses" refers to a total of "three lenses": one lens 32 (one lens) that focuses infrared light 38 onto the infrared light sensor 34, and two lenses 42 and 44 (two lenses) that focus visible light 40 onto the visible light sensors 46 and 48.

[0244] In an infrared camera 806 that incorporates an infrared light sensor 34 having infrared light 38, a bandpass optical filter 24 is positioned directly in front of the lens 32 that focuses the infrared light 38. As described above, this bandpass optical filter 24 is designed to pass only light in the wavelength range of 3.3 μm or more and 5.7 μm or less (or light near a wavelength of 3.75 μm or light near a wavelength of 5.30 μm).

[0245] The detection signal (first signal) obtained from the infrared light sensor 34 is transferred to the temperature distribution video / image generation unit 36. The temperature distribution video / image generation unit 36 ​​then generates spatially detected temperature distribution characteristics.

[0246] Furthermore, the three-lens infrared camera 806 shown in Figure 20 has a built-in function to compensate for the drop in detected temperature that occurs due to the reasons mentioned above (changes between the measurement target 20 and the infrared camera 806). Specifically, two of the three lenses (lenses 42 and 44) ​​corresponding to the visible light 40 measure the L to the measurement target (person being measured), and the detected temperature is corrected according to the measured L.

[0247] In other words, in the triple-lens infrared camera 806, the two lenses 42 and 44 that allow visible light 40 to pass through are positioned far apart from each other. Consequently, the image formation position relative to the measurement target 20 is shifted between the visible light sensors (image sensors) 46 and 48. By detecting this shift, the distance L from the measurement target 20 to the triple-lens infrared camera 806 can be calculated based on "trigonometry".

[0248] Specifically, the visible video / visible image generation units 52 and 54 use the detection signals (second signals) obtained from each visible light sensor (image sensor) 46 and 48 to generate imaging patterns for the measurement target 20. Then, the distribution calculation unit 62 for each measurement target examines the amount of difference between the two imaging patterns and calculates L from the measurement target 20 to the triple-lens infrared camera 806.

[0249] As mentioned above, the brightness of the blackbody radiation emitted from the object being measured 20 is inversely proportional to the square of L. Therefore, the temperature distribution correction unit 60 uses the value of L obtained here to correct the spatially detected temperature distribution characteristics described above and calculates the actual body temperature.

[0250] The temperature distribution correction unit 60 only provides the "corrected temperature distribution." It is difficult to identify the measurement target 20 from the temperature distribution characteristics alone. Therefore, the synthesis unit 68 overlays the contours and shape images of the measurement target 20, such as eyes, nose, and mouth, obtained by imaging with visible light, onto the "corrected temperature distribution." This synthesized image is then output to the temperature distribution output terminal 58.

[0251] Here, the contour and shape images of eyes, nose, and mouth that are superimposed on the "corrected temperature distribution" are obtained from the visible image / visible image contour shape extraction unit (including the contour of the eyes) 66. In particular, the center position of these contour and shape images of eyes, nose, and mouth must coincide with the center position of the "corrected temperature distribution". As can be seen from Figure 20, the position of the lens 32 of the infrared camera 806 and the positions of the lenses 42 and 44 of the normal cameras 816-1 and 816-2 are misaligned. Therefore, it is necessary to correct for the above-mentioned misalignment of the center position.

[0252] The combined visible light planar image / image generation unit 64 generates a visible image with corrected center position shift. This utilizes the images obtained by the visible image / visible image generation units 52 and 54, and the information L obtained from the distribution calculation unit 62 for each measurement target. The resulting visible image is then output from the visible image / visible image output terminal 56.

[0253] The correspondence between Figure 20 and Figure 2, which was explained earlier, is described below. The normal camera 816 in Figure 2 corresponds to the normal cameras 816-1 and 816-2 in Figure 20, and Figures 2 and 20 have the same infrared camera 806. Also, the visible video / visible image generation units 52 and 54 in Figure 20, the distribution calculation unit 62 for each measurement target, and the combined visible light planar video / image generation unit 64 correspond to a part of the normal camera image recognition unit 112 in Figure 2. Furthermore, the temperature distribution video / image generation unit 36 ​​in Figure 20, the temperature distribution correction unit 60, and the synthesis unit (temperature distribution with visible light contours also displayed) 68 correspond to the infrared image recognition unit 116 in Figure 2.

[0254] Furthermore, the three-lens infrared camera 806 shown in Figure 20 corresponds to the infrared camera 806 in Figure 7. The human head temperature determination unit 125 in Figure 2 corresponds to a part of the infrared camera microservice 80C in Figure 7. In addition, the detection operation setting unit 140 in Figure 2 corresponds to the mailer microservice 80H, HMD microservice 80E, individual device microservice 80F, etc. in Figure 7.

[0255] In Figure 2, the standard camera image recognition unit 112 was described as performing advanced "image recognition" ranging from "1) image analysis" to "2) person detection," "3) number of people detection," "4) head detection," and "5) detection." It was then explained that this advanced "image recognition" is performed by the AI-based image recognition class (AI Analyze Class) 2018 (Figure 13) of the sensor microservice 80A.

[0256] Figure 21 shows an example of the interaction between classes in the class structure (software architecture) of the sensor microservice 80A described in Figure 13. First, the Customims Class imports (embeds) the Custom Device Class and the AI ​​Analyze Class (processes 74 and 76). This makes it possible to use the methods defined (written) in the Custom Device Class and the AI ​​Analyze Class in the Customims Class.

[0257] Then, in step 44 (S44), the API command void setSendDataFormat (String format) issued by the application (IF-THEN) engine 90 specifies the data format to be sent. If the format argument contains a request for the number of people obtained after recognition (analysis) of the image signal from the normal camera 816, then image recognition processing using the AI ​​Analyze Class method is started.

[0258] Before starting the image recognition process, it is necessary to acquire an image signal from the camera 816 (which corresponds to either a video signal or a still image signal, and is the second signal). Therefore, the HashMap SetDeviceData() method is called in the Customims class. Then, the corresponding method pre-written in the Custom Device class is executed (S45), and the acquisition of an image signal using the camera 816 begins.

[0259] In step 46 (S46), an image signal is acquired using the normal camera 816. Next, when Stream sendSensor() is called in the Customims Class, the image signal acquired by the normal camera 816 is read into the Customims Class. The image signal read here may be saved in buffer memory 2008 (Figure 13) (S48).

[0260] The Customims Class calls a pre-defined corresponding method in the AI ​​Analyze Class (S49), saving the image signal to buffer memory 2008, and the image recognition processing side reads the image signal (S50). Similarly, the Customims Class calls a pre-defined corresponding method in the AI ​​Analyze Class to instruct the calculation of the head positions of people and the calculation of the resulting number of people (S51). The pre-defined corresponding method in the AI ​​Analyze Class then operates, and data analysis (image recognition) S52 is executed. Once the execution of the above corresponding method is complete, the number of people obtained as a result of the data analysis (image recognition) is known (S53).

[0261] In step 54 (S54), a determination is made based on the calculated number of people to determine whether or not it is necessary to obtain the temperature of their heads. Up to this point, the processing flow for the sensor microservice 80A that controls the camera 816 has been described.

[0262] The head temperature extraction process in step 55 (S55) and step 56 (S56) is performed by the coordinated processing of the sensor microservice 80A and the infrared camera microservice 80C, or by processing in the infrared camera microservice 80C using the image recognition results from the sensor microservice 80A.

[0263] FIG. 22 shows an example of an application (IF-THEN) rule setting method for simultaneously notifying multiple media when a person with a high fever is detected. In FIG. 22, not only various videos and various still images 128 obtained from the three-eye infrared camera 806, but also the position information 138 obtained from the GPS position sensor 830 are used as detection signals.

[0264] Here, the setting of the notification destination (media to be used for notification) when a person with a high fever is detected is defined (described) by the application (IF-THEN) rule 70 stored in the memory area 71 of the application (IF-THEN) engine 90.

[0265] In the example of FIG. 22, notifications are sent to both email communication / email transmission 1250 and warning screen display 1230. Here, not only the fact that "a person with a high fever has been detected" is important, but it is also necessary for the user who receives the notification to be able to "identify the detected person with a high fever". In the example of the warning screen display 1230 in FIG. 6, "the person on the right end is shaded" and can be identified. In this way, when "visualization using an image" is performed at the time of notification to the user, there is an effect that the user who receives the notification can easily identify the person with a high fever (the target to be noted).

[0266] For this "visualization" at the time of notification, the high fever person identification image 136 may be sent to the screen display microservice 80K that controls the warning screen display 1230. Also, the high fever person face image 132 may be sent to the mailer microservice 80H that controls email communication / email transmission 1250. Here, the user who receives the email is likely to be in a location far from the site. Therefore, the high fever person position information 134 obtained from the position information 138 acquired from the GPS position sensor 830 may be sent to the mailer microservice 80H.

[0267] What is particularly important here is that "the information 132 to 136 to be sent to the microservices 80H and 80K on the notification side (i.e., related to the execution (THEN) 78 in FIG. 11) is made and sent by the microservices 80C and 80J on the sensor side (i.e., related to the condition (IF) 72 in FIG. 11)". When the information required for notifying the user (i.e., the execution (THEN) 78 in FIG. 11) is created by the microservices 80C and 80J on the sensor side (i.e., related to the condition (IF) 72 in FIG. 11), it has the effect of significantly improving the processing efficiency inside the high fever detector (or the entire high fever detection system).

[0268] An example of the notification information at the time of detecting a high fever person is composed of a "warning message" and a "display image that is easy to identify a high fever person" as shown in the display screen 12 of FIG. 6. Also, the content of the above "warning message" can often be formatted in a fixed form. Therefore, as a method of "providing information to the user" in the processing performed by the execution part (THEN) 78 (FIG. 11) of the application (IF-THEN) rule 70, standard templates (fixed forms) can be used frequently. Therefore, among the microservices 80 related to "providing information to the user", the process of "creating a standard format using an existing template" can be implemented.

[0269] FIG. 23 shows the class configuration (software architecture) of the microservices related to information provision. The basic control package 2000 related to information provision includes a standard format generation class (FillFormat Class) 2019 that uses an existing template. And here, a set of a "fixed text" (for example, the above "warning message", etc.) and a "display image" (for example, the above "display image that is easy to identify a high fever person", etc.) regarding the information to be notified to the user is generated.

[0270] The created format 77, generated using a standard format generation class (FillFormat Class) 2019 (by a specified method) that utilizes this existing template, is temporarily stored in a buffer memory for saving created formats. This temporarily stored created format 77 is then displayed to the user on the screen 1230 via the device control template class (Base Device Class) 2002 or the individual device control class (Custom Device Class) 2101.

[0271] Figure 24 illustrates the method of communicating information to the user using the information provision (automatic notification) related microservices shown in Figure 23. Using Figure 22, it was explained that "information 132-136 to be sent to the notification-side microservices 80H and 80K (i.e., related to the execution (THEN) 78 in Figure 11) is created and sent by the sensor-side microservices 80C and 80J (i.e., related to the condition (IF) 72 in Figure 11)." Here, the specific processing flow is explained.

[0272] Steps 24 (S24) through 29 (S29) in Figure 24 show the process of creating and receiving the necessary information from the sensor-side microservice 80C (i.e., related to condition (IF) 72 in Figure 11). Then, steps 30 (S30) through 31 (S31) show the process of sending information created by the sensor-side microservice 80C to the notification-side microservices 4D, 80H, and 80K (i.e., related to execution (THEN) 78 in Figure 11) to perform notification processing.

[0273] In this embodiment, notification to the user is only required after a person with a high fever is detected. Therefore, the first step is to have the sensor-side microservice 80C (i.e., related to condition (IF) 72 in Figure 11) respond whether condition (IF) 72 is met, i.e., whether a person with a high fever has been detected (S27). If a person with a high fever is detected, the sensor-side microservice 80C (i.e., related to condition (IF) 72 in Figure 11) creates and sends the necessary information 128 to the notification-side microservices 4D, 80H, and 80K (i.e., related to execution (THEN) 78 in Figure 11) (S29).

[0274] In steps 24 (S24) and 28 (S28), the application (IF-THEN) engine 90 issues the API command void setSendDataFormat (String format) (Figure 15) to the sensor-side microservice 80C. This instructs the sensor-side microservice 80C to switch the content of the information it creates / sends.

[0275] While embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications are permitted without departing from the spirit of the invention. These embodiments, their variations, and combinations of embodiments are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as described in the claims. [Explanation of Symbols]

[0276] 2...Cloud servers, 6...Edge computers, 70...Application (IF-THEN) rules, 80...Microservices, 90...Application (IF-THEN) engine, 92...ifLink apps.

Claims

1. An edge computer capable of communicating with multiple devices, including IoT devices, and a cloud server, An application engine for executing multiple application rules that have been pre-loaded from the aforementioned cloud server, The system comprises a plurality of microservices, each individually provided and operating independently of the others, for applying the application rules to each of the plurality of devices and the cloud server, The aforementioned application rule is based on the principle of executing (THEN) in response to a condition (IF). The aforementioned application engine is When using the first application rule, the first microservice and the second microservice are linked by giving the first API command as the condition to the first microservice corresponding to the first device among the plurality of devices and obtaining its return value, and giving the second API command as the execution to a second microservice corresponding to a second device different from the first device or the cloud server based on the return value, Furthermore, the application engine implements multiple functions, A rule enable / disable function that allows switching the execution (THEN) on or off, An edge computer characterized by having an execution time control function that can shift the execution timing of a specific execution (THEN).

2. The edge computer according to claim 1, wherein the first microservice is connected to a buffer memory for temporarily storing video data, and further includes an AI class connected to the buffer memory for performing video data analysis processing, and sends the results of the video data analysis processing by the AI ​​class to the application engine, thereby reducing the processing load on the application engine when it processes the application rules.

3. The edge computer according to claim 1, wherein the first application rule is a rule that the application engine has received in advance via a third microservice, after the consistency between a plurality of different application rules has been verified on the cloud server.

4. The edge computer according to claim 1, characterized in that it is portable and has telephone functionality.

5. A control system for an edge computer capable of communicating with multiple devices, including IoT devices, and a cloud server, An application engine for executing multiple application rules that have been pre-loaded from the aforementioned cloud server, The system comprises a plurality of microservices, each individually provided and operating independently of the others, for applying the application rules to each of the plurality of devices and the cloud server, The aforementioned application rule is based on the principle of executing (THEN) in response to a condition (IF). The aforementioned application engine is When using the first application rule, the first API command as a condition is given to the first microservice corresponding to the first device among the multiple devices and its return value is obtained, and based on the return value, the second API command as execution is given to the second microservice corresponding to a second device different from the first device or the cloud server, thereby coordinating the multiple microservices and the second microservice. Furthermore, the application engine implements multiple functions, The rule enable / disable function allows switching the execution (THEN) on or off. The execution time control function allows for shifting the execution timing of a specific execution (THEN). An edge computer control system equipped with these features.

Images