Information collecting device, control method, and program

Abstract

To provide an information collecting device which avoids cost increase caused by increase of an amount of information to be processed, and stop of process caused by acquisition of the information to be processed, and a control method and a program thereof.SOLUTION: An imaging device which is an information collecting device for executing an inference process using a learned model, and includes: a temperature measurement unit which acquires temperature information at the imaging; an imaging unit which acquires a thermal image obtained by photographing far-infrared light being an object of the inference process; an inference processing unit which inference processes, by applying to a learned model, a thermal image and a learned parameter being set on the basis of the temperature information; and a control managing unit which sets the learned parameter on the basis of a range of the temperature of the learned parameter. When the temperature information is in a range of the temperature of the learned parameter stored in a non-volatile memory, the control managing unit selects a learned parameter on the basis of the range of the temperature of learned parameter of the temperature information. When the temperature information is not in the range of the temperature of the learned parameter, the control managing unit acquires the learned parameter from an external device such as an external server or other devices so that the temperature information is in the range.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to AI (artificial intelligence) processing according to the usage environment of an edge device. [Background technology]

[0002] In the field of IOT (Internet of Things), information gathering devices such as surveillance cameras are connected to networks such as the Internet as edge devices, and the information gathered by the edge devices is applied to a trained model in the edge device to perform AI (artificial intelligence) processing, with the results being sent to a cloud server, etc.

[0003] Patent Document 1 describes switching of trained models in edge devices depending on the usage environment of the edge devices, while Patent Document 2 describes recognition of people by AI processing from thermal images captured by infrared cameras.

[0004] When recognizing a subject using AI processing from a thermal image taken with an infrared camera as in Patent Document 2, the surface temperature of the subject is affected by the environment in which the infrared camera is used (such as outside air temperature and ambient temperature), so the environment in which the infrared camera is used must be taken into consideration.

[0005] AI processing that takes into account the usage environment of the edge device in this way is called multimodal AI processing, which combines and processes multiple types of information, such as thermal images and the usage environment of the edge device, as opposed to single-modal AI processing, which processes one type of information, such as thermal images collected by the edge device. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2020 / 105161 [Patent Document 2] JP 2022-035519 A Summary of the Invention [Problem to be solved by the invention]

[0007] Multimodal AI processing increases the processing load because more information is input to the trained model. For example, if the information about the usage environment of the edge device is temperature information at the time of shooting a video, the temperature information needs to be acquired asynchronously for each frame of the video, which increases the processing load.

[0008] In addition, storing trained models and trained parameters corresponding to various usage environments in an edge device requires a large storage capacity, which leads to increased costs. As in Patent Document 1, trained models corresponding to the usage environment of the edge device can be downloaded from an external server, but when changing the trained model, it is necessary to stop the AI ​​processing.

[0009] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize a technology that can avoid increased costs due to an increase in the amount of information used in AI processing and the suspension of AI processing due to the acquisition of information to be used in AI processing, in AI processing that takes into account the usage environment of edge devices. [Means for solving the problem]

[0010] In order to solve the above problems and achieve the object, the present invention provides an information collecting device that performs inference processing using a trained model, comprising: a communication means for communicating with an external device; a first acquisition means for acquiring first information regarding a usage environment of the information collecting device; a second acquisition means for acquiring second information to be subjected to the inference processing; a processing means for applying the second information and third information set based on the first information to the trained model to perform the inference processing; and a control means for setting the third information based on a first threshold range of the first information, wherein if the first information is within a second threshold range, the control means selects the third information based on the first threshold range of the first information, and if the first information is not within the second threshold range, the control means acquires the third information from the external device via the communication means so that the first information is within the second threshold range. Effect of the Invention

[0011] According to the present invention, in AI processing that takes into account the usage environment of an edge device, it is possible to avoid increases in costs due to an increase in the amount of information used in the AI ​​processing and the suspension of AI processing due to the acquisition of information to be used in the AI ​​processing. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing an apparatus configuration according to a first embodiment. [Diagram 2] FIG. 4 is a diagram illustrating learned parameters for each temperature range according to the first embodiment. [Diagram 3] FIG. 4 is a diagram illustrating the configuration of a threshold table according to the first embodiment. [Figure 4] 4 is a flowchart showing a control process according to the first embodiment. [Diagram 5] 4 is a flowchart showing a learned parameter switching determination process according to the first embodiment. [Figure 6] 4A to 4C are diagrams for explaining a method for determining whether to update a threshold table and a method for determining whether to change a learned parameter according to the first embodiment. [Figure 7] 4 is a flowchart showing a learned parameter replacement process according to the first embodiment. [Figure 8] FIG. 4 is a graph illustrating the relationship between the temperature and the resistance value of the NTC thermistor according to the first embodiment. [Figure 9] 6 is a flowchart showing a process for changing the temperature acquisition period according to the first embodiment. [Figure 10] FIG. 11 is a block diagram showing the configuration of an apparatus according to a second embodiment. [Figure 11] 10 is a flowchart showing a control process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] [Embodiment 1] In the first embodiment, an example will be described in which an information collection device of the present invention is applied to an imaging device capable of taking still images and moving images.

[0015] In this embodiment, an imaging device equipped with a microbolometer captures a far-infrared image (hereinafter, thermal image), and the captured thermal image and learned parameters corresponding to the usage environment of the imaging device (outside air temperature and environmental temperature) are applied to a learned model that has undergone learning processing by machine learning and deep learning of AI (artificial intelligence), and image recognition processing (hereinafter, AI processing) is performed to output a result such as a subject detection result. Then, in this embodiment, the outside air temperature and environmental temperature (hereinafter, temperature) are determined as the usage environment of the imaging device, and the learned parameters are changed according to the usage environment (for each predetermined temperature range).

[0016] The imaging device of this embodiment is, for example, a surveillance camera that is a type of Web camera called an edge device that has a function of performing AI processing on a thermal image and detecting a person's face from the thermal image and is connected to a network such as the Internet. Note that in this embodiment, the trained model used for the image recognition processing is common, and trained parameters applied to the trained model are changed.

[0017] The imaging device of this embodiment can detect people from captured thermal images by image recognition processing, and can be connected to a network as an edge device to monitor the behavior of people from a remote location.

[0018] <Device configuration> First, the configuration and functions of an image capture device 10 according to the present embodiment will be described with reference to FIG.

[0019] The imaging device 10 of this embodiment can be applied to a web camera or network camera that is communicably connected as an edge device via a network, an in-vehicle camera, a surveillance camera, a medical camera, a smart speaker with a camera function, and the like.

[0020] The imaging device 10 of this embodiment captures far-infrared light images using a thermal detector such as a microbolometer, and generates a thermal image that is the target of AI processing.

[0021] The control unit 100 includes a processor such as a CPU that performs arithmetic processing for controlling the entire imaging device 10, and the processor executes a program stored in a non-volatile memory 101, which will be described later, to realize the control processing described later. The control unit 100 may directly execute the program read from the non-volatile memory 101, or may load the program read from the non-volatile memory 101 into a work memory 102 and execute it in order to speed up processing. Note that instead of the control unit 100 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing.

[0022] The non-volatile memory 101 is an electrically erasable and recordable memory, and for example, a flash ROM is used. Constants, programs, etc. for the operation of the control unit 100 are recorded in the non-volatile memory 101. The program here means a program for executing the control process described later in this embodiment. The non-volatile memory 101 also records thermal imaging data captured by the imaging unit 107 described later and processed by the correction unit 108. The work memory 102 is a memory that can access a large amount of data faster than a flash ROM, and for example, a DRAM is used. The work memory 102 is used as a working area for loading constants, variables, programs read from the non-volatile memory 101, etc. for the operation of the control unit 100. The work memory 102 is also used as a buffer memory for temporarily storing thermal imaging data captured by the imaging unit 107 described later and processed by the correction unit 108, and thermal imaging data being processed in the correction unit 108.

[0023] The temperature measuring unit 103 is a temperature sensor that measures the outside air temperature or the environmental temperature around the imaging device 10 and outputs the measured temperature information to the control management unit 104. For example, a thermistor or an NTC (Negative Temperature Coefficient) thermistor, which is a type of thermistor, is used for the temperature measuring unit 103. The NTC thermistor has a characteristic that the resistance value decreases as the temperature increases, and can be used in a wide temperature range from -50°C to over 200°C, which covers the temperature range according to the use environment of the imaging device 10 of this embodiment. FIG. 8 illustrates the relationship between the detected temperature and the resistance value of an NTC thermistor. If the temperature around the imaging device 10 and the temperature around the subject are almost the same or a correlation between the two can be obtained, the temperature measuring unit 103 may be attached to a housing that constitutes the exterior part of the imaging device 10. In addition, when the temperature measuring unit 103 is attached to the exterior part of the imaging device 10 so as to be exposed to the outside air, it may be covered with a protective member (not shown) in consideration of the influence of direct sunlight and bad weather (rain, dew, hail, frost adhesion), etc. Moreover, the temperature measuring unit 103 may be disposed away from the housing of the image capturing device 10. The temperature measuring unit 103 is not limited to this implementation.

[0024] The control management unit 104 acquires temperature information from the temperature measurement unit 103, and manages control data such as tables and flags when performing AI processing in the inference processing unit 105 described later. The control management unit 104 includes a threshold table 1041 described later in FIG. 3, a parameter change flag 1042 described later in FIG. 4, and a table change flag 1043 described later in FIG. 5, and changes and updates parameters and temperature ranges, sets and resets flags, and the like. The control management unit 104 includes a storage area for storing the threshold table 1041, the parameter change flag 1042, and the table change flag 1043. The table 1041 and the flags 1042 and 1043 may be stored in the non-volatile memory 101 instead of being managed by the control management unit 104, and may be loaded from the non-volatile memory 101 to the work memory 102 during the startup process of the imaging device 10, and the control management unit 104 may access and manage the work memory 102.

[0025] The inference processing unit 105 includes a trained model 1051 and trained parameters 1052. The inference processing unit 105 includes a storage area for storing the trained model 1051 and the trained parameters 1052. The trained model 1051 and the trained parameters 1052 may be stored in the non-volatile memory 101 instead of being managed by the inference processing unit 105, and may be loaded from the non-volatile memory 101 to the work memory 102 during startup processing of the imaging device 10, and the inference processing unit 105 may access the work memory 102 to manage them.

[0026] The trained model 1051 is composed of, for example, a neural network. The inference processing unit 105 applies a thermal image captured by the imaging unit 107 (described later) and trained parameters 1052 set by the control management unit 104 to the trained model 1051 to execute AI processing and output a subject detection result. The subject to be detected is, for example, a person's face, body, eyes, or other organs. The AI ​​processing may be executed by a GPU (Graphics Processing Unit). The GPU is a processor capable of performing processing specialized for computer graphic calculations, and has a calculation processing capacity for performing matrix calculations and the like required for AI processing in a short time. The AI ​​processing may be performed by the control unit 100 and the GPU working together, or may be performed by the control unit 100 or the GPU alone.

[0027] The optical unit 106 includes a lens that transmits far-infrared light (for example, in a wavelength range of 8 μm to 14 μm) and forms an image on the imaging unit 107. Lenses that transmit visible light are mainly composed of silica (SiO2), which absorbs far-infrared light, and therefore germanium or a chalcogen compound is used as a material for lenses that transmit far-infrared light.

[0028] The imaging unit 107 includes a thermal detector such as a microbolometer having a focal plane array (FPA) in which light receiving elements that react to far-infrared light are arranged two-dimensionally. The microbolometer captures a thermal image by converting a temperature rise caused by the light receiving elements receiving far-infrared light into an electrical signal (voltage) as a change in resistance value. The thermal image is captured at a predetermined frame rate (for example, 30 Hz). The number of pixels of the thermal image is VGA size (640 pixels horizontally × 480 lines vertically) or more, which is a sufficient resolution for AI processing.

[0029] Thermal images obtained by a microbolometer reproduce the temperature of the subject's surface as a brightness distribution. The surface temperature of the subject is affected by the outside air temperature and environmental temperature at the time of shooting, so the feature information of the subject in the thermal image varies depending on the temperature around the subject at the time of shooting. For example, in the case of a person's face, the lower the outside air temperature, the darker the nose and cheeks will be, and the higher the outside air temperature, the brighter the entire face will be. When detecting a subject from a thermal image, it is necessary to extract feature information that takes into account the outside air temperature and environmental temperature at the time of shooting.

[0030] The correction unit 108 converts the analog electric signal of the thermal image captured by the imaging unit 107 into a digital signal, and generates thermal image data in which the offset variation and gain variation of each light receiving element of the imaging unit 107 are corrected. The correction unit 108 also performs non-uniformity correction (NUC) for each light receiving element forming a pixel after removing the offset and gain variation of each light receiving element of the imaging unit 107. Since NUC requires calibration by a temperature sensor (not shown), a shutter plate (not shown) that covers the FPA with a surface with a uniform temperature may be provided separately to obtain a thermal image for correction. The contents and order of the processing by the correction unit 108 are not limited to this implementation.

[0031] The timer 109 measures the time during which the control management unit 104 periodically acquires temperature information from the temperature measurement unit 103. The control unit 100 sets the time of the timer 109. The control unit 100 generates a temperature acquisition event for the control management unit 104 according to the set time of the timer 109, and the control management unit 104 acquires temperature information from the temperature measurement unit 103 for each temperature acquisition event. In addition, an acquisition event of a thermal image for temperature calibration of the FPA may be generated according to the time of the timer 109.

[0032] The communication unit 110 includes an interface for communicating with an external server or other external device via a network. The communication method may be a wired method such as Ethernet or a wireless method such as wireless LAN, 4G / LTE, or 5G, but is not limited to this implementation. For example, the imaging device 10 connects to an external server via a network using the communication unit 110, and transmits a frame of a thermal image captured by the imaging unit 107 and a subject detection result obtained by the inference processing unit 105 to the external server or other external device. The external server is, for example, a cloud server, and transmits at least one of a trained model and trained parameters that are not held by the imaging device 10 to the imaging device 10. In addition, for example, the imaging device 10 and an operation device (not shown) may be connected to each other using the communication unit 110, and a user operation for remotely operating the imaging device 10 may be transmitted from the operation device to the imaging device 10.

[0033] In addition, when transmitting thermal imaging data to an external server, etc., since the thermal image has brightness information, a codec (not shown) may be provided in the imaging device 10 to compress and encode the thermal image before transmitting it to the outside. The data transmitted from the imaging device 10 as an edge device is not limited to the present implementation.

[0034] FIG. 2 illustrates the relationship between the trained parameters applied to the trained model of this embodiment and the temperature range in which the trained parameters are changed.

[0035] The learned parameters 201 to 207 are parameters that are applied to each section by dividing the temperature range of 0 to 35°C into multiple sections (for example, 7 sections) at intervals of 5°C. For example, the learned parameter 201 (param_A) is a parameter that is selected in a temperature range from higher than 0°C to lower than 5°C. FIG. 2 illustrates learned parameters in a temperature range t higher than 0°C to lower than 35°C, but in reality, there are learned parameters in a temperature range lower than 0°C or higher than 35°C.

[0036] The imaging device 10 acquires temperature information t from the temperature measurement unit 103 at a predetermined cycle, the control unit 100 sets learned parameters based on the temperature information t, and the inference processing unit 105 executes AI processing. As shown in Fig. 2, the control management unit 104 includes a threshold table 1041 in which a predetermined number of learned parameters are associated with a temperature range to be applied to each learned parameter.

[0037] FIG. 3 illustrates an example of the configuration of the threshold table 1041 provided in the control management unit 104 of this embodiment.

[0038] 3(a) illustrates learned parameters registered in the threshold table 1041 of this embodiment and the temperature ranges to which the learned parameters are applied. In FIG. 3(a), a name assigned to each learned parameter is registered as a label in column 301. In column 302, the temperature ranges to which each learned parameter in column 301 is applied are described.

[0039] In this embodiment, some of the learned parameters shown in FIG. 3(a) are stored in the work memory 102, and the rest are downloaded from an external server via the communication unit 110. Therefore, in column 303, information indicating whether or not each learned parameter of the label 301 is stored in the work memory 102 is registered. In column 304, when the storage destination of the learned parameter is the work memory 102, address information indicating the storage location is registered. Note that the storage location may be offset information relative to any position information. Furthermore, when the storage destination of the learned parameter is not the work memory 102, information indicating "not exist" or "invalid" is described.

[0040] FIG. 3(b) illustrates the configuration of a threshold table of learned parameters stored in the work memory 102 in the threshold table 1041 in FIG. 3(a). In FIG. 3(b), column 305 is the name of the learned parameter stored in the work memory 102 in the threshold table 1041 in FIG. 3(a). Column 306 is the temperature range applied to each learned parameter in column 305, similar to column 302 in the threshold table 1041 in FIG. 3(a). Column 307 is address information in the work memory 102, which is the storage location of the learned parameter, similar to column 304 in the threshold table 1041 in FIG. 3(a). The table in FIG. 3(b) is updated every time a learned parameter is downloaded, and is composed of only the latest learned parameters stored in the work memory 102.

[0041] Figures 3(a) and 3(b) show an example in which the temperature range to which the learned parameters are applied is higher than 5°C and lower than 30°C, and the learned parameters from either table may be used.

[0042] Next, the control process of the imaging device 10 in this embodiment will be described with reference to FIG.

[0043] FIG. 4 is a flowchart showing the control process of the imaging device 10 of this embodiment.

[0044] 4 is realized by turning on the power of the imaging device 10, causing the control unit 100 to execute a program stored in the non-volatile memory 101 and control each component. The same applies to Figs. 5, 7, 9, and 11, which will be described later.

[0045] In step S401, the control unit 100 executes a system startup process, and proceeds to step S402. The startup process reads out a program stored in the non-volatile memory 101, and configures the work memory 102 and the memory controller (not shown). The startup program controls the system clock, and starts and initializes peripherals (interfaces), interrupts, and the timer 109.

[0046] The timer 109 generates a temperature acquisition event for acquiring temperature information by the control management unit 104. A preset initial value may be applied to the temperature acquisition event.

[0047] In step S402, the control management unit 104 acquires an initial value of the temperature information from the temperature measurement unit 103, and the process proceeds to step S403. The initial value of the temperature information may be acquired without using a temperature acquisition event of the timer 109.

[0048] In step S403, the inference processing unit 105 performs initial setting of the trained model 1051, and proceeds to step S404. The inference processing unit 105 may download an AI processing program from an external server via the communication unit 110. The inference processing unit 105 may also download trained parameters 1052 corresponding to a predetermined temperature range from an external server via the communication unit 110, and store the trained parameters 1052 in the work memory 102. In this case, the temperature information acquired in step S402 may be set to the center of the preset temperature range. The inference processing unit 105 may also select trained parameters corresponding to the temperature range that includes the temperature information acquired in step S402, and set the trained parameters 1052. The control management unit 104 updates the threshold table 1041 to reflect the trained parameters stored in the work memory 102 in step S403.

[0049] In step S404, the control unit 100 performs settings for capturing a thermal image for the imaging unit 107 and the correction unit 108, and proceeds to step S405. The settings include setting the power-on time (exposure time) and power-off time of the imaging unit 107 according to the thermal time constant of the microbolometer, generating a synchronization signal, and setting parameters related to the correction process of the correction unit 108. In addition, the settings include obtaining a thermal image for calibration of the FPA, obtaining parameters for the NUC, and the like. Note that a detailed description of the calibration of the FPA is omitted, but it is not limited to a specific process, and various processes can be applied. Also, distance measurement may be performed in the same way as with visible light. Note that a detailed description of the control of the optical unit 106 is omitted, but it is not limited to a specific process, and various processes can be applied.

[0050] In step S405, the control unit 100 determines whether or not to stop capturing thermal images by the imaging unit 107 and the correction unit 108. If the control unit 100 determines to stop capturing thermal images, it stops the operation of the imaging unit 107 and repeats the determination process of step S405. If the control unit 100 determines not to stop capturing thermal images, it proceeds to step S406. Conditions for stopping capturing images include, for example, network disconnection and abnormal heat generation of the imaging device 10, but may be other conditions.

[0051] In step S406, the control unit 100 determines whether or not to acquire a thermal image for temperature calibration of the FPA. If the control unit 100 determines to acquire a thermal image for temperature calibration of the FPA, the process proceeds to step S407. If the control unit 100 determines not to acquire a thermal image for temperature calibration of the FPA, the process proceeds to steps S411 and S420.

[0052] In step S407, the correction unit 108 acquires a thermal image for temperature calibration of the FPA, calibrates the parameters for NUC, and proceeds to step S418. The determination as to whether or not to perform temperature calibration of the bolometer may be made when a change in temperature information acquired from the temperature measurement unit 103 in response to a temperature acquisition event of the timer 109 becomes equal to or greater than a threshold value. In this case, a shutter plate (not shown) covering the FPA may be closed to capture a thermal image of a uniform temperature surface. In step S407, correction parameters are calculated so that the output of the FPA becomes uniform.

[0053] The processes enclosed by the thick lines (steps S411 to S413 and steps S420 to S423) are executed in parallel. When the parallel processes are completed, the process of step S417 is started.

[0054] In step S411, the imaging unit 107 converts the captured thermal image into a digital signal, outputs the digital signal to the correction unit 108, and the process proceeds to step S412. The imaging timing may be synchronized with a timing generator (not shown) that generates a periodic synchronization signal.

[0055] In step S412, the correction unit 108 corrects the offset components and gain component variations resulting from analog circuits such as AD conversion for the thermal imaging data output from the imaging unit 107, performs NUC, and proceeds to step S413. Note that the correction process in step S412 may be started from a part of the captured thermal imaging data without waiting for the completion of capturing the thermal image in step S411, so that the processes in steps S411 and S412 may be partially overlapped.

[0056] In step S413, the correction unit 108 executes black level clamping to reproduce the black level of the thermal image after correction. This is an offset clamping process for a target black level, but may be executed simultaneously with the correction of variations in the correction unit 108. The process of step S413 may also be executed in a cascade fashion by executing steps S411, S412, and S413 without waiting for the completion of the process of step S412.

[0057] The processing from steps S420 to S423 is a process of changing the learned parameters applied to the AI ​​processing in the imaging device 10, which is executed by the control management unit 104 and the inference processing unit 105.

[0058] In step S420, the control management unit 104 determines whether or not a request to change the learned parameters has occurred based on the value of the parameter change flag 1042. If the control management unit 104 determines that a request to change the learned parameters has occurred, the process proceeds to step S421. If the control management unit 104 determines that a request to change the learned parameters has not occurred, the process proceeds to step S417. When the parameter change flag 1042 is set to a value of 1, the learned parameters need to be changed, and when the parameter change flag 1042 is reset to a value of 0, the learned parameters do not need to be changed (or the state in which the learned parameters need to be changed has been resolved).

[0059] In step S421, the control management unit 104 locks the value of the parameter change flag 1042 determined in step S420 so that it is not changed. The change flag 1042 may be implemented as a mutex (or a semaphore) since it is set and referenced in the process of Fig. 4 and the process described later in Fig. 5. Note that the setting of the flag is not limited to this implementation form.

[0060] In step S422, the inference processing unit 105 changes the learned parameters 1052, and the process proceeds to step S423. The control unit 100 performs DMA control via the control management unit 104 to transfer the learned parameters stored in the work memory 102 in step S403 to a storage area of ​​the inference processing unit 105.

[0061] In step S423, the control management unit 104 sets the parameter change flag 1042 to a value of 0 to indicate that the learned parameters have been changed. Then, the control management unit 104 releases the lock in step S421 to enable subsequent changes to the learned parameters, and the process proceeds to step S417.

[0062] In step S417, the inference processing unit 105 applies the thermal image acquired in step S402 and the learned parameters 1052 changed as necessary in steps S420 to S423 to the learned model 1051, executes AI processing by the inference processing unit 105, and proceeds to step S418. When the imaging device 10 detects a person from a thermal image as in this embodiment, the AI ​​processing result may be transmitted to the control unit 100 as coordinates and reliability information in the thermal image, or may be temporarily stored in the work memory 102 together with the thermal image as a recognition map consisting of the accuracy level for each image region. In addition, the thermal image and / or the AI ​​processing result may be transmitted to an external server or other external device via the communication unit 110.

[0063] In step S418, the control unit 100 determines whether or not a request to end image capture has occurred. If the control unit 100 determines that a request to end image capture has occurred, the control unit 100 ends the process of FIG. 4. If the control unit 100 determines that a request to end image capture has not occurred, the control unit 100 returns the process to step S405 and repeats the process from step S405. The request to end image capture occurs, for example, when a user instructs the imaging device 10 to power off by remote control and the control unit 100 receives the instruction to power off via the communication unit 110.

[0064] In addition, in FIG. 4, the explanation of the shutdown process after the power of the imaging device 10 is turned off is omitted. However, if there is an order to stop the voltage supply or status data to be saved, the process is executed appropriately according to the command of the control unit 100.

[0065] Next, a process for determining whether or not to change the learned parameters 1052 according to this embodiment will be described with reference to FIG.

[0066] 5 is a flowchart showing the process of this embodiment for determining whether or not to change the learned parameters 1052. The process of FIG. 5 is started when a temperature acquisition event of the timer 109 is received.

[0067] In step S501, the control management unit 104 determines whether or not a request to update the threshold table 1041 has occurred based on the value of the table change flag 1043. If the control management unit 104 determines that a table update request has occurred, the process proceeds to step S502, and if the control management unit 104 determines that a table update request has not occurred, the process proceeds to step S503. When the table change flag 1043 has a value of 1, it indicates that the threshold table 1041 needs to be updated, and when the table change flag 1043 has a value of 0, it indicates that the threshold table 1041 does not need to be updated (or the state in which the update is required has been resolved).

[0068] In step S502, the control management unit 104 changes the threshold table 1041. For example, the threshold table 1041 is changed from the state shown in Fig. 3(b) to the state shown in Fig. 3(c). In the example of Fig. 3(c), the upper limit of the temperature range is updated from 30°C to 35°C, and the storage location of the newly added learned parameter is overwritten to the storage location (for example, offset address addr_0) of the work memory 102 where the deleted learned parameter (name param_11) was stored.

[0069] In step S503, the control management unit 104 acquires the temperature information t from the temperature measurement unit 103, and the process proceeds to step S504.

[0070] In step S504, the control management unit 104 compares the temperature information t acquired from the temperature measurement unit 103 in step S503 with the temperature range in the threshold table 1041 (the temperature range 306 in FIG. 3(c)), and advances the process to step S505.

[0071] In step S505, the control management unit 104 judges whether the temperature information t acquired in step S402 is included in the temperature range between the upper limit and the lower limit of the threshold table 1041 (the range of 10° C. to 35° C. of the temperature range 306 in FIG. 3(c)). If the control management unit 104 judges that the temperature information t acquired in step S402 is not included in the temperature range between the upper limit and the lower limit of the threshold table 1041, the process proceeds to step S506. If the control management unit 104 judges that the temperature information t acquired in step S402 is included in the temperature range between the upper limit and the lower limit of the threshold table 1041, the process proceeds to step S508.

[0072] In step S506, the control management unit 104 stores the failure to change the learned parameters in a log. This is a process for notifying the user of the occurrence of an abnormal operation inside the edge device. It may be determined that a change failure event has occurred when the image capture device 10 is used indoors and the learned parameters cannot be changed in time due to a sudden temperature change caused by air conditioning or the like, or when the learned parameters cannot be downloaded in time due to a communication failure.

[0073] In step S507, the control management unit 104 transmits a download request for new learned parameters to an external server via the communication unit 110, and ends the process. If a download request for learned parameters in the same temperature range has already been transmitted (if the download has not been completed in time), the download request does not need to be transmitted from the communication unit 110. The process when the change of the learned parameters fails is not limited to this implementation.

[0074] In step S508, the control management unit 104 determines whether or not the threshold table 1041 needs to be updated for the temperature information t acquired in step S503. If the control management unit 104 determines that the threshold table 1041 needs to be updated for the temperature information t acquired in step S503, the process proceeds to step S509. If the control management unit 104 determines that the threshold table 1041 does not need to be updated for the temperature information t acquired in step S503, the process proceeds to step S510. A method for determining whether to update the threshold table 1041 will be described later with reference to FIG. 6.

[0075] In step S509, similarly to step S507, the control management unit 104 transmits a download request for new learned parameters to an external server via the communication unit 110. The download of new learned parameters (processing for replacing learned parameters) will be described later with reference to FIG. 7, and is executed in the background of the processing in FIG. 5.

[0076] In step S510, the control management unit 104 determines whether or not the learned parameters applied to the AI ​​processing in the inference processing unit 105 need to be changed for the temperature information t acquired in step S503. If the control management unit 104 determines that the learned parameters need to be changed, the process proceeds to step S511. If the control management unit 104 determines that the learned parameters do not need to be changed, the process in Fig. 5 ends. A method for determining whether to change the learned parameters 1052 will be described later with reference to Fig. 6.

[0077] In step S511, the control management unit 104 determines whether the parameter change flag 1042 is in a locked state. If the control management unit 104 determines that the parameter change flag 1042 is in a locked state, the control management unit 104 waits until the locked state is released in order to avoid a collision of control requests, since the change process of the learned parameter 1052 is being executed. If the control management unit 104 determines that the locked state of the parameter change flag 1042 has been released, the process proceeds to step S512.

[0078] In step S512, the control management unit 104 transmits information for accessing the new learned parameters to the inference processing unit 105. The control management unit 104 sets, for example, the address of the storage location of the new learned parameters in the work memory 102, which is the source in the DMA transfer, and the storage location of the learned parameters 1052 in the inference processing unit 105, which is the destination (in the case of SRAM, the top address of the storage location). The master and data transfer method in the DMA transfer are not limited to this implementation. The change process of the learned parameters is performed in step S422 in FIG. 4.

[0079] In step S513, the control management unit 104 sets the parameter change flag 1042 to a value of 1, thereby setting a state in which a request to change the learned parameters has occurred in step S420 of FIG.

[0080] Next, a method of determining whether to update the threshold table 1041 in step S508 in FIG. 5 and a method of determining whether to change the learned parameters in step S510 will be described with reference to FIG.

[0081] Fig. 6(a) illustrates an example of the data configuration of a threshold table in which the temperature range between the lower and upper limits for which the learned parameters can be changed is 5° C. to 30° C. Fig. 6(b) illustrates an example of the data configuration of a threshold table in which the temperature range between the lower and upper limits for which the learned parameters can be changed is 10° C. to 35° C. Temperature information t in Fig. 6 is temperature information acquired from the temperature measurement unit 103, and is the outside air temperature or environmental temperature around the imaging device 10.

[0082] The learned parameter change determination in step S510 is the first determination, and the threshold table update determination in step S508 is the second determination. The temperature range to which the learned parameters are applied is the first threshold at which a learned parameter change request occurs in the first determination. Symbol tr is a predetermined temperature from the upper limit to the lower limit side of the temperature range or a predetermined temperature range from the lower limit to the upper limit side of the temperature range at which a threshold table update request occurs in the second determination, and is set as the second threshold. Figure 6(b) illustrates a state in which the temperature range of the threshold table 1041 has been changed by executing a learned parameter download request for a temperature range exceeding the upper limit of the temperature range in the state of Figure 6(a) by the second determination.

[0083] The first judgment in the state of FIG. 6(a) will be described. When temperature information t=19° C. is acquired within the temperature range of the second threshold in the threshold table of FIG. 6(a), the name param_13 is selected as the learned parameter to be applied to the AI ​​processing by the first judgment that compares the temperature information t with the temperature range of the first threshold. After that, when temperature information t=21° C. is acquired, the name param_14 is selected by the first judgment. In this case, since a difference occurs in the learned parameters selected by the first judgment, a change request for the learned parameter 1052 is generated in the first judgment, and the parameter change flag 1042 is set to a value of 1.

[0084] The second determination in the state of FIG. 6(a) will be described. At the time when temperature information t=19° C. is acquired in response to the temperature acquisition event of the timer 109, the temperature information t is not within the temperature range of the second threshold, so no request for updating the threshold table is generated in the second determination. When temperature information t=21° C. is acquired thereafter, the temperature information t is within the temperature range of the second threshold, so a request for updating the threshold table (replacement of the learned parameters stored in the work memory 102) is generated in the second determination. The request for updating the threshold table may be an interrupt request to the control unit 100. In response to the request for updating the threshold table, the control management unit 104 executes a learned parameter replacement process described later in FIG. 7. In the process of FIG. 7, the control unit 100 requests an external server to acquire a learned parameter (name: param_16) via the communication unit 110. FIG. 6(b) illustrates a state in which a learned parameter named param_16 is downloaded from outside, and the temperature range from the lower limit to the upper limit within which the learned parameter in the threshold table 1041 can be changed is updated to 10°C to 35°C (from param_12 to param_16). In this case, the data downloaded from outside includes the second threshold tr in addition to the learned parameter (named param_16), and the threshold tr is also updated together with the update of the threshold table 1041. In the example of FIG. 6(b), the second threshold tr is updated from 10°C to 7°C.

[0085] In the threshold table shown in Fig. 3(b), in addition to the learned parameters 305, the temperature range 306 is also rewritten. After the learned parameters (name: param_16) are replaced, the threshold table becomes as shown in Fig. 3(c). The payload when downloading the learned parameters (name: param_16) may include the temperature range (first threshold) corresponding to the downloaded learned parameters, and may be used to update the threshold table 1041 (from Fig. 3(b) to Fig. 3(c)).

[0086] The learned parameter (name param_16) is overwritten in the area where the learned parameter (name param_11) to be deleted from the work memory 102 was stored, and memory location 304 or 307 in the threshold table is updated to the top address (address addr_0) of the deleted learned parameter (name param_11).

[0087] As shown in Fig. 6(b), the second threshold tr may be different between the upper limit and the lower limit of the temperature range in which the learned parameters in the threshold table 1041 can be changed. In the example of Fig. 6(b), the second threshold at the upper limit of the temperature range is indicated by the symbol trH, and the second threshold at the lower limit is indicated by the symbol trL. Also, for example, a register (not shown) that temporarily stores the second thresholds at the upper and lower limits of the temperature range may be provided in the control management unit 104.

[0088] The first judgment in the state of Fig. 6(b) will be described. When temperature information t = 26°C is acquired, the name param_15 is selected as the learned parameter to be applied to AI processing by the first temperature judgment. Thereafter, when temperature information t = 29°C is acquired, the name param_15 is also selected as the learned parameter, so that the first judgment does not generate a change request for the learned parameter, and the parameter change flag 1042 becomes 0.

[0089] The second determination in the state of FIG. 6(b) will be described. At the time when the temperature information t=26° C. is acquired in response to the temperature acquisition event of the timer 109, the temperature information t is not within the temperature range of the second threshold, so that the second determination does not generate a request to update the threshold table. After that, when the temperature information t=29° C. is acquired, the temperature information t is within the temperature range of the second threshold, so that the second determination generates a request to update the threshold table (replacement of the learned parameters stored in the work memory 102). The control management unit 104 requests the control unit 100 to update the threshold table. In response to the request to update the threshold table, the control unit 100 executes a learned parameter replacement process described later in FIG. 7, and downloads the learned parameters from the outside. The learned parameters downloaded from the outside are overwritten in the area where the learned parameters (name param_12) to be deleted from the work memory 102 were stored, and the storage position 304 or 307 of the threshold table is updated to the top address (address addr_1) of the deleted learned parameters (name param_12).

[0090] In this manner, in this embodiment, a change in the usage environment (outside air temperature and environmental temperature) of the edge device is predicted, and learned parameters are acquired from the outside in advance to update the threshold table. This makes it possible to avoid stopping AI processing due to acquiring learned parameters from the outside when it is necessary to change the learned parameters used in AI processing due to a change in the usage environment (outside air temperature and environmental temperature) of the edge device.

[0091] The second judgment in the state of FIG. 6(a) and (b) may be made based on temperature change instead of the second threshold value. For example, a temperature measuring member whose characteristics change significantly around the Curie point (Curie temperature) is provided, and the second judgment is made based on a comparison between the characteristic change and a third threshold value (not shown). In this case, the Curie point is set near the target temperature, and a request to update the threshold table is generated when the characteristic change of the temperature measuring member becomes larger than the third threshold value. This method is useful when the temperature change becomes large and the upper or lower limit of the temperature range of the threshold table is temporarily expanded in the direction of the temperature change. The temperature measuring member whose temperature change significantly around the Curie point is, for example, a PTC (Positive Temperature Coefficient) thermistor. A PTC thermistor whose Curie point is an appropriate value may be used to directly detect the temperature change near the target temperature.

[0092] Materials other than PTC thermistors may be used. For example, in recent Japanese climate, when the outside temperature approaches 40°C, the temperature near the ground may slightly exceed 40°C due to glare from the asphalt. The second judgment may be performed by considering the comparison of the impedance change with the third threshold as a temperature change using a material whose magnetic properties change at about 40°C (the magnetic properties are lost above 40°C), such as an amorphous magnetic material such as Mn-Cu ferrite. The second judgment may be performed based on either the result of comparing the temperature information with the second threshold or the result of comparing the impedance change (temperature change) with the third threshold.

[0093] 9, the temperature acquisition cycle (first cycle T) set by the timer 109 may be changed in response to changes in the outside air temperature or the environmental temperature. For example, the setting of the timer 109 is adjusted so that the temperature acquisition cycle is shortened when the increase or decrease in temperature change is large, and the temperature acquisition cycle is lengthened when the increase or decrease in temperature change is small.

[0094] Next, the learned parameter replacement process of this embodiment will be described with reference to FIG.

[0095] Fig. 7 is a flowchart showing the learned parameter replacement process of this embodiment. The process of Fig. 7 is executed when updating the threshold table 1041 of Fig. 5 (replacing the learned parameters). In the following, an example will be described in which the control management unit 104 requests an external server to transmit the learned parameters via the communication unit 110, and downloads the learned parameters from the external server.

[0096] In step S701, the control management unit 104 checks the connection with the server through the communication unit 110.

[0097] In step S702, the control management unit 104 determines whether or not the connection to the external server is normal based on the result of the connection check with the server in step S701. If the control management unit 104 determines that the connection to the external server is normal, the process proceeds to step S706. If the control management unit 104 determines that the connection to the external server is not normal, the process proceeds to step S703.

[0098] In step S703, the control management unit 104 determines whether or not a communication failure has occurred in the connection with the external server. If the control management unit 104 determines that a communication failure has occurred in the connection with the external server and that the communication will not be restored even if the standby state is continued, the control management unit 104 executes error processing in steps S704 and S705 and ends the processing in Fig. 7. If the control management unit 104 determines that no communication failure has occurred in the connection with the external server and that the communication will be restored during the standby state, the process returns to step S701 and continues checking the connection with the external server.

[0099] In step S704, the control management unit 104 records in a log that normal processing was not performed (as a failure to replace the learned parameters). Information to be recorded in the log may include, for example, the time of occurrence, whether a communication failure occurred, whether necessary data does not exist, and the cause of the failure. The information to be recorded in the log is not limited to this implementation form.

[0100] In step S705, the control management unit 104 notifies the user that an error has occurred, and ends the processing in FIG. 7. If the error is due to the determination result in step S707 described below, an error message (such as an OSD) is superimposed on the thermal image (brightness image) and displayed on the user's operation terminal (not shown). If communication with an external server is impossible due to a communication failure or the like, a lighting member (not shown) such as an LED may be turned on or blinked in red or the like to notify the user. The lighting and blinking control may be executed by the control unit 100. The notification process in the event of a communication failure is not limited to this implementation form.

[0101] In step S706, the control management unit 104 inquires of the external server through the communication unit 110 whether or not there is data that needs to be downloaded. In the example of FIG. 6(a) where temperature information t=21° C. is acquired, the control management unit 104 inquires whether or not there is a learned parameter (param_16), and the external server replies with the presence or absence of the learned parameter (param_16). The method of inquiring of the server is not limited to this implementation.

[0102] In step S707, the control management unit 104 determines whether or not data that needs to be downloaded exists in the external server as a result of the inquiry in step S706. If the control management unit 104 determines that data that needs to be downloaded exists in the external server, the process proceeds to step S708. If the control management unit 104 determines that data that needs to be downloaded does not exist in the external server, the control management unit 104 executes error processing in steps S704 and S705 and ends the process in FIG. 7. As described in FIG. 6, this process corresponds to a case where the temperature information t acquired in step S503 in FIG. 5 is within the temperature range of the second threshold of the threshold table 1041 and the threshold table 1041 needs to be updated in step S508, but the learned parameters that need to be downloaded cannot be downloaded from the outside.

[0103] In step S708, the control management unit 104 sends a transmission request to the external server via the communication unit 110 for the learned parameters that need to be downloaded and their temperature ranges (the second threshold values ​​described with reference to FIG. 6).

[0104] In step S709, the control management unit 104 waits until the transmission of the data requested in step S708 is started, and after the transmission is started, the process proceeds to step S710.

[0105] In step S710, the control management unit 104 receives the data requested to be downloaded from the external server. The learned parameters and their temperature ranges received from the external server are stored in a predetermined address in the work memory 102. In the example of the learned parameter name param_16 in FIG. 6(b), the predetermined address corresponds to the storage location addr_0. The data received from the external server may be temporarily stored in a buffer memory (not shown) of the communication unit 110 and then transferred to the work memory 102. The data to the work memory 102 may be transferred by DMA. The series of processes from receiving data from the external server to storing it in the work memory 102 is not limited to this implementation. In step S710, the case where data reception from the external server fails is not described, but in this case, various recovery processes may be performed.

[0106] In step S711, the control management unit 104 changes the table change flag 1043 to a value of 1, and issues a request to update the threshold table 1041 using the new learned parameters.

[0107] Next, a process for changing the temperature acquisition cycle of the timer 109 in this embodiment will be described with reference to FIG.

[0108] 9 may be an interrupt process for the control unit 100 so as to be started for each temperature acquisition event of the timer 109. The process of FIG. 9 may be executed in parallel with the process of FIG.

[0109] In step S901, the control unit 100 acquires temperature information t from the temperature measurement unit 103. Note that the control unit 100 stores the previous temperature information t0 acquired from the temperature measurement unit 103 in the previous temperature acquisition event in a register (not shown).

[0110] In step S902, the control unit 100 reads the previous temperature information t0 from the register, and calculates the temperature change Δt (Δt=|t−t0|) of the current temperature t.

[0111] In step S903, the control unit 100 compares the temperature change Δt with the fourth threshold TEX or the fifth TCP. If the control unit 100 determines that the temperature change Δt exceeds the fourth threshold TEX or is less than the fifth TCP, the process proceeds to step S904. If the control unit 100 determines that the temperature change Δt is equal to or less than the fourth threshold TEX and equal to or greater than the fifth TCP, the first threshold does not need to be adjusted, and the process proceeds to step S910.

[0112] In step S904, the control unit 100 shortens the first period when the temperature change Δt exceeds a fourth threshold value TEX, and lengthens the first period when the temperature change Δt is less than a fifth threshold value TCP.

[0113] The control unit 100 calculates the change candidate value TC of the first period from the following formula 1. (Formula 1) TC = T + α (t0-t) If the temperature difference is t0-t [℃], the first period is T, and the change candidate value TC [min] (time in minutes), the unit of the coefficient α is [min / ℃]. In the case of temperature rise (Δt>TEX), the first period is 10 [min], the adjustment amount α per 1℃ is 0.5 (adjustment for 30 seconds per 1℃), the fourth threshold TEX is 2 [℃], and the temperature rises from 30 [℃] to 33 [℃], and at t0-t = -3 [℃], TC = 10 + 0.5 × (-3) = 10-1.5 = 8.5. The change candidate value TC for the first period is 8 minutes 30 seconds. The above calculation formula is an example and is not limited to this implementation form. The adjustment period may be performed in increments of a fixed value width (such that a fixed value is added / subtracted for each adjustment request).

[0114] In step S905, the control unit 100 determines whether the change candidate value TC of the first period is equal to or greater than the second period set in advance. If the control unit 100 determines that the change candidate value TC of the first period is equal to or greater than the second period, the control unit 100 proceeds to step S906. If the control unit 100 determines that the change candidate value TC of the first period is less than the second period, the control unit 100 proceeds to step S907.

[0115] In step S906, the control unit 100 changes the first period to the second period, and proceeds to step S910. The second period defines an upper limit of the temperature acquisition period (first period) (first period<second period).

[0116] In step S907, the control unit 100 determines whether the change candidate value TC of the first period is equal to or less than a preset third period. If the control unit 100 determines that the change candidate value TC of the first period is equal to or less than the third period, the control unit 100 proceeds to step S908. If the control unit 100 determines that the change candidate value TC of the first period exceeds the third period, the control unit 100 proceeds to step S909.

[0117] In step S908, the control unit 100 changes the first period to a third period, and proceeds to step S910. The third period defines a lower limit of the temperature acquisition period (first period) (first period>third period).

[0118] In step S909, the control unit 100 substitutes the change candidate value TC for the first period T into the first period T, and the process proceeds to step S910.

[0119] In step S910, the control unit 100 substitutes the current temperature information t for the previous temperature information t0, and ends the process of FIG.

[0120] According to the above-described first embodiment, there is no need to store learned parameters corresponding to various usage environments in the imaging device 10 as an edge device, and therefore no increase in storage capacity or cost is required.

[0121] In addition, by predicting changes in the edge device's usage environment (temperature at the time of shooting) and obtaining learned parameters from an external source in advance to update the threshold table, it is possible to avoid stopping AI processing due to obtaining learned parameters from an external source when it is necessary to change the learned parameters used for AI processing due to changes in the edge device's usage environment (outside temperature and environmental temperature).

[0122] [Embodiment 2] In the second embodiment, an example in which the information collecting device of the present invention is applied to a recording device will be described.

[0123] In this embodiment, an example is described in which the learned parameters are changed based on humidity information when the sound recording device records. According to the second embodiment, it is possible to change the learned parameters according to the usage environment of the sound recording device without stopping the AI ​​processing in the edge device such as the sound recording device.

[0124] Sound is air vibration, and it seems that a state where air contains moisture and has high density is more favorable for sound propagation than a state where air density is low and humidity is low. However, low-frequency sounds have relatively large energy when viewed over the entire frequency range, so there is little change due to humidity. However, high-frequency sounds have short wavelengths and little energy, so their energy is absorbed by the moisture in the air and attenuates during propagation. Therefore, in this embodiment, the recording device converts the analog sound signal collected by the sound collection unit of the operating sound of a machine operating in a factory or the like into a digital signal by an AD converter based on the change in the amount of attenuation of the high-frequency range due to the change in humidity, and generates recorded data. Then, AI processing is performed on the recorded data generated at a specified humidity acquisition period, and a failure analysis process is performed to analyze the failure of the machine from the operating sound.

[0125] The audio recording device 20 of this embodiment can be applied to web cameras, network cameras, vehicle-mounted cameras, surveillance cameras, medical cameras, smart speakers, and the like with audio recording functions that are communicatively connected via a network as edge devices.

[0126] The configuration and functions of the recording device 20 of the second embodiment will be described with reference to FIG.

[0127] The recording device 20 of this embodiment collects and records the operating sounds of machines operating in a factory or the like using a sound collection unit such as a microphone.

[0128] The control unit 1000, non-volatile memory 1001, work memory 1002, timer 1009, and communication unit 1010 are similar to the control unit 100, non-volatile memory 101, work memory 102, timer 109, and communication unit 110 in Figure 1 of embodiment 1, except that the temperature is replaced by humidity and the thermal image is replaced by sound.

[0129] The humidity measurement unit 1003 includes a humidity sensor and an AD converter (not shown). The humidity sensor has a structure in which a moisture-sensitive material is sandwiched between electrodes, and converts the change in moisture absorption / desorption of the moisture-sensitive material into a change in resistance or capacitance, and outputs an electrical signal. The AD converter generates humidity information by converting the analog electrical signal (voltage value) generated by the humidity sensor into a digital signal.

[0130] The control management unit 1004 acquires humidity information from the humidity measurement unit 1003, and manages control data such as tables and flags when AI processing is executed in the inference processing unit 1005 described later. Like the control management unit 104 shown in FIG. 1 of the first embodiment, the control management unit 1004 includes a threshold table 10041, a parameter change flag 10042, and a table change flag 10043, and changes or updates parameters and humidity ranges, sets or resets flags, etc. These table 10041 and flags 10042 and 10043 may not be managed by the control management unit 1004, but may be stored in the work memory 1002 or the like so that the control management unit 1004 can access them.

[0131] The inference processing unit 1005 includes a trained model 10051 and trained parameters 10052. The inference processing unit 1005 executes a fault analysis process by AI processing in which the recording data generated by the sound collection unit 1011 described later and the trained parameters 10052 selected by the control unit 1000 are applied to the trained model, and outputs a fault detection result. The detected fault is, for example, a fault in a machine operating in a factory or the like. In this case, the inference processing unit 1005 may include a GPU (Graphics Processing Unit) that can perform efficient calculations by processing more microcodes in parallel. When performing multiple inference processes using a trained model, such as deep learning or machine learning, it is useful to perform the process using a GPU. In this embodiment, the inference process may be performed by the control unit 1000 and the GPU working together, or may be performed only by the control unit 1000 or the GPU.

[0132] The sound collecting unit 1011 is a microphone that converts sound into an electric signal and outputs it. The frequency characteristics of the microphone are such that it can collect sounds up to a range (for example, about 100 to 400 KHz) beyond the audible range (20 Hz to 20,000 Hz) for failure analysis, so an industrial ultrasonic microphone (ultrasonic sensor) or the like may be used. Also, a microphone array configuration using multiple microphones may be used. The sound collecting unit 1011 of this embodiment is not limited to a microphone.

[0133] The AMP / AGC 1012 includes a preamplifier that amplifies the analog sound signal output from the sound collection unit 1011, and an AGC (Auto Gain Control) that prevents distortion due to saturation.

[0134] The filter 1013 is an anti-aliasing filter for preventing aliasing when an analog sound signal is converted into a digital signal at a predetermined period.

[0135] The AD converter 1014 samples the analog sound signal at a predetermined cycle and converts it into a digital signal to generate recording data. The analog sound signal is limited by the filter 1013 to a band half the sampling frequency.

[0136] The data transmitted to an external server or other external device may be the sound recording data recorded in the sound recording device 20 and the AI ​​processing results (fault analysis results). The sound recording data may be prepared separately with the band limited for human monitoring (band limiting filter not shown). Also, a codec (not shown) may be provided in the sound recording device 20, and the sound recording data may be compressed and encoded before being transmitted to the outside. The data transmitted from the sound recording device 20 as an edge device may be subjected to quantization to reduce the amount of information on the high frequency side.

[0137] Next, the control process of the recording device 20 of the second embodiment will be described with reference to FIG.

[0138] FIG. 11 is a flowchart showing the control process of the recording device 20 of the second embodiment.

[0139] In step S1101, the control unit 1000 executes a system startup process, and the process proceeds to step S1102. The startup process is similar to step S401 in FIG.

[0140] The timer 1009 generates a humidity acquisition event for acquiring humidity information by the control management unit 1004. The humidity acquisition event may be set to a preset initial value.

[0141] In step S1102, the control unit 1000 waits while energizing the sound collection unit 1011, the AMP / AGC 1012, the filter 1013, and the AD converter 1014 in order to quickly stabilize the operation of the analog circuit.

[0142] In step S1103, the control management unit 1004 acquires an initial value of the humidity information from the humidity measurement unit 1003, and the process proceeds to step S1104. The initial value of the humidity information may be acquired without using the humidity acquisition vent of the timer 109.

[0143] In step S1104, the inference processing unit 1005 performs the initial setting of the trained model, and the process proceeds to step S1105. This process is the same as the case where the temperature information is replaced with the humidity information in step S403 of FIG. 4 in the first embodiment. In the present embodiment, when the trained model is common, only the trained parameters are changed according to the humidity information. In addition, in this process, the control management unit 1004 may initialize the threshold table 10041. The configuration of the threshold table is the same as that in FIG. 3(a) and (b). The temperature range 302 (306) in FIG. 3 is the humidity range, and the unit of the parameter is from [°C] to [%]. However, the humidity range to which the trained parameters are applied is 0[%] to 100[%], and when divided every 5%, 5 types of data from 20 types of trained parameters are stored in the work memory 1002 as in the first embodiment.

[0144] In step S1105, the control unit 1000 judges whether or not to stop recording. If the control unit 1000 judges to stop recording, it stops generating sound data and repeats the judgment process of step S1105. If the control unit 1000 judges not to stop recording, it advances the process to step S1106. Conditions for stopping recording include, for example, a network disconnection or abnormal heat generation of the recording device 20, but are not limited to this implementation form.

[0145] The processes enclosed by the thick lines (step S1110 and steps S1120 to S1123) are executed in parallel. When the parallel processes are completed, the process of step S1130 starts.

[0146] In step S1110, the audio recording data for a predetermined period is temporarily stored (buffered) in a storage unit such as the work memory 1002 as a target for AI processing. The storage unit may have multiple storage areas for buffering the audio recording data, and may be assigned to a storage area separate from the storage area of ​​the trained model used for AI processing, so that the AI ​​processing of the audio recording data in each storage area is superimposed and executed. If there is audio recording data that can be AI processed in addition to the current audio recording data, the processing of step S1130 may be started only after the processing of the data that has been started in parallel is completed, without waiting for the buffering of the current audio recording data.

[0147] The processing from steps S1120 to S1123 is processing for changing the learned parameters used for the AI ​​processing of the sound recording device 20 as an edge device, and is similar to steps S420 to S423 in FIG. 4 of the first embodiment.

[0148] In step S1130, the inference processing unit 1005 executes AI processing by applying the recording data buffered in step S1110 and the learned parameters changed as necessary in steps S1120 to S1123 to the learned model 10051, and then proceeds to step S1131.

[0149] The failure analysis result output by the AI ​​processing of this embodiment is, for example, an error code indicating an abnormal sound of a machine gear or a change in a press sound, and may be transmitted to the control unit 1000. The failure analysis result and recorded data for human monitoring (which may be buffered data or recorded data sampled separately for monitoring) may be transmitted to an external server or other external device via the communication unit 1010. In this case, the data for monitoring may be transmitted to a user's operation terminal (not shown). Also, only the failure analysis result when a failure is diagnosed (when a failure is estimated to have occurred) may be transmitted. Note that the data transmitted from the recording device 20 as an edge device is not limited to this implementation form.

[0150] In step S1131, the control unit 1000 determines whether or not a request to end recording has occurred. If the control unit 1000 determines that a request to end recording has occurred, the control unit 1000 ends the process of FIG. 11. If the control unit 1000 determines that a request to end recording has not occurred, the control unit 1000 returns the process to step S1105 and repeats the process from step S1105. A request to end recording occurs, for example, when a user instructs the recording device 20 to be turned off by remote control and the control unit 1000 receives the instruction to turn off the power via the communication unit 1010.

[0151] In FIG. 11, the explanation of the shutdown process after the power of the recording device 20 is turned off is omitted, but if there is an order to stop the voltage supply or status data to be saved, the process is executed appropriately according to the command of the control unit 1000.

[0152] The process of changing the learned parameters and the process of updating the threshold table are the same as those in the case where the temperature information is replaced with the humidity information in the process of Fig. 5. The process of replacing the learned parameters is the same as the process of Fig. 7.

[0153] According to the above-described second embodiment, there is no need for the recording device 20 serving as an edge device to store learned parameters corresponding to various usage environments, and therefore there is no increase in storage capacity or cost.

[0154] In addition, by predicting changes in the edge device's usage environment (outside air temperature and environmental temperature) and obtaining learned parameters from an external source in advance to update the threshold table, it is possible to avoid stopping AI processing due to obtaining learned parameters from an external source when it is necessary to change the learned parameters used for AI processing due to a change in the edge device's usage environment (humidity during recording).

[0155] In the above-mentioned embodiments 1 and 2, examples have been described in which learned parameters that are not held by the edge device are acquired from outside, but it is also possible to acquire the learned model together with the learned parameters from outside, or to acquire only the learned model from outside.

[0156] Furthermore, the edge devices exemplified in the above-mentioned first and second embodiments may be stationary devices or mobile devices such as a drone.

[0157] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0158] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0159] The disclosure of this specification includes the following information collection device, control method, and program. [Configuration 1] An information collection device that performs inference processing using a trained model, A communication means for communicating with an external device; A first acquisition means for acquiring first information related to a usage environment of the information collection device; A second acquisition means for acquiring second information to be subjected to the inference process; A processing means for applying the second information and third information set based on the first information to the trained model to execute the inference process; A control means for setting the third information based on a first threshold range of the first information, When the first information is within a range of a second threshold, the control means selects the third information based on a first threshold range of the first information; An information collecting device characterized in that, if the first information is not within the range of the second threshold, the third information is acquired from the external device via the communication means so that the first information is within the range of the second threshold. [Configuration 2] a storage means for storing the third information within the range of the second threshold value; When the first information is not within the range of the second threshold, the control means acquires the third information and the range of the first threshold corresponding to the third information from the external device via the communication means so that the first information is within the range of the second threshold; The information collecting device according to configuration 1, characterized in that the third information stored in the memory means and the first threshold range corresponding to the third information are replaced with the third information acquired from the external device and the first threshold range corresponding to the third information. [Configuration 3] The control means a threshold table in which a plurality of pieces of third information are associated with a range of the first threshold applied to each piece of the third information; The threshold table includes: The information collecting device according to configuration 2, further comprising at least one of information indicating whether the third information is stored in the storage means and information indicating a storage location of the third information, in addition to the range of the first threshold applied to each of the third information. [Configuration 4] when the first information is not within a predetermined range from an upper limit to a lower limit of the range of the second threshold, the control means acquires from the external device third information that is greater than the upper limit and a range of the first threshold applied to the third information; The information collecting device according to configuration 2 or 3, characterized in that if the first information is not within a predetermined range from the lower limit to the upper limit of the range of the second threshold, third information smaller than the upper limit and the range of the first threshold applied to the third information are obtained from the external device. [Configuration 5] when the amount of change in the first information exceeds an upper limit threshold of the first information, the control means acquires from the external device third information that is greater than the upper limit and a range of the first threshold applied to the third information; The information collecting device according to configuration 2 or 3, characterized in that when the amount of change in the first information is smaller than a lower limit threshold of the first information, third information smaller than the lower limit and the range of the first threshold applied to the third information are acquired from the external device. [Configuration 6] the control means stores in the storage means third information that is greater than the upper limit and a first threshold range that is applied to the third information, which are acquired from the external device, and deletes the third information of the lower limit and the first threshold range that is applied to the third information, which are stored in the storage means; The information collecting device according to configuration 4 or 5, characterized in that the third information smaller than the lower limit and the range of the first threshold applied to the third information acquired from the external device are stored in the storage means, and the third information of the lower limit and the range of the first threshold applied to the third information stored in the storage means are deleted. [Configuration 7] the first acquisition means acquires the first information at a first period; The control means shortens the first period when the amount of change in the first information is greater than a predetermined threshold value, 7. The information collecting device according to any one of configurations 1 to 6, wherein the first period is lengthened when an amount of change in the first information is smaller than a predetermined threshold value. [Configuration 8] 8. The information collecting device according to claim 7, wherein the control means changes the first period to a value between an upper limit and a lower limit of the first period. [Configuration 9] The information collecting device according to any one of configurations 1 to 8, wherein the control means acquires the third information and the first threshold range corresponding to the third information from the external device, and also acquires the second threshold range. [Configuration 10] the information collecting device is an imaging device, the first information is temperature information at the time of photographing, The second information is a thermal image captured using far-infrared light, The third information is a learned parameter, The inference process using the trained model is a process of detecting a subject from the thermal image, The first threshold range is a range of temperatures of the trained parameters to be applied to the trained model; The information collecting device according to any one of configurations 2 to 6, characterized in that the second threshold range is a width from the upper or lower limit of the temperature range of the learned parameters stored in the memory means. [Configuration 11] the information collecting device is a recording device; the first information is humidity information at the time of recording, the second information being an operating sound of a machine; The third information is a learned parameter, The inference process using the trained model is a process for detecting a fault from the operation sound, The first threshold range is a humidity range of the trained parameters to be applied to the trained model; The information collecting device according to any one of configurations 2 to 6, characterized in that the second threshold range is a width from an upper or lower limit of the humidity range of the learned parameter stored in the memory means. [Configuration 12] A method for controlling an information collection device that executes an inference process using a trained model, comprising: The information collection device includes a communication means for communicating with an external device; A first acquisition means for acquiring first information related to a usage environment of the information collection device; and a second acquisition means for acquiring second information to be subjected to the inference process, The control method includes: Setting third information to be applied to the trained model based on a range of a first threshold of the first information; and applying the second information and the third information to the trained model to execute the inference process. In the setting step, when the first information is within a range of a second threshold, the third information is selected based on the range of a first threshold of the first information; a control method characterized in that, if the first information is not within the range of the second threshold, the third information is obtained from the external device via the communication means so that the first information is within the range of the second threshold. [Configuration 13] 12. A program for causing a computer to function as the information collection device according to any one of claims 1 to 11. [Explanation of symbols]

[0160] 10...imaging device (edge ​​device), 20...audio recording device (edge ​​device), 100, 1000...control unit (CPU), 103...temperature measurement unit, 1003...humidity measurement unit, 104, 1004...control management unit, 105, 1005...inference processing unit, 110, 1010...communication unit

Claims

1. An information collection device that performs inference processing using a trained model, A communication means for communicating with an external device; A first acquisition means for acquiring first information related to a usage environment of the information collection device; A second acquisition means for acquiring second information to be subjected to the inference process; A processing means for applying the second information and third information set based on the first information to the trained model to execute the inference process; a control means for setting the third information based on a range of a first threshold value of the first information, When the first information is within a range of a second threshold, the control means selects the third information based on a range of a first threshold of the first information; An information collecting device characterized in that, if the first information is not within the range of the second threshold, the third information is acquired from the external device via the communication means so that the first information is within the range of the second threshold.

2. a storage means for storing the third information within the range of the second threshold value; When the first information is not within the range of the second threshold, the control means acquires the third information and the range of the first threshold corresponding to the third information from the external device via the communication means so that the first information is within the range of the second threshold; The information collecting device according to claim 1, characterized in that the third information stored in the memory means and the first threshold range corresponding to the third information are replaced with the third information acquired from the external device and the first threshold range corresponding to the third information.

3. The control means a threshold table in which a plurality of pieces of third information are associated with a range of the first threshold applied to each piece of third information, The threshold table includes: The information collection device according to claim 2, characterized in that in addition to the range of the first threshold applied to each of the third information, the information includes at least one of information indicating whether the third information is stored in the storage means and information indicating the storage location of the storage means.

4. when the first information is not within a predetermined range from an upper limit to a lower limit of the range of the second threshold, the control means acquires from the external device third information that is greater than the upper limit and a range of the first threshold applied to the third information; The information collecting device according to claim 2, characterized in that if the first information is not within a predetermined range from the lower limit to the upper limit of the second threshold range, third information smaller than the upper limit and the first threshold range to be applied to the third information are obtained from the external device.

5. when the amount of change in the first information exceeds an upper limit threshold of the first information, the control means acquires third information that is greater than the upper limit and a range of first thresholds applied to the third information from the external device; The information collecting device according to claim 2, characterized in that when the amount of change in the first information is smaller than a lower limit threshold of the first information, third information smaller than the lower limit and the range of the first threshold applied to the third information are obtained from the external device.

6. the control means stores in the storage means third information that is greater than the upper limit and a range of a first threshold value that is applied to the third information, which are acquired from the external device, and deletes the third information of the lower limit and the range of a first threshold value that is applied to the third information, which are stored in the storage means; The information collecting device according to claim 4, characterized in that the third information smaller than the lower limit acquired from the external device and the first threshold range applied to the third information are stored in the memory means, and the third information of the lower limit and the first threshold range applied to the third information stored in the memory means are deleted.

7. the first acquisition means acquires the first information at a first period; The control means shortens the first period when an amount of change in the first information is greater than a predetermined threshold value, 2. The information collecting device according to claim 1, wherein the first period is lengthened when an amount of change in the first information is smaller than a predetermined threshold value.

8. 8. The information collecting device according to claim 7, wherein the control means changes the first period to a value between an upper limit and a lower limit of the first period.

9. The information collecting device according to claim 1, characterized in that the control means acquires the third information and the first threshold range corresponding to the third information from the external device, and also acquires the second threshold range.

10. the information collecting device is an imaging device, the first information is temperature information at the time of photographing, the second information is a thermal image captured using far-infrared light, the third information is a learned parameter, The inference process using the trained model is a process of detecting a subject from the thermal image, The first threshold range is a range of temperatures of the learned parameters to be applied to the learned model; 3. The information collecting device according to claim 2, wherein the second threshold range is a width from an upper limit or a lower limit of the range of temperatures of the learned parameters stored in the storage means.

11. the information collecting device is a recording device; the first information is humidity information at the time of recording, the second information being an operating sound of a machine; the third information is a learned parameter, The inference process using the trained model is a process for detecting a fault from the operation sound, the first threshold range is a humidity range of the learned parameters to be applied to the learned model; 3. The information collecting device according to claim 2, wherein the second threshold range is a width from an upper limit or a lower limit of the humidity range of the learned parameter stored in the storage means.

12. A method for controlling an information collection device that executes an inference process using a trained model, comprising: The information collection device includes a communication means for communicating with an external device; A first acquisition means for acquiring first information related to a usage environment of the information collection device; and a second acquisition means for acquiring second information to be subjected to the inference process, The control method includes: Setting third information to be applied to the trained model based on a range of a first threshold of the first information; and applying the second information and the third information to the trained model to execute the inference process. In the setting step, when the first information is within a range of a second threshold, the third information is selected based on the range of a first threshold of the first information; A control method characterized in that, if the first information is not within the range of the second threshold, the third information is obtained from the external device via the communication means so that the first information is within the range of the second threshold.

13. A program for causing a computer to function as the information collecting device according to any one of claims 1 to 11.

Images