Device, training device, and method

Abstract

The present invention provides a device and a method that can assess the status of a subject while protecting the privacy of the subject.　An abnormal state detection device 100 according to the present disclosure comprises: a future image generation unit 102 that, on the basis of a thermal camera image (low-resolution image) including a subject P taken by a thermal camera 300 (low-resolution camera) at a point in time (one point in time; first point in time) that includes an n-th frame, estimates an estimated state of the subject P that will be included in a thermal camera image at the (n+1)-th frame (next point in time; second point in time); a thermal camera comparison image input unit 103 that acquires an output state of the subject P included in the thermal camera image (low-resolution image) at the (n+1)-th frame (next point in time); and an abnormal state detection unit 104 that assesses the state of the subject P on the basis of the estimated state and the output state.

Description

Apparatus, learning device, and method

[0001] The present invention relates to a device for processing images captured by a camera, and a learning device and method for performing such processing.

[0002] Patent Document 1 describes a configuration in which, when abnormal behavior such as a fall of a wandering elderly person is detected, a TV camera image at the time of or immediately after the abnormal behavior is detected is transmitted to a contact personal computer.

[0003] Japanese Patent Application Publication No. 11-73582

[0004] However, there is a problem that high-resolution cameras such as TV camera images cannot protect privacy.

[0005] Therefore, an object of the present invention is to provide a device, a learning device, and a method that can determine the situation of a subject while protecting the subject's privacy.

[0006] The device disclosed herein includes an estimation unit that estimates an estimated state of a subject included in a low-resolution image at a next point in time from a low-resolution image including the subject photographed by a low-resolution camera at a certain point in time; an acquisition unit that acquires an output state of the subject included in the low-resolution image at the next point in time; and a judgment unit that judges the state of the subject based on the estimated state and the output state.

[0007] According to the present disclosure, it is possible to determine whether a subject is in an abnormal state while protecting the subject's privacy.

[0008] FIG. 1 is a diagram illustrating a system configuration including an abnormal condition detection device 100 according to the present disclosure. FIG. 2 is a block diagram illustrating the functional configuration of the abnormal condition detection device 100 according to the present disclosure. FIG. 3 is a flowchart illustrating the operation of the abnormal condition detection device 100 according to the present disclosure. FIG. 4 is a schematic diagram illustrating anomaly detection based on errors in position coordinates of keypoints. FIG. 5 is a diagram illustrating a system configuration for a learning process of a generative model 106a and a keypoint estimation model 107a according to the present disclosure. FIG. 6 is a block diagram illustrating the functional configuration of a learning device 200. FIG. 7 is a flowchart illustrating the operation of the learning device 200. FIG. 8 is a schematic diagram illustrating the learning process. FIG. 9 is a diagram illustrating an example of the hardware configuration of the abnormal condition detection device 100 and the learning device 200 according to an embodiment of the present disclosure.

[0009] The present disclosure will be described with reference to the accompanying drawings. Whenever possible, the same parts are designated by the same reference numerals and redundant description will be omitted.

[0010] Fig. 1 is a diagram showing a system configuration including an abnormal condition detection device 100 according to the present disclosure. As shown in the figure, the abnormal condition detection device 100 acquires image information from a thermal camera 300. Then, the abnormal condition detection device 100 detects an abnormal condition of a subject P based on the image information. In Fig. 1, the subject P is shown to have fallen, which is an abnormal condition.

[0011] In this disclosure, the thermal camera 300 refers to a thermography camera, which is a camera for displaying and visualizing thermal images or videos. A typical thermal camera 300 is a low-resolution camera that displays high-temperature areas in red and low-temperature areas in blue.

[0012] The abnormal condition detection device 100 of the present disclosure uses the keypoint estimation model 107a and the generative model 106a to detect an abnormal condition of the subject P from an image captured by the thermal camera 300 when the subject P is included in the image. In the present disclosure, an abnormal condition refers to, for example, the subject P having fallen, but may also include other conditions.

[0013] The generative model 106a is a model that estimates the position coordinates of key points in the (n+1)th frame based on the position coordinates of key points in the nth frame of a photographed subject P.

[0014] The keypoint estimation model 107a is an estimation model for estimating keypoints of a subject. The keypoint estimation model 107a learns the movements of the subject P in advance and estimates the keypoints. In the present disclosure, keypoints refer to the positions of human parts such as the eyes, shoulders, hands, and feet of the subject P, as well as the joint points of the human.

[0015] The keypoint estimation model 107a and the generative model 106a are trained by a training device 200. The training will be described later.

[0016] 2 is a block diagram showing the functional configuration of the abnormal condition detection device 100 of the present disclosure. As shown in the figure, the abnormal condition detection device 100 includes a control unit, a comparison image input unit, and an estimation unit. The estimation unit includes an inference thermo camera image input unit 101 and a future image generation unit 102. The comparison image input unit includes a comparison thermo camera image input unit 103. The control unit includes an abnormal condition detection unit 104 and a future image-comparison image comparison unit 105.

[0017] The inference thermo camera image input unit 101 is a part that inputs a thermo camera image (low resolution image) from the thermo camera 300.

[0018] The future image generation unit 102 is a part that derives the positions of key points of the subject P contained in the thermo camera images in future frames from the input thermo camera images. For example, the future image generation unit 102 inputs the input thermo camera images (frames n-2 to n) into the generative model 106a, and receives from the generative model 106a a position coordinate image of the predicted key points in the n+1 frame.

[0019] The comparison thermo camera image input unit 103 is a part that inputs a thermo camera image (n+1 frame) for comparison from the thermo camera 300 and outputs the position coordinates of the key points of the subject P using the key point estimation model 107a.

[0020] The future image-comparison image comparison unit 105 is a part that compares the position coordinates of the keypoints output from the comparison thermo camera image input unit 103 with the position coordinates of the estimated keypoints output from the future image generation unit 102.

[0021] The abnormal condition detection unit 104 is a part that outputs, based on the comparison result of the future image-comparison image comparison unit 105, that if the error is equal to or greater than a threshold, it indicates an abnormal condition, and if it is less than the threshold, it indicates that the condition is normal. Based on the output result, the operator of the abnormal condition detection device 100 can decide whether to rescue the subject P. This output destination may be a display, or may be a device such as a PC via a communication line.

[0022] Next, the operation of the abnormal condition detection device 100 configured as described above will be described. FIG. 3 is a flowchart showing the operation of the abnormal condition detection device 100 of the present disclosure. The inference thermo camera image input unit 101 acquires low-resolution thermo camera images of the subject P (S101). Here, for example, images of three frames, the n-2, n-1, and n-th frames, are acquired, but the number of frames is not limited to three. Four or more, or two or less, may also be used.

[0023] The future image generation unit 102 executes the generative model 106a stored in the trained generative model storage unit 106 on the positions of the keypoints in the acquired thermo camera images (S102).The future image generation unit 102 then inputs the position coordinates of the keypoints obtained from the thermo camera images of the three frames (n-2, n-1, and n) into the generative model 106a and obtains an output (S103).This output is the position coordinates of the estimated keypoints in the n+1 frame.

[0024] Meanwhile, the comparison thermocamera image input unit 103 acquires a thermocamera image of the actual next frame (frame n+1) (S104).The comparison thermocamera image input unit 103 executes the keypoint estimation model 107a in the trained keypoint estimation model storage unit 107 and outputs the position coordinates of the subject P's keypoints in the actual next frame (frame n+1) (S105).

[0025] The future image-comparison image comparator 105 compares the predicted keypoint position coordinates of subject P in the n+1 frame with the actual keypoint position coordinates of subject P in the n+1 frame, and if the error is equal to or greater than a threshold, the abnormal state detector 104 outputs an abnormality, and if the error is less than the threshold, it outputs a normal state (S106). Typically, subject P has multiple keypoints, such as the head, shoulders, and hands. The determination here is based on the number of errors equal to or greater than a threshold between the predicted position coordinates of each keypoint and the actual position coordinates of each keypoint, but other determination methods may also be used.

[0026] As shown in Figure 4, the position coordinates of the predicted keypoints in the n+1th frame are obtained from the actual thermo camera images of the n-2th, n-1st, and nth frames. At the same time, the position coordinates of the actual keypoints in the n+1th frame are obtained. Then, by comparing the keypoints in the n+1th frame predicted by the future image generation unit 102 with the keypoints in the n+1th frame actually measured by the comparison thermo camera image input unit 103, it is possible to determine whether an abnormality exists.

[0027] Next, the learning process of the generative model 106a and the keypoint estimation model 107a will be described. Fig. 5 is a diagram showing a system configuration for the learning process of the generative model 106a and the keypoint estimation model 107a of the present disclosure. As shown in the figure, the learning device 200 learns the keypoint estimation model 107a and the generative model 106a based on images including the subject P taken from the thermal camera 300 and the RGB camera 400.

[0028] 6 is a block diagram showing the functional configuration of the learning device 200. The learning device 200 includes a learning thermo camera image input unit 201, an RGB camera image input unit 202, a person keypoint position output unit 203, and an acquisition unit 204. The trained generative model storage unit 106 and the trained keypoint estimation model storage unit 107 are located outside the learning device 200, but may also be provided inside.

[0029] The learning thermo camera image input unit 201 is a part that inputs a thermo camera image of a subject P taken for learning purposes from the thermo camera 300. The subject P may be a person for learning, or may be the same as the subject P in FIG.

[0030] The RGB camera image input unit 202 is a part that inputs an RGB camera image of the subject P captured for learning purposes from the RGB camera 400 .

[0031] The person keypoint position output unit 203 is a unit that acquires and outputs the position coordinates of the keypoints of the subject P based on the RGB camera image. For example, the person keypoint position output unit 203 acquires keypoints using a known body part / joint position detection model (e.g., a model such as Openpose). As described above, in addition to body parts such as the eyes, shoulders, hands, and feet, joint points are acquired as keypoints.

[0032] The acquisition unit 204 acquires training thermo camera images and the position coordinates of the corresponding keypoints, learns the keypoint estimation model 107a and the generative model 106a, and stores them in the trained generative model storage unit 106 and the trained keypoint estimation model storage unit 107, respectively.

[0033] The operation of the learning device 200 configured as described above will be described. Fig. 7 is a flowchart showing the operation of the learning device 200. The learning thermo camera image input unit 201 inputs a learning thermo camera image captured by the thermo camera 300, and the RGB camera image input unit 202 simultaneously acquires an RGB camera image captured by the RGB camera 400 capturing an image of the subject P in a normal state at the same frame rate (S201).

[0034] The person keypoint position output unit 203 executes a detection model (for example, a known model such as OpenPose) of the body parts and joint positions of the subject P on the acquired RGB camera image (S202). That is, the person keypoint position output unit 203 uses the detection model to acquire and output the position coordinates of the keypoints of the subject P.

[0035] The person keypoint position output unit 203 transfers the position coordinates (position labels) of person keypoints detected from the RGB camera image to the training thermocamera image (frames n-2, n-1, n) of the training thermocamera image input unit 201. That is, the person keypoint position output unit 203 performs a process of learning (corresponding) the positions of the keypoints in the training thermocamera image by overlaying the keypoints on the training thermocamera image and outputting the result to the acquisition unit 204 (S203). Note that, during transfer, it is preferable to perform position correction by linear shifting based on the relative positions of the two cameras.

[0036] The acquisition unit 204 then learns a keypoint estimation model 107a (e.g., OpenPose) for thermocamera images that estimates keypoints of people from thermocamera images based on the training thermocamera images and the person keypoints transferred from the RGB camera images to the thermocamera images, and stores the model in the trained keypoint estimation model storage unit 107 (S204). This detection model is a convolutional neural network (CNN) that learns OpenPose for thermocamera images. In other words, the acquisition unit 204 learns a keypoint estimation model for estimating the position coordinates of keypoints from thermocamera images.

[0037] Furthermore, the acquisition unit 204 acquires the position coordinates of the key points of the subject P in the actual next frame (n+1th frame) of the RGB camera image (S205).

[0038] The acquisition unit 204 stores the position coordinates (n, n-1, n-2, and n+1 frames) of the key points in the RGB camera images as training sample data (S206). That is, the acquisition unit 204 stores multiple training sample data including the position coordinates of the training key points in the n, n-1, n-2, and n+1 frames. This training sample data may be for the same subject P or may be for different subjects. Furthermore, various images, such as images from different time periods, may be used as training sample data.

[0039] The acquisition unit 204 then executes the least squares method using the training sample data to determine the weighting coefficients w1, w2, and w3 in the following regression equation, trains a generative model that predicts the position coordinates of keypoints in the n+1 frame, and stores the trained generative model in the trained generative model storage unit 106 (S207): p_n+1=b+w1p_n-2+w2p_n-1+w3p_n... (1) Here, p_n-2 is the position coordinate of the keypoint in the n-2 frame. Similarly, p_n-1, p_n, and p_n+1 are the position coordinates of keypoints included in the n-1, n, and n+1 frames, respectively. w1 to w3 are weighting coefficients. b is a bias.

[0040] For the sake of convenience, the position coordinates of one keypoint have been described, but there are multiple keypoints for subject P, and a generative model (regression equation) exists for each keypoint. Each generative model is trained using the least squares method so that the MSE (Mean Squared Error) is minimized.

[0041] 8 is a schematic diagram of the learning process. As shown in FIG. 8, the acquisition unit 204 estimates the position coordinates of keypoints in frame n+1 from actual frames (frames n, n-1, and n-2) of the RGB camera image using the generative model 106a. Meanwhile, the acquisition unit 204 acquires the actual frame n+1 of the RGB camera image. The acquisition unit 204 uses the estimated keypoints in frame n+1 and the actual keypoints in frame n+1 to train the generative model 106a so as to reduce the error between them.

[0042] Next, a description will be given of the effects of the abnormal condition detection device 100 and learning device 200 of the present disclosure. The abnormal condition detection device 100 of the present disclosure includes a future image generation unit 102 that estimates an estimated state of subject P included in a thermocamera image at the n+1th frame (next time point: second time point) from a thermocamera image (low-resolution image) including subject P captured by a thermocamera 300 (low-resolution camera) at a time point including the nth frame (one time point: first time point), a comparison thermocamera image input unit 103 that acquires the output state of subject P included in the thermocamera image (low-resolution image) at the n+1th frame (next time point), and an abnormal condition detection unit 104 that determines the state of subject P based on the estimated state and the output state.

[0043] According to this configuration, by using a low-resolution camera such as the thermal camera 300 to photograph the subject P and check his / her condition, it is possible to detect abnormal conditions while protecting the subject P's privacy.

[0044] In the present disclosure, the estimated state is the estimated position coordinates (estimated position coordinates) of the key points of the subject P, and the output state is the actual position coordinates (output position coordinates) of the key points of the subject P.

[0045] When the future image generation unit 102 of the present disclosure receives the position coordinates of the key points of the subject P estimated from the thermal camera image, it uses the generation model 106a to estimate the position coordinates of the key points of the subject P included in the low-resolution image at the next point in time (n+1 frame).

[0046] According to this configuration, it is possible to determine whether the subject P is in a normal state or an abnormal state based on the position coordinates of the key points of the subject P. In other words, it is possible to estimate the position coordinates of the key points at the next time point, which is a future state, and determine that an abnormality exists if the estimated position coordinates deviate from the actual position coordinates.

[0047] Furthermore, the future image generation unit 102 of the present disclosure uses the keypoint estimation model 107a, which is the output model, to output the output position coordinates of the keypoints of the subject P included in the low-resolution image at the next time point.

[0048] The low-resolution camera of the present disclosure is a thermography camera or an infrared camera, but may be any other camera that can protect personal privacy.

[0049] The generative model 106 a is trained based on time-series changes in the position coordinates of key points of a training subject photographed by a high-resolution camera such as the RGB camera 400 .

[0050] Furthermore, the keypoint estimation model 107a of the present disclosure is trained based on low-resolution images captured by a thermal camera and position coordinates of keypoints of the training subject, which are derived based on high-resolution images captured by an RGB camera in synchronization with the thermal camera. The training is performed by transferring (associating) the position coordinates of keypoints of the training subject captured by the RGB camera to the thermal camera image.

[0051] This configuration makes it possible to derive the position coordinates of key points of a subject P contained in a low-resolution image, such as a thermal camera image. Furthermore, by training the key point estimation model 107a, it is possible to accurately learn the position coordinates of key points of a person according to the low-resolution image, even in an environment containing, for example, lighting, electrical appliances (such as a PC), and other thermal objects. Therefore, it is possible to estimate the position coordinates of key points of a person from a low-resolution thermal camera while distinguishing them from other thermal objects, and to detect abnormal conditions based on the estimation.

[0052] In addition, in the present disclosure, the learning device 200 includes a learning thermo camera image input unit 201 that acquires thermo camera images captured by the thermo camera 300, an RGB camera image input unit 202 that acquires RGB camera images captured by the RGB camera 400 in synchronization with the thermo camera, and an acquisition unit 204 that functions as a learning unit that learns a key point estimation model 107a that outputs the position coordinates of the key points of the subject from the thermo camera image based on the thermo camera image and the position coordinates of the key points of the subject derived based on the RGB camera image.

[0053] This configuration makes it possible to train a keypoint estimation model 107a that derives the position coordinates of keypoints of a subject from thermal camera images. This can be used to detect abnormal conditions using a thermal camera.

[0054] Next, the configuration of the device and method of the present disclosure will be described.

[0055] [1] An apparatus comprising: an estimation unit that estimates an estimated state of a subject of photography included in a low-resolution image at a next point in time from a low-resolution image including the subject photographed by a low-resolution camera at a certain point in time; an acquisition unit that acquires an output state of the subject of photography included in the low-resolution image at the next point in time; and a judgment unit that judges the state of the subject of photography based on the estimated state and the output state.

[0056] [2] The device according to [1], wherein the estimated state is estimated position coordinates of key points of the subject, and the output state is output position coordinates of key points of the subject.

[0057] [3] The device described in [2], wherein when the estimation unit receives estimated position coordinates of key points of the subject, it estimates the estimated position coordinates using a generative model that estimates estimated position coordinates of key points of the subject included in a low-resolution image at a next time point.

[0058] [4] The device according to [2] or [3], wherein the estimation unit outputs the output position coordinates using an output model that outputs the output position coordinates of key points of the subject included in the low-resolution image at the next time point.

[0059] [5] The device according to any one of [1] to [4], wherein the low-resolution camera is a thermographic camera or an infrared camera.

[0060] [6] The device according to [3], wherein the generative model is trained based on time-series changes in position coordinates of key points of a training subject photographed by a high-resolution camera.

[0061] [7] The device described in [4], wherein the output model is trained based on low-resolution images taken by the low-resolution camera and position coordinates of key points of the training subject derived based on high-resolution images taken by the high-resolution camera in synchronization with the low-resolution camera.

[0062] [8] The device according to [7], wherein the output model is trained by associating position coordinates of key points of the training photographer photographed by the high-resolution camera with the low-resolution image.

[0063] [9] A method comprising: an estimation step of estimating an estimated state of a subject to be photographed that is included in a low-resolution image at a next point in time from a low-resolution image including the subject photographed by a low-resolution camera at a certain point in time; an acquisition step of acquiring an output state of the subject to be photographed that is included in the low-resolution image at the next point in time; and a determination step of determining the state of the subject to be photographed based on the estimated state and the output state.

[0064] The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of hardware and / or software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are connected directly or indirectly (e.g., via wire, wirelessly, etc.) and these multiple devices. The functional block may also be realized by combining the single device or multiple devices with software.

[0065] Functions include, but are not limited to, judgment, determination, assessment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.

[0066] For example, the abnormal condition detection device 100 and the learning device 200 according to an embodiment of the present disclosure may function as a computer that performs processing of the abnormal condition detection method and learning method of the present disclosure. Fig. 9 is a diagram illustrating an example of the hardware configuration of the abnormal condition detection device 100 according to an embodiment of the present disclosure. The learning device 200 also has a similar hardware configuration. The abnormal condition detection device 100 and the learning device 200 described above may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

[0067] In the following description, the term "apparatus" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the abnormal state detection apparatus 100 and the learning apparatus 200 may be configured to include one or more of the apparatuses shown in the figures, or may be configured to exclude some of the apparatuses.

[0068] Each function of the abnormal state detection device 100 and the learning device 200 is realized by loading specified software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communication via the communication device 1004, and control at least one of reading and writing data in the memory 1002 and storage 1003.

[0069] The processor 1001, for example, runs an operating system to control the entire computer. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the above-mentioned future image generation unit 102, future image-comparison image comparison unit 105, abnormal state detection unit 104, person keypoint position output unit 203, acquisition unit 204, etc. may be realized by the processor 1001.

[0070] The processor 1001 also loads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes in accordance with the programs. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. For example, the future image generation unit 102, the future image-comparison image comparison unit 105, the abnormal state detection unit 104, the person keypoint position output unit 203, and the acquisition unit 204 may be implemented by a control program stored in the memory 1002 and running on the processor 1001, and similarly may be implemented for other functional blocks. While the above-described various processes have been described as being executed by a single processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may also be transmitted from a network via a telecommunications line.

[0071] The memory 1002 is a computer-readable recording medium and may be configured, for example, by at least one of a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be referred to as a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing the abnormal state detection method and learning method according to an embodiment of the present disclosure.

[0072] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other appropriate medium including at least one of memory 1002 and storage 1003.

[0073] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as a network device, network controller, network card, or communication module. The communication device 1004 may include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc. to implement at least one of frequency division duplex (FDD) and time division duplex (TDD). For example, the inference thermo camera image input unit 101 and the comparison thermo camera image input unit 103 described above may be implemented by the communication device 1004. The communication device 1004 may be implemented with a transmitter and a receiver that are physically or logically separated.

[0074] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. Note that the input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0075] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0076] Furthermore, abnormal condition detection device 100 and learning device 200 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, processor 1001 may be implemented using at least one of these pieces of hardware.

[0077] The notification of information is not limited to the aspects / embodiments described in the present disclosure and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI) and Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB) and System Information Block (SIB))), other signals, or a combination thereof. Furthermore, the RRC signaling may be referred to as an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.

[0078] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0079] Input and output information may be stored in a specific location (for example, memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.

[0080] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0081] The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0082] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0083] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0084] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0085] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0086] In addition, terms explained in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings.

[0087] Furthermore, the information, parameters, etc. described in the present disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information. For example, a radio resource may be indicated by an index.

[0088] The names used for the above parameters are not limiting in any way, and furthermore, the mathematical formulas etc. using these parameters may differ from those explicitly disclosed in this disclosure.

[0089] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0090] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using one or more wires, cables, and / or printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0091] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0092] As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0093] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

[0094] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0095] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0096] 100...Abnormal state detection device, 200...Learning device, 107a...Keypoint estimation model, 106a...Generation model, 101...Inference thermo camera image input unit, 102...Image generation unit, 103...Comparison thermo camera image input unit, 104...Abnormal state detection unit, 105...Image-comparison image comparison unit, 106...Learned generative model storage unit, 107...Learned keypoint estimation model storage unit, 300...Thermal camera, 400...RGB camera, 201...Learning thermo camera image input unit, 202...RGB camera image input unit, 203...Person keypoint position output unit, 204...Acquisition unit.

Claims

1. An apparatus comprising: an estimation unit that estimates an estimated state of a subject of photography included in a low-resolution image at a next point in time from a low-resolution image including the subject photographed by a low-resolution camera at a certain point in time; an acquisition unit that acquires an output state of the subject of photography included in the low-resolution image at the next point in time; and a judgment unit that judges the state of the subject of photography based on the estimated state and the output state.

2. The device of claim 1, wherein the estimated state is estimated position coordinates of key points of the subject, and the output state is output position coordinates of key points of the subject.

3. The device according to claim 2, wherein when the estimation unit receives estimated position coordinates of key points of the subject, it estimates the estimated position coordinates using a generative model that estimates estimated position coordinates of key points of the subject included in a low-resolution image at a next point in time.

4. The device according to claim 2, wherein the estimation unit outputs the output position coordinates using an output model that outputs the output position coordinates of key points of the subject included in the low-resolution image at the next time point.

5. The device of claim 1, wherein the low-resolution camera is a thermographic camera or an infrared camera.

6. The device according to claim 3, wherein the generative model is trained based on time-series changes in position coordinates of key points of a training subject photographed by a high-resolution camera.

7. The device described in claim 4, wherein the output model is trained based on low-resolution images taken by the low-resolution camera and position coordinates of key points of a training subject derived based on high-resolution images taken by a high-resolution camera in synchronization with the low-resolution camera.

8. The device according to claim 7, wherein the output model is trained by associating position coordinates of key points of a training photographer photographed by the high-resolution camera with the low-resolution image.

9. A learning device comprising: an input unit that acquires low-resolution images taken by a low-resolution camera; an input unit that acquires high-resolution images taken by a high-resolution camera in synchronization with the low-resolution camera; and a learning unit that learns an output model that outputs position coordinates of key points of a subject from the low-resolution images based on the low-resolution images and position coordinates of key points of a subject for training derived based on the high-resolution images.

10. A method comprising: an estimation step of estimating an estimated state of a subject of photography included in a low-resolution image at a next point in time from a low-resolution image including the subject of photography taken by a low-resolution camera at a certain point in time; an acquisition step of acquiring an output state of the subject of photography included in the low-resolution image at the next point in time; and a judgment step of judging the state of the subject of photography based on the estimated state and the output state.

Images