Model generation system

Abstract

The present invention appropriately estimates the trajectory of a person even if low-resolution images captured by a thermo-camera are used.　A trajectory estimation system 10 comprises: an acquisition unit 11 that acquires time-series images obtained by being captured by a thermo-camera; and an estimation unit 12 that, by sequentially inputting the time-series images acquired by the acquisition unit 11 to a model that recursively uses time-series images generated by machine learning, estimates the trajectory of a person appearing in the time-series images.

Description

Technical Field

[0001] The present invention relates to a trajectory estimation system for estimating a person's trajectory and a model generation system for generating a model used in the trajectory estimation system.

Background Art

[0002] Patent Document 1 shows detecting a person shown in each image of a video and estimating the person's trajectory. The camera that captures the images may be an infrared camera capable of photographing a heat source.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Many services starting from a person's location information have been proposed. For example, in commercial facilities, there are efforts to utilize marketing by visualizing the flow of customers who come to the store. It is conceivable to use the trajectory of a person estimated by the above-described conventional technology in the above efforts.

[0005] For example, when a person is imaged at high resolution in a public place, there is a risk of privacy problems such as the ability to identify an individual from the captured image. Therefore, it is conceivable to use a low-resolution image captured by a thermal camera, which is difficult to identify an individual from, for detecting a person's trajectory. However, when the resolution of the image is low, it becomes difficult to estimate the position of a person from the image. In particular, when a plurality of people are shown in the image, it becomes difficult to estimate the position of a person from the image.

[0006] One embodiment of the present invention has been made in view of the above, and aims to provide a trajectory estimation system and a model generation system that can appropriately estimate the trajectory of a person even when using low-resolution images captured by a thermal camera. [Means for solving the problem]

[0007] To achieve the above objective, a trajectory estimation system according to one embodiment of the present invention comprises an acquisition unit that acquires time-series images obtained by imaging with a thermal camera, and an estimation unit that estimates the trajectory of a person in the time-series images by sequentially inputting the time-series images acquired by the acquisition unit into a model that recursively uses time-series images generated by machine learning.

[0008] In a trajectory estimation system according to one embodiment of the present invention, a model is used that recursively uses time-series images generated by machine learning to estimate the trajectory of a person captured in the time-series images. As a result of this estimation, the trajectory estimation system according to one embodiment of the present invention can appropriately estimate the trajectory of a person even when using low-resolution images captured by a thermal camera.

[0009] A model generation system according to one embodiment of the present invention comprises a learning acquisition unit that acquires time-series learning images obtained by imaging with a thermal camera, and learning trajectory information indicating the trajectory of a person in the time-series learning images, and a model generation unit that performs machine learning based on the time-series learning images and learning trajectory information acquired by the learning acquisition unit, and generates a model that recursively uses the time-series images and is used for the trajectory of a person in the time-series images.

[0010] According to a model generation system of one embodiment of the present invention, it is possible to generate a model used in a trajectory estimation system. [Effects of the Invention]

[0011] According to one embodiment of the present invention, even when using low-resolution images captured by a thermal camera, the trajectory of a person can be appropriately estimated. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows the configuration of a server which is a trajectory estimation system and a model generation system according to an embodiment of the present invention. [Figure 2] This is an example of an image that makes up a video used by a trajectory estimation system to estimate a person's trajectory. [Figure 3] This figure shows an example of an image used to estimate a person's trajectory and the estimation result. [Figure 4] This flowchart shows the processes performed by the trajectory estimation system according to the present invention. [Figure 5] This is a flowchart showing the process performed by the model generation system according to an embodiment of the present invention. [Figure 6] This figure shows the hardware configuration of a server that is a trajectory estimation system and a model generation system according to an embodiment of the present invention. [Modes for carrying out the invention]

[0013] Hereinafter, embodiments of the trajectory estimation system and model generation system according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant explanations are omitted.

[0014] Figure 1 shows the server 1, which is the trajectory estimation system 10 and model generation system 20 according to this embodiment. The trajectory estimation system 10 is a system (device) that analyzes time-series images (video) obtained by imaging with a thermal camera 100 and estimates the trajectory of a person captured in the time-series images. That is, the trajectory estimation system 10 is a system that tracks a person from time-series images obtained by imaging with a thermal camera 100. The estimated trajectory is, for example, the trajectory in the space being imaged by the thermal camera (thermography) 100, or the trajectory in the image corresponding to that space. The estimated trajectory may also be indicated by the relative position to the position of the thermal camera 100. The space in which the trajectory of a person is estimated is indoors. However, the space may be other than indoors.

[0015] The video footage used by the trajectory estimation system 10 to estimate a person's trajectory is low-resolution video footage. Figure 2 shows an example of the images that make up the video footage used by the trajectory estimation system 10 to estimate a person's trajectory. Note that the images in Figure 2 also show the portion of the person detected from the image (the portion indicated by the rectangle in the image).

[0016] The image in Figure 2(a) has a resolution of 12 pixels vertically x 16 pixels horizontally. The image in Figure 2(b) has a resolution of 18 pixels vertically x 24 pixels horizontally. The image in Figure 2(c) has a resolution of 24 pixels vertically x 32 pixels horizontally. The resolution of these images is less than 1 / 10th the resolution of a typical commercially available infrared thermal camera.

[0017] As shown in Figure 2, the above images are of such low quality that it is difficult to smoothly identify the same person even when comparing two images of the same person. Furthermore, the above images are of such low quality that individual identification is impossible by visual inspection.

[0018] Thus, since the moving images used by the trajectory estimation system 10 are of low resolution, it is possible to estimate the trajectory of a person using moving images in places where it is not appropriate to identify an individual. That is, the trajectory estimation system 10 can realize a sensing technology that takes privacy into consideration. For example, the trajectory estimation system 10 estimates the trajectory (human behavior) of a person in a residence such as a home or a public facility such as a hotel, which is an area with high privacy concerns. The trajectory estimated by the trajectory estimation system 10 can be utilized for any service or the like triggered by the person's position. For example, the estimated trajectory may be used for proposing an interaction using the person's trajectory.

[0019] The estimation of the trajectory of a person by the trajectory estimation system 10 is performed using a model generated by machine learning. The model generation system 20 is a system (device) that generates a model used for estimating the trajectory of a person by the trajectory estimation system 10.

[0020] As the server 1 which is the trajectory estimation system 10 and the model generation system 20 according to this embodiment, a conventional server device or a computer such as a PC (personal computer) can be used. Also, the server 1 may be a computer system including a plurality of computers. Further, the server 1 may have a communication function and be able to transmit and receive information to and from other devices.

[0021] The thermal camera 100 is a device that performs imaging and acquires a moving image used by the trajectory estimation system 10. The thermal camera 100 continuously images the space of the imaging target and acquires a moving image of the thermal distribution in the imaging target. As described above, the moving image captured by the thermal camera 100 is a low-resolution moving image. The thermal camera 100 is pre-positioned and fixedly installed at a position where it can image a person related to the trajectory that is the estimation target of the trajectory estimation system 10. The thermal camera 100 has a communication function (for example, a communication function of a wireless LAN (Local Area Network)), and transmits the moving image obtained by imaging to the server 1 by the communication function. As the functional unit for performing imaging of the thermal camera 100, a module of a conventional thermal camera (for example, an MLX90640 thermal camera unit) can be used.

[0022] The learning thermal camera 200 is a device that performs imaging and acquires a moving image used by the model generation system 20. The learning thermal camera 200 has the same functions as the thermal camera 100 and acquires a moving image similar to the moving image obtained by the imaging of the thermal camera 100. A moving image similar to the moving image obtained by the imaging of the thermal camera 100 is, for example, a moving image whose resolution and frame rate are the same as those of the moving image obtained by the imaging of the thermal camera 100. Note that a moving image similar to the moving image obtained by the imaging of the thermal camera 100 does not necessarily have to have the same resolution and frame rate, and any moving image can be used as long as it can be appropriately used to generate a model for estimating the trajectory of a person from the moving image obtained by the imaging of the thermal camera 100.

[0023] The learning thermal camera 200 may be the same as or of the same type as the thermal camera 100. The learning thermal camera 200 is pre-positioned and fixedly installed at a position where it can image a person related to the trajectory for creating a model. The learning thermal camera 200 has a communication function (for example, a communication function of a wireless LAN), and transmits the moving image obtained by imaging to the server 1 by the communication function.

[0024] The learning RGB camera 201 is a device that takes images and acquires moving images to be used by the model generation system 20. As will be described later, the moving images obtained by the learning RGB camera 201 are used to identify the trajectory of a person. The learning RGB camera 201 continuously takes images of the space targeted by the learning thermal camera 200 at the same timing as when the learning thermal camera 200 takes images, and acquires moving images of the target area in visible light.

[0025] The video images captured by the learning RGB camera 201 are of a resolution that allows for the identification of a person's trajectory. Unlike the video images captured by the thermal camera 100 and the learning thermal camera 200, the video images captured by the learning RGB camera 201 do not need to be of low resolution. The learning RGB camera 201 is pre-positioned and fixedly installed at a location corresponding to the location where the learning thermal camera 200 is installed, as described above. The learning RGB camera 201 has a communication function (e.g., wireless LAN communication function) and transmits the video images obtained through imaging to the server 1 using this communication function. A conventional RGB camera module can be used as the learning RGB camera 201 (the function unit that performs imaging).

[0026] Next, the functions of the trajectory estimation system 10 and the model generation system 20 according to this embodiment will be described. As shown in Figure 1, the trajectory estimation system 10 according to this embodiment comprises an acquisition unit 11 and an estimation unit 12.

[0027] The acquisition unit 11 is a functional unit that acquires time-series images obtained by imaging with the thermal camera 100. The acquisition unit 11 may acquire images with a vertical and horizontal pixel count of 32 or less as each image in the time series.

[0028] The acquisition unit 11 receives and acquires a video, which is a time-series image transmitted from the thermal camera 100. As shown in the example in Figure 2, the images that make up the video, i.e., each image in the time series, may be low-resolution images with 32 or fewer pixels in each direction (vertical and horizontal). However, each image in the time series may be any other type of video, as long as it is a time-series image obtained by imaging with the thermal camera and can be used to estimate the trajectory of a person. The acquisition unit 11 may also acquire time-series images obtained by imaging with the thermal camera 100 by a method other than those described above. The acquisition unit 11 outputs the acquired video to the estimation unit 12.

[0029] The estimation unit 12 is a functional unit that estimates the trajectory of a person in a time-series image by sequentially inputting the time-series images acquired by the acquisition unit 11 into a model that recursively uses time-series images generated by machine learning. The model may include a recurrent neural network. The estimation unit 12 estimates the person's trajectory as follows, for example.

[0030] The estimation unit 12 inputs and stores the model generated by the model generation system 20. An example of a model used by the estimation unit 12 to estimate a person's trajectory is described below. The model is a trained model generated by machine learning. The model takes a time-series of images as input and outputs information indicating the trajectory of the person in the time-series of images. Specifically, the model takes the time-series of images in chronological order, starting with the oldest, and outputs an identifier (ID) that identifies the person in the image and information indicating the person's position for each input image.

[0031] For example, the image shown on the left side of Figure 3 is the image input to the model, and the position of the rectangle and the number in the upper left corner of the rectangle in the image shown on the right side of Figure 3 are the information indicating the position of a person and the identifier that identifies the person, which are output from the model. By arranging the information indicating the position of a person output from the model in chronological order for each identifier that identifies that person, the trajectory of that person is obtained. The person identifier is assigned by the model so that the same person has the same identifier across images, and different people have different identifiers.

[0032] The model is constructed using a neural network. Furthermore, the model includes a recurrent neural network and recursively uses the input time-series images to calculate the position of the person it outputs. Specifically, each time an image is input, the model uses the information of that image to calculate the position of the person in that image, but this calculation uses the results of calculations performed within the model on previously input images.

[0033] The model's input layer includes neurons for inputting images that make up the moving image. For example, the model's input layer includes neurons that receive the pixel values ​​of a number of pixels in the image. The model's output layer includes neurons for outputting information indicating the trajectory of a person. For example, the model's output layer includes neurons that output the identifier of a person in the input image and the position of that person (for example, the rectangular area in the image in which the person is located, as shown in Figure 2).

[0034] The above model may be a conventional pre-trained model, such as DeepSORT which tracks people. However, the model may be any other model that recursively uses time-series images generated by machine learning and can be used to estimate the trajectory of a person in the time-series images.

[0035] The pre-trained model used in the estimation unit 12 is intended to be used as a program module that is part of artificial intelligence software. The model is used, for example, in a computer equipped with hardware such as a CPU (Central Processing Unit) and memory, and the computer's CPU operates according to instructions from the model stored in memory. For example, the computer's CPU operates in accordance with the instructions to input information to the model, perform calculations according to the model, and output a result from the model. Specifically, the computer's CPU operates in accordance with the instructions to input information to the input layer of the neural network, perform calculations based on the pre-trained weighting coefficients in the neural network, and output a result from the output layer of the neural network.

[0036] The estimation unit 12 receives video footage from the acquisition unit 11. The estimation unit 12 sequentially inputs the video footage received from the acquisition unit 11 into a pre-stored model and performs calculations according to the model, obtaining information indicating the time-series position of the person in the video footage as output from the model. The obtained information indicating the time-series position of the person is the result of the estimation of the person's trajectory by the estimation unit 12. The acquisition of time-series images by the acquisition unit 11 and the estimation of the person's trajectory by the estimation unit 12 may be performed in real time.

[0037] The estimation unit 12 outputs information indicating the estimation result of the person's trajectory. For example, the estimation unit 12 transmits the information indicating the estimation result to the user's terminal. Alternatively, the estimation unit 12 may output the information indicating the estimation result in a format recognizable by the user. For example, the estimation unit 12 may display the information indicating the estimation result on a display device provided by the server 1. The information indicating the estimation result may be, for example, an image obtained by imaging with the thermal camera 100, on which information indicating the person's position (for example, the display of the rectangle in the figure) and the person's ID (for example, the number in the upper left corner of the rectangle in the figure) are superimposed. Furthermore, the information indicating the estimation result may be anything other than the above, as long as it indicates the estimated trajectory of the person. The output of the information indicating the estimation result from the estimation unit 12 may be performed by a method other than those described above and to an output destination other than those described above. The above describes the functions of the trajectory estimation system 10 according to this embodiment.

[0038] As shown in Figure 1, the model generation system 20 according to this embodiment includes a learning acquisition unit 21 and a model generation unit 22.

[0039] The learning acquisition unit 21 is a functional unit that acquires time-series learning images obtained by imaging with a thermal camera (learning thermal camera 200), and learning trajectory information showing the trajectories of people in the time-series learning images. The learning acquisition unit 21 may also acquire time-series learning images that include images showing lights. The learning acquisition unit 21 may also acquire time-series trajectory estimation images, which are of a different type from the time-series learning images and are obtained by imaging corresponding to the time-series learning images, and estimate the trajectories of people in the acquired time-series trajectory estimation images to acquire learning trajectory information. The learning acquisition unit 21 acquires each piece of information as follows, for example.

[0040] The time-series training images and training trajectory information acquired by the training data acquisition unit 21 are training data (teaching data) used to generate the model. The training data acquisition unit 21 acquires a sufficient number of time-series training images and training trajectory information to generate an appropriate model.

[0041] The learning acquisition unit 21 receives and acquires a time-series of learning images, which are moving images, transmitted from the learning thermal camera 200. As described above, the images that make up the moving images acquired by the learning acquisition unit 21, that is, each learning image in the time series, are low-resolution images similar to the images that make up the moving images acquired by the acquisition unit 11.

[0042] Furthermore, when imaging is performed by the learning thermal camera 200, the camera may be placed in the space being imaged by the learning thermal camera 200 so that at least one of the time-series learning images acquired by the learning acquisition unit 21 includes an image of a light that could be a heat source. In other words, at least one of the time-series learning images may be an image of a person and a light. By making the time-series learning images such images, the estimation accuracy of the model can be improved.

[0043] Furthermore, each time-series training image corresponds to a time-series image acquired by the acquisition unit 11 and may be any other image as long as it is used appropriately for model generation. The training acquisition unit 21 may also acquire time-series training images obtained by imaging with the training thermal camera 200 by a method other than those described above.

[0044] The learning acquisition unit 21 acquires not only time-series learning images (videos) but also learning trajectory information that shows the trajectory of people in the video. The learning trajectory information is a label for model training. The learning trajectory information is, for example, an identifier (ID) that identifies the person in each learning image for each timing related to the video, and information indicating the position of that person. The learning trajectory information is the same type of information as the information output from the model. The time-series learning images and learning trajectory information acquired by the learning acquisition unit 21 correspond to the time-series images acquired by the acquisition unit 11 and the information output from the model, respectively, as training data used to generate the model. The learning acquisition unit 21 acquires learning trajectory information, for example, as follows:

[0045] The learning acquisition unit 21 receives and acquires video footage transmitted from the learning RGB camera 201 in order to acquire learning trajectory information. The video footage transmitted from the learning RGB camera 201 is visible light video footage (RGB video footage), which is of a different type from the video footage (thermal distribution video footage) transmitted from the learning thermal camera 200. The video footage transmitted from the learning RGB camera 201 does not necessarily have to be visible light video footage (RGB video footage); it is sufficient if it can identify the trajectory of the person captured in the video for the purpose of generating the model. For example, the video footage used to acquire learning trajectory information may be thermal distribution video footage with a higher resolution than the thermal distribution video footage obtained by imaging with the thermal camera 100, and may be high-resolution video footage in which the position of the person can be appropriately identified.

[0046] The video transmitted from the learning RGB camera 201 is, for example, captured over the same period as the video transmitted from the learning thermal camera 200, and has the same frame rate as the video transmitted from the learning thermal camera 200. This is to associate each time-series learning image with each position of the person that constitutes the person's trajectory, as shown by the learning trajectory information, as will be described later. However, if this association is possible, the video transmitted from the learning RGB camera 201 does not need to be as described above.

[0047] The learning acquisition unit 21 estimates the trajectory of the person in the video from the acquired video and generates learning trajectory information. For example, the learning acquisition unit 21 detects the position of the person in each image that makes up the video and assigns an identifier (ID) to identify the person. The person identifier is the same for the same person across images, and different identifiers are assigned for different people.

[0048] The detection of a person's position and the assignment of an identifier can be performed using conventional methods, such as conventional person detection models (YOLO (You Only Look Once)). As mentioned above, the video images obtained by the training RGB camera 201 do not need to be low resolution, unlike the video images obtained by the thermal camera 100 and the training thermal camera 200, so the position of a person can be detected using conventional methods.

[0049] The learning acquisition unit 21 generates a person's trajectory and identifier, which will become learning trajectory information, based on the person's trajectory and identifier obtained as described above. Since the learning thermal camera 200 and the learning RGB camera 201 are positioned at different locations, the relative position of the person with respect to the learning thermal camera 200 is different from the relative position of the person with respect to the learning RGB camera 201. Therefore, the learning acquisition unit 21 may store the positional relationship (relative position) between the learning thermal camera 200 and the learning RGB camera 201 in advance, and correct the obtained person's trajectory based on the stored positional relationship so that it becomes the trajectory of the position detected by the learning thermal camera 200. The above correction based on the positional relationship is, for example, a linear shift correction, which can be performed by conventional methods.

[0050] The learning acquisition unit 21 associates each learning image in the time series with each position of the person that constitutes the person's trajectory, as shown by the learning trajectory information. The learning acquisition unit 21 also associates the learning image with the person's position for the same time period.

[0051] The learning acquisition unit 21 may acquire learning trajectory information by methods other than those described above. For example, it may acquire learning trajectory information corresponding to time-series learning images obtained by imaging with the learning thermal camera 200, which has been prepared in advance by any method. In this case, the learning acquisition unit 21 does not need to acquire the video images obtained by imaging with the learning RGB camera 201. The learning acquisition unit 21 outputs the acquired video images obtained by imaging with the learning thermal camera 200 and the learning trajectory information to the estimation unit 12.

[0052] The model generation unit 22 is a functional unit that performs machine learning based on time-series training images and training trajectory information acquired by the training acquisition unit 21, and recursively uses the time-series images to generate a model used for the trajectory of the person depicted in the time-series images. The model generation unit 22 generates a model, for example, as follows:

[0053] The model generation unit 22 receives time-series training images and training trajectory information corresponding to those time-series training images from the training acquisition unit 21. The model generation unit 22 uses the input time-series training images as input to the model and the input training trajectory information as the model's output (ground truth) to perform machine learning and generate a model. When using time-series training images as input to the model, the model generation unit 22 inputs the time-series training images in chronological order, starting with the oldest. When using training trajectory information as input to the model, the model generation unit 22 ensures that the identifier of the person and the information indicating the person's position at the time corresponding to the training image input to the model correspond to the model's output. The machine learning training itself, i.e., the updating of the model parameters, can be performed in the same way as conventional machine learning training.

[0054] In the model, the number of past images used recursively (the number of past images traced back from the input image) is set in advance. Low-resolution images make it difficult to capture a person's physical characteristics. Therefore, using too many past images makes it difficult to associate images with the same person because the person's posture and orientation will differ too much. For this reason, the number of past images used recursively may be kept to a minimum. For example, the number of past images used recursively may be set to 10 through parameter tuning.

[0055] The model generation unit 22 outputs the generated model to the trajectory estimation system 10. In the trajectory estimation system 10, the estimation unit 12 inputs the model, stores it, and uses it to estimate the trajectory of the person described above. The above describes the functions of the model generation system 20 according to this embodiment.

[0056] Next, the processes performed by Server 1 according to this embodiment (the methods performed by Server 1) will be explained using the flowcharts in Figures 4 and 5. First, the processes performed by the trajectory estimation system 10 according to this embodiment will be explained using the flowchart in Figure 4.

[0057] In this process, first, the acquisition unit 11 acquires a time-series of images obtained by imaging with the thermal camera 100 (S01). Next, the estimation unit 12 inputs the time-series images sequentially into a model and estimates the trajectory of the person captured in the time-series images (S02). Subsequently, the estimation unit 12 outputs information indicating the estimation result (S03). The above is the process executed by the trajectory estimation system 10 according to this embodiment.

[0058] Next, the process performed by the model generation system 20 according to this embodiment will be explained using the flowchart in Figure 5. This process is performed before the process is performed by the trajectory estimation system 10 according to this embodiment.

[0059] In this process, first, the learning acquisition unit 21 acquires time-series learning images obtained by imaging with the learning thermal camera 200 and time-series RGB images obtained by imaging with the learning RGB camera 201 (S11). Next, the learning acquisition unit 21 estimates the trajectory of a person from the time-series RGB images and generates learning trajectory information (S12). Subsequently, the model generation unit 22 performs machine learning based on the time-series learning images and learning trajectory information to generate a model (S13). The generated model is output to the trajectory estimation system 10, stored by the estimation unit 12, and used for estimating the trajectory of a person as described above. The above is the process executed by the model generation system 20 according to this embodiment.

[0060] As described above, in the trajectory estimation system 10 according to this embodiment, the trajectory of a person is estimated using low-resolution images captured by the thermal camera 100, as shown in Figure 2. Normally, it is difficult to detect the position of a person and estimate their trajectory from such low-resolution images. In particular, when people overlap in the image, the trajectory of a person is an estimation.

[0061] In response to this, the trajectory estimation system 10 according to this embodiment uses a model that recursively uses time-series images generated by machine learning to estimate the trajectory of a person in the time-series images. Through this estimation, the trajectory estimation system 10 according to this embodiment can appropriately estimate the trajectory of a person even when using low-resolution images captured by the thermal camera 100. In particular, it can appropriately estimate the trajectory of a person even when people overlap in the image. Furthermore, by using low-resolution images, it is possible to estimate the trajectory of a person while considering privacy and not identifying individuals.

[0062] Furthermore, as mentioned above, each time-series image used to estimate a person's trajectory may be an image with 32 or fewer pixels in both the horizontal and vertical directions. However, each time-series image used to estimate a person's trajectory may have a resolution other than that specified above.

[0063] Furthermore, a model including a recurrent neural network, such as DeepSORT, may be used to estimate the trajectory of a person. With this configuration, time-series images can be used appropriately and reliably to estimate the trajectory of a person. However, the model used to estimate the trajectory of a person does not necessarily have to include a recurrent neural network; any model that recursively uses time-series images generated by machine learning will suffice.

[0064] The model generation system 20 according to this embodiment can generate a model used in the trajectory estimation system 10. Furthermore, the time-series training images used for model generation may include images showing lights. This configuration can improve the estimation accuracy of the generated model. However, the time-series training images used for model generation do not necessarily have to include images showing lights.

[0065] Furthermore, as mentioned above, in model generation, time-series trajectory estimation images (for example, time-series RGB images) may be acquired, and the trajectory of the person depicted in the acquired time-series trajectory estimation images may be estimated to obtain trajectory estimation images. With this configuration, training trajectory information can be easily and appropriately acquired, and as a result, a model can be easily and appropriately generated. However, the trajectory estimation images do not necessarily have to be acquired in the manner described above; they can be acquired by any method.

[0066] In this embodiment, Server 1 includes a trajectory estimation system 10 and a model generation system 20. However, the trajectory estimation system 10 and the model generation system 20 may be implemented independently of each other. That is, the trajectory estimation system 10 and the model generation system 20 may be implemented on separate devices.

[0067] The block diagrams used in the description of the above embodiments show functional units. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or it may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired or wireless connections). A functional block may also be realized by combining the above one device or the above multiple devices with software.

[0068] Functions include, but are not limited to, judgment, decision, determination, calculation, calculation, processing, derivation, investigation, exploration, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, assumption, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating (mapping), and assigning. For example, a functional block (configuration part) that enables transmission is called a transmitting unit or transmitter. In all cases, as mentioned above, the method of implementation is not particularly limited.

[0069] For example, Server 1 in one embodiment of the present disclosure may function as a computer that performs information processing according to the present disclosure. Figure 6 is a diagram showing an example of the hardware configuration of Server 1 according to one embodiment of the present disclosure. The Server 1 described above may be physically configured as a computer device including a processor 1001, memory 1002, storage 1003, communication device 1004, input device 1005, output device 1006, bus 1007, etc.

[0070] In the following explanation, the term "device" can be replaced with "circuit," "device," "unit," etc. The hardware configuration of Server 1 may include one or more of the devices shown in the diagram, or it may be configured to omit some of the devices.

[0071] Each function in Server 1 is realized by loading predetermined software (programs) onto hardware such as the processor 1001 and memory 1002, which allows the processor 1001 to perform calculations, control communication by the communication device 1004, and control at least one of the reading and writing of data in the memory 1002 and storage 1003.

[0072] The processor 1001 controls the entire computer, for example, by running an operating system. The processor 1001 may be composed of a central processing unit (CPU) that includes interfaces with peripheral devices, control units, arithmetic units, registers, etc. For example, each of the functions in the server 1 described above may be implemented by the processor 1001.

[0073] Furthermore, the processor 1001 reads programs (program code), software modules, data, etc., from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes accordingly. The program used is one that causes the computer to execute at least a part of the operations described in the above embodiment. For example, each function in the server 1 may be implemented by a control program stored in the memory 1002 and running on the processor 1001. Although the above-described processes have been executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may also be transmitted from the network via a telecommunications line.

[0074] Memory 1002 is a computer-readable recording medium and may consist of at least one of the following: ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. Memory 1002 may also be called a register, cache, main memory, etc. Memory 1002 can store executable programs (program code), software modules, etc., for carrying out information processing according to one embodiment of this disclosure.

[0075] Storage 1003 is a computer-readable recording medium and may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disc, a digital multipurpose disc, a Blu-ray® disc), a smart card, flash memory (e.g., a card, a stick, a key drive), a floppy® disk, a magnetic strip, etc. Storage 1003 may also be called an auxiliary storage device. The storage medium provided by Server 1 may be, for example, a database, server, or other suitable medium including at least one of memory 1002 and storage 1003.

[0076] The communication device 1004 is hardware (transceiver / receiver device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as a network device, network controller, network card, communication module, etc.

[0077] The input device 1005 is an input device that accepts input from an external source (e.g., a keyboard, mouse, microphone, switch, button, sensor, etc.). The output device 1006 is an output device that outputs to an external source (e.g., a display, speaker, LED lamp, etc.). The input device 1005 and the output device 1006 may be configured as an integrated unit (e.g., a touch panel).

[0078] Furthermore, each device, such as the processor 1001 and memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or different buses may be configured for each device.

[0079] Furthermore, server 1 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array), and some or all of each functional block may be realized by such hardware. For example, processor 1001 may be implemented using at least one of these hardware components.

[0080] The processing procedures, sequences, flowcharts, etc., of each aspect / embodiment described herein may be reordered, provided they are consistent with each other. For example, the methods described herein present various step elements in an exemplary order and are not limited to that specific order.

[0081] Input and output information may be stored in a specific location (e.g., memory) or managed using a management table. Input and output information may be overwritten, updated, or appended to. Output information may be deleted. Input information may be transmitted to other devices.

[0082] The determination may be made by a value represented by 1 bit (0 or 1), by a boolean value (true or false), or by a numerical comparison (for example, a comparison with a predetermined value).

[0083] Each aspect / embodiment described herein may be used individually, in combination, or switched between as needed during implementation. Furthermore, notification of specific information (e.g., notification that "X is") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).

[0084] Although the present disclosure has been described in detail above, it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the intent and scope of the present disclosure as defined by the claims. Therefore, the descriptions in the present disclosure are illustrative and not intended to be restrictive in any way.

[0085] Software should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, and so on, whether they are called software, firmware, middleware, microcode, hardware description languages, or by any other name.

[0086] Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL)) and wireless technology (such as infrared or microwave), then at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0087] The terms “system” and “network” as used in this disclosure are interchangeable.

[0088] Furthermore, the information, parameters, etc., described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0089] As used in this disclosure, the terms “determining” and “determining” may encompass a wide variety of actions. “Determining” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiry (e.g., searching in a table, database, or other data structure), and ascertaining. “Determining” may also include, for example, receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and accessing (e.g., accessing data in memory). Furthermore, "judgment" and "decision" can include considering something as having been "judged" or "decided" after resolving, selecting, choosing, establishing, comparing, etc. In other words, "judgment" and "decision" can include considering something as having been "judged" or "decided" after some action. Also, "judgment (decision)" can be reinterpreted as "assuming," "expecting," or "considering."

[0090] The terms “connected,” “coupled,” or any variation thereof, mean any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” with each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, “connection” may be reinterpreted as “access.” As used in this disclosure, two elements may be considered to be “connected” or “coupled” with each other using at least one of one or more wires, cables, and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain, and optical (both visible and invisible) domain.

[0091] In this disclosure, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on."

[0092] Any reference to elements using the designations “first,” “second,” etc., as used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Accordingly, references to the first and second elements do not imply that only two elements may be employed, or that the first element must precede the second element in any way.

[0093] Where the terms “include,” “including,” and variations thereof are used in this disclosure, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to mean exclusive OR.

[0094] In this disclosure, if articles are added through translation, such as a, an, and the in English, this disclosure may include the fact that the noun following these articles is plural.

[0095] In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combine" may be interpreted similarly to "different."

[0096] The population status determination system and model generation system of this disclosure have the following configurations. [1] An acquisition unit that acquires time-series images obtained by imaging with a thermal camera, An estimation unit estimates the trajectory of a person in a time-series image by sequentially inputting the time-series images acquired by the acquisition unit into a model that recursively uses time-series images generated by machine learning. A trajectory estimation system equipped with the following features. [2] The acquisition unit acquires images with a vertical and horizontal pixel count of 32 or less as each image in the time series, as the trajectory estimation system according to [1]. [3] The model is a trajectory estimation system according to [1] or [2], which includes a recurrent neural network. [4] A learning acquisition unit that acquires time-series learning images obtained by imaging with a thermal camera, and learning trajectory information showing the trajectory of a person in the time-series learning images, A model generation unit performs machine learning based on the time-series training images and training trajectory information acquired by the aforementioned training acquisition unit, and recursively uses the time-series images to generate a model used for the trajectory of a person depicted in the time-series images. A model generation system equipped with the following features. [5] The model generation system described in [4], wherein the learning acquisition unit acquires the time-series learning images including images in which a light is visible. [6] The model generation system according to [4] or [5], wherein the learning acquisition unit acquires a time-series trajectory estimation image obtained by imaging which is of a different type from the time-series learning image and corresponds to the time-series learning image, estimates the trajectory of a person in the acquired time-series trajectory estimation image, and acquires the learning trajectory information. [Explanation of symbols]

[0097] 1...Server, 10...Trajectory estimation system, 11...Acquisition unit, 12...Estimation unit, 20...Model generation system, 21...Training acquisition unit, 22...Model generation unit, 100...Thermal camera, 200...Training thermal camera, 201...Training RGB camera, 1001...Processor, 1002...Memory, 1003...Storage, 1004...Communication device, 1005...Input device, 1006...Output device, 1007...Bus.

Claims

1. A learning acquisition unit that acquires time-series learning images obtained by imaging with a thermal camera, and learning trajectory information showing the trajectory of a person captured in the time-series learning images, A model generation unit performs machine learning based on the time-series training images and training trajectory information acquired by the aforementioned training acquisition unit, and recursively uses the time-series images to generate a model used for the trajectory of a person depicted in the time-series images. Equipped with, The aforementioned learning acquisition unit is a model generation system that acquires time-series trajectory estimation images in visible light obtained by imaging corresponding to the time-series learning images, estimates the trajectory of a person depicted in the acquired time-series trajectory estimation images, and acquires the learning trajectory information.

2. The model generation system according to claim 1, wherein the learning acquisition unit acquires the time-series learning images, including images in which a light is visible.

Images