Trajectory estimation system and model generation system

Abstract

The present invention appropriately estimates the trajectory of a person even if low-resolution images captured by a thermo-camera are used.　A trajectory estimation system 10 comprises: an acquisition unit 11 that acquires time-series images obtained by being captured by a thermo-camera; and an estimation unit 12 that, by sequentially inputting the time-series images acquired by the acquisition unit 11 to a model that recursively uses time-series images generated by machine learning, estimates the trajectory of a person appearing in the time-series images.

Description

Trajectory estimation system and model generation system

[0001] The present invention relates to a trajectory estimation system that estimates the trajectory of a person, and a model generation system that generates a model used in the trajectory estimation system.

[0002] Patent Literature 1 discloses a method for detecting a person in each image of a video and estimating the person's trajectory. The camera for capturing the images may be a far-infrared camera that can capture images of heat sources.

[0003] JP 2010-63001 A

[0004] Many services based on people's location information have been proposed. For example, commercial facilities are working to visualize the movement of customers and use this information for marketing purposes. It is conceivable that the trajectories of people estimated using the conventional technology described above could be used for these efforts.

[0005] For example, capturing a person in a public place with high resolution can lead to privacy issues, such as the possibility of identifying the individual from the captured image. Therefore, it is conceivable to use low-resolution images captured by a thermal camera, which make it difficult to identify the individual from the image, to detect the person's trajectory. However, low image resolution makes it difficult to estimate the position of the person from the image. This is particularly difficult when multiple people are captured in the image.

[0006] One embodiment of the present invention has been made in consideration of the above, and aims to provide a trajectory estimation system and a model generation system that can appropriately estimate a person's trajectory even when using low-resolution images captured by a thermal camera.

[0007] In order to achieve the above object, a trajectory estimation system according to one embodiment of the present invention includes an acquisition unit that acquires time-series images obtained by imaging with a thermal camera, and an estimation unit that estimates the trajectory of a person appearing in the time-series images by sequentially inputting the time-series images acquired by the acquisition unit into a model that recursively uses the time-series images generated by machine learning.

[0008] In a trajectory estimation system according to an embodiment of the present invention, a model that recursively uses time-series images generated by machine learning is used to estimate the trajectories of people appearing in the time-series images. With this estimation, the trajectory estimation system according to an embodiment of the present invention can appropriately estimate the trajectories of people even when using low-resolution images captured by a thermal camera.

[0009] A model generation system according to one embodiment of the present invention includes a learning acquisition unit that acquires time-series learning images obtained by imaging with a thermal camera and learning trajectory information indicating the trajectories of people appearing in the time-series learning images, and a model generation unit that performs machine learning based on the time-series learning images and learning trajectory information acquired by the learning acquisition unit, and recursively uses the time-series images to generate a model to be used for the trajectories of people appearing in the time-series images.

[0010] According to a model generation system according to an embodiment of the present invention, it is possible to generate a model used in a trajectory estimation system.

[0011] According to one embodiment of the present invention, even when a low-resolution image captured by a thermal camera is used, the trajectory of a person can be appropriately estimated.

[0012] FIG. 1 is a diagram showing the configuration of a server that is a trajectory estimation system and a model generation system according to an embodiment of the present invention. FIG. 2 is an example of images that constitute a moving image used to estimate a person's trajectory by the trajectory estimation system. FIG. 3 is a diagram showing examples of images used to estimate a person's trajectory and estimation results. FIG. 4 is a flowchart showing processing executed by the trajectory estimation system according to an embodiment of the present invention. FIG. 5 is a flowchart showing processing executed by the model generation system according to an embodiment of the present invention. FIG. 6 is a diagram showing the hardware configuration of a server that is a trajectory estimation system and a model generation system according to an embodiment of the present invention.

[0013] Hereinafter, embodiments of a trajectory estimation system and a model generation system according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals and duplicated explanations will be omitted.

[0014] FIG. 1 shows a server 1 that is a trajectory estimation system 10 and a model generation system 20 according to this embodiment. The trajectory estimation system 10 is a system (device) that analyzes time-series images (video) captured by a thermal camera 100 and estimates the trajectory of a person appearing in the time-series images. That is, the trajectory estimation system 10 is a system that tracks a person from the time-series images captured by the thermal camera 100. The estimated trajectory is, for example, a trajectory in the space of the object captured by the thermal camera (thermography) 100, or a trajectory in an image corresponding to the space. The estimated trajectory may be indicated by a relative position with respect to the position of the thermal camera 100. The space in which the person's trajectory is estimated is indoors. However, the space may be other than indoors.

[0015] The moving images used to estimate a person's trajectory by the trajectory estimation system 10 are low-resolution moving images. Fig. 2 shows an example of images that make up the moving images used to estimate a person's trajectory by the trajectory estimation system 10. Note that the image shown in Fig. 2 also shows the part of the person detected from the image (the part indicated by a rectangle in the image).

[0016] The resolution of the image in Fig. 2(a) is 12 pixels (vertical) x 16 pixels (horizontal). The resolution of the image in Fig. 2(b) is 18 pixels (vertical) x 24 pixels (horizontal). The resolution of the image in Fig. 2(c) is 24 pixels (vertical) x 32 pixels (horizontal). The resolution of these images is 1 / 10 or less of the resolution of a typical commercially available infrared thermal camera.

[0017] As shown in Figure 2, the image quality of the above-mentioned images is so low that it is not possible to easily identify the same person when comparing two images of the same person. Also, the image quality of the above-mentioned images is so low that it is not possible to visually identify the individual.

[0018] As described above, the video images used by the trajectory estimation system 10 have low resolution, making it possible to estimate a person's trajectory using video images in places where it is not appropriate to identify individuals. In other words, the trajectory estimation system 10 can realize a sensing technology that takes privacy into consideration. For example, the trajectory estimation system 10 estimates a person's trajectory (person's behavior) in an area where privacy is a high concern, such as a residence, such as a home, or a public facility, such as a hotel. The trajectory estimated by the trajectory estimation system 10 can be used for any service triggered by the person's position. For example, the estimated trajectory may be used to suggest an interaction using the person's trajectory.

[0019] The trajectory estimation system 10 estimates a trajectory of a person using a model generated by machine learning. The model generation system 20 is a system (device) that generates a model used for the trajectory estimation system 10 to estimate a trajectory of a person.

[0020] The server 1, which is the trajectory estimation system 10 and the model generation system 20 according to this embodiment, may be a conventional server device or a computer such as a PC (personal computer). The server 1 may also be a computer system including multiple computers. The server 1 may also have a communication function and be able to send and receive information to and from other devices.

[0021] The thermal camera 100 is a device that captures images and acquires moving images used by the trajectory estimation system 10. The thermal camera 100 continuously captures images of the space of the imaging target and acquires moving images of the heat distribution in the imaging target. As described above, the moving images captured by the thermal camera 100 are low-resolution moving images. The thermal camera 100 is pre-positioned and fixedly installed in a position where it can capture images of the person related to the trajectory that is the estimation target of the trajectory estimation system 10. The thermal camera 100 has a communication function (e.g., a wireless LAN (Local Area Network) communication function) and transmits the moving images obtained by imaging to the server 1 via this communication function. A conventional thermal camera module (e.g., an MLX90640 thermal camera unit) can be used as the thermal camera 100 (the functional unit that captures images).

[0022] The training thermo camera 200 is a device that captures images and acquires moving images to be used by the model generation system 20. The training thermo camera 200 has the same functions as the thermo camera 100 and acquires moving images similar to those acquired by imaging with the thermo camera 100. Moving images similar to those acquired by imaging with the thermo camera 100 are, for example, moving images that have the same resolution and frame rate as those acquired by imaging with the thermo camera 100. Note that moving images similar to those acquired by imaging with the thermo camera 100 do not necessarily have to have the same resolution and frame rate, as long as they are moving images that can be used to appropriately generate a model for estimating a person's trajectory from the moving images acquired by imaging with the thermo camera 100.

[0023] The learning thermo camera 200 may be the same as the thermo camera 100 or may be of the same type as the thermo camera 100. The learning thermo camera 200 is positioned in advance and fixedly installed in a position where it can capture images of people related to the trajectory for creating a model. The learning thermo camera 200 has a communication function (e.g., a wireless LAN communication function) and transmits the moving images obtained by capturing images to the server 1 via this communication function.

[0024] The training RGB camera 201 is a device that captures images and acquires moving images to be used by the model generation system 20. As will be described later, the moving images acquired by the training RGB camera 201 are used to identify a person's trajectory. The training RGB camera 201 continuously captures images of the space of the target imaged by the training thermal camera 200 at the same timing as when the training thermal camera 200 captures images, and acquires moving images of the target imaged using visible light.

[0025] The video captured by the learning RGB camera 201 has a resolution that allows the trajectory of a person to be identified. Unlike the video captured by the thermal camera 100 and the learning thermal camera 200, the video captured by the learning RGB camera 201 does not need to be low resolution. The learning RGB camera 201 is positioned in advance and fixedly installed at a position that corresponds to the position where the learning thermal camera 200 will be installed, as described above. The learning RGB camera 201 has a communication function (e.g., a wireless LAN communication function), and transmits the video captured by the image capture to the server 1 via this communication function. A conventional RGB camera module can be used as the learning RGB camera 201 (the functional unit that captures images).

[0026] Next, a description will be given of the functions of the trajectory estimation system 10 and the model generation system 20 according to this embodiment. As shown in FIG.

[0027] The acquisition unit 11 is a functional unit that acquires time-series images obtained by imaging with the thermal camera 100. The acquisition unit 11 may acquire, as each time-series image, an image with 32 or less pixels in each of the vertical and horizontal directions.

[0028] The acquisition unit 11 receives and acquires moving images, which are time-series images transmitted from the thermal camera 100. As in the example shown in FIG. 2 , the images constituting the moving image, i.e., each image in the time series, may be low-resolution images with 32 or fewer pixels in each direction. Note that the images in the time series may be other than those described above, as long as they are time-series images obtained by imaging with a thermal camera and can be used to estimate a person's trajectory. The acquisition unit 11 may also acquire time-series images obtained by imaging with the thermal camera 100 by a method other than those described above. The acquisition unit 11 outputs the acquired moving images to the estimation unit 12.

[0029] The estimation unit 12 is a functional unit that estimates the trajectory of a person appearing in the time-series images acquired by the acquisition unit 11 by sequentially inputting the time-series images into a model that recursively uses the time-series images generated by machine learning. The model may include a recurrent neural network. The estimation unit 12 estimates the trajectory of a person, for example, as follows.

[0030] The estimation unit 12 receives and stores a model generated by the model generation system 20. An example of a model used by the estimation unit 12 to estimate a person's trajectory will be described. The model is a trained model generated by machine learning. The model receives a time series of images as input and outputs information indicating the trajectory of a person appearing in the time series of images. Specifically, the model receives the time series of images in chronological order from the oldest to the newest, and outputs, for each input image, an identifier (ID) that identifies a person appearing in the image and information indicating the position of the person.

[0031] For example, the image shown on the left side of Figure 3 is the image input to the model, and the position of the rectangle and the number in the upper left corner of the rectangle shown in the image on the right side of Figure 3 are information indicating the position of a person output from the model and an identifier identifying the person. The information indicating the position of a person output from the model is arranged in chronological order for each identifier identifying the person, resulting in a trajectory of the person. The person's identifier is assigned by the model so that the same identifier is assigned to the same person between images, and different identifiers are assigned to different people.

[0032] The model is composed of a neural network. Furthermore, the model includes a recurrent neural network, and recursively uses input time-series images to calculate the position of the person to be output. Specifically, each time an image is input, the model uses information about that image to calculate the position of the person appearing in that image, and this calculation uses the results of calculations performed within the model on previously input images.

[0033] The input layer of the model is provided with neurons for inputting images constituting a moving image. For example, the input layer of the model is provided with neurons for inputting pixel values ​​of the pixels of the image, the number of which is the same as the number of pixels in the image. The output layer of the model is provided with neurons for outputting information indicating the trajectory of a person. For example, the output layer of the model is provided with neurons for outputting an identifier of a person appearing in the input image and the position of the person (for example, a rectangular area in which the person appears in the image as shown in Figure 2).

[0034] The model may be a conventional trained model, such as DeepSORT, which tracks people. However, the model may be any other model that recursively uses time-series images generated by machine learning and can be used to estimate the trajectory of a person appearing in the time-series images.

[0035] The trained model used in the estimation unit 12 is expected to be used as a program module that is part of artificial intelligence software. The model is used, for example, in a computer equipped with hardware such as a CPU (Central Processing Unit) and memory, and the CPU of the computer operates according to instructions from the model stored in the memory. For example, the CPU of the computer operates in accordance with the instructions to input information to the model, perform calculations according to the model, and output results from the model. Specifically, the CPU of the computer operates in accordance with the instructions to input information to the input layer of a neural network, perform calculations based on trained weighting coefficients in the neural network, and output results from the output layer of the neural network.

[0036] The estimation unit 12 receives an input of a moving image from the acquisition unit 11. The estimation unit 12 sequentially inputs images of the moving image received from the acquisition unit 11 into a pre-stored model, performs calculations according to the model, and obtains, as an output from the model, information indicating the time-series positions of people appearing in the moving image. The obtained information indicating the time-series positions of people is the result of estimation of the person's trajectory by the estimation unit 12. The acquisition of time-series images by the acquisition unit 11 and the estimation of the person's trajectory by the estimation unit 12 may be performed in real time.

[0037] The estimation unit 12 outputs information indicating the estimation result of the person's trajectory. For example, the estimation unit 12 transmits the information indicating the estimation result to the user's terminal. Alternatively, the estimation unit 12 may output the information indicating the estimation result in a format recognizable by the user. For example, the estimation unit 12 may display the information indicating the estimation result on a display device provided in the server 1. The information indicating the estimation result may be, for example, information indicating the person's position (e.g., a rectangular display in the figure) and the person's ID (e.g., a number in the upper left corner of the rectangle in the figure) superimposed on a moving image obtained by imaging with the thermal camera 100, as shown in the images on the right side of Figures 2 and 3. Furthermore, the information indicating the estimation result may be other than the above as long as it indicates the estimated person's trajectory. The information indicating the estimation result from the estimation unit 12 may be output by a method other than the above or to an output destination other than the above. The above is the function of the trajectory estimation system 10 according to this embodiment.

[0038] As shown in FIG. 1, a model generation system 20 according to this embodiment includes a learning acquisition unit 21 and a model generation unit 22 .

[0039] The learning acquisition unit 21 is a functional unit that acquires time-series learning images obtained by imaging with a thermal camera (learning thermal camera 200) and learning trajectory information that indicates the trajectories of people appearing in the time-series learning images. The learning acquisition unit 21 may acquire time-series learning images that include images in which a light is captured. The learning acquisition unit 21 may acquire time-series trajectory estimation images that are of a different type from the time-series learning images and that are obtained by imaging corresponding to the time-series learning images, and estimate the trajectories of people appearing in the acquired time-series trajectory estimation images to acquire learning trajectory information. The learning acquisition unit 21 acquires each piece of information, for example, as follows.

[0040] The time-series learning images and learning trajectory information acquired by the learning acquisition unit 21 are learning data (teacher data) used to generate a model. The learning acquisition unit 21 acquires a sufficient number of time-series learning images and learning trajectory information to generate an appropriate model.

[0041] The learning acquisition unit 21 receives and acquires moving images that are time-series learning images transmitted from the learning thermo camera 200. As described above, the images that make up the moving images acquired by the learning acquisition unit 21, i.e., the time-series learning images, are low-resolution images similar to the images that make up the moving images acquired by the acquisition unit 11.

[0042] Furthermore, when the learning thermo camera 200 captures images, the camera may be installed in the space to be imaged by the learning thermo camera 200, so that at least one of the time-series learning images acquired by the learning acquisition unit 21 includes an image showing a light that could be a heat source. That is, at least one of the time-series learning images may be an image showing a person and a light. By using such images for the time-series learning images, the estimation accuracy of the model can be improved.

[0043] Note that the time-series learning images may be other than those described above as long as they correspond to the time-series images acquired by the acquisition unit 11 and are appropriately used for generating a model. The learning acquisition unit 21 may also acquire the time-series learning images obtained by imaging with the learning thermo camera 200 by a method other than those described above.

[0044] The learning acquisition unit 21 acquires video images, which are time-series learning images, and also acquires learning trajectory information indicating the trajectories of people appearing in the video images. The learning trajectory information is a label for model learning. The learning trajectory information is, for example, an identifier (ID) that identifies a person appearing in each learning image constituting the video at each timing and information indicating the position of the person. The learning trajectory information is the same type of information as the information output from the model. The time-series learning images and learning trajectory information acquired by the learning acquisition unit 21 correspond to the time-series images acquired by the acquisition unit 11 and the information output from the model, respectively, as learning data used to generate the model. The learning acquisition unit 21 acquires the learning trajectory information, for example, as follows.

[0045] The learning acquisition unit 21 receives and acquires the video images transmitted from the learning RGB camera 201 to acquire the learning trajectory information. The video images transmitted from the learning RGB camera 201 are video images using visible light (RGB video images), and are of a different type from the video images (thermal distribution video images) transmitted from the learning thermal camera 200. The video images transmitted from the learning RGB camera 201 do not necessarily have to be video images using visible light (RGB video images), as long as they can identify the trajectory of a person captured for model generation. For example, the video images used to acquire the learning trajectory information may be video images of thermal distribution with a higher resolution than the video images of thermal distribution captured by the thermal camera 100, and may be video images that allow the position of a person to be appropriately identified at a high resolution.

[0046] The moving images transmitted from the learning RGB camera 201 are, for example, captured during the same period as the moving images transmitted from the learning thermal camera 200, and are moving images at the same frame rate as the learning thermal camera 200. This is to associate each learning image in time series with each position of a person that constitutes the person's trajectory indicated by the learning trajectory information, as will be described later. However, as long as this association is possible, the moving images transmitted from the learning RGB camera 201 do not have to be as described above.

[0047] The learning acquisition unit 21 estimates the trajectories of people appearing in the acquired video images and generates learning trajectory information. For example, the learning acquisition unit 21 detects the position of people appearing in each image constituting the video images and assigns an identifier (ID) that identifies the people. The person's identifier is the same for the same person between images, and different identifiers are assigned for different people.

[0048] The detection of a person's position and the assignment of an identifier may be performed by a conventional method, for example, a method using a conventional person detection model (YOLO (You Only Look Once)). As described above, the video images obtained by the learning RGB camera 201 do not need to be low resolution, unlike the video images obtained by the above-mentioned thermal camera 100 and learning thermal camera 200, so the position of a person can be detected by a conventional method.

[0049] The learning acquisition unit 21 generates a person's trajectory and identifier, which become learning trajectory information, based on the person's trajectory and identifier obtained as described above. Because the learning thermo camera 200 and the learning RGB camera 201 are positioned at different positions, the relative position of the person with respect to the learning thermo camera 200 differs from the relative position of the person with respect to the learning RGB camera 201. Therefore, the learning acquisition unit 21 may pre-store the positional relationship (relative position) between the learning thermo camera 200 and the learning RGB camera 201, and correct the obtained person's trajectory to match the trajectory of the position detected by the learning thermo camera 200 based on the stored positional relationship. The correction based on the positional relationship is, for example, a linear shift correction, and may be performed using a conventional method.

[0050] The learning acquisition unit 21 associates each learning image in time series with each position of a person that constitutes the person's trajectory indicated by the learning trajectory information. The learning acquisition unit 21 associates learning images with the positions of people that relate to the same time.

[0051] The learning acquisition unit 21 may acquire the learning trajectory information by a method other than the above. For example, the learning trajectory information may be acquired by inputting learning trajectory information corresponding to time-series learning images acquired by imaging with the learning thermo camera 200, which has been prepared in advance by any method. In this case, the learning acquisition unit 21 does not need to acquire moving images acquired by imaging with the learning RGB camera 201. The learning acquisition unit 21 outputs the acquired moving images acquired by imaging with the learning thermo camera 200 and the learning trajectory information to the estimation unit 12.

[0052] The model generation unit 22 is a functional unit that performs machine learning based on the time-series learning images and learning trajectory information acquired by the learning acquisition unit 21, and recursively uses the time-series images to generate a model that is used for the trajectories of people appearing in the time-series images. The model generation unit 22 generates a model, for example, as follows.

[0053] The model generation unit 22 receives time-series training images and training trajectory information corresponding to the time-series training images from the training acquisition unit 21. The model generation unit 22 performs machine learning using the received time-series training images as input to the model and the received training trajectory information as the model output (correct answer), to generate a model. When the time-series training images are used as input to the model, the model generation unit 22 inputs the time-series training images in order from oldest to newest, similar to the input of time-series images to the model by the estimation unit 12. Furthermore, when the training trajectory information is used as input to the model, the model generation unit 22 ensures that the identifier of a person and information indicating the position of the person at the time corresponding to the training images input to the model correspond to the model output. The above-described machine learning training itself, i.e., the updating of the model parameters, may be performed in the same manner as conventional machine learning training.

[0054] In the model, the number of past images to be recursively used (the number of past images going back from the input image) is set in advance. It is difficult to acquire a person's external features from low-resolution images. Therefore, if too many past images are used, the person's posture and orientation will be too different, making it difficult to link the images to the same person. For this reason, the number of past images to be recursively used may be kept to a minimum. The number of past images to be recursively used may be set to 10, for example, by parameter tuning.

[0055] The model generation unit 22 outputs the generated model to the trajectory estimation system 10. In the trajectory estimation system 10, the estimation unit 12 inputs and stores the model and uses it to estimate the trajectory of the person described above. The functions of the model generation system 20 according to this embodiment have been described above.

[0056] Next, the processing executed by the server 1 according to this embodiment (the operation method performed by the server 1) will be described using the flowcharts of Figures 4 and 5. First, the processing executed by the trajectory estimation system 10 according to this embodiment will be described using the flowchart of Figure 4.

[0057] In this process, first, the acquisition unit 11 acquires time-series images captured by the thermal camera 100 (S01). Next, the estimation unit 12 inputs the time-series images into a model in order, and estimates the trajectories of people appearing in the time-series images (S02). Next, the estimation unit 12 outputs information indicating the estimation results (S03). The above is the process executed by the trajectory estimation system 10 according to this embodiment.

[0058] Next, the processing executed by the model generation system 20 according to this embodiment will be described with reference to the flowchart of Fig. 5. This processing is performed before the processing is executed by the trajectory estimation system 10 according to this embodiment.

[0059] In this process, first, the learning acquisition unit 21 acquires time-series learning images obtained by imaging with the learning thermo camera 200 and time-series RGB images obtained by imaging with the learning RGB camera 201 (S11). Next, the learning acquisition unit 21 estimates a person's trajectory from the time-series RGB images to generate learning trajectory information (S12). Next, the model generation unit 22 performs machine learning based on the time-series learning images and the learning trajectory information to generate a model (S13). The generated model is output to the trajectory estimation system 10, stored by the estimation unit 12, and used to estimate the person's trajectory as described above. The above is the process executed by the model generation system 20 according to this embodiment.

[0060] As described above, in the trajectory estimation system 10 according to this embodiment, a low-resolution image such as that shown in Fig. 2 captured by the thermal camera 100 is used to estimate a person's trajectory. Normally, it is difficult to detect the position of a person and estimate the person's trajectory from such a low-resolution image. In situations where people overlap in the image, the person's trajectory is particularly difficult to estimate.

[0061] In response to this, the trajectory estimation system 10 according to the present embodiment uses a model that recursively uses time-series images generated by machine learning to estimate the trajectories of people appearing in the time-series images. This estimation allows the trajectory estimation system 10 according to the present embodiment to appropriately estimate the trajectories of people even when using low-resolution images captured by the thermal camera 100. In particular, it is possible to appropriately estimate the trajectories of people even when people overlap in the images. Furthermore, using low-resolution images makes it possible to track people's trajectories while taking privacy into consideration and not identifying individuals.

[0062] Furthermore, as described above, each of the time-series images used to estimate a person's trajectory may be an image having a pixel count of 32 or less in both the vertical and horizontal directions. However, each of the time-series images used to estimate a person's trajectory may have a resolution other than the above.

[0063] Furthermore, the model used to estimate a person's trajectory may include a recurrent neural network, such as DeepSORT. This configuration allows time-series images to be used to estimate a person's trajectory appropriately and reliably, thereby enabling appropriate and reliable estimation of the person's trajectory. However, the model used to estimate a person's trajectory does not need to include a recurrent neural network; it may be any model that recursively uses time-series images generated by machine learning.

[0064] The model generation system 20 according to this embodiment can generate a model to be used in the trajectory estimation system 10. Furthermore, the time-series learning images used to generate the model may include images in which a light is captured. This configuration can improve the estimation accuracy of the generated model. However, the time-series learning images used to generate the model do not have to include images in which a light is captured.

[0065] Furthermore, as described above, in generating a model, time-series trajectory estimation images (for example, time-series RGB images) may be acquired, and the trajectories of people appearing in the acquired time-series trajectory estimation images may be estimated to acquire the trajectory estimation images. This configuration allows for easy and appropriate acquisition of training trajectory information, thereby allowing for easy and appropriate generation of a model. However, the trajectory estimation images do not necessarily need to be acquired in the manner described above, and may be acquired by any method.

[0066] In the present embodiment, the server 1 includes the trajectory estimation system 10 and the model generation system 20. However, the trajectory estimation system 10 and the model generation system 20 may be implemented independently of each other. That is, the trajectory estimation system 10 and the model generation system 20 may be implemented by separate devices.

[0067] The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of hardware and / or software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (e.g., wired, wireless, etc.) and these multiple devices. The functional block may also be realized by combining software with the single device or multiple devices.

[0068] Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.

[0069] For example, the server 1 according to an embodiment of the present disclosure may function as a computer that performs information processing according to the present disclosure. Fig. 6 is a diagram illustrating an example of a hardware configuration of the server 1 according to an embodiment of the present disclosure. The server 1 described above may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

[0070] In the following description, the term "device" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the server 1 may be configured to include one or more of the devices shown in the drawings, or may be configured to exclude some of the devices.

[0071] Each function in server 1 is realized by loading specified software (programs) onto hardware such as processor 1001 and memory 1002, causing processor 1001 to perform calculations, control communication via communication device 1004, and control at least one of reading and writing data in memory 1002 and storage 1003.

[0072] The processor 1001, for example, runs an operating system to control the entire computer. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, each function of the server 1 described above may be realized by the processor 1001.

[0073] The processor 1001 also reads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes in accordance with these. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. For example, each function of the server 1 may be implemented by a control program stored in the memory 1002 and running on the processor 1001. While the above-described various processes have been described as being executed by one processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may also be transmitted from a network via a telecommunications line.

[0074] The memory 1002 is a computer-readable recording medium and may be configured, for example, by at least one of a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for performing information processing according to an embodiment of the present disclosure.

[0075] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The storage medium provided in server 1 may be, for example, a database including at least one of memory 1002 and storage 1003, a server, or other appropriate medium.

[0076] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.

[0077] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0078] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0079] The server 1 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0080] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0081] Input and output information may be stored in a specific location (for example, memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0082] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0083] The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0084] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0085] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0086] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0087] As used in this disclosure, the terms "system" and "network" are used interchangeably.

[0088] Furthermore, the information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information.

[0089] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0090] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using one or more wires, cables, and / or printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0091] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0092] As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0093] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

[0094] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0095] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0096] The population status determination system and model generation system disclosed herein have the following configuration. [1] A trajectory estimation system comprising: an acquisition unit that acquires time-series images captured by a thermal camera; and an estimation unit that estimates trajectories of people appearing in the time-series images by sequentially inputting the time-series images acquired by the acquisition unit into a model that recursively uses the time-series images generated by machine learning. [2] The trajectory estimation system described in [1], wherein the acquisition unit acquires, as each image in the time series, an image with 32 or fewer pixels in each direction. [3] The trajectory estimation system described in [1] or [2], wherein the model includes a recurrent neural network. [4] A model generation system comprising: a learning acquisition unit that acquires time-series learning images obtained by imaging with a thermal camera and learning trajectory information indicating trajectories of people appearing in the time-series learning images; and a model generation unit that performs machine learning based on the time-series learning images and learning trajectory information acquired by the learning acquisition unit, to recursively use the time-series images and generate a model to be used for the trajectories of people appearing in the time-series images. [5] The model generation system described in [4], wherein the learning acquisition unit acquires the time-series learning images including images containing a light. [6] The model generation system described in [4] or [5], wherein the learning acquisition unit acquires time-series trajectory estimation images that are different in type from the time-series learning images and obtained by imaging corresponding to the time-series learning images, estimates the trajectories of people appearing in the acquired time-series trajectory estimation images, and acquires the learning trajectory information.

[0097] 1...server, 10...trajectory estimation system, 11...acquisition unit, 12...estimation unit, 20...model generation system, 21...learning acquisition unit, 22...model generation unit, 100...thermal camera, 200...learning thermal camera, 201...learning RGB camera, 1001...processor, 1002...memory, 1003...storage, 1004...communication device, 1005...input device, 1006...output device, 1007...bus.

Claims

1. A trajectory estimation system comprising: an acquisition unit that acquires time-series images captured by a thermo camera; and an estimation unit that estimates the trajectory of a person appearing in the time-series images by sequentially inputting the time-series images acquired by the acquisition unit into a model that recursively uses the time-series images generated by machine learning.

2. A trajectory estimation system according to claim 1, wherein the acquisition unit acquires, as each image in the time series, an image having 32 or less pixels in each direction.

3. The trajectory estimation system of claim 1, wherein the model includes a recurrent neural network.

4. A model generation system comprising: a learning acquisition unit that acquires time-series learning images obtained by imaging with a thermo camera and learning trajectory information showing the trajectories of people appearing in the time-series learning images; and a model generation unit that performs machine learning based on the time-series learning images and learning trajectory information acquired by the learning acquisition unit, and recursively uses the time-series images and generates a model to be used for the trajectories of people appearing in the time-series images.

5. A model generation system according to claim 4, wherein the learning acquisition unit acquires the time series of learning images including images in which a light is captured.

6. A model generation system as described in claim 4, wherein the learning acquisition unit acquires time-series images for trajectory estimation that are of a different type from the time-series learning images and that have been obtained by imaging that corresponds to the time-series learning images, estimates trajectories of people appearing in the acquired time-series images for trajectory estimation, and acquires the learning trajectory information.

Images