Path estimation system, model generation system, path estimation method, and model generation method

Abstract

The present invention appropriately estimates a path of a person even when low resolution images captured by a plurality of thermal cameras are used.　A path estimation system 10 comprises: an acquisition unit 11 that acquires, for each of a plurality of thermal cameras 100 which have respective imaging ranges that at least partially overlap, time-series images which are captured by imaging during the same time period by the plurality of thermal cameras 100; a conversion unit 12 that converts each of the time-series images in order to obtain, from the images, positions in a coordinate system which is shared by the plurality of thermal cameras 100; and an estimation unit 13 that, for each of the plurality of thermal cameras 100, estimates a path of a person by sequentially inputting the converted time-series images into a model that is generated by machine learning and that recursively uses time-series images, compares positions at common timings in the paths of a person which are estimated for the respective thermal cameras 100, in the shared coordinate system, and connects the paths which are estimated for the respective thermal cameras 100 with regard to the same person.

Description

Trajectory estimation system, model generation system, trajectory estimation method, and model generation method

[0001] The present invention relates to a trajectory estimation system that estimates the trajectory of a person, a model generation system that generates a model used in the trajectory estimation system, and a trajectory estimation method and model generation method corresponding thereto.

[0002] Patent Literature 1 discloses a method for detecting a person in each image of a video and estimating the person's trajectory. The camera for capturing the images may be a far-infrared camera that can capture images of heat sources.

[0003] JP 2010-63001 A

[0004] Many services based on people's location information have been proposed. For example, commercial facilities are working to visualize the movement of customers and use this information for marketing purposes. It is conceivable that the trajectories of people estimated using the conventional technology described above could be used for these efforts.

[0005] For example, capturing a person in a public place with high resolution may raise privacy concerns, such as the possibility of identifying the individual from the captured image. Therefore, it is conceivable to use low-resolution images captured by a thermal camera, which make it difficult to identify the individual from the image, to detect the person's trajectory. However, low image resolution makes it difficult to estimate the position of the person from the image. This is particularly difficult when multiple people are captured in the image. It is also difficult to estimate the trajectory of a single person from each of the images captured by multiple thermal cameras.

[0006] One embodiment of the present invention has been made in consideration of the above, and aims to provide a trajectory estimation system and a model generation system that can appropriately estimate a person's trajectory even when using low-resolution images captured by multiple thermal cameras.

[0007] In order to achieve the above object, a trajectory estimation system according to one embodiment of the present invention comprises: an acquisition unit that acquires time-series images for each of the multiple thermo cameras obtained by capturing images at the same time period using multiple thermo cameras that have at least a partial common imaging range; a conversion unit that performs conversion on each of the time-series images for each of the multiple thermo cameras acquired by the acquisition unit to determine a position in a coordinate system common to the multiple thermo cameras; and an estimation unit that estimates the trajectories of people appearing in the time-series images in the common coordinate system by sequentially inputting the time-series images converted by the conversion unit into a model that recursively uses time-series images generated by machine learning for each of the multiple thermo cameras, and compares the positions of common times in the common coordinate system of the trajectories of people estimated for each thermo camera, and connects the trajectories estimated for the same person for each thermo camera.

[0008] In a trajectory estimation system according to one embodiment of the present invention, time-series images from each of a plurality of thermal cameras are transformed to determine a position in a coordinate system common to the plurality of thermal cameras. Next, a model recursively using the time-series images generated by machine learning is used to estimate the trajectories of people captured in the time-series images in the common coordinate system from the transformed time-series images for each of the plurality of thermal cameras. Next, common timing positions of the person trajectories estimated for each of the thermal cameras in the common coordinate system are compared, and the trajectories estimated for the same person for each of the thermal cameras are linked. Because the person trajectories for each of the thermal cameras are estimated using the transformed time-series images as described above, the estimated trajectories can be linked in a common coordinate system. Therefore, the estimated trajectories can be linked appropriately. As a result, the trajectory estimation system according to one embodiment of the present invention can appropriately estimate a person's trajectory even when using low-resolution images captured by multiple thermal cameras.

[0009] A model generation system according to one embodiment of the present invention comprises a learning acquisition unit that acquires time-series learning images obtained by imaging with a thermal camera and learning trajectory information indicating the trajectories of people appearing in the time-series learning images; a learning conversion unit that performs conversion on each of the time-series learning images acquired by the learning acquisition unit to determine a position in a coordinate system common to multiple thermal cameras, including the thermal camera; and a model generation unit that performs machine learning based on the time-series learning images converted by the learning conversion unit and the learning trajectory information acquired by the learning acquisition unit, and recursively uses the time-series images to generate a model that is used to estimate the trajectory of people appearing in the time-series images in the common coordinate system.

[0010] According to a model generation system according to an embodiment of the present invention, it is possible to generate a model used in a trajectory estimation system.

[0011] Incidentally, one embodiment of the present invention can be described not only as an invention of a trajectory estimation system and a model generation system as described above, but also as an invention of a trajectory estimation method and a model generation method as described below. These are essentially the same invention, just in different categories, and have similar functions and effects.

[0012] That is, a trajectory estimation method according to one embodiment of the present invention includes: an acquisition step in which a trajectory estimation system acquires time-series images for each of a plurality of thermo cameras obtained by imaging at the same time period using the plurality of thermo cameras that have at least a partial common imaging range; a conversion step in which the trajectory estimation system converts each of the time-series images for each of the plurality of thermo cameras acquired in the acquisition step to determine a position in a coordinate system common to the plurality of thermo cameras; and an estimation step in which the trajectory estimation system estimates the trajectories of people appearing in the time-series images in the common coordinate system by sequentially inputting the time-series images converted in the conversion step into a model that recursively uses the time-series images generated by machine learning for each of the plurality of thermo cameras, and compares the positions of common times in the common coordinate system of the trajectories of people estimated for each thermo camera, and connects the trajectories estimated for each thermo camera for the same person.

[0013] Furthermore, a model generation method according to one embodiment of the present invention includes a learning acquisition step in which a model generation system acquires time-series learning images obtained by imaging with a thermal camera and learning trajectory information indicating the trajectories of people appearing in the time-series learning images; a learning transformation step in which the model generation system performs a transformation on each of the time-series learning images acquired in the learning acquisition step to determine a position in a coordinate system common to multiple thermal cameras including the thermal camera; and a model generation step in which the model generation system performs machine learning based on the time-series learning images converted in the learning transformation step and the learning trajectory information acquired in the learning acquisition step, to recursively use the time-series images and generate a model to be used for estimating the trajectory of people appearing in the time-series images in the common coordinate system.

[0014] According to one embodiment of the present invention, even when low-resolution images captured by multiple thermal cameras are used, the trajectory of a person can be appropriately estimated.

[0015] FIG. 1 is a diagram showing the configuration of a server that is a trajectory estimation system and a model generation system according to an embodiment of the present invention. FIG. 2 is an example of images that constitute a moving image used to estimate a person's trajectory by the trajectory estimation system, and the images after conversion. FIG. 3 is a diagram showing a space in which a person's trajectory is estimated, and the arrangement of thermal cameras in the space. FIG. 4 is an example of images that constitute a moving image used to generate a model by the model generation system, and the images after conversion. FIG. 4 is a flowchart showing a trajectory estimation method that is processing executed by the trajectory estimation system according to an embodiment of the present invention. FIG. 5 is a flowchart showing a model generation method that is processing executed by the model generation system according to an embodiment of the present invention. FIG. 6 is a diagram showing the hardware configuration of a server that is a trajectory estimation system and a model generation system according to an embodiment of the present invention.

[0016] Hereinafter, embodiments of a trajectory estimation system, a model generation system, a trajectory estimation method, and a model generation method according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated explanations will be omitted.

[0017] FIG. 1 shows a server 1 that is a trajectory estimation system 10 and a model generation system 20 according to this embodiment. The trajectory estimation system 10 is a system (device) that analyzes time-series images (videos) captured by a thermal camera 100 and estimates the trajectory of a person appearing in the time-series images. In other words, the trajectory estimation system 10 is a system that tracks a person from the time-series images captured by the thermal camera 100. The space in which the person's trajectory is estimated is indoors. However, the space may be other than indoors. The estimated trajectory is, for example, a trajectory in the space of the object captured by the thermal camera (thermography) 100. Specifically, the estimated trajectory is a trajectory on a floor plan of the indoor space.

[0018] The moving images used to estimate a person's trajectory by the trajectory estimation system 10 are low-resolution moving images. Figures 2(a) and 2(b) show examples of images that make up the moving images used to estimate a person's trajectory by the trajectory estimation system 10. Note that the images shown in Figures 2(a) and 2(b) also show parts of the person (parts indicated by rectangles in the images).

[0019] The resolution of the images in Figures 2(a) and (b) is 24 pixels vertically by 32 pixels horizontally. This image resolution is 1 / 10 or less of the resolution of a typical infrared thermal camera available on the market. Note that the trajectory estimation system 10 may perform downsampling to reduce the image resolution. For example, an image of 24 pixels vertically by 32 pixels horizontally may be downsampled to an image of 18 pixels vertically by 24 pixels horizontally, or an image of 12 pixels vertically by 16 pixels horizontally.

[0020] As shown in Figures 2(a) and 2(b), the image quality of the above images is so low that it is not possible to easily identify the same person when comparing two images of the same person. Also, the image quality of the above images is so low that it is not possible to visually identify the individual.

[0021] As described above, the video images used by the trajectory estimation system 10 have low resolution, making it possible to estimate a person's trajectory using video images in places where it is not appropriate to identify individuals. In other words, the trajectory estimation system 10 can realize a sensing technology that takes privacy into consideration. For example, the trajectory estimation system 10 estimates a person's trajectory (person's behavior) in areas where privacy is a high concern, such as a home, a public facility such as a hotel, or a changing room. The trajectory estimated by the trajectory estimation system 10 can be used for any service triggered by the person's position. For example, the estimated trajectory may be used to suggest an interaction that utilizes the person's trajectory. Alternatively, the trajectory estimation system 10 may be used as a substitute for a surveillance camera to track suspicious individuals, etc.

[0022] The trajectory estimation system 10 estimates the trajectory of a person captured in time-series images taken by multiple thermal cameras 100 at the same time period for each thermal camera 100. The multiple thermal cameras 100 share at least a portion of their imaging range. The trajectory estimation system 10 connects the trajectories estimated for the same person by each thermal camera 100.

[0023] Depending on the space in which a person's trajectory is to be estimated, it may be difficult to capture the entire space with a single thermal camera 100. For example, it is difficult to capture the entire indoor corridor that bends at a right angle, as shown in the plan view of FIG. 3( a), with a single thermal camera 100. By providing multiple thermal cameras 100 in such a space, it is possible to reduce or eliminate positions where a person's trajectory cannot be estimated. For example, as shown in FIG. 3( a), two thermal cameras 100 can be installed in an indoor corridor that bends at a right angle, at an angle that captures an image from above and diagonally below the head of the person 300 whose trajectory is to be estimated, as shown in the side view of the space in FIG. 3( b), thereby estimating the person's trajectory in the corridor.

[0024] The trajectory estimation system 10 estimates a trajectory of a person using a model generated by machine learning. The model generation system 20 is a system (device) that generates a model used for estimating a trajectory of a person by the trajectory estimation system 10.

[0025] The server 1, which is the trajectory estimation system 10 and the model generation system 20 according to this embodiment, may be a conventional server device or a computer such as a PC (personal computer). The server 1 may also be a computer system including multiple computers. The server 1 may also have a communication function and be able to send and receive information to and from other devices.

[0026] The thermal camera 100 is a device that captures images and acquires moving images used by the trajectory estimation system 10. The thermal camera 100 continuously captures images of the space of the imaging target and acquires moving images of the heat distribution in the imaging target. As described above, the moving images captured by the thermal camera 100 are low-resolution moving images. The thermal camera 100 is pre-positioned and fixedly installed in a position where it can capture images of the person related to the trajectory that is the estimation target of the trajectory estimation system 10. The thermal camera 100 has a communication function (e.g., a wireless LAN (Local Area Network) communication function) and transmits the moving images obtained by imaging to the server 1 via this communication function. A conventional thermal camera module (e.g., an MLX90640 thermal camera unit) can be used as the thermal camera 100 (the functional unit that captures images).

[0027] As described above, multiple thermal cameras 100 are used to estimate a person's trajectory using the trajectory estimation system 10. The multiple thermal cameras 100 are installed in locations where their imaging ranges are at least partially shared. Note that the location at which a thermal camera 100 is installed does not need to be a location where its imaging range is partially shared with all other thermal cameras 100, as long as its imaging range is partially shared with any of the other thermal cameras 100. The multiple thermal cameras 100 capture images during the same time period. The same time period is a pre-set time period, and is the time period during which the person's trajectory is to be estimated. In the video images captured by the thermal cameras 100, the time at which each image in the video was captured can be determined by the server 1 (the same applies to subsequent video images).

[0028] The training thermo camera 200 is a device that captures images and acquires moving images to be used by the model generation system 20. The training thermo camera 200 has the same functions as the thermo camera 100 and acquires moving images similar to those acquired by imaging with the thermo camera 100. Moving images similar to those acquired by imaging with the thermo camera 100 are, for example, moving images that have the same resolution and frame rate as those acquired by imaging with the thermo camera 100. Note that moving images similar to those acquired by imaging with the thermo camera 100 do not necessarily have to have the same resolution and frame rate, as long as they are moving images that can be used to appropriately generate a model for estimating a person's trajectory from the moving images acquired by imaging with the thermo camera 100.

[0029] The learning thermo camera 200 may be the same as the thermo camera 100 or may be of the same type as the thermo camera 100. The learning thermo camera 200 is positioned in advance and fixedly installed in a position where it can capture images of people related to the trajectory for creating a model. The learning thermo camera 200 has a communication function (e.g., a wireless LAN communication function) and transmits the moving images obtained by capturing images to the server 1 via this communication function.

[0030] The training RGB camera 201 is a device that captures images and acquires moving images to be used by the model generation system 20. As will be described later, the moving images acquired by the training RGB camera 201 are used to identify a person's trajectory. The training RGB camera 201 continuously captures images of the space of the target imaged by the training thermal camera 200 at the same timing as when the training thermal camera 200 captures images, and acquires moving images of the target imaged using visible light.

[0031] The video captured by the learning RGB camera 201 has a resolution that allows the trajectory of a person to be identified. Unlike the video captured by the thermal camera 100 and the learning thermal camera 200, the video captured by the learning RGB camera 201 does not need to be low-resolution. The learning RGB camera 201 is pre-positioned and fixedly installed at a position corresponding to the position where the learning thermal camera 200 is installed, as described above. For example, the learning thermal camera 200 and the learning RGB camera 201 are installed side by side. The learning RGB camera 201 has a communication function (e.g., a wireless LAN communication function) and transmits the video captured by the image capture to the server 1 via this communication function. A conventional RGB camera module can be used as the learning RGB camera 201 (the functional unit that captures images).

[0032] The learning thermo camera 200 and the learning RGB camera 201 are installed in corresponding positions and are used in combination. A plurality of learning thermo cameras 200 and learning RGB cameras 201 may be used, as with the thermo camera 100. However, the number of learning thermo cameras 200 and learning RGB cameras 201 does not necessarily have to be multiple, and only one of each may be used.

[0033] Next, a description will be given of the functions of the trajectory estimation system 10 and the model generation system 20 according to this embodiment. As shown in FIG. 1 , the trajectory estimation system 10 according to this embodiment includes an acquisition unit 11, a conversion unit 12, and an estimation unit 13.

[0034] The acquisition unit 11 is a functional unit that acquires time-series images of each of the plurality of thermal cameras 100, which are obtained by imaging at the same time period using the plurality of thermal cameras 100 that share at least a part of their imaging range. The acquisition unit 11 may acquire images with 32 or less pixels in each of the vertical and horizontal directions as each of the time-series images.

[0035] The acquisition unit 11 receives and acquires moving images, which are time-series images transmitted from each thermal camera 100. As shown in the examples of FIGS. 2( a) and 2(b), the images constituting the moving image, i.e., each image in the time series, may be low-resolution images with 32 or fewer pixels in each direction. Note that the time-series images may be other than those described above, as long as they are time-series images obtained by imaging with each thermal camera 100 and can be used to estimate a person's trajectory. The acquisition unit 11 may also acquire time-series images obtained by imaging with each thermal camera 100 by methods other than those described above. The acquisition unit 11 outputs the acquired moving images to the conversion unit 12.

[0036] The conversion unit 12 is a functional unit that performs conversion on each of the time-series images of the multiple thermal cameras 100 acquired by the acquisition unit 11 in order to determine a position in a coordinate system common to the multiple thermal cameras 100 from the image. The conversion unit 12 may perform a projective transformation as the above conversion. The conversion unit 12 converts the time-series images, for example, as follows:

[0037] The common coordinate system is, for example, a coordinate system having coordinate axes along a horizontal plane, that is, the position determined from the image is the position on the horizontal plane, i.e., the position as viewed from the vertical direction (bird's-eye view).

[0038] The thermo camera 100 is installed so as to capture an image obliquely downward, for example. That is, the image obtained by imaging with the thermo camera 100 is an image of a horizontal surface taken obliquely. The conversion by the conversion unit 12 converts the image into an image of the horizontal surface taken from the front, i.e., from the vertical direction, for example.

[0039] The transformation unit 12 inputs the moving image from the acquisition unit 11. The transformation unit 12 performs projective transformation as the transformation on each image constituting the input moving image. FIGS. 2(c) and 2(d) show examples of images after projective transformation. FIG. 2(c) is an image obtained by transforming the image in FIG. 2(a), and FIG. 2(d) is an image obtained by transforming the image in FIG. 2(b). Note that the images shown in FIGS. 2(c) and 2(d) also show the parts of people detected from the images (parts indicated by rectangles in the images).

[0040] The conversion unit 12 stores information for performing the projective transformation in advance and performs the projective transformation of the image based on the information. The projective transformation itself may be performed by a conventional method. The projective transformation may also be performed by a method set using an RGB image, as described below.

[0041] The transformation performed by the transformation unit 12 may be a transformation other than projective transformation, as long as it is a transformation for determining positions in a coordinate system common to the multiple thermal cameras 100 from the images. The transformation unit 12 outputs a moving image composed of the transformed images for each thermal camera 100 to the estimation unit 13.

[0042] The estimation unit 13 is a functional unit that estimates the trajectory of a person appearing in the time-series images in a common coordinate system by sequentially inputting the time-series images converted by the conversion unit 12 into a model that recursively uses time-series images generated by machine learning for each of the multiple thermal cameras 100, compares the common timing positions of the person's trajectory estimated for each thermal camera 100 in the common coordinate system, and connects the trajectories estimated for the same person for each thermal camera. The model may include a recurrent neural network. The estimation unit 13 estimates the person's trajectory, for example, as follows.

[0043] The estimation unit 13 receives and stores a model generated by the model generation system 20. An example of a model used by the estimation unit 13 to estimate a person's trajectory will be described. The model is a trained model generated by machine learning. The model receives a time series of images as input and outputs information indicating the trajectory of a person appearing in the time series of images. Specifically, the model receives the time series of images in chronological order from the oldest to the newest, and outputs, for each input image, an identifier (ID) that identifies a person appearing in the image and information indicating the position of the person.

[0044] For example, the image shown in FIG. 2(c) or FIG. 2(d) is an image input to the model. Information indicating the position of a rectangle that is the position of a person in the image shown in FIG. 2(c) or FIG. 2(d), and an identifier that identifies the person associated with that position, are information output from the model. The information indicating the position of a person output from the model is arranged in chronological order for each identifier that identifies the person, resulting in a trajectory of the person. The person's identifier is assigned by the model so that the same identifier is assigned to the same person between images, and different identifiers are assigned to different people.

[0045] The model is composed of a neural network. Furthermore, the model includes a recurrent neural network, and recursively uses input time-series images to calculate the position of the person to be output. Specifically, each time an image is input, the model uses information about that image to calculate the position of the person appearing in that image, and this calculation uses the results of calculations performed within the model on previously input images.

[0046] The input layer of the model is provided with neurons for inputting images constituting a moving image. For example, the input layer of the model is provided with neurons for inputting pixel values ​​of the pixels of the image, the number of which is the same as the number of pixels in the image. The output layer of the model is provided with neurons for outputting information indicating the trajectory of a person. For example, the output layer of the model is provided with neurons for outputting an identifier of a person appearing in the input image and the position of the person (for example, a rectangular area in which the person appears in the image, as shown in Figures 2(c) and (d)).

[0047] The model may be a conventional trained model, such as DeepSORT, which tracks people. However, the model may be any other model that recursively uses time-series images generated by machine learning and can be used to estimate the trajectory of a person appearing in the time-series images.

[0048] The trained model used in the estimation unit 13 is expected to be used as a program module that is part of artificial intelligence software. The model is used, for example, in a computer equipped with hardware such as a CPU (Central Processing Unit) and memory, and the CPU of the computer operates according to instructions from the model stored in the memory. For example, the CPU of the computer operates in accordance with the instructions to input information to the model, perform calculations according to the model, and output results from the model. Specifically, the CPU of the computer operates in accordance with the instructions to input information to the input layer of a neural network, perform calculations based on trained weighting coefficients in the neural network, and output results from the output layer of the neural network.

[0049] The estimation unit 13 inputs the converted moving images for each thermal camera 100 from the conversion unit 12. For each thermal camera 100, the estimation unit 13 sequentially inputs the images of the moving images input from the conversion unit 12 into a pre-stored model, performs calculations according to the model, and obtains information indicating the time-series positions of people appearing in the moving images as output from the model. The obtained information indicating the time-series positions of people is the result of estimation of the person's trajectory by the estimation unit 13 for each thermal camera 100. Note that the estimated position of a person does not have to be the position of the entire person, but may be the position of a specific part of the person (for example, the person's feet).

[0050] As described above, it is difficult to identify individuals from images captured by the thermal camera 100. Therefore, it is not possible to determine whether the trajectories of a person captured by each thermal camera 100 are of the same person across the thermal cameras 100. The estimation unit 13 connects the trajectories estimated for the same person captured by each thermal camera as follows.

[0051] The trajectory estimation results for each thermal camera 100 are in a common coordinate system, for example, a coordinate system having coordinate axes along a horizontal plane as described above. Furthermore, the trajectory estimation results for each thermal camera 100 are based on the position of the thermal camera 100. To enable comparison of trajectory estimation results between thermal cameras 100, the estimation unit 13 assigns the trajectory estimation results for each thermal camera 100 to a coordinate system including the entire imaging range of the multiple thermal cameras 100. The coordinate system including the entire imaging range of the multiple thermal cameras 100 is similar to the coordinate system of the trajectory estimation results for each thermal camera 100, for example, a coordinate system having coordinate axes along a horizontal plane as described above. Specifically, the coordinate system including the entire imaging range of the multiple thermal cameras 100 is a coordinate system on a horizontal plane within a building where a person can move, as shown in FIG. 3(a).

[0052] The estimation unit 13 stores in advance, for each thermal camera 100, the positional correspondence between a coordinate system that includes the entire imaging range of multiple thermal cameras 100 and the coordinate system of the estimated trajectory results for each thermal camera 100, and performs the above-mentioned assignment using this correspondence.

[0053] The estimation unit 13 uses the person's trajectory after assignment to connect trajectories estimated for the same person for each thermal camera. The estimation unit 13 compares the positions of common timings of the person's trajectories estimated for each thermal camera 100. For example, the estimation unit 13 detects, through this comparison, whether two trajectories associated with different thermal cameras 100 are at the same position at the same time. The same timing does not necessarily have to be exactly the same, as long as it can be considered to be the same (for example, within a certain time difference (error)). Furthermore, the same position does not necessarily have to be exactly the same, as long as it can be considered to be the same (for example, within a certain distance (error) between positions).

[0054] The estimation unit 13 connects two trajectories associated with different thermal cameras 100 that are at the same position at the same time as the trajectory of the same person. The estimation unit 13 may also connect two trajectories associated with different thermal cameras 100 that are at the same position for a predetermined consecutive time as the trajectory of the same person. The estimation unit 13 continues connecting trajectories until no more trajectories can be connected. The trajectory after connection is the final trajectory estimation result by the estimation unit 13. The acquisition of time-series images by the acquisition unit 11 and the estimation of the person's trajectory by the estimation unit 13 may be performed in real time.

[0055] The estimation unit 13 outputs information indicating the estimation result of the person's trajectory. For example, the estimation unit 13 transmits information indicating the estimation result to a user's terminal. Alternatively, the estimation unit 13 may output information indicating the estimation result in a format that can be recognized by the user. For example, the estimation unit 13 may display information indicating the estimation result on a display device provided in the server 1. Furthermore, the information indicating the estimation result may be other than the above as long as it indicates the estimated trajectory of the person. The information indicating the estimation result from the estimation unit 13 may be output by a method other than the above or to an output destination other than the above. These are the functions of the trajectory estimation system 10 according to this embodiment.

[0056] As shown in FIG. 1, a model generation system 20 according to this embodiment includes a learning acquisition unit 21, a learning conversion unit 22, and a model generation unit 23.

[0057] The learning acquisition unit 21 is a functional unit that acquires time-series learning images obtained by imaging with a thermal camera (learning thermal camera 200) and learning trajectory information that indicates the trajectories of people appearing in the time-series learning images. The learning acquisition unit 21 may acquire time-series learning images that include images in which a light is captured. The learning acquisition unit 21 may acquire time-series trajectory estimation images that are of a different type from the time-series learning images and that are obtained by imaging corresponding to the time-series learning images, and estimate the trajectories of people appearing in the acquired time-series trajectory estimation images to acquire learning trajectory information. The learning acquisition unit 21 acquires each piece of information, for example, as follows.

[0058] The time-series learning images and learning trajectory information acquired by the learning acquisition unit 21 are learning data (teacher data) used to generate a model. The learning acquisition unit 21 acquires a sufficient number of time-series learning images and learning trajectory information to generate an appropriate model.

[0059] The learning acquisition unit 21 receives and acquires moving images that are time-series learning images transmitted from the learning thermo camera 200. As described above, the images that make up the moving images acquired by the learning acquisition unit 21, i.e., the time-series learning images, are low-resolution images similar to the images that make up the moving images acquired by the acquisition unit 11.

[0060] Furthermore, when the learning thermo camera 200 captures images, the camera may be installed in the space to be imaged by the learning thermo camera 200, so that at least one of the time-series learning images acquired by the learning acquisition unit 21 includes an image showing a light that could be a heat source. That is, at least one of the time-series learning images may be an image showing a person and a light. By using such images for the time-series learning images, the estimation accuracy of the model can be improved.

[0061] Note that the time-series learning images may be other than those described above as long as they correspond to the time-series images acquired by the acquisition unit 11 and are appropriately used for generating a model. The learning acquisition unit 21 may also acquire the time-series learning images obtained by imaging with the learning thermo camera 200 by a method other than those described above.

[0062] The learning acquisition unit 21 acquires video images, which are time-series learning images, and also acquires learning trajectory information indicating the trajectories of people appearing in the video images. The learning trajectory information is a label for model learning. The learning trajectory information is, for example, an identifier (ID) that identifies a person appearing in each learning image that constitutes the video at each timing and information indicating the position of the person. The learning trajectory information is the same type of information as the information output from the model.

[0063] Therefore, the training trajectory information is information indicating the trajectory of a person in the above-mentioned common coordinate system, for example, a coordinate system having coordinate axes along a horizontal plane as described above. The time-series training images and training trajectory information acquired by the training acquisition unit 21 correspond to the time-series images acquired by the acquisition unit 11 and information output from the model, respectively, as training data used to generate the model. The training acquisition unit 21 acquires the training trajectory information, for example, as follows.

[0064] To acquire learning trajectory information, the learning acquisition unit 21 receives and acquires moving images transmitted from the learning RGB camera 201. The moving images transmitted from the learning RGB camera 201 are moving images using visible light (RGB moving images), and are of a different type from the moving images (moving images of heat distribution) transmitted from the learning thermal camera 200. Figures 4(a) and (b) show examples of images constituting the moving images transmitted from the learning RGB camera 201.

[0065] The moving images transmitted from the training RGB camera 201 do not necessarily have to be moving images using visible light (RGB moving images), but may be any images that can identify the trajectory of a person captured for model generation. For example, the moving images used to acquire training trajectory information may be moving images of thermal distribution with a higher resolution than the moving images of thermal distribution captured by the thermal camera 100, and may be high-resolution moving images that can appropriately identify the position of a person.

[0066] The moving images transmitted from the learning RGB camera 201 are, for example, captured during the same period as the moving images transmitted from the learning thermal camera 200, and are moving images at the same frame rate as the learning thermal camera 200. This is to associate each learning image in time series with each position of a person that constitutes the person's trajectory indicated by the learning trajectory information, as will be described later. However, as long as this association is possible, the moving images transmitted from the learning RGB camera 201 do not have to be as described above.

[0067] The learning acquisition unit 21 generates learning trajectory information by estimating the trajectories of people appearing in the acquired video images, for example, as follows. The learning acquisition unit 21 performs a transformation on each of the acquired time-series images from the learning RGB camera 201. This transformation is intended to determine positions in the above-mentioned common coordinate system from the time-series images from the learning RGB camera 201. This transformation corresponds to the transformation performed by the transformation unit 12.

[0068] The learning RGB camera 201 is installed so as to capture, for example, an image obliquely downward. That is, the image obtained by the learning RGB camera 201 is an image of a horizontal surface taken obliquely. The conversion by the learning acquisition unit 21 converts the image into an image of the horizontal surface taken from the front, i.e., from the vertical direction.

[0069] The learning acquisition unit 21 performs projective transformation as the transformation. For example, the projective transformation may be performed as follows. A landmark for projective transformation is provided in advance within the imaging range of the learning RGB camera 201. The shape, color, etc. of the landmark are known in advance. For example, a rectangular mat 400 of a specific color is provided as the landmark, as shown in FIG. 4( b). Because the learning RGB camera 201 captures the landmark from diagonally above, the landmark does not appear rectangular in the image captured by the learning RGB camera 201 as it would appear if captured from a vertical direction. The learning acquisition unit 21 performs projective transformation so that the landmark captured in the image has the same shape as if captured from a vertical direction. For example, in the example shown in FIG. 4( b), projective transformation is performed so that the four corner positions 400a of the mat 400 become the four corners of a rectangle when the mat 400 is captured from a vertical direction.

[0070] Figures 4(c) and (d) show examples of images after projective transformation. Figure 4(c) is an image obtained by transforming the image in Figure 4(a), and Figure 4(d) is an image obtained by transforming the image in Figure 4(b). In the projectively transformed image in Figure 4(d), the mat 410 is a rectangle when photographed from the vertical direction, and the four corner positions 410a of the mat 400 are the four corner positions of the rectangle.

[0071] The learning acquisition unit 21 detects the position of a person appearing in each image constituting the moving image after projective transformation, and assigns an identifier (ID) to identify the person. The person's identifier is the same for the same person across images, and different identifiers are assigned for different people.

[0072] The detection of a person's position and the assignment of an identifier may be performed by a conventional method, for example, a method using a conventional person detection model (YOLO (You Only Look Once)). As described above, the video images obtained by the learning RGB camera 201 do not need to be low resolution, unlike the video images obtained by the above-mentioned thermal camera 100 and learning thermal camera 200, so the position of a person can be detected by a conventional method.

[0073] The learning acquisition unit 21 generates a person's trajectory and identifier, which become learning trajectory information, based on the person's trajectory and identifier obtained as described above. Because the learning thermo camera 200 and the learning RGB camera 201 are positioned at different positions, the relative position of the person with respect to the learning thermo camera 200 differs from the relative position of the person with respect to the learning RGB camera 201. Therefore, the learning acquisition unit 21 may pre-store the positional relationship (relative position) between the learning thermo camera 200 and the learning RGB camera 201, and correct the obtained person's trajectory to match the trajectory of the position detected by the learning thermo camera 200 based on the stored positional relationship. The correction based on the positional relationship is, for example, a linear shift correction, and may be performed using a conventional method.

[0074] The learning acquisition unit 21 associates each learning image in time series with each position of a person that constitutes the person's trajectory indicated by the learning trajectory information. The learning acquisition unit 21 associates learning images with the positions of people that relate to the same time.

[0075] The learning acquisition unit 21 may acquire the learning trajectory information by a method other than the above. For example, the learning acquisition unit 21 may input and acquire learning trajectory information corresponding to time-series learning images obtained by imaging with the learning thermo camera 200, which has been prepared in advance by any method. In this case, the learning acquisition unit 21 does not need to acquire moving images obtained by imaging with the learning RGB camera 201. The learning acquisition unit 21 outputs the acquired moving images obtained by imaging with the learning thermo camera 200 to the learning conversion unit 22. The learning acquisition unit 21 outputs the acquired learning trajectory information to the model generation unit 23.

[0076] The learning conversion unit 22 is a functional unit that performs conversion for each of the time-series learning images acquired by the learning acquisition unit 21 to determine a position in a coordinate system common to multiple thermo cameras, including the thermo camera (learning thermo camera 200). The learning conversion unit 22 converts the time-series learning images, for example, as follows.

[0077] The conversion is for generating an image to be used for generating a model, which corresponds to the image after conversion by the conversion unit 12 of the trajectory estimation system 10. The conversion corresponds to the conversion by the conversion unit 12. The learning conversion unit 22 performs the same conversion as that by the conversion unit 12 of the trajectory estimation system 10.

[0078] The learning thermo camera 200 is installed, for example, so as to capture an image obliquely downward. That is, the image obtained by imaging with the learning thermo camera 200 is an image of a horizontal surface taken obliquely. The conversion by the learning conversion unit 22 converts the image into an image of the horizontal surface taken from the front, i.e., from the vertical direction, for example.

[0079] The learning transformation unit 22 inputs the moving images obtained by the learning thermo camera 200 from the learning acquisition unit 21. The learning transformation unit 22 performs projective transformation as the transformation on each of the learning images that make up the input moving images. The learning transformation unit 22 stores information for performing the projective transformation in advance, and performs projective transformation of the images based on the information. The projective transformation itself may be performed by a conventional method.

[0080] The projective transformation by the learning conversion unit 22 may be performed based on the projective transformation by the learning acquisition unit 21 on the image captured by the learning RGB camera 201. For example, if the correspondence between the image captured by the learning RGB camera 201 and the image after transformation of that image is the same as the correspondence between the learning image captured by the learning thermal camera 200 and the image after transformation of that image, the projective transformation by the learning conversion unit 22 may be performed in the same manner as the projective transformation by the learning acquisition unit 21. Furthermore, if there is a deviation in the above correspondence, the learning conversion unit 22 may perform a projective transformation that corrects the deviation. Furthermore, the transformation by the conversion unit 12 of the trajectory estimation system 10 may also be performed based on the projective transformation by the learning acquisition unit 21 on the image captured by the learning RGB camera 201, as described above.

[0081] The transformation performed by the training transformation unit 22 may be any transformation other than projective transformation, as long as it is a transformation for determining positions in a coordinate system common to multiple thermo cameras from the training images. The training transformation unit 22 outputs a moving image composed of the transformed training images to the model generation unit 23.

[0082] The model generation unit 23 is a functional unit that performs machine learning based on the time-series learning images converted by the learning conversion unit 22 and the learning trajectory information acquired by the learning acquisition unit 21, and recursively uses the time-series images to generate a model that is used to estimate the trajectory of a person appearing in the time-series images in a common coordinate system. The model generation unit 23 generates a model, for example, as follows.

[0083] The model generation unit 23 inputs the converted time-series training images from the training conversion unit 22. The model generation unit 23 inputs training trajectory information corresponding to the time-series training images from the training acquisition unit 21. The model generation unit 23 performs machine learning using the input converted time-series training images as input to the model and the input training trajectory information as the model output (correct answer), to generate a model. When the converted time-series training images are input to the model, the model generation unit 23 inputs the converted time-series training images in chronological order, similar to the input of converted time-series images to the model by the estimation unit 13. Furthermore, when the training trajectory information is input to the model, the model generation unit 23 ensures that the identifier of a person and information indicating the position of the person at the time corresponding to the converted training images input to the model correspond to the model output. The above machine learning training itself, i.e., the update of the model parameters, may be performed in the same manner as conventional machine learning training.

[0084] The combination of the converted time series training images and training trajectory information used simultaneously to generate the model corresponds to the training thermal camera 200 and the training RGB camera 201, which are respectively installed at corresponding positions.

[0085] In the model, the number of past images to be recursively used (the number of past images going back from the input image) is set in advance. It is difficult to acquire a person's external features from low-resolution images. Therefore, if too many past images are used, the person's posture and orientation will be too different, making it difficult to link the images to the same person. For this reason, the number of past images to be recursively used may be kept to a minimum. The number of past images to be recursively used may be set to 10, for example, by parameter tuning.

[0086] The model generation unit 23 outputs the generated model to the trajectory estimation system 10. In the trajectory estimation system 10, the estimation unit 13 inputs and stores the model and uses it to estimate the trajectory of the person described above. The functions of the model generation system 20 according to this embodiment have been described above.

[0087] Next, the processing executed by the server 1 according to this embodiment (the operation method performed by the server 1) will be described using the flowcharts of Figures 5 and 6. First, the trajectory estimation method, which is the processing executed by the trajectory estimation system 10 according to this embodiment, will be described using the flowchart of Figure 5.

[0088] In this process, first, the acquisition unit 11 acquires time-series images for each of the multiple thermo cameras 100, obtained by capturing images at the same time period using the multiple thermo cameras 100 that share at least a portion of their imaging range (S01, acquisition step). Next, the conversion unit 12 performs a transformation on each of the time-series images for each of the multiple thermo cameras 100 to determine a position in a coordinate system common to the multiple thermo cameras 100 (S02, conversion step). Next, the estimation unit 13 sequentially inputs the transformed time-series images for each of the multiple thermo cameras 100 into a model, and estimates the trajectories of people captured in the time-series images in the common coordinate system (S03, estimation step). Next, the estimation unit 13 compares the common timing positions of the person's trajectory estimated for each thermo camera 100 in the common coordinate system, and connects the trajectories estimated for the same person for each thermo camera (S04, estimation step). Next, the estimation unit 13 outputs information indicating the estimation results (S05). The above is the trajectory estimation method, which is the processing executed by the trajectory estimation system 10 according to this embodiment.

[0089] Next, a model generation method, which is processing executed by the model generation system 20 according to this embodiment, will be described using the flowchart in Fig. 6. This processing is performed before processing is executed by the trajectory estimation system 10 according to this embodiment.

[0090] In this process, the learning acquisition unit 21 first acquires time-series learning images obtained by the learning thermo camera 200 and time-series RGB images obtained by the learning RGB camera 201 (S11, learning acquisition step). The learning acquisition unit 21 then estimates a person's trajectory from the time-series RGB images to generate learning trajectory information (S12, learning acquisition step). The learning conversion unit 22 then performs a conversion on each of the time-series learning images to determine a position in a coordinate system common to multiple thermo cameras (S13, learning conversion step). The model generation unit 23 then performs machine learning based on the converted time-series learning images and learning trajectory information to generate a model (S14). The generated model is output to the trajectory estimation system 10, stored by the estimation unit 13, and used to estimate the person's trajectory. The model generation method, which is the processing executed by the model generation system 20 according to this embodiment, has been described above.

[0091] As described above, in the trajectory estimation system 10 according to this embodiment, a person's trajectory is estimated using low-resolution images such as those shown in FIGS. 2A and 2B captured by a thermal camera 100. Normally, it is difficult to detect the position of a person and estimate the person's trajectory from such low-resolution images. Estimating a person's trajectory is particularly difficult when people overlap in the image. Furthermore, when using images from multiple thermal cameras 100, it is difficult to combine the person trajectories estimated for each thermal camera 100.

[0092] In response to this, in the trajectory estimation system 10 according to the present embodiment, a transformation is performed on each of the time-series images from each of the multiple thermal cameras 100 to determine a position in a coordinate system common to the multiple thermal cameras 100. Next, a model that recursively uses the time-series images generated by machine learning is used to estimate the trajectories of the person appearing in the time-series images in the common coordinate system from the transformed time-series images for each of the multiple thermal cameras 100. Next, the common timing positions of the person's trajectories estimated for each of the thermal cameras 100 in the common coordinate system are compared, and the trajectories estimated for the same person for each of the thermal cameras 100 are linked. As described above, the estimation of the person's trajectory for each of the thermal cameras 100 is performed using the transformed time-series images, so the estimated trajectories can be linked in a common coordinate system.

[0093] By using the above estimation, the trajectory estimation system 10 according to this embodiment can appropriately estimate a person's trajectory even when using low-resolution images captured by the thermal camera 100. In particular, it can appropriately estimate a person's trajectory even when people overlap in the image. Furthermore, using low-resolution images makes it possible to track a person's trajectory while taking privacy into consideration and not identifying the individual.

[0094] Furthermore, the estimated trajectories can be appropriately linked. As a result, the trajectory estimation system 10 according to this embodiment can appropriately estimate the trajectory of a person even when using low-resolution images captured by multiple thermal cameras 100.

[0095] This makes it possible to track people's trajectories without identifying them, even in buildings where a single camera cannot capture the entire route. By tracking people regardless of the indoor structure using privacy-conscious sensing technology, it is possible to develop people flow utilization services such as security surveillance in private spaces.

[0096] Furthermore, as described above, the image transformation may be a projective transformation. With this configuration, an appropriately transformed image can be obtained, and the person's trajectory can be appropriately estimated. However, the image transformation does not have to be a projective transformation, and any transformation can be used to determine the position in a coordinate system common to multiple thermal cameras 100 from the image.

[0097] Furthermore, as described above, each of the time-series images used to estimate a person's trajectory may be an image having a pixel count of 32 or less in both the vertical and horizontal directions. However, each of the time-series images used to estimate a person's trajectory may have a resolution other than the above.

[0098] Furthermore, the model used to estimate a person's trajectory may include a recurrent neural network, such as DeepSORT. This configuration allows time-series images to be used to estimate a person's trajectory appropriately and reliably, thereby enabling appropriate and reliable estimation of the person's trajectory. However, the model used to estimate a person's trajectory does not need to include a recurrent neural network; it may be any model that recursively uses time-series images generated by machine learning.

[0099] The model generation system 20 according to this embodiment can generate a model to be used in the trajectory estimation system 10. Furthermore, the time-series learning images used to generate the model may include images in which a light is captured. This configuration can improve the estimation accuracy of the generated model. However, the time-series learning images used to generate the model do not have to include images in which a light is captured.

[0100] Furthermore, as described above, in generating a model, time-series trajectory estimation images (for example, time-series RGB images) may be acquired, and the trajectories of people appearing in the acquired time-series trajectory estimation images may be estimated to acquire the trajectory estimation images. This configuration allows for easy and appropriate acquisition of training trajectory information, thereby allowing for easy and appropriate generation of a model. However, the trajectory estimation images do not necessarily need to be acquired in the manner described above, and may be acquired by any method.

[0101] In the present embodiment, the server 1 includes the trajectory estimation system 10 and the model generation system 20. However, the trajectory estimation system 10 and the model generation system 20 may be implemented independently of each other. That is, the trajectory estimation system 10 and the model generation system 20 may be implemented by separate devices.

[0102] The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of hardware and / or software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (e.g., wired, wireless, etc.) and these multiple devices. The functional block may also be realized by combining software with the single device or multiple devices.

[0103] Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.

[0104] For example, the server 1 according to an embodiment of the present disclosure may function as a computer that performs information processing according to the present disclosure. Fig. 7 is a diagram illustrating an example of a hardware configuration of the server 1 according to an embodiment of the present disclosure. The server 1 described above may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

[0105] In the following description, the term "device" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the server 1 may be configured to include one or more of the devices shown in the drawings, or may be configured to exclude some of the devices.

[0106] Each function in server 1 is realized by loading specified software (programs) onto hardware such as processor 1001 and memory 1002, causing processor 1001 to perform calculations, control communication via communication device 1004, and control at least one of reading and writing data in memory 1002 and storage 1003.

[0107] The processor 1001, for example, runs an operating system to control the entire computer. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, each function of the server 1 described above may be realized by the processor 1001.

[0108] The processor 1001 also reads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes in accordance with these. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. For example, each function of the server 1 may be implemented by a control program stored in the memory 1002 and running on the processor 1001. While the above-described various processes have been described as being executed by one processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may also be transmitted from a network via a telecommunications line.

[0109] The memory 1002 is a computer-readable recording medium and may be configured, for example, by at least one of a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for performing information processing according to an embodiment of the present disclosure.

[0110] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The storage medium provided in server 1 may be, for example, a database including at least one of memory 1002 and storage 1003, a server, or other appropriate medium.

[0111] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.

[0112] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0113] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0114] The server 1 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0115] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0116] Input and output information may be stored in a specific location (for example, memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.

[0117] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0118] The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0119] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0120] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0121] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0122] As used in this disclosure, the terms "system" and "network" are used interchangeably.

[0123] Furthermore, the information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information.

[0124] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0125] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using one or more wires, cables, and / or printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0126] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0127] As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0128] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

[0129] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0130] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0131] The trajectory estimation system, model generation system, trajectory estimation method, and model generation method disclosed herein have the following configuration: [1] A trajectory estimation system comprising: an acquisition unit that acquires time-series images for each of a plurality of thermo cameras obtained by capturing images at the same time period using the plurality of thermo cameras that have at least a partial common imaging range; a conversion unit that performs a conversion on each of the time-series images for each of the plurality of thermo cameras acquired by the acquisition unit to determine a position in a coordinate system common to the plurality of thermo cameras; and an estimation unit that estimates a trajectory of a person appearing in the time-series images in the common coordinate system by sequentially inputting the time-series images converted by the conversion unit into a model that recursively uses the time-series images generated by machine learning for each of the plurality of thermo cameras, and connects the trajectories estimated for the same person for each thermo camera by comparing positions at common times in the common coordinate system for the person's trajectory estimated for each thermo camera. [2] The trajectory estimation system described in [1], wherein the conversion unit performs a projective transformation as the transformation. [3] The trajectory estimation system according to [1] or [2], wherein the acquisition unit acquires, as each image in the time series, an image having 32 or less pixels in each of the vertical and horizontal directions. [4] The trajectory estimation system according to any of [1] to [3], wherein the model includes a recurrent neural network. [5] A model generation system comprising: a training acquisition unit that acquires time series training images acquired by imaging with a thermal camera and training trajectory information indicating the trajectories of people appearing in the time series training images; a training conversion unit that performs conversion from the image to determine a position in a coordinate system common to multiple thermal cameras including the thermal camera, for each of the time series training images acquired by the training acquisition unit; and a model generation unit that performs machine learning based on the time series training images converted by the training conversion unit and the training trajectory information acquired by the training acquisition unit, and recursively uses the time series images to generate a model used to estimate the trajectory of people appearing in the time series images in the common coordinate system. [6] The model generation system according to [5], wherein the learning acquisition unit acquires the time-series learning images including images in which a light is captured.[7] The model generation system according to [5] or [6], wherein the learning acquisition unit acquires time-series images for trajectory estimation that are different in type from the time-series images for learning and that have been obtained by imaging corresponding to the time-series images for learning, estimates trajectories of people appearing in the acquired time-series images for trajectory estimation, and acquires the learning trajectory information. [8] A trajectory estimation method including: an acquisition step in which a trajectory estimation system acquires time-series images for each of a plurality of thermo cameras obtained by imaging at the same time period using the plurality of thermo cameras that have at least a partial common imaging range; a transformation step in which the trajectory estimation system performs a transformation on each of the time-series images for each of the plurality of thermo cameras acquired in the acquisition step to determine a position in a coordinate system common to the plurality of thermo cameras; and an estimation step in which the trajectory estimation system estimates a trajectory of a person appearing in the time-series images in the common coordinate system by sequentially inputting the time-series images converted in the transformation step into a model that recursively uses time-series images generated by machine learning for each of the plurality of thermo cameras, and connecting the trajectories estimated for the same person for each thermo camera by comparing positions at common times in the common coordinate system for the trajectories of the person estimated for each thermo camera. [9] A model generation method including: a learning acquisition step in which a model generation system acquires time-series learning images obtained by imaging with a thermal camera and learning trajectory information indicating the trajectories of people appearing in the time-series learning images; a learning transformation step in which the model generation system performs a transformation on each of the time-series learning images acquired in the learning acquisition step to determine a position in a coordinate system common to a plurality of thermal cameras including the thermal camera; and a model generation step in which the model generation system performs machine learning based on the time-series learning images converted in the learning transformation step and the learning trajectory information acquired in the learning acquisition step, to recursively use the time-series images and generate a model used to estimate the trajectory of people appearing in the time-series images in the common coordinate system.

[0132] 1...server, 10...trajectory estimation system, 11...acquisition unit, 12...conversion unit, 13...estimation unit, 20...model generation system, 21...learning acquisition unit, 22...learning conversion unit, 23...model generation unit, 100...thermal camera, 200...learning thermal camera, 201...learning RGB camera, 1001...processor, 1002...memory, 1003...storage, 1004...communication device, 1005...input device, 1006...output device, 1007...bus.

Claims

1. A trajectory estimation system comprising: an acquisition unit that acquires time-series images for each of a plurality of thermo cameras obtained by imaging at the same time period using multiple thermo cameras that have at least a partial common imaging range; a conversion unit that performs conversion on each of the time-series images for each of the multiple thermo cameras acquired by the acquisition unit to determine a position in a coordinate system common to the multiple thermo cameras; and an estimation unit that estimates the trajectory of a person appearing in the time-series images in the common coordinate system by sequentially inputting the time-series images converted by the conversion unit into a model that recursively uses time-series images generated by machine learning for each of the multiple thermo cameras, and compares the common timing positions of the person's trajectory estimated for each thermo camera in the common coordinate system, and connects the trajectories estimated for the same person for each thermo camera.

2. The trajectory estimation system according to claim 1, wherein the transformation unit performs a projective transformation as the transformation.

3. The trajectory estimation system according to claim 1, wherein the acquisition unit acquires, as each image in the time series, an image having 32 or less pixels in each of the vertical and horizontal directions.

4. The trajectory estimation system of claim 1, wherein the model comprises a recurrent neural network.

5. A model generation system comprising: a learning acquisition unit that acquires time-series learning images obtained by imaging with a thermal camera and learning trajectory information that indicates the trajectories of people appearing in the time-series learning images; a learning conversion unit that performs conversion on each of the time-series learning images acquired by the learning acquisition unit to determine a position in a coordinate system common to multiple thermal cameras including the thermal camera; and a model generation unit that performs machine learning based on the time-series learning images converted by the learning conversion unit and the learning trajectory information acquired by the learning acquisition unit, and recursively uses the time-series images to generate a model that is used to estimate the trajectory of people appearing in the time-series images in the common coordinate system.

6. A model generation system according to claim 5, wherein the learning acquisition unit acquires the time series of learning images including images in which a light is captured.

7. A model generation system as described in claim 5, wherein the learning acquisition unit acquires time-series images for trajectory estimation that are of a different type from the time-series learning images and that have been obtained by imaging that corresponds to the time-series learning images, estimates the trajectories of people appearing in the acquired time-series images for trajectory estimation, and acquires the learning trajectory information.

8. A trajectory estimation method comprising: an acquisition step in which a trajectory estimation system acquires time-series images for each of a plurality of thermo cameras obtained by imaging at the same time period using the plurality of thermo cameras that have at least a partial common imaging range; a transformation step in which the trajectory estimation system performs transformation on each of the time-series images for each of the plurality of thermo cameras acquired in the acquisition step to determine a position in a coordinate system common to the plurality of thermo cameras; and an estimation step in which the trajectory estimation system estimates the trajectory of a person appearing in the time-series images in the common coordinate system by sequentially inputting the time-series images converted in the transformation step into a model that recursively uses time-series images generated by machine learning for each of the plurality of thermo cameras, and connects the trajectories estimated for the same person for each thermo camera by comparing common timing positions in the common coordinate system for the trajectories of the person estimated for each thermo camera.

9. A model generation method comprising: a learning acquisition step in which a model generation system acquires time-series learning images obtained by imaging with a thermal camera and learning trajectory information indicating the trajectories of people appearing in the time-series learning images; a learning transformation step in which the model generation system performs a transformation on each of the time-series learning images acquired in the learning acquisition step to determine a position in a coordinate system common to multiple thermal cameras including the thermal camera; and a model generation step in which the model generation system performs machine learning based on the time-series learning images converted in the learning transformation step and the learning trajectory information acquired in the learning acquisition step, to recursively use the time-series images and generate a model to be used for estimating the trajectory of people appearing in the time-series images in the common coordinate system.

Images