Image processing device, image processing method, and computer readable medium

Abstract

The present invention discloses an image processing device, an image processing method and a computer-readable medium. In the image processing device, an acquisition unit acquires a motion image including multiple images. A density estimation unit estimates the density distribution of a target object based on images in the multiple images. A position estimation unit estimates the position of the target object in the image based on the density distribution. A measurement unit measures the flow of the target object based on the position of the target object in the multiple images.

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method, and a computer-readable medium, and more particularly to an analysis technology for moving images captured by a video camera, a security camera, etc. Background Art

[0002] In recent years, a device has been proposed for analyzing the flow of a target object (e.g., a person) (e.g., the number of people or the direction of people) from an image capture area of ​​a moving image captured by a camera or the like, that is, for analyzing the flow of people. In order to measure the flow of the target object, it is necessary to detect the target object from the moving image. Japanese Patent Laid-Open No. 2007-201556 discloses a method for detecting a human area included in an image captured by a fixed-point camera, and measuring the number of people. Summary of the invention

[0003] According to one embodiment of the present invention, an image processing device includes: an acquisition unit for acquiring a motion image including multiple images; a density estimation unit for estimating the density distribution of a target object based on an image in the multiple images; a position estimation unit for estimating the position of the target object in the image based on the density distribution; and a measurement unit for measuring the flow of the target object based on the position of the target object in the multiple images.

[0004] According to another embodiment of the present invention, an image processing method includes: obtaining a motion image including multiple images; estimating a density distribution of a target object based on an image in the multiple images; estimating a position of the target object in the image based on the density distribution; and measuring a flow of the target object based on the position of the target object in the multiple images.

[0005] Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings). BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a diagram illustrating one example of a hardware configuration of an image processing apparatus.

[0007] Figure 2 is a diagram illustrating one example of a functional structure of an image processing apparatus.

[0008] Figure 3 is a diagram illustrating one example of a processing flow of the image processing apparatus.

[0009] Figure 4A and 4B is a diagram illustrating an example of a method of extracting a partial image from an input image.

[0010] Figures 5A to 5D It is a diagram used to explain the central area and blank area of ​​a part of the image.

[0011] Fig. 6A and 6B is a diagram illustrating an example of density distribution estimation of people in an input image.

[0012] Figures 7A to 7C This is a diagram illustrating an example of estimating the position coordinates of a person from density distribution.

[0013] Figures 8A to 8C This is a diagram illustrating an example of a process for calculating the position coordinates of a person from a position map.

[0014] Fig. 9 is a diagram illustrating a matching method for matching position coordinates of a person at different times.

[0015] Figures 10A to 10C is a diagram illustrating an example of a cost matrix for matching and a matching result.

[0016] Fig.11 is a diagram illustrating an example of a process for calculating a motion trajectory of a person through matching.

[0017] Fig. 12A and 12B This is a diagram illustrating a matching method based on the estimated position of a person.

[0018] Fig.13A and 13B This is a diagram illustrating a method of compensating a movement trajectory when the movement trajectory cannot be continued.

[0019] Fig.14A and 14B This is a diagram illustrating an example of a method of measuring flow rate from a person's movement trajectory.

[0020] Fig.15 This is a diagram illustrating an example of a method of measuring flow rate from a person's movement trajectory. DETAILED DESCRIPTION

[0021] Each embodiment will be described in detail below with reference to the accompanying drawings. It should be noted that these embodiments are by no means intended to limit the scope of the present invention. Many features are described in these embodiments, but no invention needs to include the limitation of all these features, and these features can be combined arbitrarily. Further, in the accompanying drawings, the same or similar configurations will be given the same reference numerals, and repeated descriptions will be omitted.

[0022] In order to accurately measure the flow of a target object in a scene with a group of target objects (a crowded scene), for example, in the case of capturing an image of a group of people, it is necessary to accurately detect the position of the target object in such a scene. On the other hand, Japanese Patent Laid-Open No. 2007-201556 is configured to detect an area occupied by a person to detect the presence of a person. In this method, when the overlap of people becomes larger, the detection accuracy of people will decrease. Therefore, this method makes the accuracy of measuring flow in a crowded scene very low.

[0023] An embodiment of the present invention can improve the accuracy of measuring the flow of a target object in a crowded scene.

[0024] Figure 1 An example of a hardware structure of an image processing apparatus 100 according to an embodiment of the present invention is illustrated. The image processing apparatus 100 includes a control device 11, a storage device 12, a calculation device 13, an input device 14, an output device 15, and an I / F (Interface) device 16.

[0025] The control device 11 is a device configured to control the entire image processing apparatus 100. The storage device 12 stores programs and data required for the operations performed by the control device 11. The calculation device 13 is configured to perform necessary calculation processing under the control of the control device 11. For example, the calculation device 13 can be configured to perform neural network calculations to be described later. The input device 14 is a human-machine interface device or the like, and is configured to receive input via an operation of a user. The output device 15 is a display device or the like, and is configured to present the processing results generated by the image processing apparatus 100 to the user.

[0026] The I / F device 16 may be a universal serial bus, or optical cables, or wired interfaces such as Wi-Fi or Via the I / F device 16, other devices may be connected to the image processing apparatus 100. For example, an image capturing device such as a camera may be connected to the I / F device 16, so that the image processing apparatus 100 may obtain the captured image via the I / F device 16. As another example, the image processing apparatus 100 may send the processing result to the external device via the I / F device 16. As yet another example, the image processing apparatus 100 may obtain a program, data, etc. required for operation via the I / F device 16.

[0027] The functions of the image processing device 100 to be described later can be implemented by, for example, a processor (e.g., control device 11) operating according to a program on a memory (e.g., storage device 12). The storage device 12 or other recording medium can store such a program. However, at least a part of the functions of the image processing device 100 to be described later can be implemented by hardware configured for this at least part of the functions. In addition, the image processing device according to the present invention can be implemented as a combination of multiple devices connected to each other via a network, for example.

[0028] Figure 2 An example of a functional structure of the image processing apparatus 100 according to an embodiment of the present invention is illustrated. The image processing apparatus 100 includes an obtaining unit 201, a density estimating unit 202, a position estimating unit 203, a tracking unit 204, and a measuring unit 205 as functional elements.

[0029] The acquisition unit 201 is configured to acquire a moving image including a plurality of images. The moving image thus acquired by the acquisition unit 201 is analyzed to measure the flow of the target object. The flow of the target object may be the flow of the target object on the image or the flow of the target object in the real space estimated via image analysis. Note that the target object to be analyzed is not particularly limited, and examples of the target object include people, vehicles such as bicycles and motorcycles, vehicles such as cars and trucks, and animals such as livestock.

[0030] The moving image may be a stream, a moving image file, a series of image files stored per frame, a moving image stored on a medium, etc., and these include a plurality of images as frame images. These plurality of images may be, for example, images captured at different times by an image capture device installed in the same place. The acquisition unit 201 may obtain the moving image from a solid-state image sensor such as a CMOS sensor or a CCD sensor, or from an image capture device such as a camera equipped with such a solid-state image sensor. In addition, the acquisition unit 201 may obtain the moving image data from a storage device such as a hard disk or an SSD, or from a storage medium, etc.

[0031] The density estimation unit 202 is configured to estimate the density distribution of the target object whose flow is to be measured in the image obtained by the acquisition unit 201. The density distribution indicates the portion of the image where the target object (e.g., a person's head) for flow measurement is estimated to exist. The density distribution may indicate the region in the image where the target object is estimated to exist. The density distribution may, for example, be a uniform distribution centered at a position where the target object exists with a high probability and may correspond to the size of the target object. In addition, the density distribution may be any distribution having a maximum value at a position where the target object exists with a high probability. The latter example includes a Gaussian distribution. The density distribution may indicate the probability of the target object existing. The density estimation unit 202 is capable of outputting a density map as information representing the density distribution, which represents the density of the target object at each position in the image. Please note that the resolution of the density map is equal to or less than the resolution of the image.

[0032] In the case where the target objects overlap each other in the image, the density estimation unit 202 may calculate a density distribution in which the density distributions respectively corresponding to the target objects overlap. The density distribution may be standardized so that the sum of the densities in the density distribution corresponding to one target object is 1. In this case, the sum of the densities of the density distribution generated by the density estimation unit 202 corresponds to the number of target objects in the image.

[0033] Different methods can be used as methods for the density estimation unit 202 to estimate density distribution. For example, the density estimation unit 202 can input an image, or features extracted from an image, or both an image and a feature into a neural network. Examples of such neural networks include convolutional neural networks, deconvolutional neural networks, autoencoders to which convolutional and deconvolutional neural networks are connected, networks with shortcuts such as U-Net, etc. The neural network used by the density estimation unit 202 is trained so that when an image is input to the neural network, the neural network will output the density distribution of the target object. For example, the training can be implemented by using images for training and training data representing the density distribution of images such as the above. For example, the training can be implemented in such a way that the sum of the output values ​​of the neural network is equal to the number of target objects, that is, the output value represents the density of the target object at each position.

[0034] The position estimation unit 203 is configured to estimate the position of the target object in the image based on the density distribution obtained from the density estimation unit 202. For example, the position estimation unit 203 may be configured to output the coordinates of the position of the target object thus estimated. The coordinates of the position of the target object are representative points of the target object in the image. The representative point of the target object may be the center of the head of a person being measured as the target object.

[0035] The method by which the position estimation unit 203 estimates the position is not particularly limited, and the position estimation unit 203 can, for example, estimate the center of gravity of the density distribution as the position of the target object. On the other hand, the density estimation unit 202 may provide an unclear boundary of the density distribution corresponding to a target object, and such a boundary will have various shapes. In addition, due to the overlap of the density distribution corresponding to a target object, there is the possibility that such overlapping density distribution has a complex shape. In addition, in certain cases, the density estimation unit 202 may provide an unexpected output. In one embodiment, the position estimation unit 203 is configured to estimate the position of the target object by using a neural network. The use of the neural network makes it possible to obtain a result of position estimation with a predetermined fixed amount of calculation regardless of the density distribution of the input. Therefore, the position estimation unit 203 is able to process various density distributions output by the density estimation unit 202.

[0036] For example, the neural network used by the position estimation unit 203 can be configured so that, in the case of a density map representing a density distribution as an input, the neural network outputs a position map that represents the estimated result of the two-class classification of each position on the density map by probability. In this way, the position estimation unit 203 can perform such a two-class classification: for each position in the image, it is distinguished whether the position represents a representative point of the target object. In this two-class classification, in the case where the position on the density map represents the position coordinates of the target object, the estimated result of the position is 1, and, in the case where the position on the density map does not represent the position coordinates of the target object, the estimated result of the position is 0. The position map 701 can represent the probability of whether each position on the density map is a position representing the position coordinates of the target object. Such a position map is so sparse that most elements are 0. Therefore, since it is possible to handle the two-class classification problem of sparse output, it becomes easier to implement the training of a neural network for generating such a position map.

[0037] Examples of such a neural network include a convolutional neural network, a deconvolutional neural network, an autoencoder in which convolutional and deconvolutional neural networks are connected, a network with a shortcut such as U-Net, etc. In addition, the neural network used by the position estimation unit 203 may have an output layer whose value range is greater than 0 but less than 1. Examples of such an output layer include a step function, a Sigmoid function, a SoftMax function, etc. A neural network having such an output layer can provide an output suitable for a two-class classification problem.

[0038] The neural network used by the position estimation unit 203 is pre-trained so that when the density map is input to the neural network, the neural network will output the position of the target object. Such training can be achieved, for example, by using a density map for training and training data representing the density map in which elements corresponding to the positions of people are 1 and other elements are 0.

[0039] The measuring unit 205 is configured to measure the flow of the target object based on the position of the target object in the plurality of images estimated by the position estimating unit 203. For example, the measuring unit 205 may measure the flow rate of the target object or the flow direction of the target object. The flow rate may be, for example, the number of target objects passing through a pre-set measurement area or measurement line. For example, the measuring unit 205 may be configured to measure the number of target objects moving from one area to another area, wherein the areas are divided by the measurement area. In the case where the measurement area is a line (measurement line), the measuring unit 205 may be configured to measure the number of target objects passing through the measurement line.

[0040] Such a measurement area may be preset based on the image capture scene. In this case, the measurement unit 205 may be configured to obtain a setting value of the predetermined measurement area from a memory such as the storage device 12. In addition, the measurement unit 205 may be configured to obtain a setting value input by a user through operation of a human-machine interface device connected to the input device 14, etc., with reference to the image displayed on the output device 15.

[0041] The measurement area may be one or more measurement areas. In addition, the shape of the measurement area is not particularly limited, and may be, for example, a broken line, a curve, a polygon, a circle, an ellipse as appropriate, or any shape consisting of a closed curve. In addition, the measurement area may be a line with a small thickness or a line with a large thickness (or a longitudinal rectangular shape). Such a measurement area may be set according to a freely selectable guide shape that may be a line, a rectangular shape, etc. For example, a shape according to such a guide shape may be provided at a specific position of the image. Such a guide shape may be one or more guide shapes. In addition, the guide shape may be extended by expanding its width, etc.

[0042] The method of measuring the flow of the target object based on the position of the target object is not particularly limited, but the present embodiment is configured such that the measuring unit 205 measures the flow by using the tracking result of the target object obtained by the tracking unit 204. The tracking unit 204 is configured to track the position of the target object. The tracking unit 204 may be configured to track the position of the target object based on the position of the target object estimated by the position estimating unit 203. For example, based on the position of the target object estimated for a plurality of images, the tracking unit 204 may be configured to perform tracking so that the tracking unit 204 tracks the change in the position over time of what is estimated to be the same target object.

[0043] In one embodiment, the tracking unit 204 is configured to obtain a motion trajectory of the target object by tracking the position coordinates of the target object based on the position coordinates of the target object output by the position estimation unit 203. The tracking unit 204 can associate the position of the target object in a first image at a first time in the multiple images with the position of the target object in a second image at a second time in the multiple images. By repeating such association, the tracking unit 204 can obtain the motion trajectory of the target object.

[0044] For example, the tracking unit 204 may be configured to perform tracking by using a list of position coordinates of the target object in images at different times. For example, the density estimation unit 202 and the position estimation unit 203 perform the above-mentioned processing on each of a plurality of images captured at the same place and with the same viewing angle but at different times, thereby preparing a list of position coordinates of the target object for each of the plurality of images. The tracking unit 204 may be configured to obtain a motion trajectory of the target object by matching the position coordinates of the target object on a list of position coordinates for one time with the position coordinates of the target object on a list of position coordinates for another time. In this case, the measuring unit 205 may measure the flow of the target object, for example, by determining whether the motion trajectory crosses a measurement line. The following will refer to Figure 3 A specific processing example is described.

[0045] Will refer to Figure 3 A processing example of the image processing device 100 according to the present embodiment will be described. In the following, the measurement of the flow of people will be described as an example. More specifically, the number of people passing through a preset measurement area will be measured.

[0046] In step S301, the acquisition unit 201 acquires a motion image. In addition, the acquisition unit 201 acquires a plurality of images for measuring flow from the motion image. For example, the acquisition unit 201 may be configured to acquire a plurality of images captured at a plurality of times between the measurement start time and the measurement end time. Note that the acquisition unit 201 may be configured to acquire a frame image from an image capture device. In the following, each of the plurality of images acquired in step S301 is regarded as an input image. The processing of steps S302 and S303 will be performed on each input image.

[0047] The acquisition unit 201 may extract one or more partial images from the input image. The acquisition unit 201 may extract one or more partial images from each of the multiple images, the one or more partial images belonging to a part of the image. By using a partial image smaller in size than the input image, the amount of processing after step S302 may be reduced. Such a configuration will be helpful for real-time measurement of flow. The area (partial area) of the input image from which the partial image will be extracted may be set in advance. The area (partial area) from which the partial image is to be extracted may be set by the user. For example, it may be configured such that the user operates a human-machine interface device connected to the input device 14, etc., so as to set a partial area on the input image displayed on the output device 15. The setting method of the area from which the partial image is to be extracted and the specific shape of the area are not particularly limited. For example, the acquisition unit 201 may automatically determine the partial area to be extracted based on the area specified by the user. As described above, the acquisition unit 201 may obtain the user's designation of the measurement area. In this case, the acquisition unit 201 may automatically determine the partial area to be extracted based on the measurement area so specified. For example, the acquisition unit 201 may extract the partial image from the input image in such a way that the partial image includes the measurement area.

[0048] In addition, if Figure 4A As illustrated, the obtaining unit 201 may be configured to divide the entire input image 400 into a plurality of sub-regions. In this case, the obtaining unit 201 may extract a partial image from each of one or more sub-regions, the one or more sub-regions being selected from the plurality of sub-regions. Figure 4A As illustrated, if the measurement area 401 is set in the input image 400, a partial image to be used for measuring the flow rate can be extracted from a rectangular area surrounded by a thick frame and including the measurement area 401. Figure 4B As illustrated, one or more rectangular regions including the measurement region 403 may be set in a portion of the input image 402. From the rectangular regions thus set, a partial image to be used for measuring the flow rate may be extracted.

[0049] although Figure 4A and 4B The partial image in the embodiment is a rectangle, but the shape of the partial image may be, for example, a polygon, a circle, an ellipse as appropriate, or an arbitrary shape consisting of a closed curve. In addition, in the case where the shape of the measurement area is not a line segment but a broken line, a curve, a polygon, a circle, an ellipse as appropriate, or an arbitrary shape consisting of a closed curve, the partial image can be extracted in a similar manner.

[0050] In an input image obtained by an image capture device such as a camera, a person to be measured will be depicted to have different sizes depending on where the person is located. Therefore, the size of the partial image may vary depending on the position in the image. For example, the size of the partial image may be set in such a way that the ratio between the size of the partial image and the size of the person will become a roughly constant, and the partial image may be extracted based on the size. Here, the size of the person may be the size of a human body part including the head and shoulders of the person, the size of the head of the person, the size of the entire human body, etc. Figure 4A and 4B An example of changing the size of a portion of an image in this manner is illustrated. Figure 4A and 4B The farther the area where the person is drawn is located (ie, the upper part of the input image), the smaller the size of the partial image.

[0051] When the partial images are set in this way, the processing of steps S302 and S303 will be performed on each of the partial images in the input image. That is, the density estimation unit 202 can estimate the density distribution of the target object in each partial image, and the position estimation unit 203 can estimate the position of the target object from the density distribution of each partial image. On the other hand, the accuracy of the estimation results of steps S302 and S303 will be lower in the boundary area (peripheral area) around the partial image. Therefore, in one embodiment, it can be configured to define a central area in the partial image so that the flow rate of the person detected in the central area can be measured in step S305. This configuration measures the flow rate by using the detection results of high-precision people, so that the accuracy of the measurement can be improved.

[0052] like Figure 5A As illustrated, an example of a method of defining a central area in a partial image includes defining a central area 501 and a blank area 502 inside a partial image 500, wherein the blank area 502 is surrounded by a boundary line of the partial image 500 and a boundary line of the central area 501. Figure 5A In the example illustrated in , the central area 501 is a rectangle, but the shape of the central area 501 is not particularly limited. For example, the shape of the central area 501 can be any shape that can be included in the partial image 500, such as a polygon, a circle, an ellipse, or any closed curve.

[0053] The size of the blank area can be set according to the size of the person whose flow measurement is to be performed. For example, the blank area can be set to have a size that can contain a human body part required for detecting the person. Figure 5B An example of this is shown in Figure 5BIn the embodiment of the present invention, the blank area 502 of the partial image 500 is set to a size that can include a human body part required for detecting the person 503 or the person 504 for which flow measurement is to be performed. More specifically, the size of the blank area is set to a size that can include the head and shoulders of the person. As another example, the blank area can be set in such a way that the blank area is large enough to include the head of the person but not the shoulders, or large enough to include the whole body of the person.

[0054] In the case where a plurality of partial images are extracted from an input image, the regions of the partial images may be set in a manner such that adjacent partial images overlap each other. For example, the obtaining unit 201 may divide the image into a plurality of regions. For each of one or more regions selected from the plurality of regions, the obtaining unit 201 may extract a partial image from the region (central region) and a blank region surrounding the region. Figure 5C An example of such a segmentation is shown in Figure 5C , two adjacent partial images 505 and 506 partially overlap each other. More specifically, partial images 505 and 506 are extracted in such a manner that there is no gap between central area 507 of partial image 505 and central area 508 of partial image 506. Even if a person 510 to be detected is in blank area 512 of partial image 505, person 510 can be detected from central area 508 of partial image 506 adjacent to partial image 505.

[0055] Use as Figure 5C In the illustrated configuration where the central areas of the partial images are adjacent to each other without gaps, the person to be measured can be detected in any of the central areas of the partial images, thereby improving the accuracy of measurement. An example of such a method of extracting partial images is that an input image is divided into a plurality of areas, and a partial image is extracted from each area (corresponding to the central area) and the blank areas surrounding the areas. In addition, a segmentation method that can cover the measurement area with the necessary minimum area while ensuring the blank areas of the partial images can be determined by optimization calculations, etc. Figure 5D , along a measurement area 521 set in an input image 520, a plurality of partial images 522 each having a blank area are extracted.

[0056] In the following explanation, the word "input image" means the input image itself or a partial image. That is, the processing of steps S302 and S303 can be performed on each of the input image or the partial image in the input image.

[0057] In step S302, as described above, the density estimation unit 202 estimates the density distribution of the person to be measured from the input image obtained by the obtaining unit 201 in this way. Fig. 6AIn the illustrated example, the density estimation unit 202 estimates the density distribution of the head position of the person 601 in the input image 600. Figure 6B As illustrated, the density estimation unit 202 generates a density map 602 representing a density distribution 603 of the head positions of the person 601 .

[0058] Please note that the input image used by the density estimation unit 202 is not limited to one, and the density estimation unit 202 can use multiple input images to estimate the density distribution. For example, the density estimation unit 202 can estimate the density distribution by inputting multiple input images into the neural network. The multiple input images used by the density estimation unit 202 can be, for example, images of multiple different frames contained in a motion image. In addition, the density distribution output by the density estimation unit 202 is not limited to one. For example, the density estimation unit 202 can be configured to output multiple density distributions corresponding to multiple input images respectively. In the case where the density estimation unit 202 is configured to output multiple density distributions, the subsequent position estimation unit 203 can receive all density distributions or receive one or more selected density distributions.

[0059] In step S303, as described above, the position estimation unit 203 estimates the position coordinates of the person from the density distribution output by the density estimation unit 202. For example, the position estimation unit 203 estimates the position coordinates of the person by inputting the following information into the neural network: Fig. 7A The density map 700 shown in the example may be output as Figure 7B The illustrated position map 701. The position map 701 illustrates a case where the position coordinates of a person match the position of an element on the position map 701. In this case, the value (probability value) of an element corresponding to a person will become larger. Therefore, the position of such an element can be determined as the position coordinates of a person.

[0060] On the other hand, Figure 7C As shown in the illustrated position graph 702, when the position coordinates of a person do not match the positions of elements on the position graph 702 (i.e., the positions of the grid-like points), the values ​​(probability values) of the multiple elements corresponding to a person will become larger. Therefore, there are some situations where multiple positions adjacent to each other on the input image are estimated to represent the position of the target object. In this case, the position estimation unit 203 can calculate the position of the target object based on the multiple positions adjacent to each other on the input image, which have been estimated to represent the position of the target object. For example, the position estimation unit 203 can determine the position coordinates of the person by integrating the probability values ​​separated between the multiple elements into one point.

[0061] For example, Figures 8A to 8CAs illustrated, the position estimation unit 203 may obtain the map 801 by applying 3×3 max pooling to the position map 800 output from the neural network. Then, the position estimation unit 203 may compare the position map 800 and the map 801 for each element. In this case, the position estimation unit 203 determines the coordinates of the element whose value on the position map 800 matches the value on the map 801 as the position coordinates of the person (as illustrated in the map 802). The kernel size of max pooling is not limited to 3×3, and may be other sizes such as 5×5. The processing of max pooling may be performed separately, or the processing of max pooling may be performed by adding a max pooling layer next to the output layer of the neural network used by the position estimation unit 203.

[0062] How to integrate multiple probability values ​​is not limited to this method, and the position estimation unit 203 can use other maximum value detection methods such as watershed. In addition, the position estimation unit 203 can use such a method: multiple elements for which probability values ​​are detected are clustered, and the center coordinates of each cluster are used as the position coordinates of the person. Examples of clustering methods include hierarchical clustering and k-means clustering. The center coordinates of the cluster can be a simple average of the coordinates of the elements in the cluster, the coordinates of the element with the highest probability among the elements in the cluster, or a weighted average of the existence probabilities of the coordinates of the elements in the cluster.

[0063] In the position map output from the neural network used by the position estimation unit 203, there may be a slight probability that an element that does not correspond to the position coordinates of a person will act as noise. Figures 8A to 8C In the example of integrating the probability values ​​of adjacent elements into one point, the coordinates of the element with slight noise will be erroneously determined as the position coordinates of a person. Therefore, it can be configured that: the position map output from the neural network is corrected so that the slight probability equal to or less than the threshold value to be regarded as noise is corrected to 0, and the position estimation unit 203 performs the process of integrating the probability values ​​into one point and determining the position coordinates of a person.

[0064] In step S304, the tracking unit 204 tracks the position coordinates of the person based on the position coordinates output by the position estimation unit 203. As described above, by performing the processes of steps S302 and S303 on each input image captured at different times, a list of position coordinates of the person included in the input images captured at different times can be obtained. Fig. 9A position coordinate list 900 for time t1 and a position coordinate list 901 for time t2 later than time t1 are illustrated. The positions of people detected from the input image are shown in the position coordinate lists 900 and 901. By matching (associating) the positions of people between the position coordinate list 900 and the position coordinate list 901, the tracking unit 204 can obtain a motion trajectory 903 of the person from time t1 to time t2.

[0065] The matching is not limited to a specific method. For example, the tracking unit 204 can perform matching in a manner that reduces the sum of costs calculated based on the matching results. Examples of algorithms for matching include Hungarian matching, linear programming, neural networks, etc. Examples of reference costs include distances between people, etc. In addition, costs based on the similarity of image information of people between input images can be used. In addition, more than one type of cost can be used, and a combination of costs of multiple documents can be used. For example, the cost can be calculated based on both the distance between people and the similarity of image information of people between input images. The tracking unit 204 can perform matching in a manner that minimizes the cost.

[0066] Next, we will refer to Figures 10A-10C Describe an example of using the distance between people as a cost. Fig. 10A The input image 1000 illustrated in shows the position coordinates A, B, and C of a person. In addition, the position coordinates b, c, and d are detected from the image of the previous frame of the input image. Note that in this example, the position coordinates b and the position coordinates B are the position coordinates of one person, and the position coordinates c and the position coordinates C are the position coordinates of another person. Therefore, the matching result including the position coordinate combinations (b, B) and (c, C) is corrected.

[0067] On the other hand, in the case where the tracking unit 204 performs matching in a manner that minimizes the sum of the distances between people, there is a possibility of obtaining the position coordinate combinations (c, B) and (b, C). In view of this, the tracking unit 204 can be configured to define the cost in such a way that the sum of the cost corresponding to the distance bB and the cost corresponding to the distance cC is less than the sum of the cost corresponding to the distance cB and the cost corresponding to the distance Cb. In one embodiment, the cost for the distance d is less than half of the cost corresponding to the distance 2d. For example, the cost can be defined using such a distance scale that emphasizes the long distance Cb. Examples of such distance scales include squared Euclidean distance, etc.

[0068] In addition, a person drawn at position coordinate A of input image 1000 exists at position coordinate a outside the image capture range when the image of the previous frame is captured, and the person moves to the image capture range when input image 1000 is captured. Further, the person drawn at position coordinate d of the image of the previous frame moves to position coordinate D outside the image capture range when input image 1000 is captured. In this case, position coordinates a and D to be matched with position coordinates A and d cannot be detected in input image 1000 and the image of the previous frame. In this case, if all position coordinates detected within the image capture range of input image 1000 are matched in a manner that minimizes the sum of distances between people, there is a possibility of erroneously obtaining an incorrect matching result. For example, in Fig. 10A In the illustrated example, position coordinate combinations (A, b), (B, c), and (C, d) will be erroneously obtained.

[0069] In one embodiment, the tracking unit 204 may be configured to match in a manner that does not match isolated position coordinates such as position coordinates A and d. For example, a maximum distance r for matching may be set. In this case, the tracking unit 204 may match in a manner that does not match position coordinates farther than the maximum distance r.

[0070] Fig. 10B and 10C Illustrate the cost matrix and matching results for Hungarian matching to prevent matching of isolated location coordinates. Fig. 10B In the above equation, the B, C, b, and c distributions correspond to Fig. 10A The location coordinates B, C, b, and c in the . The location coordinates A and d are isolated location coordinates whose distances from all other location coordinates are equal to or longer than the maximum distance r. Therefore, in order to prevent the location coordinates A and d from matching with other location coordinates, the cost for the location coordinates A and d is changed from Fig. 10B The tracking unit 204 may generate such a cost matrix based on the squared Euclidean distance, for example. Fig. 10B In the example of , the position coordinate combinations (B, b) and (C, c) highlighted with bold frames are obtained in such a way as to minimize the sum of the costs.

[0071] exist Fig. 10C In another example shown, A, B, C, b, c, and d correspond to Fig. 10A The position coordinates A, B, C, b, c and d in (hereinafter referred to as actual coordinates). Fig. 10CIn the illustrated example, first, the tracking unit 204 generates a cost matrix of the actual coordinates based on, for example, the squared Euclidean distance. Second, the tracking unit 204 adds three virtual ones (i.e., columns corresponding to ×1, ×2, and ×3) as many as the position coordinates A, B, and C, and the position coordinates A, B, and C are the actual coordinates detected from the input image 1000. Further, the tracking unit 204 adds three virtual ones (i.e., rows corresponding to X1, X2, and X3) as many as the position coordinates b, c, and d, and the position coordinates b, c, and d are the actual coordinates detected from the image of the previous frame. Then, the tracking unit 204 sets the cost between a virtual one and any actual coordinate by using a value r1 greater than the maximum distance r, and sets the cost between actual coordinates whose distance therebetween is longer than the maximum distance r by using a value r2 greater than the ratio r1. That is, r < r1 < r2. The tracking unit 204 can generate such a cost matrix based on, for example, the squared Euclidean distance. By performing matching by using such a cost matrix, an isolated actual coordinate will not be matched with other actual coordinates but with a virtual one. In addition, redundant virtual ones will be matched with each other. In Fig. 10C FIG. 211, the result of the matching performed by using the cost matrix is indicated by a thick frame. The tracking unit 204 can obtain the matching result by excluding the matching results of these virtual ones.

[0072] In yet another example, the tracking unit 204 can be configured to group position coordinates whose distance therebetween is within the maximum distance r with each other before performing the matching. In this case, the tracking unit 204 can perform grouped matching so that the position coordinates in the same group are matched with each other. For example, the tracking unit 204 can separately generate a cost matrix as illustrated in Fig. 10B FIG. 214 or 10C for each group, and perform the matching by using the cost matrix. By such grouping, isolated position coordinates A and d as illustrated in Fig. 10A FIG. 215 can be excluded. By such a method, the processing speed can be improved in the case where the number of people (i.e., the number of detected position coordinates) is large. The grouping is not limited to a particular method, and examples of the grouping method include a method of searching for unlinked independent figures by using a cost matrix or an adjacent figure including an isolated position coordinate. In addition, clustering methods such as hierarchical clustering and k-means clustering can be used.

[0073] As described above, the tracking unit 204 can match the position coordinates of a person detected from the input image with the position coordinates of a person detected from the image captured at a previous time. On the other hand, the tracking unit 204 can perform the matching by referring to the matching result obtained previously.

[0074] In one embodiment, the tracking unit 204 is configured to match the position coordinates at time t2 with the position coordinates at time t3 later than time t2 by referring to the matching result of the position coordinates at time t1 and the position coordinates at time t2. Fig.11 In the example of FIG. 1 , the tracking unit 204 performs matching to match the matching result 1100 of matching the position coordinates between time t1 and time t2 and the position coordinate list 1101 for time t3. Through such processing, the motion trajectory 1103 of the person from time t1 to time t2 and then to time t3 can be obtained. By repeating such processing for the position coordinates at the following time, the motion trajectory of the same person over a long period of time can be obtained.

[0075] The tracking unit 204 can perform matching by performing processing in such a manner that the moving direction or moving speed of the person at time t2 is used in addition to the position coordinates at time t2 and the position coordinates at time t3. That is, the tracking unit 204 performs matching by preferentially matching the position coordinates at time t2 and the position coordinates at time t3 that do not contradict the moving direction or moving speed of the person at time t2.

[0076] For example, the tracking unit 204 may be configured to predict the position of the target object in the input image at time t3 based at least on the position of the target object in the input image at time t2. Fig. 12A = represents the movement of the position coordinates of the person between time t2 and time t3. In this example, the position coordinates b and c are detected at time t2, and the position coordinates B and C are detected at time t3. In such a case, there is a possibility of obtaining a wrong match (B, c) because the person is too close. The tracking unit 204 uses the predicted position coordinates b* and c* of the person at time t3 (such as Fig. 12B As illustrated), matching can be performed to improve matching accuracy.

[0077] The tracking unit 204 may calculate the predicted position coordinates b* and c* of the person at time t3 from the motion trajectory at the time before time t3. For example, the tracking unit 204 may calculate the predicted position coordinates b* and c* of the person by using the motion trajectory obtained based on the matching results at and before time t2. Fig. 12BIn the illustrated example, the position coordinate b' at time t1 matches the position coordinate b at time t2, and the position coordinate c' at time t1 matches the position coordinate c at time t2. In this example, the tracking unit 204 can calculate the predicted position coordinate 1202 (b*, c*) at time t3 from the motion trajectory from time t1 to time t2. The calculation of the predicted position coordinate is not limited to a particular method. For example, the tracking unit 204 can calculate the predicted position coordinate at time t3 by using the position coordinates at time t1 and at time t2 according to linear extrapolation.

[0078] According to the above method, the tracking unit 204 performs matching 1204 to match the predicted position coordinates 1202 (b*, c*) and the position coordinates 1203 (B, C) detected at time t3. In this way, instead of the position coordinates b and c detected at time t2, the tracking unit 204 matches the predicted position coordinates b* and c* at time t3 with the position coordinates B and C detected at time t3. Since it is more desirable that the predicted position coordinates b* and c* are closer to the position coordinates B and C than the position coordinates b and c, according to this method, the matching accuracy can be improved. Based on the matching result of matching the predicted position coordinates b* and c* with the position coordinates B and C, the tracking unit 204 can match the position coordinates B and C with the position coordinates b and c. In this way, a motion trajectory 1205 from time t1 to time t2 and then to time t3 can be obtained.

[0079] Note that the calculation of the predicted position coordinates at time t3 may use position coordinates at multiple times prior to time t3 instead of using position coordinates at two times (time t1 and time t2).

[0080] For some reason, there is a possibility that the input image to be provided to the acquisition unit 201 is lost, the density distribution of the person by the density estimation unit 202 is lost, or the position estimation unit 203 cannot detect the position coordinates of the person. Fig.13A In the example shown, the person is not detected at position 1304 at time t3, but is re-detected at time t4. In the case where the person's motion trajectory is interrupted, it will be erroneously determined that the person has not crossed the lines 1301 and 1302 representing the measurement area. To avoid such an event, Fig. 13B As shown, the tracking unit 204 can compensate for the interrupted motion trajectory by using the predicted position coordinates 1305 at time t3. In this way, even if the motion trajectory is interrupted, it can be determined that the person has passed through the measurement area. The predicted position coordinates 1305 at time t3 can be calculated as described above.

[0081] For example, the tracking unit 204 may predict that the person will be present at the predicted position coordinates 1305 at time t3, and based on the prediction, generate a motion trajectory of the person that passes through the predicted position coordinates 1305. Note that in the case where the motion trajectory thus generated does not match the position coordinates detected after time t3, the tracking unit 204 excludes the predicted position coordinates 1305 from the motion trajectory.

[0082] In the case where the motion trajectory of a person is interrupted, the estimation of the position coordinates of the person is not limited to the above method. For example, the tracking unit 204 can estimate the position coordinates of the person by using a target object tracking method such as template matching. In addition, even if the position coordinates of the person cannot be detected in two or more frames, the motion trajectory of the person can be tracked by repeatedly calculating the predicted position coordinates.

[0083] In addition, the tracking unit 204 may be configured to match the position coordinates of the person at time t2 and the position coordinates of the person at time t4. In this case, the tracking unit 204 may calculate the predicted position coordinates of the person at time t4 from the position coordinates of the person at time t2 (and at a time prior to time t2 such as time t1). As described above, by matching these predicted position coordinates with the position coordinates of the person detected at time t4, the motion trajectory of the person up to time t2 may be matched with the position coordinates of the person detected at time t4. In this case, the motion trajectory of the person from the position coordinates detected at time t2 to the position coordinates detected at time t4 may be calculated. In this case, the tracking unit 204 may further calculate the predicted position coordinates of the person at time t3. In addition, the tracking unit 204 may calculate the motion trajectory of the person from the position coordinates detected at time t2 through the predicted position coordinates of the person at time t3 to the position coordinates detected at time t4.

[0084] Please note that Fig. 9 In the illustrated example, the position coordinate lists 900 and 901 for matching do not specify the persons corresponding to the respective positions. However, the persons corresponding to the respective positions may be specified by image recognition processing, etc. In this case, the tracking unit 204 may perform matching by matching the position coordinates of the persons at different times with reference to the information for identifying the persons.

[0085] In step S305, the measuring unit 205 measures the flow of people based on the motion track of the people output from the tracking unit 204. For example, the measuring unit 205 can measure the flow of people by determining whether the motion track of the people output from the tracking unit 204 passes through a preset measurement area. Please note that the setting method of the measurement area is as described above. Figure 4A and 4BAs described above, the measurement areas 401 and 403 may be set when the area from which the partial image is to be extracted is set in step S301.

[0086] The following will refer to Figures 14A-14B , describes an example of a method for estimating human flow when the measurement area is linear. Fig.14A An input image 1400 at time t1 and a line 1401 serving as a measurement area are illustrated. Also, position coordinates 1403 of a person 1402 and position coordinates 1405 of a person 1404 are detected from the input image 1400. Similarly, Fig. 14B An input image 1406 at time t2 is illustrated. Position coordinates 1407 of a person 1402 and a motion track 1409 from position coordinates 1403 to position coordinates 1407 are detected from the input image 1406. Also, position coordinates 1408 of a person 1404 and a motion track 1410 from position coordinates 1405 to position coordinates 1408 are detected from the input image 1406.

[0087] exist Fig. 14B In the illustrated example, the measurement unit 205 may determine that the motion trajectory 1409 crosses the line 1401, and thus may determine that the person 1402 crosses the line 1401. Similarly, the measurement unit 205 may determine that the motion trajectory 1410 also crosses the line 1401, and thus may determine that the person 1404 also crosses the line 1401. In this case, the measurement unit 205 may determine that the direction in which the person 1404 crosses the line 1401 is opposite to the direction in which the person 1402 crosses the line 1401. For example, the intersection determination of the motion trajectory and the line may be implemented by a geometric method such as the intersection determination of one line and another line.

[0088] In one embodiment, when a person moves from one area separated by the measurement area to another area, the measurement unit 205 determines that the person has passed through the measurement area. On the other hand, when the person exists in the measurement area, the measurement unit 205 does not determine that the person has passed through the measurement area. That is, the measurement unit 205 measures the number of people who move from one area to another area across the measurement area, but the number of people measured does not include the number of people who move from one area to the measurement area but have not yet moved to another area.

[0089] For example, Fig.15The line 1500 used as the measurement area and the motion trajectory 1501 of the person are illustrated. In this example, the measurement unit 205 can determine that the person crosses the line 1500 from bottom to top at three points a, d, and f, and can determine that the person crosses the line 1500 from top to bottom at two points c and e. On the other hand, the measurement unit 205 can determine that the person does not cross the line 1500 at points b and g (where the person is exactly on the line 1500). As described above, the line 1500 may be a line with a large thickness (that is, a rectangular area), and in this case, the measurement unit 205 does not determine that the person crosses the line 1500 while the person is still on the line 1500.

[0090] By determining the movement trajectory of each person in this way, the measurement unit 205 can measure the flow of people. Fig.15 In the illustrated example, the measuring unit 205 may measure the cumulative number of people crossing the line 1500 from bottom to top and the cumulative number of people crossing the line 1500 from top to bottom as the flow of people in the respective directions. Fig.15 As described, by determining that a person has not passed through the measurement area while the person is still on the measurement area, repeated counting of the accumulated number of passages caused by slight movements of the person around the measurement area can be avoided.

[0091] The measurement unit 205 may be configured to calculate the difference between the cumulative number of people crossing the measurement area in their respective movement directions as the flow of people. More specifically, the measurement unit 205 may calculate the difference between the number of people crossing the measurement area from one area to another area and the number of people crossing the measurement area from the other area to the one area. Fig.15 In the case of the person represented by the motion trajectory 1501, the cumulative number of crossing the line 1500 from bottom to top is 3 times, and the cumulative number of crossing the line 1500 from top to bottom is 2 times. The measurement unit 205 can determine that a person passes the line 1500 from bottom to top by calculating the difference between the cumulative crossing numbers in each motion direction.

[0092] As described above, the shape of the measurement area is not particularly limited. For example, in the case where the measurement area is a closed curve surrounding a certain area, the measurement unit 205 can measure the flow of people entering the measurement area and the flow of people leaving the measurement area.

[0093] According to the present embodiment, even if a target object in an image overlaps with another target object, the density distribution can still be estimated and the position of the target object can be estimated from the density distribution. Therefore, compared with the configuration of directly detecting the target object from the image, for example, as in Japanese Patent Laid-Open No. 2007-201556, the configuration according to the present embodiment can improve the accuracy of the position estimation of the target object in a crowded scene. In addition, since the position map representing the position estimation result is highly sparse (for example, the value of most positions is 0), it may not be easy to use a neural network to directly estimate the position of the target object from the image. On the other hand, according to the present embodiment, the position estimation result can be obtained by solving a two-class classification problem for each position on the density map, thereby making position estimation easier. In one embodiment, in order to achieve these effects, the acquisition unit 201, the density estimation unit 202 and the position estimation unit 203 perform the processing of steps S301 to S303, and the processing of steps S304 and S305 performed by the tracking unit 204 and the measurement unit 205 can be selectively omitted.

[0094] In addition, according to the present embodiment, the position coordinates of the target object are tracked between frames, so that the flow of the target object can be accurately measured. In one embodiment, in order to achieve such an effect, the tracking unit 204 and the measuring unit 205 perform the processing of steps S304 and S305, and the processing of steps S301 to S303 performed by the obtaining unit 201, the density estimation unit 202 and the position estimation unit 203 can be selectively omitted.

[0095] Other embodiments

[0096] Some embodiments may also be implemented by reading out and executing computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more completely referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more of the above-mentioned (one or more) embodiments and / or a system or device including one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more of the above-mentioned (one or more) embodiments, and by a method performed by a computer of the system or device, for example, by reading out and executing computer executable instructions from a storage medium to perform the functions of one or more of the above-mentioned (one or more) embodiments and / or controlling one or more circuits to perform the functions of one or more of the above-mentioned (one or more) embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read out and execute computer executable instructions. Computer executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage device of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD)) TM ), one or more of flash memory devices, memory cards, etc.

[0097] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0098] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An image processing device, comprising: An obtaining unit, configured to obtain a motion image including a plurality of images; a density estimation unit for estimating a density map using a first neural network, the density map representing density distribution of a plurality of objects at each position on an image in the plurality of images; a position estimation unit, configured to estimate a plurality of position coordinates of respective representative points representing the plurality of objects in the image based on the density map; as well as a measuring unit, configured to measure the flows of the plurality of objects based on the estimation results of the plurality of images by the position estimation unit, in, The position estimation unit is configured to perform two-class classification on each position of the density map by using a second neural network different from the first neural network to generate a position map indicating respective probabilities of whether each position on the density map represents each position of the representative point, wherein the first neural network and the second neural network are trained using different types of training data from each other, The position estimation unit is configured to estimate the position coordinates of the respective representative points by probability of adjacent positions to each other.

2. The image processing apparatus according to claim 1, wherein: The density map represents an area in the image where the object is estimated to exist.

3. The image processing apparatus according to claim 1, wherein: The obtaining unit is further used to extract one or more partial images belonging to a part of the image from each of the multiple images, and The density estimation unit is further configured to estimate a density map of the object in each of the one or more partial images.

4. The image processing apparatus according to claim 3, wherein: The obtaining unit is further configured to extract the partial image from the image in such a manner that the partial image includes a measurement region in which flows of the plurality of objects are to be measured.

5. The image processing apparatus according to claim 4, wherein: The obtaining unit is further configured to obtain a designation of the measurement region via a user, and to set a region from which the partial image is to be extracted based on the designation.

6. The image processing apparatus according to claim 3, wherein: The obtaining unit is further configured to divide the image into a plurality of regions, and to extract the partial image from each of one or more regions selected from the plurality of regions and a blank region surrounding the region.

7. The image processing apparatus according to any one of claims 1 to 6, wherein: The measuring unit is further configured to measure the number of objects moving from a first area to a second area, the first area and the second area being separated by a measuring area in which the flow of the plurality of objects is to be measured.

8. The image processing apparatus according to claim 7, wherein: The measuring unit is further configured to measure the number of objects that move from the first area across the measuring area to the second area, and exclude the number of objects that move from the first area to the measuring area instead of to the second area from the measured number of objects.

9. The image processing apparatus according to claim 7, wherein: The measuring unit is further configured to calculate a difference between the number of objects moved from the first area to the second area and the number of objects moved from the second area to the first area.

10. The image processing device according to any one of claims 1 to 6, further comprising an associating unit for associating the position coordinates of an object in a first image at a first time in the multiple images with the position coordinates of an object in a second image at a second time in the multiple images.

11. The image processing apparatus according to claim 10, wherein: The association unit is further used for: predicting, based at least on the position coordinates of the object in the second image, the position coordinates of the object in the third image at a third time later than the second time; as well as The position coordinates of the object in the second image are associated with the position coordinates of the object in the third image based on the predicted position coordinates of the object in the third image at the third time and the position coordinates of the object in the third image.

12. The image processing apparatus according to any one of claims 1 to 6, wherein: The first neural network is trained using training data representing a density distribution of an image.

13. The image processing apparatus according to any one of claims 1 to 6, wherein: The second neural network is trained using the training data.

14. An image processing method, comprising: obtaining a motion image including a plurality of images; estimating a density map using a first neural network, the density map representing a density distribution of a plurality of objects at each location on an image in the plurality of images; estimating a plurality of position coordinates of respective representative points of the plurality of objects in the image based on the density map, wherein each position of the density map is classified into two categories by using a second neural network different from the first neural network to generate a position map representing respective probabilities of whether each position on the density map represents each position of the representative points, wherein the first neural network and the second neural network are trained using training data of different types from each other, and the position coordinates of the respective representative points are estimated by the probability of adjacent positions to each other; as well as The flows of the plurality of objects are measured based on the estimation results of the plurality of images. 15 . A computer-readable medium storing a program which, when executed by a computer, causes the computer to perform the image processing method according to claim 14 .

Images