Image processing device, image processing method, and program

Abstract

To reduce the processing load in measuring the flow of an object to be measured.SOLUTION: A detection line indicating the position for measuring the flow of an object to be measured is set in a moving image. Multiple partial images set in the vicinity of the detection line are extracted from respective images in the moving image. The partial images are used to measure the flow of the object to be measured passing through the detection line.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program, and more particularly to a technique for analyzing moving images captured by a video camera, a security camera, or the like. [Background technology]

[0002] In recent years, devices have been proposed that analyze the flow of objects to be measured in a shooting area, such as people flow, from moving images captured by a camera, etc., and more specifically, analyze the amount and direction of people flow. Patent Document 1 discloses a congestion estimation device that divides an image into multiple patches and determines whether people are moving or staying in each patch. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-110152 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method described in Patent Document 1 analyzes people flow using the entire image, so if a high-resolution image with a large number of pixels is used, the time required to analyze people flow increases.

[0005] An object of the present invention is to reduce the processing load when measuring the flow of a measurement target. [Means for solving the problem]

[0006] An image processing apparatus according to an embodiment of the present invention includes the following configuration. A setting means for setting a detection line on an input image; Based on the position of the detection line set in the input image by the setting means, , set to include the detection line One or more sub-images A pattern of partial images showing each arrangement of the setThe extraction means extracts the partial image by executing a plurality of extraction methods each having a different pattern of the partial image. Extract multiple patterns from a partial image The extraction means; A selection means for selecting one or more partial images of a plurality of patterns from among the one or more partial images of the plurality of patterns extracted by executing the plurality of extraction methods; One or more portions of the pattern selected by said selection means image a measuring means for measuring at least one of a flow rate and a passing direction of a predetermined object passing through the detection line set by the setting means using the has. Effect of the Invention

[0007] The processing load when measuring the flow of the measurement target can be reduced. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram showing an example of a hardware configuration of an image processing apparatus. [Diagram 2] FIG. 2 is a diagram showing an example of the functional configuration of an image processing apparatus. [Diagram 3] FIG. 4 is a diagram showing an example of a processing flow of the image processing apparatus. [Figure 4] 11A and 11B are diagrams showing an example of a method for extracting partial images from divided regions that intersect with a detection line. [Diagram 5] 11A to 11C are diagrams showing an example of a method for extracting partial images of various sizes. [Figure 6] 11A and 11B are diagrams showing an example of a method for extracting a partial image along a detection line. [Figure 7] 11A and 11B are diagrams showing an example of a method for arranging an extraction region on a detection line. [Figure 8] 11A and 11B are diagrams showing an example of a method for arranging an extraction region on a rasterized detection line. [Figure 9] 11A and 11B are diagrams showing an example of a method for arranging a plurality of extraction regions on a detection line. [Figure 10] 11A and 11B are diagrams showing an example of a method for arranging an extraction region on an extended detection line. [Figure 11]FIG. 13 illustrates an example of a method for extracting sub-images along various detection lines. [Figure 12] 11A and 11B are diagrams showing the relationship between the arrangement order of extraction regions and extracted partial images. [Figure 13] FIG. 13 is a diagram showing the relationship between an extraction region to be initially placed and a partial image to be extracted. [Figure 14] FIG. 11 is a diagram showing an example of a search method for extracting a partial image. [Figure 15] 11A and 11B are diagrams showing an example of a method for evaluating a partial image extraction method. [Figure 16] 4A and 4B are diagrams for explaining the center region and margin region of an extraction region. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] 1 shows an example of a hardware configuration of an image processing device 100 according to an embodiment of the present invention. The image processing device 100 includes a control unit 11, a storage unit 12, a calculation unit 13, an input unit 14, an output unit 15, and an I / F unit 16.

[0011] The control unit 11 is a device that controls the entire image processing device 100. The memory unit 12 holds programs and data necessary for the operation of the control unit 11. The calculation unit 13 executes necessary calculation processing based on the control from the control unit 11. For example, the calculation unit 13 may perform neural network calculations described later. The input unit 14 is a human interface device or the like, and acquires inputs by user operations. The output unit 15 is a display or the like, and presents the processing results generated by the image processing device 100 to the user.

[0012] The I / F unit 16 is a wired interface such as a universal serial bus, Ethernet (registered trademark), or an optical cable, or a wireless interface such as Wi-Fi or Bluetooth (registered trademark). Other devices can be connected to the image processing device 100 via the I / F unit 16. For example, an imaging device such as a camera can be connected to the I / F unit 16, and the image processing device 100 can acquire an image via the I / F unit 16. As another example, the image processing device 100 can transmit a processing result to the outside via the I / F unit 16. As a further example, the image processing device 100 can acquire a program or data required for operation via the I / F unit 16.

[0013] The functions of the image processing device 100 described below can be realized, for example, by a processor (e.g., control unit 11) operating in accordance with a program on a memory (e.g., storage unit 12). The storage unit 12 or other storage media can store such programs. However, at least some of the functions of the image processing device 100 described below may be realized by dedicated hardware. Furthermore, an image processing device according to one embodiment of the present invention may be composed of multiple devices connected via a network, for example.

[0014] The image processing device 100 may be a general computer. On the other hand, the image processing device 100 may be an imaging device such as a digital camera or a network camera. The image processing device 100 can acquire input by a user's operation from an information processing device such as a computer or a smartphone connected via the I / F unit 16 and a network such as the Internet. At this time, the image processing device 100 may generate a user interface for requesting user input and transmit it to such an information processing device for display. The image processing device 100 can output the processing results generated by the image processing device 100, such as the extraction result of a partial image or the measurement result of the flow of the measurement target, to such an information processing device.

[0015] FIG. 2 shows an example of the functional configuration of an image processing device 100 according to an embodiment of the present invention. The image processing device 100 has, as its functional configuration, an acquisition unit 201, a setting unit 202, an extraction unit 203, and a measurement unit 204. The image processing device according to this embodiment can measure the flow of a measurement target based on a moving image. In recent years, the performance of cameras has improved, making it easy to obtain high-resolution images, but the size of such high-resolution images tends to be large, and it takes a long time to measure the flow of a measurement target. To address this issue, high-speed processing is achieved by setting multiple partial images in the vicinity of the position where the flow rate measurement is performed.

[0016] The acquisition unit 201 acquires a moving image including a plurality of images. The moving image acquired by the acquisition unit 201 is subjected to an analysis for measuring the flow of the measurement object. The flow of the measurement object may be the flow of the measurement object on the image, or may be the flow of the measurement object in real space estimated by image analysis. The object of analysis (measurement object) is not particularly limited, and examples thereof include a person, a vehicle such as a bicycle or a motorbike, a vehicle such as a car or a truck, or an animal such as livestock.

[0017] The moving image may be, for example, streaming, a moving image file, a series of image files saved for each frame, or a moving image saved in a medium, which includes a plurality of frame images. Each of the plurality of images may be captured at a different time by an imaging device at the same location, for example. The acquisition unit 201 may acquire the moving image from a solid-state imaging element such as a CMOS sensor or a CCD sensor, or an imaging device such as a camera equipped with such a solid-state imaging element. The acquisition unit 201 may also acquire the moving image data from a storage device such as a hard disk or an SSD, or a recording medium, etc.

[0018] The setting unit 202 sets a detection line indicating a position for measuring the flow of the measurement object in the moving image. In this embodiment, the flow rate of the measurement object passing through the set detection line, or the direction of the flow of the measurement object, etc. are measured. The flow rate may be the total number of the measurement objects passing through the set detection line, or the number per a given time period.

[0019] The shape of the detection line is not particularly limited, and may be, for example, any broken line, curve, polygon, circle, or ellipse, or a figure consisting of any closed curve. The detection line may be one or more. For example, multiple detection lines that are not connected to each other may be set, or multiple detection lines that intersect with each other may be set. When multiple detection lines are set, the flow of the measurement target passing through each detection line can be measured.

[0020] Such a detection line may be set based on an image acquired by the acquisition unit 201. For example, the setting unit 202 may acquire a detection line set by a user by the user operating a human interface device or the like connected to the input unit 14 while referring to an image displayed on the output unit 15. The setting unit 202 may also automatically set a detection line based on an area designated by the user. As a specific example, the setting unit 202 may set a line passing through the center of one area designated by the user or a line passing between two areas designated by the user as the detection line. Furthermore, the setting unit 202 may set the detection line according to an operation via the I / F unit 16 instead of a human interface device connected to the input unit 14.

[0021] As yet another alternative method, the setting unit 202 may set the detection line according to a setting value that is stored in advance in the storage unit 12 or another device connected via the I / F unit 16 .

[0022] The setting unit 202 may display the detection line thus set to the user. For example, the setting unit 202 can output information indicating the position of the detection line to the output unit 15 or another device connected via the I / F unit 16. For example, the detection line can be displayed superimposed on the image acquired by the acquisition unit 201.

[0023] The extraction unit 203 extracts a plurality of partial images set in the vicinity of the detection line set by the setting unit 202 from each of the plurality of images in the moving image acquired by the acquisition unit 201. The partial images thus extracted are used for measuring the flow. A specific extraction method will be described later.

[0024] The extraction unit 203 may display the result of the partial image extraction process, for example, the extracted partial image or the position of the partial image in the image acquired by the acquisition unit 201, to the user. Furthermore, when a detection line is set by the user operating a human interface device or the like connected to the input unit 14, the extraction unit 203 may display the result of the partial image extraction process based on the detection line to the user by cooperating with the setting unit 202. The setting unit 202 can output the result of the extraction process to the output unit 15 or another device connected via the I / F unit 16. Examples of display of the detection line and the result of the partial image extraction process are shown in Figs. 4(A)-(B), 5(A)-(D), and 6(A)-(B).

[0025] The measuring unit 204 measures the flow of the measurement target passing through the detection line by using the partial image. That is, the measuring unit 204 can measure the flow of the measurement target based on the detection line set by the setting unit 202 and the partial image extracted by the extraction unit 203. For example, the measuring unit 204 can measure the flow rate of the measurement target that moves from one area separated by the detection line to the other area crossing the detection line. Also, when the detection line is a closed curve surrounding a predetermined area, the measuring unit 204 can measure the flow rate of the measurement target flowing into this area and the flow rate of the measurement target flowing out of this area.

[0026] Various methods can be used to measure the flow. For example, a method of detecting and tracking a person to be measured, or a method of directly calculating the flow rate by estimating the position, moving direction, moving speed, etc. of the person to be measured can be used. Examples of algorithms for implementing such measurement methods include a matching method, a method using optical flow, a method using machine learning, and a method using a neural network. Furthermore, a combination of these methods can also be used.

[0027] For measuring the flow, a single partial image may be used, or multiple partial images may be used simultaneously. When multiple partial images are used, partial images taken at the same time may be used, or partial images taken at different times may be used.

[0028] Specific processing methods by the measurement unit 204 include, for example, the following methods. First, the measurement unit 204 estimates the position of the measurement target around the detection line at time t1 by inputting each partial image at time t1 into a neural network. Similarly, the measurement unit 204 estimates the position of the measurement target around the detection line at time t2 by inputting each partial image at time t2 into this neural network. This neural network can be trained to estimate the position of the measurement target (e.g., a person's head) in an image from the image. In addition, as another method for improving the estimation accuracy, a neural network trained to estimate the density distribution of the measurement target in an image from the image and a neural network trained to estimate the position of the measurement target from the density distribution may be used in combination. With this method, the measurement unit 204 can estimate the position of the measurement target in each of the different regions independently using partial images extracted from the region.

[0029] Next, the measurement unit 204 estimates the trajectory of the measurement object from time t1 to time t2 by matching the estimated position of the measurement object at time t1 with the estimated position of the measurement object at time t2. As a matching method, a method of minimizing a cost according to the distance between the measurement objects to be matched can be used, for example, the Hungarian matching method. When the trajectory estimated in this way intersects with the detection line, it can be determined that one measurement object has passed the detection line. Such matching processing and trajectory estimation may be performed simultaneously based on the positions of the measurement objects detected from each partial image.

[0030] However, the method of measuring the flow is not limited to the above method. The measuring unit 204 may estimate the trajectory of the measurement object in each region or measure the flow of the measurement object using partial images extracted from each different region independently. For example, the measuring unit 204 may not only estimate the position of the measurement object for each partial image, but also estimate the trajectory of the measurement object for each partial image. Also, partial images at the same position at time t1 and time t2 may be input to a neural network to estimate the position, moving direction, moving speed, etc. of the measurement object, thereby estimating the flow of the measurement object.

[0031] An example of processing by the image processing device 100 according to this embodiment will be described with reference to Fig. 3. An example of measuring the flow rate of people (people flow measurement) will be described below, but other measurement objects may be used, and other indices of flow, such as flow direction, may be measured.

[0032] In step S301, the acquisition unit 201 acquires a moving image as described above. Note that the acquisition unit 201 may successively acquire frame images constituting the moving image from an imaging device or another device such as a storage device.

[0033] In step S302, the setting unit 202 sets the detection line as described above.

[0034] In step S303, the extraction unit 203 extracts a partial image from the image acquired by the acquisition unit 201, based on the detection line set by the setting unit 202. A specific extraction method will be described later.

[0035] In step S304, the measurement unit 204 measures the flow of people as described above, based on the detection line set by the setting unit 202 and the partial image extracted by the extraction unit 203.

[0036] (How to extract a partial image) Hereinafter, a method in which the extraction unit 203 extracts a partial image in step S303 will be described in detail. Hereinafter, one image in the moving image (for example, one frame of the moving image) acquired by the acquisition unit 201 is referred to as an input image. The method of extracting a partial image from an input image is not limited to a specific method. For example, the extraction unit 203 extracts an area near the detection line as a partial image, and the measurement unit 204 performs measurement based on this partial image, thereby reducing the processing load compared to when the measurement unit 204 performs measurement based on the entire input image. In one embodiment, the extraction unit 203 extracts a partial image that is a part of the input image so that the entire detection line is included. However, it is not necessary that all partial images include the detection line. Depending on the measurement method used by the measurement unit 204, the measurement accuracy of the flow rate may be improved by further using a partial image extracted from an area close to the detection line.

[0037] Also, as described above, at least a part of the measurement process by the measurement unit 204 (e.g., person position estimation or flow rate estimation process) may be performed for each of the multiple partial images. For this reason, multiple partial images may be extracted from the input image. For example, the extraction unit 203 can extract a number of partial images based on the position of the detection line from one of the multiple images in the moving image. That is, in one embodiment, a different number of partial images are extracted depending on the set detection line. In this case, the extraction unit 203 can extract multiple partial images, each of which is a part of the input image, such that each part of the detection line is included in one of the multiple partial images.

[0038] The measurement unit 204 may perform at least a part of the measurement process (e.g., person position estimation or flow rate estimation process) on an image of a predetermined size. For example, when the measurement unit 204 estimates the position of a person using a neural network, a partial image of a certain size can be input to the neural network. Therefore, in one embodiment, the extraction unit 203 extracts a partial image of a size according to the setting. As described later, the size of the partial image may differ depending on the position on the input image from which the partial image is extracted.

[0039] On the other hand, by reducing the number of partial images to be extracted, it is possible to further reduce the load of the measurement process by the measurement unit 204. For example, by reducing overlapping parts between partial images, it is possible to reduce the number of partial images while ensuring that each part of the detection line is included in one of the partial images.

[0040] Furthermore, as described above, the setting unit 202 may set detection lines of various shapes, multiple detection lines, or intersecting detection lines, and it is desirable for the extraction unit 203 to extract partial images corresponding to such various detection lines.

[0041] Although the method by which the extraction unit 203 extracts partial images does not need to satisfy all of the above requirements, several methods for extracting partial images are described below. In the following methods, the extraction unit 203 sets one or more extraction regions in the image region of the moving image based on the position of the detection line. Then, the extraction unit 203 can extract, as partial images, each part included in each of the extraction regions in one image of the multiple images.

[0042] At this time, the extraction unit 203 sets the extraction area so that a set of one or more extraction areas includes the detection line. By setting the extraction area in this manner, it becomes possible to detect a measurement target passing through the detection line using any of the partial images. On the other hand, it is not essential that the set of one or more extraction areas includes the entire detection line. For example, there may be a portion on the detection line where a measurement target such as a person is not detected, such as an obstruction or an obstacle. The extraction unit 203 can detect the portion where the measurement target is not detected in this manner. In this case, it is not necessary to set the extraction area in the portion where the measurement target is not detected. For this reason, the extraction unit 203 may set the extraction area so that the set of one or more extraction areas includes a portion of the detection line excluding the portion set as the portion where the measurement target is not detected.

[0043] In the following, a case where the extraction region is rectangular will be described, but the shape of the extraction region is not particularly limited. In addition, each of the multiple extraction regions may have a different shape.

[0044] 4(A)-(B) show a method for extracting one or more partial images, each having a certain size, based on a detection line. In these examples, a plurality of regions are set in an image region of a moving image. For example, a plurality of regions can be set in an input image or a part thereof. A set of the plurality of regions includes the detection line. Then, a region among the plurality of regions where the detection line intersects is set as an extraction region, and a partial image is extracted from this extraction region.

[0045] 4A, the input image 400 is divided into grid-like regions of a certain size. In this case, the extraction unit 203 can extract a partial image from each of the grids (i.e., extraction regions) through which the detection line passes.

[0046] In Fig. 4(B), the extraction unit 203 sets a bounding box 403 that contains the detection line 402. The bounding box 403 in the example of Fig. 4(B) is the smallest rectangle that contains the detection line 402 and matches the input image 400 in the vertical and horizontal directions. Also, in the example of Fig. 4(B), the bounding box 403 is divided into a grid of a fixed size. In this case, the extraction unit 203 can extract partial images from each of the grids (extraction regions) through which the detection line passes.

[0047] According to the methods of Fig. 4(A)-(B), partial images can be extracted in accordance with various detection lines. Although the input image 400 or the bounding box 403 is divided into a square lattice in Fig. 4(A)-(B), the shape of the regions is not particularly limited. Furthermore, the size of each region may be set in advance, and the number of regions (length x width) may be set in advance. Furthermore, the size or shape of each region may differ from each other. Such a configuration can also be adopted in other examples such as that shown in Fig. 5.

[0048] In the examples of FIGS. 4(A)-(B), each position of the input image 400 or the bounding box 403 is included in one of a plurality of grids. That is, each grid is tiled so as to include the input image 400 or the bounding box 403. On the other hand, each grid may include an area outside the input image 400 or the bounding box 403, or the grids may overlap each other. Thus, in one embodiment, a plurality of areas (e.g., grid-like areas) are set so that a set of the multiple areas includes the bounding box 403 and becomes part of the image area of ​​the moving image, and an extraction area is selected from these areas.

[0049] Also, it is not necessary to divide the entire input image 400 or the bounding box 403 into a plurality of regions. As an example, only a portion of the input image 400 or the bounding box 403 necessary for flow rate analysis, for example, a portion other than an area where an obstacle that a person cannot pass through exists, may be divided into a plurality of regions. Such a configuration can also be adopted in another example as shown in FIG.

[0050] 5(A) to (I) show an example of extracting partial images of different sizes according to their positions in an input image. For example, the size of an extraction region from which a partial image is extracted can be set according to the position on the image where the extraction region is set, or the size of a measurement target on the image in the extraction region.

[0051] As a specific example, the size of a person (measurement target) may vary depending on the position on an input image captured by a camera or the like. In this case, the size of each extraction region can be determined so that the ratio between the size of the extraction region and the size of the person (measurement target) on the image in the extraction region is approximately constant. In this case, each partial image is resized to an image of a constant size, and the measurement unit 204 can measure the flow of people based on the resized image. With this configuration, the size of the person included in each image used by the measurement unit 204 for measurement can be made approximately constant, thereby improving the measurement accuracy. This configuration can be used, for example, when the measurement unit 204 measures the flow rate by inputting an image of a constant size into a neural network.

[0052] The size of a person may be, for example, the size of a part including the head and shoulders of a person, the size of the person's head, or the size of the whole body of a person. The size of the extraction region may be determined according to the size of a person appearing at the position of the extraction region, in which case the ratio between the size of the extraction region and the size of the person can be made constant regardless of the actual size of the person. The size of a person at a specific position can be determined using the size of a person near this position, detected, for example, by person recognition processing of an input image.

[0053] Also, people appear smaller in positions where subjects are farther from the camera (e.g., at the top of the input image), and larger in positions where subjects are closer to the camera (e.g., at the bottom of the input image). Therefore, the size of the extraction region may be determined according to the position of the extraction region, and may be smaller at the top of the input image and larger at the bottom.

[0054] FIG. 5(A) shows an example in which an input image 500 is divided into rectangular regions of various sizes, and partial images are extracted from regions intersecting with a detection line 501. FIG. 5(B) shows an example in which a bounding box 503 including a detection line 502 set in the same manner as in FIG. 4(B) is divided into rectangular regions of various sizes, and partial images are extracted from regions intersecting with the detection line 502. In FIGS. 5(A) and 5(B), a plurality of regions are set so that the ratio of the size of the region to the size of the person is approximately constant. The positions and sizes of the plurality of regions shown in FIG. 5(A) may be set in advance. The region shown in FIG. 5(B) may be set by repeatedly arranging a group of rectangular regions in the horizontal direction along the bottom edge of the bounding box 503, and arranging further groups of rectangular regions in the horizontal direction on top of the group of rectangular regions. In this case, the size of each rectangular region can be set according to the size of a person moving near the group of rectangular regions, or the vertical position of the group of rectangular regions.

[0055] When detection line 502 shown in Fig. 5(B) is extended like detection line 504 in Fig. 5(C), a method of changing the size of the area so as to include detection line 504 can be considered. On the other hand, by including detection line 504 in an area whose size depends on the position of extended detection line 504 as in Fig. 5(D), it is possible to maintain the ratio between the size of the partial image and the size of the person before and after the extension.

[0056] 5(D), the area occupied by the arranged region group may be larger than the bounding box 506 that contains the extended detection line 504. In this case, the relative positional relationship between the region group and the bounding box 506 can be determined arbitrarily so that the region group contains the bounding box 506.

[0057] 5(E) to 5(I) are examples of methods for setting the relative positional relationship between the region group and the bounding box 506. In these examples, the relative positional relationship is set based on an upper margin 507, a lower margin 508, a left margin 509, or a right margin 510, which indicate the distance between the boundary of the region group and the boundary of the bounding box.

[0058] In the flow measurement in step S304, the result of detection of a person to be measured from the partial image can be used. Since the detection of a person is performed based on the information contained in the partial image, the more information available for detection, the higher the accuracy of person detection and the higher the accuracy of flow measurement are expected to be. Here, the highest accuracy of person detection is achieved at the center of the partial image. On the other hand, since information from the area beyond the boundary of the partial image cannot be used, the amount of information available for person detection decreases as the area moves away from the center of the partial image, and the detection accuracy tends to decrease. Therefore, the accuracy of flow measurement in step S304 tends to be lower in areas close to the boundary of the partial image. By providing the above-mentioned margins evenly around the bounding box and moving the detection line as far away from the boundary of the partial image as possible, the accuracy of person detection near the detection line is improved, and the accuracy of flow measurement is also expected to improve.

[0059] For example, in FIG. 5(E), the positions of the region groups are set so that the upper margin 507 and the lower margin 508 are equal in width, and the left margin 509 and the right margin 510 are equal in width.

[0060] In the example of FIG. 5(F), the widths of the left margin 509 and the right margin 510 are equal, and the ratio of the widths of the upper margin 507 and the lower margin 508 is equal to the ratio of the sizes of the area located at the top and the area located at the bottom. Here, the area located at the top can be the area 511 located at the center of the upper side of the bounding box 506, and the area located at the bottom can be the area 512 located at the center of the lower side of the bounding box 506. FIG. 5(G) shows a case where the midpoints of the upper and lower sides of the bounding box 506 are located at the boundary between the two areas. In this example, the ratio of the widths of the upper margin 507 and the lower margin 508 is equal to the ratio of the sizes of the area located at the top and the area located at the bottom. In the case of FIG. 5(G), the size of the area located at the top can be the size of the area 516 or the area 517 located at the center of the upper side of the bounding box 506. As another example, the size of the region located at the top may be determined from the size of region 516 and the size of region 517, for example, the average size. Similarly, the size of region 518 or region 519 located at the center of the top side of bounding box 506, or the size determined from the size of region 518 and the size of region 519, may be used as the size of the region located at the bottom.

[0061] In Fig. 5(H), the midpoint of the bottom side of bounding box 506 is located on the boundary between the two regions, and in Fig. 5(I), the midpoint of the top side of bounding box 506 is located on the boundary between the two regions. In the example of Fig. 5(H), the width of top margin 507 can be determined in the same way as in the example of Fig. 5(F), and the width of bottom margin 508 can be determined in the same way as in the example of Fig. 5(G). Also, in the example of Fig. 5(I), the width of top margin 507 can be determined in the same way as in the example of Fig. 5(G), and the width of bottom margin 508 can be determined in the same way as in the example of Fig. 5(F).

[0062] 4 and 5, partial images are extracted from areas that intersect with the detection line among the multiple set areas. Alternatively, the extraction unit 203 may set one or more extraction areas such that the one or more extraction areas are arranged along the detection line, and extract partial images from each of the set extraction areas.

[0063] Figures 6(A)-(B) show an example of such an extraction method. In these examples, a plurality of extraction regions are set so that a collection of the plurality of extraction regions encompasses a detection line 601. In Figure 6(A), a plurality of extraction regions of the same shape are arranged along the detection line 601, and in Figure 6(B), a plurality of extraction regions of varying sizes are arranged along the detection line 601. The size of each extraction region in Figure 6(B) can be set so that the ratio between the size of a partial image extracted from the extraction region and the size of a person is approximately constant, as in Figures 5(A)-(I).

[0064] Even with this configuration in which extraction regions are arranged along the detection line, it is possible to set a group of extraction regions that completely encompass the detection line so that all measurement subjects passing along the detection line can be measured. With this configuration, it is possible to accommodate various detection lines, while reducing the number of extracted partial images, and it is expected that the measurement process using the partial images by the measurement unit 204 will be faster.

[0065] When setting the extraction region group so as to include the detection line, for example, as shown in Fig. 7(A), the extraction regions can be arranged so that each of the extraction regions includes the detection line 700. Fig. 7(A) shows extraction region 701 arranged so as to include an end point of the detection line 700, and extraction region 702 arranged midway along the detection line 700.

[0066] Under the constraint that the extraction region includes the detection line 700, various methods of setting the extraction region are possible. A specific setting method is a method of arranging the extraction region so that the representative point of the extraction region is located on the detection line 700. The representative point of the extraction region can be defined as a point at a predetermined relative position with respect to the extraction region. In the example of FIG. 7(B), a representative point 704 of a figure formed by the boundary line of the extraction region 703 is defined, and the extraction region 703 is set so that this representative point 704 is located on the detection line 700. In this example, the center of gravity of the figure formed by the boundary line of the extraction region 703 is used as the representative point 704 of the extraction region 703. However, the representative point is not limited to the center of gravity, and may be a vertex or a midpoint of a side. Furthermore, the representative point of the extraction region is not limited to one, and each extraction region may have multiple representative points. By using a constraint based on the relationship between the representative point of the extraction region and the detection line 700 in this way, it is possible to easily determine the arrangement of the extraction region.

[0067] The method of arranging the representative point of the extraction region on the detection line is not particularly limited. For example, when the detection line is described by mathematical equations, for example, a straight line, an algebraic curve, or a piecewise polynomial such as a spline curve, the extraction region can be arranged so that the coordinates of the representative point of the extraction region satisfy these equations. Another method is to refer to a table that stores the coordinates of several points on the detection line. In this case, the coordinates of points on the detection line that are not included in the table can be obtained by interpolation or the like from the coordinates of nearby points. When such a table is used, the extraction region can be arranged so that the coordinates of the representative point of the extraction region match the coordinates on the detection line obtained based on the table.

[0068] As another method, the extraction area can be set so that the representative point of the extraction area is located on a pixel representing the rasterized detection line. In the example of FIG. 8(A), the detection line is rasterized on the grid points, and the extraction area is arranged so that the representative point of the extraction area coincides with the grid point thus obtained. The input image, which is a raster image, has a structure in which pixels are arranged on grid points. Therefore, by defining the extraction area based on the grid points and arranging the representative point of the extraction area on the rasterized detection line, it is possible to tile the extraction area group so as to include the detection line as shown in FIG. 8(B). The algorithm used for rasterization is not particularly limited. In addition, when the pixel value such as the shade or hue of each pixel differs in the result of rasterization, it may be determined whether or not to arrange the representative point of the extraction area on the grid point according to the pixel value, or the representative point of the extraction area may be arranged on the grid point regardless of the magnitude of the pixel value. Although an example in which the detection line is a line segment is shown in FIG. 8(A) to (B), such a method is applicable even when the detection line has any shape, such as a broken line or a curve, regardless of whether the detection line is expressed by a mathematical equation or not.

[0069] 9(A) to 9(E) show specific examples of methods for arranging extraction regions along the detection line. Here, a case will be described in which an extraction region is arranged so that its representative point is located on the detection line. In FIG. 9(A), an extraction region 902 is newly arranged adjacent to an extraction region 901 already placed on the detection line 900, so that its center of gravity, which is the representative point of the extraction region, is located on the detection line 900. This extraction region 902 includes a part of the detection line 900. In FIG. 9(B), an extraction region 904 is newly arranged adjacent to an extraction region 903 already placed on the detection line 900, so that its center of gravity, which is the representative point of the extraction region, is located on the detection line 900. This extraction region 904 also includes a part of the detection line 900.

[0070] Depending on the shape of the detection line, when the extraction regions are arranged by the above method, the entire detection line may not be included in the extraction region group. For example, in the example of FIG. 9(C) where only the center of gravity is used as the representative point, if the extraction region 907 is arranged adjacent to the extraction region 906 already placed on the detection line 905, the detection line cannot be included. In such a case, a new extraction region can be arranged so that another representative point is located on the detection line 905. For example, as in the example of FIG. 9(D), the extraction region 908 adjacent to the extraction region 906 can be arranged so that the midpoint of the left side, which is one of the representative points of the extraction region 908, is located on the detection line 905. In this example, an extraction region 909 adjacent to the extraction region 908 is further arranged so that the center of gravity, which is the representative point, is located on the detection line 905.

[0071] Also, in the example of Fig. 9(C), an additional extraction region adjacent to the other extraction regions may be set so that the detection line can be included in the group of extraction regions. For example, as shown in Fig. 9(E), extraction region 910 may be positioned so as to include detection line 905 that could not be included. In this example, the position of extraction region 910 is uniquely determined by extraction region 906 and extraction region 907. In this case, it is not necessary to use the representative point of extraction region 910.

[0072] As a further example, the extraction unit 203 may set the extraction region so that the representative point of the extraction region is located on an extension of the detection line. In this way, by virtually extending at least a part of the detection line and arranging the extraction region along the virtually extended detection line, the extraction region group can be arranged so as to encompass the entire detection line. In FIG. 10(A), the extraction region 1001, the extraction region 1002, and the extraction region 1003 are arranged along the detection line 1000. Furthermore, the extraction region 1004 is also arranged along the line 1005 obtained by virtually extending the detection line 1000. In this example, the extraction region 1004 is arranged so that the representative point of the extraction region 1004 is located on the line 1005. In this example, the detection line 1000 is encompassed by four extraction regions. According to this method, even if the length of the detection line is insufficient, the extraction region group can be laid out so as to encompass the detection line.

[0073] In the example of Figure 10(B), after extraction area 1007 is placed on detection line 1006, which is a broken line, extraction area 1009 is placed along line 1008, which is a virtual extension of one of the line segments that make up detection line 1006, and then extraction area 1010 is placed.

[0074] The method of virtually extending the detection line is not particularly limited, but for example, a detection line that is a line segment can be extended in the same direction as the line segment, and a detection line that is a curved line can be extended along a tangent at an end point of the detection line, or can be extended by connecting another arbitrary line segment.

[0075] As described above, when there is a portion on the detection line where a person or other measurement target is not detected, such as an obstruction or an obstacle, an extraction region does not need to be arranged in such a portion. For example, there may be a gap between the extraction regions equal to the size of the obstruction or obstacle.

[0076] Depending on the shape of the detection line, when the extraction regions are arranged according to the above method, the extraction regions may overlap each other. For example, when an extraction region A is arranged and then an extraction region B is arranged so that the representative point is located on the detection line, the extraction regions A and B may overlap each other. In the example of FIG. 11(A), the extraction regions are arranged counterclockwise starting from an extraction region 1101 along a detection line 1100, which is a quadrangle consisting of four line segments. Then, the last extraction region 1102 is arranged on an extension line of one of the line segments that constitute the detection line 1100. In this example, an overlap region 1103 occurs between the extraction regions 1101 and 1102, and the overlap region 1103 includes a part of the detection line 1100.

[0077] In such a case, the overlap of extraction regions can be eliminated by moving extraction region 1102 by the overlap width between extraction region 1101 and extraction region 1102. As shown in Fig. 11(B), the last placed extraction region 1102 is shifted upward by the overlap width while maintaining contact with extraction region 1104 placed immediately before extraction region 1102. This process can eliminate overlap region 1103, and can prevent double flow measurement in overlap region 1103 even if a person to be measured is present in overlap region 1103.

[0078] As another method, when the extracted regions overlap, the overlap of the extracted regions may be eliminated by moving the extracted regions according to user instructions via a human interface device or the like connected to the input unit 14 or via the I / F unit 16.

[0079] On the other hand, there are cases where overlapping of extraction regions is unavoidable. For example, as shown in FIG. 11C, when extraction regions are arranged counterclockwise starting from extraction region 1107 along detection line 1105, the overlapping of extraction regions cannot be eliminated even by moving the last extraction region 1108.

[0080] In this way, when the first partial image and the second partial image have an overlapping portion, the measuring unit 204 can exclude the overlapping portion in the first partial image or the second partial image from the measurement target. For example, in the flow measurement in step S304, a method can be adopted in which the entire partial image from one of the overlapping extraction regions is used, while a portion of the partial image from the other region corresponding to the overlapping region is not used. As a method for not using the overlapping region in the partial image in the flow measurement, for example, a method of filling the overlapping region of the partial image with a pattern that is not recognized as a person (measurement target) in step S304 can be mentioned. As such a pattern, a pattern in which a human-like shape cannot be recognized, such as a monochromatic pattern such as black or white, can be mentioned. Since a human-like shape cannot be recognized from an area filled with such a pattern, the flow of people is not measured from the image in this area.

[0081] In the example of FIG. 11(C), the partial image from extraction area 1107 is used as is, while the overlap area 1109 of the partial image from extraction area 1108 can be filled in with black. If a person to be measured exists in overlap area 1109, the person can be recognized from the overlap area 1109 in the partial image from extraction area 1107. On the other hand, in the partial image from extraction area 1108, the overlap area 1109 is filled in with black, so the person is not recognized. With this configuration, even if a person to be measured exists in overlap area 1109, it is possible to prevent the person from being recognized in multiple partial images in a duplicated manner and to prevent the person from being measured in a duplicated manner.

[0082] As another method, the extraction unit 203 can input the position of the overlap region 1109 to the measurement unit 204. In this case, the measurement unit 204 can measure the flow so as not to measure people in the overlap region in an overlapping manner. For example, the measurement unit 204 can prevent people from being measured in an overlapping manner by excluding detection results from the overlap region 1109 from among detection results for partial images from the extraction region 1108.

[0083] On the other hand, as shown in Fig. 11(D), if the overlapping area does not include the detection line, the effect of the overlapping area on the flow measurement is considered to be small, and therefore the partial images from the overlapping extraction areas can be used as they are in the flow measurement in step S304.

[0084] When the partial images include overlapping regions in this way, the extraction unit 203 may notify the user of this via the output unit 15 or another device connected via the I / F unit 16.

[0085] When arranging the extraction regions along the detection line, changing the arrangement order may change the position of the extraction regions. For example, in Fig. 12(A)-(B), the extraction regions are arranged along the detection line consisting of the line segments 1200, 1201, and 1202. In the example of Fig. 12(A), the extraction regions are arranged starting from the line segment 1200, followed by the line segments 1201 and 1202 in that order. In this example, five partial images are finally obtained. On the other hand, in the example of Fig. 12(B), the extraction regions are arranged starting from the line segment 1202, followed by the line segments 1201 and 1200 in that order. In this example, four partial images are finally obtained. Thus, when Fig. 12(A) and Fig. 12(B) are compared, the flow rate measurement on the detection line can be performed using fewer partial images in the case of Fig. 12(B).

[0086] In such a case, since the flow rate measurement process in step S304 can be performed faster with a smaller number of partial images, a method of arranging the extraction regions that reduces the number of partial images can be adopted. For example, when the results shown in Figures 12(A) and 12(B) are obtained, the result of Figure 12(B), which has a smaller number of partial images, can be adopted.

[0087] In this way, when the order of arranging the extraction regions is reversed, a series of partial images different from that in the forward direction may be obtained. The method of changing the arrangement order is not limited to this method. For example, even if an extraction region is arranged in both directions along the detection line starting from an extraction region arranged in the middle of the detection line, a series of partial images different from that in the forward direction and the reverse direction may be obtained. Hereinafter, one or more extraction regions thus set based on the detection line are referred to as a setting pattern of the extraction region. One setting pattern can indicate the arrangement of each of a plurality of extraction regions set to include the detection line, for example. That is, the extraction unit 203 can generate a plurality of setting patterns each indicating the setting of one or more extraction regions by repeating the setting of one or more extraction regions based on the position of the detection line.

[0088] Similarly, a different series of partial images may be obtained depending on the position of the initial extraction region placed on the detection line. For example, as shown in Figures 13(A) to 13(C), by repeatedly placing the extraction region while changing the position of the initial extraction region placed on the detection line, it is possible to search for an optimal setting pattern of the extraction region.

[0089] In FIG. 13(A), an extraction area 1302 on a detection line 1300 is placed first, followed by an extraction area 1303 on an extension line 1301 of one line segment constituting the detection line 1300, and an extraction area 1304 on the detection line 1300. In FIG. 13(B), an extraction area 1305 on the detection line 1300 is placed first, followed by an extraction area 1306 and an extraction area 1307 on the detection line 1300. In this example, the extraction areas can be placed without using an extended detection line. In FIG. 13(C), an extraction area 1309 is placed first on an extension line 1308 of one line segment of the detection line 1300, followed by an extraction area 1310 and an extraction area 1311 on the detection line 1300. In this way, by changing the position of the extraction area placed first, the placement of the extraction area group also changes.

[0090] In searching for a setting pattern of an extraction area, a set of points obtained by rasterizing a detection line can be used as a candidate for a starting point, as shown in FIG. 14(A). In this case, by determining a representative point of the extraction area in advance, as shown in FIG. 14(B), the number of repetitions can be limited to the range of (number of candidates for starting points) × (number of representative points), so that the setting pattern can be searched efficiently. In addition, as shown in FIG. 14(C), the search range can be expanded by using a set of points obtained by extending and then rasterizing the detection line as a candidate for a starting point. In the example of FIG. 14(C), when an extraction area is arranged so that a representative point is located at the end of the extended detection line, the detection line is extended so that the end of the detection line before extension overlaps the boundary of the extraction area.

[0091] In this way, when a plurality of setting patterns are obtained, the extraction unit 203 can select one of the plurality of setting patterns. The extraction unit 203 can select a setting pattern based on various criteria or based on a user instruction. For example, the extraction unit 203 can select one of the plurality of setting patterns according to the number of extraction areas set in each setting pattern as described above. Specifically, the extraction unit 203 can evaluate the number of extraction areas for each of the setting patterns of the plurality of extraction areas, and adopt the setting pattern with the smallest number of extraction areas. On the other hand, the adoption criterion of the setting pattern is not limited to the number of extraction areas. For example, the setting pattern may be selected based on another criterion such as the following, or two or more criteria may be used in combination. As an example, when there are a plurality of setting patterns with the smallest number of extraction areas, the extraction unit 203 may select one of them based on an index other than the number of extraction areas.

[0092] An example of an index other than the number of extraction regions is the sum of the areas of the overlapping parts between the extraction regions. In this case, the extraction unit 203 can preferentially adopt a setting pattern of the extraction regions that has a smaller sum of the areas of the overlapping parts. By reducing the overlapping regions, an efficient target range can be set for the analysis process, and there is an effect of preventing the same object from being analyzed in duplicate. In addition, when the size of the extraction regions differs from one extraction region to another, the sum of the areas of the extraction regions Ssum or the size Smin of the smallest one of the extraction regions can be used as an index. When it is considered that the larger the partial image, the larger the amount of information, the extraction unit 203 can preferentially adopt a setting pattern that has a larger Ssum or Smin. In addition, a lower limit may be set for the value of Smin so that a partial image that is too small is not used. This method is effective, for example, when the ratio of the size of the extraction region extraction region to the size of the person is almost constant and there is a strong tendency for a person who is far away to appear small, such as when the input image is a wide-angle image. Furthermore, if the size of the partial image affects the processing time of the flow measurement in step S304, the extraction unit 203 can preferentially adopt a setting pattern that reduces Ssum. In this case, too, a lower limit can be set for the value of Smin so that an extremely small partial image is not used.

[0093] As an additional index, the length L of the portion of the detection line included in each extraction region can be used. As described above, by moving the detection line as far away as possible from the boundary of the extraction region, it is expected that the accuracy of detecting people near the detection line will improve, and the accuracy of flow measurement will also improve. In light of this tendency, this length L can be used as an evaluation index for the setting pattern of the extraction region.

[0094] For example, in the example of Fig. 15(A), the length of the portion of detection line 1500 included in extraction region 1501 is shorter than the other extraction regions, with length L = 30. On the other hand, in the example of Fig. 15(B), the lengths of the portions of detection line 1500 included in extraction regions 1502 and 1503 are equal, with length L = 60. Furthermore, comparing the minimum value Lmin of L for each extraction region in Fig. 15(A) and Fig. 15(B), Lmin = 30 in Fig. 15(A) and Lmin = 60 in Fig. 15(B).

[0095] The arrangement in FIG. 15(B) is more balanced than the arrangement in FIG. 15(A). That is, in FIG. 15(A), since there is no detection line near the central region in the extracted region 1501 where the accuracy of the flow measurement is high, the central region of the partial image from this extracted region cannot be used for flow measurement. On the other hand, in FIG. 15(B), the central region can be used for flow measurement in all partial images. In this way, by adopting a setting pattern in which Lmin is larger as in FIG. 15(B), it is expected that the balance of accuracy will be improved and the measurement accuracy will also be improved. In this way, the extraction unit 203 can preferentially adopt a setting pattern in which Lmin is larger.

[0096] When multiple setting patterns of extraction regions are obtained, the extraction unit 203 can also show each setting pattern, the extraction result according to each setting pattern, or the above-mentioned index for each setting pattern to the user via the output unit 15, etc. In this case, the user may select the setting pattern to be used for the flow rate measurement in step S304 by operating a human interface device or the like connected to the input unit 14, or by operating via the I / F unit 16.

[0097] As another measure against the tendency for the accuracy of the flow rate measurement in step S304 to be low in boundary regions of partial images, a central region may be defined in the extraction region, and the flow rate measurement in step S304 may be performed in the central region.

[0098] In such a case, one extraction region may include a central region and a margin region around the central region. As a method of defining the central region in the extraction region, for example, as shown in FIG. 16(A), a method of defining a central region 1601 and a margin region 1602 surrounded by the boundary line of the extraction region 1600 and the boundary line of the central region 1601 inside the extraction region 1600 may be used. In the example of FIG. 16(A), the central region 1601 is rectangular, but the shape of the central region 1601 is not particularly limited. For example, the shape of the central region 1601 may be any shape included inside the extraction region 1600, such as a polygon, a circle, an ellipse, or any closed curve.

[0099] The size of the margin area can be set according to the size of the person who is the subject of flow measurement. For example, the margin area can be set to a size that can include a portion necessary for detecting a person. FIG. 16(B) shows an example. In FIG. 16(B), the margin area 1602 of the extraction area 1600 is set to a size that can include a portion necessary for detecting the person 1603 or person 1604 who is the subject of flow measurement. More specifically, the size of the margin area is set so that it can include the person's head and shoulders. As another example, the margin area may be set so that only the person's head or the person's entire body is included in the margin area.

[0100] When setting a margin area, the extraction area for extracting a partial image can be set so that adjacent extraction areas overlap each other. On the other hand, the extraction area can be set so that the central area of ​​a first extraction area included in the extraction area and the central area of ​​a second extraction area included in the extraction area are adjacent to each other without overlapping each other. For example, in FIG. 16C, the central area 1607 of the extraction area 1605 and the central area 1608 of the extraction area 1606 are adjacent to each other so that there is no gap between the central area 1607 and the central area 1608. On the other hand, the margin area 1610 of the extraction area 1605 and the margin area 1611 of the extraction area 1606 overlap each other. With this configuration, when a person 1609 to be detected is in the margin area 1610 of the extraction area 1605, the person 1609 can be detected from the central area 1608 of the extraction area 1606 adjacent to the extraction area 1605.

[0101] The method of setting the extraction area described with reference to Figs. 4 to 15 can also be used to extract a partial image including such a margin area. For example, a partial image can be extracted from each of the extraction areas shown in Figs. 4 to 15 and their surrounding areas. In this case, the image extracted from the extraction area shown in Figs. 4 to 15 corresponds to the image in the central area 1601 of the extraction area 1600 in Fig. 16. Moreover, the image extracted from the surrounding area of ​​the extraction area shown in Figs. 4 to 15 corresponds to the image in the margin area 1602 of the extraction area 1600 in Fig. 16. Fig. 16(D) shows an example of an extraction area 1613 for extracting a partial image including a margin area from an input image 1614, the extraction area 1613 being provided along a detection line 1612.

[0102] Providing a margin area is also effective in improving the detection accuracy of a person passing through the detection line near the edge of the central area. In one embodiment, the measurement unit 204 can detect a person who passes through the detection line from the central area to the margin area between time t1 and time t2. For example, the measurement unit 204 can estimate the trajectory of a person between time t1 and time t2 using partial images at times t1 and t2 extracted from the same extraction area. In addition, when this trajectory intersects with the detection line in the central area of ​​the same partial image, it can be determined that the person has passed the detection line in this partial image. By providing a margin area in this way, the accuracy of flow measurement near the edge of the central area can be improved when a measurement target such as a person moves at high speed or when the frame rate of a moving image is low.

[0103] According to this method, the measuring unit 204 can measure the flow of the measurement object passing through the detection line using the partial image extracted from the extraction region independently for each extraction region. For example, the measuring unit 204 can measure the flow of the measurement object passing through the detection line in the center region of the first extraction region and the flow of the measurement object passing through the detection line in the center region of the second extraction region independently. Then, the flow measurement result for the entire detection line can be obtained by combining the flow measurement results in each extraction region. Note that, when the trajectory of the person intersects with the detection line in the margin region of the partial image, it is not necessary to determine that the person has passed the detection line in this partial image. This method can prevent the same person from being measured twice in adjacent partial images.

[0104] In the above description, partial images are extracted so that each position of the detection line is included in one of the partial images. On the other hand, a margin area may be provided around the detection line, and the extraction area may be set so that the set of extraction areas includes the detection line and the margin area of ​​the detection line. This method also improves the possibility of detecting the positions of a person before and after passing through the detection line when the person (measurement target) moves at high speed or when the frame rate of the video is low, thereby improving the accuracy of flow measurement.

[0105] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0106] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0107] 100: image processing device, 201: acquisition unit, 202: setting unit, 203: extraction unit, 204: measurement unit

Claims

1. A setting means for setting a detection line on an input image; an extraction means for extracting from the input image a partial image pattern that indicates the arrangement of each of a set of one or more partial images that are set so as to include the detection line, based on the position of the detection line that is set in the input image by the setting means, the extraction means executing a plurality of extraction methods that extract different partial image patterns, and a selection means for selecting one or more partial images of a plurality of patterns from among the one or more partial images of the plurality of patterns extracted by executing the plurality of extraction methods; a measuring means for measuring at least one of a flow rate and a passing direction of a predetermined object passing through the detection line set by the setting means, using one or more partial images of the pattern selected by the selection means; 13. An image processing device comprising:

2. A setting means for setting a detection line in an input image; an extraction means for extracting one or more partial images from the input image based on the position of the detection line set in the input image by the setting means, the extraction means executing a plurality of extraction methods each having a different pattern of the extracted partial images; a selection means for selecting one or more partial images of a plurality of patterns from among the one or more partial images of the plurality of patterns extracted by executing the plurality of extraction methods, based on the number of partial images in each of the plurality of patterns; a measuring means for measuring at least one of a flow rate and a passing direction of a predetermined object passing through the detection line set by the setting means, using one or more partial images of the pattern selected by the selection means; 13. An image processing device comprising:

3. The setting means sets one or more detection lines, the detection lines being expressed by at least one of a broken line, a curve, a polygon, and a closed curve.

3. The image processing device according to claim 1, wherein the image processing device further comprises:

4. A setting means for setting a detection line on an input image; an extraction means for extracting one or more partial images from the input image based on the position of the detection line set in the input image by the setting means; a measuring means for measuring at least one of a flow rate and a passing direction of a predetermined object passing through the detection line set by the setting means, using one or more partial images extracted by the extraction means; having the setting means sets one or more detection lines, the detection lines being expressed by at least one of a broken line, a curved line, a polygon, and a closed curve; The extraction method for extracting the partial image executed by the extraction means includes a method for extracting, as the partial image, an image corresponding to a partial area included in an area including the detection line in the input image, excluding an area in which the predetermined object is not detected.

13. An image processing device comprising:

5. A setting means for setting a detection line in an input image; an extraction means for extracting one or more partial images from the input image based on the position of the detection line set in the input image by the setting means; a measuring means for measuring at least one of a flow rate and a passing direction of a predetermined object passing through the detection line set by the setting means, using one or more partial images extracted by the extraction means; having the setting means sets one or more detection lines, the detection lines being expressed by at least one of a broken line, a curved line, a polygon, and a closed curve; When the partial images are multiple and a first partial image and a second partial image among the multiple partial images have an overlapping area, the measurement means measures the partial image by using a partial area in which any of the multiple partial images is moved so that the overlapping area disappears.

13. An image processing device comprising:

6. A setting means for setting a detection line in an input image; an extraction means for extracting one or more partial images from the input image based on the position of the detection line set in the input image by the setting means; a measuring means for measuring at least one of a flow rate and a passing direction of a predetermined object passing through the detection line set by the setting means, using one or more partial images extracted by the extraction means; having the setting means sets one or more detection lines, the detection lines being expressed by at least one of a broken line, a curved line, a polygon, and a closed curve; When the partial images are multiple and a first partial image and a second partial image among the multiple partial images have an overlapping area, the measurement means measures the overlapping area by excluding the first partial image or the second partial image.

13. An image processing device comprising:

7. The extraction method for extracting the partial image executed by the extraction means includes the steps of: setting an area including the detection line in the input image based on the position of the detection line; A method for extracting an image corresponding to a partial area included in the set area as the partial image is also included.

7. The image processing device according to claim 1, wherein the image processing device further comprises: a first input unit;

8. The extraction method for extracting the partial image executed by the extraction means includes the steps of: A plurality of regions are set in the input image such that a set of the plurality of regions includes the detection line; A method for extracting an image corresponding to a partial area included in an area intersecting the detection line among the plurality of areas as the partial image. The image processing device according to claim 7 .

9. 9. The image processing apparatus according to claim 8, wherein the extraction method for extracting the partial image executed by the extraction means includes a method for setting the plurality of regions so as to include a bounding box that includes the detection line.

10. 10. The image processing device according to claim 1, wherein the extraction method for extracting the partial image executed by the extraction means includes a method for extracting an image corresponding to a partial area included in a plurality of areas arranged along the detection line as the partial image.

11. 11. The image processing apparatus according to claim 10, wherein the extraction method for extracting the partial images executed by the extraction means includes a method for setting the plurality of regions such that representative points of the regions are located on the detection line.

12. 12. The image processing device according to claim 10, wherein the extraction method for extracting the partial image executed by the extraction means includes a method for setting the plurality of regions so that representative points of the regions are located on pixels representing a rasterized detection line.

13. 13. The image processing device according to claim 10, wherein an extraction method for extracting the partial image executed by the extraction means includes a method for setting the plurality of regions so that representative points of the regions are located on extensions of the detection line.

14. 14. The image processing apparatus according to claim 11, wherein the representative point is a position corresponding to a center of gravity of a figure represented by the boundary line of the region.

15. 15. The image processing device according to claim 1, wherein the extraction method for extracting the partial image executed by the extraction means includes a method for extracting, as the partial image, an image corresponding to a partial area included in an area in the input image that includes the detection line, excluding an area in which the specified object is not detected.

16. the extraction method for extracting the partial image executed by the extraction means includes a method for extracting a partial image including the detection line in the input image and configured from a central region and a margin region surrounding the central region, The partial images are extracted such that the central regions are adjacent to each other without overlapping with each other; The measuring means measures at least one of a flow rate and a passing direction in the central area of ​​the predetermined object detected from a partial image including the margin area.

16. The image processing device according to claim 1, wherein the image processing device further comprises:

17. 17. The image processing device according to claim 1, wherein the size of the partial image extracted by the extraction means is set based on at least one of the position of an area including the partial image in the input image and the size of the specified object in the input image.

18. 18. The image processing apparatus according to claim 1, wherein the measuring means measures at least one of a flow rate and a passing direction of the predetermined object detected in the one or more partial images.

19. 20. The image processing device according to claim 18, characterized in that, when there are a plurality of partial images, the measurement means measures at least one of the flow rate and the direction of passage of the specified object detected in the partial image independently for each of the plurality of partial images.

20. The image processing device of any one of claims 1 to 19, characterized in that, when there are multiple partial images and a first partial image and a second partial image among the multiple partial images have an overlapping area, the measurement means measures using a partial area in which one of the multiple partial images has been moved so that the overlapping area disappears, or measures by excluding the overlapping area from the first partial image or the second partial image.

21. A setting step of setting a detection line in an input image; an extraction step of extracting from the input image a partial image pattern indicating each arrangement of a set of one or more partial images set so as to include the detection line based on a position of the detection line set in the input image in the setting step, the extraction step performing a plurality of extraction methods which extract different partial image patterns, and a selection step of selecting one or more partial images of a plurality of patterns from among the one or more partial images of the plurality of patterns extracted by executing the plurality of extraction methods; a measuring step of measuring at least one of a flow rate and a passing direction of a predetermined object passing through the detection line set in the setting step, using one or more partial images of the pattern selected in the selection step; 13. An image processing method comprising:

22. A setting step of setting a detection line in an input image; an extraction step of extracting one or more partial images from the input image based on the position of the detection line set in the setting step, the extraction step executing a plurality of extraction methods in which the partial images to be extracted have different patterns; a selection step of selecting one or more partial images of any one of a plurality of patterns from among the one or more partial images of the plurality of patterns extracted by executing the plurality of extraction methods, based on the number of partial images in each of the plurality of patterns; a measuring step of measuring at least one of a flow rate and a passing direction of a predetermined object passing through the detection line set in the setting step, using one or more partial images of the pattern selected in the selection step; 13. An image processing method comprising:

23. A setting step of setting a detection line in an input image; an extraction step of extracting one or more partial images from the input image based on the position of the detection line set in the input image in the setting step; a measuring step of measuring at least one of a flow rate and a passing direction of a predetermined object passing through the detection line set in the setting step, using one or more partial images extracted in the extracting step; having In the setting step, one or more detection lines are set, the detection lines being expressed by at least one of a broken line, a curved line, a polygon, and a closed curve; The image processing method is characterized in that in the extraction step, an image corresponding to a partial area included in an area in the input image that includes the detection line, excluding an area in which the specified object is not detected, is extracted as the partial image.

24. A setting step of setting a detection line in an input image; an extraction step of extracting one or more partial images from the input image based on the position of the detection line set in the input image in the setting step; a measuring step of measuring at least one of a flow rate and a passing direction of a predetermined object passing through the detection line set in the setting step, using one or more partial images extracted in the extracting step; having In the setting step, one or more detection lines are set, the detection lines being expressed by at least one of a broken line, a curved line, a polygon, and a closed curve; In the measuring step, when the partial images are multiple and a first partial image and a second partial image among the multiple partial images have an overlapping area, the measurement is performed using a partial area in which any of the multiple partial images is moved so that the overlapping area disappears.

13. An image processing method comprising:

25. A setting step of setting a detection line in an input image; an extraction step of extracting one or more partial images from the input image based on the position of the detection line set in the input image in the setting step; a measuring step of measuring at least one of a flow rate and a passing direction of a predetermined object passing through the detection line set in the setting step, using one or more partial images extracted in the extracting step; having In the setting step, one or more detection lines are set, the detection lines being expressed by at least one of a broken line, a curved line, a polygon, and a closed curve; In the measuring step, when the partial images are multiple and a first partial image and a second partial image among the multiple partial images have an overlapping area, the overlapping area is excluded from the first partial image or the second partial image and then the measurement is performed.

13. An image processing method comprising:

26. A program for causing a computer to function as the image processing device according to any one of claims 1 to 20.

Images