Image processing apparatus, method, and non-transitory computer-readable storage medium

Abstract

Image processing apparatus, method, and non-transitory computer-readable storage medium. The image processing apparatus includes a setting unit configured to set, in a display screen, a patch region based on a detection line set in the display screen, the patch region including a detection region serving as a region for detecting a position of an object and a counting region surrounded by the detection region, and a counting unit configured to count, from among a moving vector of an object moving within the display screen in the detection region or an intersection between a trajectory based on the moving vector and the detection line in the detection region, an intersection located in the counting region.

Description

TECHNICAL FIELD

[0001] The present application relates to an image processing apparatus, an image processing method, and a non-transitory computer-readable storage medium, and particularly relates to a moving image analysis technology. BACKGROUND

[0002] In recent years, an apparatus that analyzes a flow of people in a photographing region based on an image captured by a camera or the like has been proposed. As a specific method for analyzing the flow of people, there is a method of counting the number of people who have passed through a detection line that is a virtual line drawn on an image.

[0003] In a facility where many people gather, by counting the number of people who enter and exit from a gate of the facility, it is possible to estimate the number of people who stay inside the facility. In addition, it is desirable to identify a gate at which the degree of congestion is high or a gate at which a jam has occurred, thereby preventing accidents and criminal activities due to congestion.

[0004] In Khurram Shafique and Mubarak Shah. A Non-Iterative Greedy Algorithm for Multi-frame Point Correspondence. IEEE transactions on pattern analysis and machine intelligence (2005) (hereinafter, referred to as Non-Patent Literature 1), a method is disclosed in which tracking of a plurality of points is performed by associating points detected at each time of day on a plurality of frames with each other. Japanese Patent Application Publication No. 2018-180619 (hereinafter, referred to as Patent Literature 1) discloses dividing an image that is continuous in time order into a plurality of local regions to obtain a plurality of divided regions, extracting the number of objects and movement features for each divided region, and integrating the extracted number and movement features, thereby estimating a flow of objects in the divided region.

[0005] The technology disclosed in Non-Patent Literature 1 performs detection of points on the entire screen without narrowing the detection range for points corresponding to the position of the detection line, and thus has a problem of prolonging the processing time for counting the number of passages. In order to count the number of passages at a high speed, it is conceivable to perform people flow analysis only for the surroundings of the detection line, and to determine the intersection between the people flow and the detection line.

[0006] The technology disclosed in Patent Literature 1 adopts a method of estimating the number of object passes for each divided region. Based on this concept, by counting only the number of object passes in the divided region through which the detection line passes, and summing the results, the number of object passes for the entire detection line can be obtained. Here, because the number of passes can be counted independently for each divided region, the number of people can be counted at high speed by making the processing for each divided region in parallel.

[0007] However, according to Patent Literature 1, the method for dividing the image remains the same regardless of the position of the detection line. When the method for dividing the image remains the same, the following problem can arise: depending on the relationship between the detection line and the divided region, the counting result of the number of passes can not be accurate. Figure 1 An example showing this problem is shown.

[0008] Figure 1 An example of an input image 100 in which a detection line 101 is set by a user is shown. In this case, in order to count the number of object passes for the detection line 101, the number of object passes is counted in each of the divided regions 102, 103, and 104. Each of the arrows 105, 106, 107, and 108 is a movement vector indicating the movement of a given object from the position at time tl (the start point) to the position at time t2 (the end point). For the object shown by the arrow 105, in which the start point and the end point are both located within the divided region 104, the pass of the object can be detected in the divided region 104. However, in any of the divided regions 102, 103, and 104, the pass of the object corresponding to the arrow 106, in which the start point and the end point are located in different divided regions, and the passes of the objects corresponding to the arrows 107 and 108, in which at least one of the start point and the end point is located outside the divided region, cannot be detected. This can lead to the following problem: the number of passes for the entire detection line is smaller than the actual number of passes. SUMMARY

[0009] The present application provides a technology for counting the number of objects passing through a detection line more accurately.

[0010] According to a first aspect of the present application, there is provided an image processing apparatus including: a setting unit configured to set a sheet region in a display screen based on a detection line set in the display screen, the sheet region including a detection region serving as a region for detecting the position of an object and a counting region surrounded by the detection region; and a counting unit configured to count, from among the intersection points between the movement vectors or trajectories based on the movement vectors of an object moving within the display screen in the detection region and the detection line in the detection region, the intersection points located in the counting region.

[0011] According to a second aspect of the present application, there is provided an image processing method performed by an image processing apparatus, the image processing method including: setting a patch region in a display screen based on a detection line set in the display screen, the patch region including a detection region serving as a region for detecting a position of an object and a counting region surrounded by the detection region; and counting, among intersection points between a moving vector of the object moving within the display screen in the detection region or a trajectory based on the moving vector and the detection line in the detection region, an intersection point located in the counting region.

[0012] According to a third aspect of the present application, there is provided a non-transitory computer readable storage medium storing a computer program for causing a computer to function as: a setting unit configured to set a patch region in a display screen based on a detection line set in the display screen, the patch region including a detection region serving as a region for detecting a position of an object and a counting region surrounded by the detection region; and a counting unit configured to count, among intersection points between a moving vector of the object moving within the display screen in the detection region or a trajectory based on the moving vector and the detection line in the detection region, an intersection point located in the counting region.

[0013] Other features of the present application will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a diagram for explaining a problem of a conventional technique.

[0015] Figure 2 is a block diagram showing an exemplary hardware configuration of the image processing apparatus 200.

[0016] Figure 3 is a block diagram showing an exemplary functional configuration of the image processing apparatus 200.

[0017] Figure 4 is a flowchart of processing performed by the image processing apparatus 200.

[0018] Figure 5A is a diagram showing a patch region.

[0019] Figure 5B is a diagram explaining setting of a patch region based on a detection line.

[0020] Figures 6A-6D is a diagram explaining an effect provided by a patch region according to the first embodiment.

[0021] Figures 7A-7Eis a diagram illustrating an effect provided by the sheet region according to the first embodiment.

[0022] Figure 8A is a diagram showing an example of a sheet region in which a count region has substantially the same size as a detection region.

[0023] Figure 8B is a diagram showing an example of a sheet region in which a count region is formed to be smaller than Figure 8A the count region shown in

[0024] Figures 9A-9C is a diagram illustrating a sequence of determining a position of a sheet region.

[0025] Figure 10 is a diagram illustrating estimation of a moving vector of a person in an image. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but not all of these features are required in order to implement the invention. A plurality of these features can be combined as appropriate. Furthermore, in the drawings, the same or similar components are denoted by the same reference numerals, and repeated description is omitted.

[0027] [First Embodiment]

[0028] In the present embodiment, a description will be given of an image processing device that counts the number of objects (passing number) passing through a detection line set in a display screen when a moving image obtained by imaging a monitoring target region is displayed in the display screen. Although a case where the "object" (i.e., a count target) is a "person" will be described below, the "object" is not limited to a "person" and can be another moving body.

[0029] First, an exemplary hardware configuration of an image processing device according to the present embodiment will be described with reference to the block diagram of Figure 2 Figure 2 The image processing device 200 shown in

[0030] The control device 21 includes a processor such as a CPU and a memory device such as a RAM or a ROM. The processor executes various types of processing using a computer program and data stored in the memory device. Thus, the control device 21 performs overall operation control of the image processing device 200 and executes or controls various types of processing, which will be described later when the image processing device 200 executes the processing.

[0031] ​The storage device 22 is a nonvolatile memory in which an operating system (OS) and computer programs and data for causing the control device 21 (processor) to execute or control various types of processing are stored, which will be described later when the image processing apparatus 200 executes the processing. The computer programs and data stored in the storage device 22 are loaded into a memory device (RAM) under the control of the control device 21 (processor) and then processed by the control device 21 (processor).

[0032] The calculation device 23 performs various types of calculation processing based on instructions from the control device 21. The input device 24 is a user interface such as a keyboard, a mouse, or a touch panel screen, and allows various instructions to be input into the control device 21 (processor) by being operated by a user.

[0033] The output device 25 is a display device including a liquid crystal screen or a touch panel screen as a display screen, and can display the results of processing by the control device 21 using images, characters, and the like. The interface (I / F) 26 is a communication interface for data communication with an image pickup apparatus that picks up images of a monitoring target region, an apparatus that holds a moving image obtained by picking up the monitoring target region, an apparatus that holds various computer programs and data, and the like. The image processing apparatus 200 performs data communication with these apparatuses via the I / F 26. The I / F 26 is, for example, a wired interface such as a universal serial bus, Ethernet (registered trademark), or an optical cable, or a wireless interface such as Wi-Fi or Bluetooth (registered trademark).

[0034] Next, an exemplary functional configuration of the image processing apparatus 200 according to the present embodiment will be described with reference to the block diagram of Figure 3 Although the functional units shown in Figure 3 are described as executors of processing, in fact, the functions of the functional units are realized by the control device 21 (processor) or the calculation device 23 executing a computer program for causing the processor or the calculation device 23 to execute the functions of the functional units. Thus, the functional units shown in Figure 3 may be realized by software (computer programs) or may be realized by hardware.

[0035] The detection line setting unit 301 sets a detection line in the display screen of the output device 25. The image acquisition unit 302 acquires images (picked-up images) of each frame in a moving image obtained by picking up a monitoring target region from the outside via the I / F 26. The images of each frame in the moving image are displayed in the display screen of the output device 25.

[0036] The patch extraction unit 303 sets a patch region on the display screen based on the detection line set on the display screen of the output device 25, and extracts a partial image located in the set patch region from the captured image displayed in the display screen as a "patch".

[0037] The estimation unit 304 estimates, for each patch region set by the patch extraction unit 303 (independently), a movement vector of a person in the patch region who moves in the display screen of the output device 25.

[0038] The coordinate tracking unit 305 acquires, for each patch region set by the patch extraction unit 303 (independently), a trajectory of a person in the display screen of the output device 25 based on the movement vector of the person estimated by the estimation unit 304 for the patch region.

[0039] The passage count unit 306 counts, for each patch region set by the patch extraction unit 303 (independently), the number of passages of a person (passage number) based on the movement vector of the person estimated by the estimation unit 304 or the trajectory of the person acquired by the coordinate tracking unit 305. Then, the passage count unit 306 determines the sum of the passage numbers counted for each patch region as a passage number for the entire detection line.

[0040] Next, the processing performed by the image processing device 200 for counting the number of passages of a person (passage number) for the detection line set in the display screen of the output device 25 will be described with reference to the flowchart of Figure 4

[0041] <Step S401>

[0042] ​The detection line setting unit 301 sets a detection line in the display screen of the output device 25. The method for setting the detection line is not limited to any particular setting method. For example, the moving image obtained by imaging the monitoring target region can be reproduced and displayed in the display screen of the output device 25, and the user can set the detection line in the display screen while viewing the moving image by operating the input device 24. Alternatively, for example, the detection line setting unit 301 can cause the moving image obtained by imaging the monitoring target region to be reproduced and displayed in the display screen of the output device 25, detect an image region of an object related to the entry and exit of a person (such as a door or a ticket gate) in the display screen, and set the detection line in the vicinity of the image region. Alternatively, for example, the detection line setting unit 301 can set the detection line at the end portion (one or more of the upper end portion, the lower end portion, the left end portion, and the right end portion of the display screen) of the display screen of the output device 25. Alternatively, for example, the detection line setting unit 301 can cause the moving image obtained by imaging the monitoring target region to be reproduced and displayed in the display screen of the output device 25, detect a high-traffic region in the display screen, and set the detection line in the vicinity of the region.

[0043] <Step S402>

[0044] The image acquisition unit 302 acquires, from the outside via the I / F 26, the images (shot images) of the respective frames in the moving image obtained by imaging the monitoring target region as input images. As described above, the images of the respective frames in the moving image are sequentially displayed in the display screen of the output device 25.

[0045] Here, the moving image is, for example, a streaming or a video file, a series of image files saved for each frame, an animation saved in a medium, or the like. The source of the moving image is not limited to any particular source. For example, the image acquisition unit 302 can acquire the images of the respective frames in the moving image obtained by imaging the monitoring target region by a solid-state imaging element such as a CMOS sensor or a CCD sensor or an imaging device on which the solid-state imaging element is mounted. Alternatively, for example, the image acquisition unit 302 can acquire the images of the respective frames in the moving image obtained by imaging the monitoring target region from a server device, a storage device such as a hard disk or an SSD, or a recording medium.

[0046] <Step S403>

[0047] The slice extraction unit 303 sets a slice region on the display screen of the output device 25 based on the detection line set in step S401, and extracts a partial image in the slice region from the input image displayed in the display screen as a "slice".

[0048] Here, the reference will be made toFigure 5A A piece region is described. As shown in Figure 5A Fig. 5, the piece region 500 is a region having a configuration in which a "count region 502 serving as a region for counting the number of people who have passed through a detection line is provided inside a detection region 501 serving as a region for detecting the position of a person." The center position of the count region 502 is the same as the center position of the detection region 501, and the count region 502 is a reduced region similar to the detection region 501. That is, the count region 502 has a size (number of pixels) smaller than the size of the detection region 501, and is surrounded by the detection region 501. Hereinafter, the relationship between the count region 502 and the detection region 501 is referred to as "the count region 502 is truly surrounded by the detection region 501."

[0049] Next, the setting of a piece region based on a detection line will be described with reference to Figure 5B In this embodiment, the piece extraction unit 303 determines the positions of piece regions in the display screen so that the entire detection line is covered by the count regions of the piece regions, and the count regions do not overlap each other. In Figure 5B , the positions of six piece regions are determined so that the entire detection line 511 on the input image 510 is covered by the count regions of the piece regions.

[0050] Here, a description will be given regarding the reason why the counting accuracy of the number of passes is improved due to the count regions of the piece regions being truly surrounded by the detection regions, with reference to Figures 6A-6D Fig. 6. Figure 6A A case is shown in which the entire region of a piece region is constituted by a detection region and a count region (the count region and the detection region are completely identical to each other), and such a piece region is set in the display screen. In Figure 6A , the entire detection line 601 is covered by the piece region 604 (= count region) and the piece region 603 (= count region). The arrow 602 indicates a movement vector from the position of the person at time tl (the start point) to the position of the person at time t2 (the end point). In Figure 6A , the arrow 602 intersects the detection line 601, and further indicates that there is a person who has just crossed the detection line 601 between time tl and time t2. Here, the start point of the arrow 602 (the position of the person at time tl) is located in the piece region 604, and the end point of the arrow 602 (the position of the person at time t2) is located in the piece region 603. Therefore, if the estimation of the movement vector of the person and the tracking of the person are performed for each piece region, the pass of the person corresponding to the arrow 602 cannot be detected within either of the piece region 603 and the piece region 604. That is, in the case shown in Figure 6A , omission of counting of the number of passes occurs.

[0051] Figure 6BThe case where a piece region that is surrounded by the detection region is set in the display screen is shown. In Figure 6B , the entire detection line 611 is covered by the count region 618 of the piece region 616 (including the detection region 617 and the count region 618) and the count region 615 of the piece region 613 (including the detection region 614 and the count region 615). The arrow 612 indicates a movement vector from the position of the person at time t1 (the start point) to the position of the person at time t2 (the end point). In Figure 6B , the arrow 612 intersects the detection line 611, and further indicates that the person has just crossed the detection line 611 between time t1 and time t2. Here, the start point (the position of the person at time t1) and the end point (the position of the person at time t2) of the arrow 612 are both located in both the piece region 613 and the piece region 616. Therefore, if the estimation of the movement vector of the person and the tracking of the person are performed for each piece region, the passage of the person corresponding to the arrow 612 can be detected, and thus omission of counting of the number of passages does not occur.

[0052] Figure 6C The piece region 613 in Figure 6B is shown. The detection line 621 is a portion (a line segment) of the above-described detection line 611 that overlaps the count region 615 of the piece region 613. As described above, the start point and the end point of the arrow 612 are located in the piece region 613, and thus the arrow 612 can be recognized in the piece region 613. As a result, it can be determined in the count region 615 of the piece region 613 whether the arrow 612 intersects the detection line 621. Because the arrow 612 intersects the detection line 621 in the count region 615 of Figure 6C , it is determined that one person has crossed the detection line 621 between time t1 and time t2.

[0053] Figure 6D The piece region 616 in Figure 6B is shown. The detection line 631 is a portion (a line segment) of the above-described detection line 611 that overlaps the count region 618 of the piece region 616. As described above, because the start point and the end point of the arrow 612 are located in the piece region 616, the arrow 612 can be recognized in the piece region 616. As a result, it can be determined in the count region 618 of the piece region 616 whether the arrow 612 intersects the detection line 631. Because the arrow 612 does not intersect the detection line 631 in the count region 618 of Figure 6D , it is determined that no person has crossed the detection line 631 between time t1 and time t2.

[0054] Because the "number of passages = 1" as the counting result in Figure 6C and the "number of passages = 0" as the counting result in Figure 6Dthe count result in the count region is "the number of passes = 0", so the sum of these results "the number of passes = 1 + 0 = 1" can be obtained, and it can be estimated that one person has passed the detection line between the time t1 and the time t2. This is equal to the true number of passes.

[0055] Next, reference will be made to FIG. 12 showing another example. Figures 7A-7E A description will be given regarding the reason why the counting accuracy of the number of passing persons is improved because the count region of the sheet region is completely surrounded by the detection region. Figure 7A A case is shown in which the entire region of the sheet region is constituted by the detection region and the count region (the count region and the detection region are completely coincident with each other), and such a sheet region is set in the display screen. In this case, the entire detection line 701 is covered by the sheet region 705 (= the count region) and the sheet region 706 (= the count region). The arrows 702, 703, and 704 respectively indicate the movement vectors from the positions (start points) of the persons A, B, and C at the time t1 to the positions (end points) of the persons A, B, and C at the time t2. In this case, the arrow 702 intersects the detection line 701, and further indicates that the person A has crossed the detection line 701 between the time t1 and the time t2. In this case, the arrow 703 intersects the detection line 701, and further indicates that the person B has crossed the detection line 701 between the time t1 and the time t2. In this case, the arrow 704 intersects the detection line 701, and further indicates that the person C has crossed the detection line 701 between the time t1 and the time t2. Figure 7A Figure 7A In this case, the arrow 702 intersects the detection line 701, and further indicates that the person A has crossed the detection line 701 between the time t1 and the time t2. In this case, the arrow 703 intersects the detection line 701, and further indicates that the person B has crossed the detection line 701 between the time t1 and the time t2. In this case, the arrow 704 intersects the detection line 701, and further indicates that the person C has crossed the detection line 701 between the time t1 and the time t2. Figure 7A Figure 7A In this case, the arrow 703 intersects the detection line 701, and further indicates that the person B has crossed the detection line 701 between the time t1 and the time t2. In this case, the arrow 704 intersects the detection line 701, and further indicates that the person C has crossed the detection line 701 between the time t1 and the time t2.

[0056] Here, the end point of the arrow 702 (the position of the person A at the time t2) is located in the sheet region 706, and the start point of the arrow 702 (the position of the person A at the time t1) does not belong to any of the sheet region 705 and the sheet region 706. The start point of the arrow 703 (the position of the person B at the time t1) is located in the sheet region 705, and the end point of the arrow 703 (the position of the person B at the time t2) does not belong to any of the sheet region 705 and the sheet region 706. The start point and the end point of the arrow 704 (the positions of the person C at the times t1 and t2) do not belong to any of the sheet region 705 and the sheet region 706.

[0057] Therefore, if the estimation of the movement vectors of the persons and the tracking of the persons are performed for each sheet region, in either of the sheet region 705 and the sheet region 706, the passing of the persons A, B, and C corresponding to the arrows 702, 703, and 704, respectively, cannot be detected. That is, in this case as shown in FIG. 12, the omission of the counting of the number of passes (the number of passes is erroneously counted as 0) will occur. Figure 7A

[0058] ​​​Figure 7B The case where a piece region that is surrounded by the count region is detected in the display screen is shown. In Figure 7B In the case where the entire detection line 711 is covered by the count region 717 of the piece region 715 (including the detection region 716 and the count region 717), the count region 720 of the piece region 718 (including the detection region 719 and the count region 720), and the count region 723 of the piece region 721 (including the detection region 722 and the count region 723), the detection line 711 is shown.

[0059] Figure 7C The piece region 715 in Figure 7B is shown. The detection line 731 is a portion (line segment) of the above-described detection line 711 that overlaps the count region 717 of the piece region 715. The arrow 713 indicates a movement vector from the position of the person at time tl (the start point) to the position of the person at time t2 (the end point). The arrow 714 indicates a movement vector from the position of the person at time tl (the start point) to the position of the person at time t2 (the end point). Since the start point and the end point of the arrow 713 are located in the piece region 715, the arrow 713 can be recognized in the piece region 715. As a result, it is possible to determine whether the arrow 713 intersects the detection line 731 in the count region 717 of the piece region 715. Since the start point and the end point of the arrow 714 are located in the piece region 715, the arrow 714 can be recognized in the piece region 715. As a result, it is possible to determine whether the arrow 714 intersects the detection line 731 in the count region 717 of the piece region 715. In Figure 7C In the case where the arrow 713 intersects the detection line 731 and the arrow 714 intersects the detection line 731, it is determined that two persons have crossed the detection line 731 between time tl and time t2.

[0060] Figure 7D The piece region 718 in Figure 7B is shown. The detection line 732 is a portion (line segment) of the above-described detection line 711 that overlaps the count region 720 of the piece region 718. Since no movement vector has been detected from the piece region 718, it is determined that the number of persons who have crossed the detection line 732 between time tl and time t2 is 0.

[0061] Figure 7E The piece region 718 in Figure 7BThe detection line 733 is a portion (line segment) of the above-described detection line 711 that overlaps with the counting region 723 of the tile region 721. The arrow 712 indicates a movement vector from the position of the person at time t1 (the start point) to the position of the person at time t2 (the end point). Since the start point and the end point of the arrow 712 are located in the tile region 721, the arrow 712 can be recognized in the tile region 721. As a result, it is possible to determine whether the arrow 712 intersects with the detection line 733 in the counting region 723 of the tile region 721. In Figure 7E In this case, the arrow 712 intersects with the detection line 733, and thus it is determined that one person has crossed the detection line 733 between time t1 and time t2.

[0062] Since it is possible to independently obtain "the number of crossings = 2" as the counting result in the tile region 701, "the number of crossings = 0" as the counting result in the tile region 702, and "the number of crossings = 1" as the counting result in the tile region 703, it is possible to obtain the sum of the results "the number of crossings = 2 + 0 + 1 = 3" and estimate that three people have crossed the detection line between time t1 and time t2. This is equal to the true number of crossings. Figure 7C Figure 7D Figure 7E

[0063] Therefore, as described in the examples of Figures 6A-6D and Figures 7A-7E by setting the counting region of the tile region to be completely surrounded by the detection region, it is possible to prevent omission of counting the number of people crossing the detection line. As a result, it is possible to improve the counting accuracy.

[0064] Next, a description will be given of a determination method by which the tile extraction unit 303 determines the position of the tile region based on the detection line. As described above, in the present embodiment, the center position of the counting region in the tile region coincides with the center position of the detection region, and the counting region is a reduced region similar to the detection region.

[0065] First, the tile extraction unit 303 determines the size of the counting region in the tile region. To this end, the tile extraction unit 303 determines a ratio r1 obtained by dividing the size of the detection region in the tile region by the size of the counting region of the tile region.

[0066] There is a trade-off for the ratio r1. Figure 8A An example of a tile region in which the counting region has substantially the same size as the detection region is shown. The tile region 803 includes a detection region 804 and a counting region 805. The end point of the arrow 802 is located outside the detection region 804, and thus the crossing of the arrow 802 over the detection line 801 cannot be detected. Next, Figure 8B An example of a tile region in which the counting region is smaller than the detection region is shown. The tile region 806 includes a detection region 807 and a counting region 808. The end point of the arrow 802 is located within the counting region 808, and thus the crossing of the arrow 802 over the detection line 801 can be detected.​​​Figure 8A The piece region 813 includes a detection region 814 and a count region 815. The ratio of the size of the detection region 814 to the size of the count region 815 is greater than the ratio of the size of the detection region 804 to the size of the count region 805. Therefore, both the start point and the end point of the arrow 812 are included in the detection region 814, and the passage of the arrow 812 over the detection line 811 can be detected.

[0067] An advantage of increasing the ratio r1 is that omission of counting of the passage can be prevented. However, a disadvantage is that the amount of processing increases due to an increase in the number of piece regions required to cover the entire detection line.

[0068] A method for determining the ratio r1 includes a method of determining the ratio r1 based on the distance moved by the person between the time t1 and the time t2. In the case of assuming that the person moves at the maximum speed v, the maximum movement amount of the person in the real world between the time t1 and the time t2 is determined. The maximum movement amount in the real world is converted into a pixel count on the display screen (maximum travel pixel count). By determining the ratio r1 in such a way that the pixel count between the boundary of the detection region and the boundary of the count region is greater than the maximum travel pixel count, the number of persons moving at the maximum speed v or less can be counted without omission.

[0069] Another method for determining the ratio r1 includes a method based on a statistical value in a learning period. A learning period is set, and movement vectors of persons that have been estimated by the estimation unit 304 during the learning period are collected. Then, from the collected set of movement vectors, movement vectors having a size (size of the movement vector) ranked in the top p% (for example, p = 5) are deleted, and the length of the movement vector having the largest size among the remaining movement vectors is defined as l. Then, by determining the ratio r1 in such a way that the pixel count between the boundary of the detection region and the boundary of the count region is l, persons of at least (100-p)% of the movement vectors can be counted without omission.

[0070] Thus, by determining the ratio r1 based on the moving speed of the person, most of the movement vectors can be counted without omission while minimizing the number of piece regions required to cover the entire detection line. Note that the method for determining the ratio r1 is not limited to the above-described method, and various methods can be employed.

[0071] Once the ratio r1 has been determined, the patch extraction unit 303 determines the longitudinal dimension and the lateral dimension of the count region in the patch region as "the longitudinal dimension and the lateral dimension obtained by dividing the longitudinal dimension and the lateral dimension of the detection region in the patch region by r1". Once the dimensions of the count region in the patch region have been determined, the patch region to be used subsequently is confirmed as a patch region having "a count region having a longitudinal dimension and a lateral dimension obtained by dividing the longitudinal dimension and the lateral dimension of the detection region by r1 disposed inside the detection region" in configuration.

[0072] Next, the patch extraction unit 303 determines the position of one or a plurality of patch regions set on the display screen of the output device 25 so that all portions of the detection line are covered by the count region of the patch region.

[0073] An example of the method for determining the position of the patch region in the display screen of the output device 25 is a method in which the position of each patch region is determined so that the ratio between the size of the detection region in the patch region and the size of the person appearing in the patch region is substantially constant. By employing this method, the variation in the size of the person appearing in the patch region is reduced, and thus it is possible to improve the accuracy of the movement vector estimation, which will be described later.

[0074] In this method, the position of the patch region is determined from the end of the detection line. The angle formed by the detection line and the positive direction of the X axis (the axis in the horizontal direction) of the display screen is defined as θ. As shown in FIG. 21, when θ is 45 degrees or more and 135 degrees or less, the position of the patch region is determined from the lower side of the detection line. As shown in FIG. 22, when θ is 0 degrees or more and less than 45 degrees, the position of the patch region is determined from the left side of the detection line. As shown in FIG. 23, when θ is more than 135 degrees and less than 180 degrees, the position of the patch region is determined from the right side of the detection line. Figure 9A Figure 9B Figure 9C

[0075] In the case where it is assumed that the ratio r1 obtained by dividing the size of the detection region by the size of the count region is fixed, and the ratio r2 obtained by dividing the size of the detection region by the size of the person in the display screen is fixed, the ratio obtained by dividing the size of the count region by the size of the person in the display screen is r2 / r1, and is fixed. Thus, by sequentially repeatedly determining the position of the patch region from the end of the detection line in such a manner as to satisfy all of the following conditions 1 to 3, it is possible to determine the positions of all the patch regions.

[0076] Condition 1... The end point of the remaining line segment of the detection line that has not been covered is in contact with the boundary of the count region of the patch region.

[0077] ​​​Condition 2... The detection line or a straight line obtained by extending the detection line passes through the center of the counting region in the sheet region.

[0078] Condition 3... A ratio obtained by dividing the size of the counting region in the sheet region by the size of the person appearing in the counting region is approximately equal to r2 / r1 (approximately constant).

[0079] The size of the person appearing in the counting region can be estimated, for example, by providing a cue manually in advance, or by providing a learning period in advance and determining the size statistically from the person detected during the learning period. Note that the method for determining the position of the sheet region is not limited to the above-described method. For example, as another method, the position of each sheet region can be determined in such a manner that each sheet region is a region including a fixed number of pixels, regardless of where the sheet region is located on the display screen. Alternatively, for example, the position of each sheet region can be determined in such a manner that the counting regions in the sheet regions partially overlap each other. In this case, in order to prevent multiple counting when summing up the number of passes obtained in the sheet regions in step S407 (to be described later), measures such as averaging the number of passes for the overlapping part of the counting regions need to be taken.

[0080] Once the position of the sheet region has been determined so that the entire detection line is covered by the counting region, the sheet extraction unit 303 sets the sheet region at the determined position, and extracts a partial image located within the sheet region among the input image displayed in the display screen as a sheet. Then, the sheet extraction unit 303 adjusts the size of each extracted sheet to an appropriate size (e.g., the size determined by the estimation unit 304).

[0081] <Step S404>

[0082] The estimation unit 304 estimates the moving vector of the person in the sheet (in the detection region) for each sheet acquired by the sheet extraction unit 303 in step S403, based on the position of the person (independently). For the estimation of the moving vector of the person in the image, various methods can be used. The estimation of the moving vector will be described with reference to Figure 10 Examples thereof will be described.

[0083] In Figure 10In the illustrated method, the estimation unit 304 estimates a list of person positions at each time ti from the patches. The person position is defined as a position of a specific part of a person, including, for example, a center of a head or a center of a torso of a person. For example, the estimation unit 304 receives a patch 1001 extracted from an input image of a frame at time tl as input, and estimates a list of person positions (see rectangle 1002) in the patch 1001 at time tl. A mark 1008 in the rectangle 1002 that schematically indicates a person position in the list is a mark indicating a person position in the patch 1001. In addition, the estimation unit 304 receives a patch 1003 extracted from an input image of a frame at time t2 as input, and estimates a list of person positions (see rectangle 1004) in the patch 1003 at time t2. A mark 1009 in the rectangle 1004 that schematically indicates a person position in the list is a mark indicating a person position in the patch 1003. Subsequently, the estimation unit 304 performs a best match between a list of person positions corresponding to time t(i-1) and a list of person positions corresponding to time ti. A "movement vector at time ti" is obtained, which has a person position at time t(i-1) associated as a result of the best match as a starting point and a person position at time ti as a terminal point. For example, the estimation unit 304 receives a list of person positions corresponding to time tl (a position of the mark 1008 in the rectangle 1002) and a list of person positions corresponding to time t2 (a position of the mark 1009 in the rectangle 1004) as input, and obtains a list of movement vectors 1011 at time t2 (see rectangles 1005 in the list that schematically indicate movement vectors).

[0084] Details of each process will be described below. Details of estimation of a person position will be described. For estimation of a person position, various methods can be used. For example, an object detector that recognizes a position of a specific part of a person can be used for estimation of a person position. The object detector outputs a position of an object in a rectangular format or a mask image format. A machine learning method such as deep learning can be used to learn the object detector. A representative point such as a center of an output result can be used as a position of a person.

[0085] For the estimation of the person position, a method that utilizes a density distribution can be used. The density distribution is a representation of positions in a given input image where a person is estimated to exist. The density distribution can be, for example, a uniform distribution that corresponds to the size of the number of persons to be counted, or can be an arbitrary distribution that has a maximum value at a position where a person is highly likely to exist. Examples of the latter include a Gaussian distribution. A density distribution estimator for estimating the density distribution can be learned using a machine learning method such as deep learning. Furthermore, from the density distribution estimated by the density distribution estimator, a person position estimator for estimating the position of a person can be learned using a machine learning method such as deep learning. That is, the density distribution can be estimated from the input image using the density distribution estimator, and the position of a person can be estimated from the density distribution using the person position estimator.

[0086] In Figure 10 , the following method is employed for the estimation of the person position at time ti: in this method, a slice at time ti is received as input. However, the following estimator can be used for the estimation of the person position: a slice at a plurality of times is received as input, or a person position at a plurality of times is estimated. For example, in order to estimate the person position at time ti, the following object detector can be used: it receives both a slice at time t(i-1) and a slice at time ti as input, and outputs the position of a person at time t(i-1) and the position of a person at time ti.

[0087] Details of the best match between the list of person positions corresponding to time t(i-1) and the list of person positions corresponding to time ti will be described. In the matching of person positions, it is desirable to minimize the total sum of costs at the time of matching. Examples of algorithms for performing such matching include Hungarian matching, linear programming, and neural networks. As the cost at the time of matching, the cost can be defined, for example, based on the distance between a pair of person positions associated by matching or the difference between feature amounts of local images. By implementing the matching in such a way that the total sum of costs is minimized, the best match can be made in consideration of the overall best.

[0088] By using the person position of the person at time t(i-1) and the person position of the person at time ti associated as a result of this best match as the starting point and the end point, respectively, a vector pointing from the starting point to the end point is obtained as the movement vector of this person at time ti.

[0089] <Step S405>

[0090] The coordinate tracking unit 305 tracks the position of the person based on the movement vector of the person estimated by the estimation unit 304 for each slice acquired by the slice extraction unit 303 (independently).

[0091] The result of tracking the position of the person is called a trajectory. The trajectory can be generated by connecting the movement vectors together. For example, assume that "movement vector A of time ti which starts from the person position at time t(i-1) and ends at the person position at time ti" and "movement vector B of time t(i+1) which starts from the person position at time ti and ends at the person position at time t(i+1)" are estimated. If the end point of the movement vector A and the start point of the movement vector B are the same position (person position), the coordinate tracking unit 305 determines the connected movement vector A and movement vector B as a trajectory. The trajectory is given an ID for distinguishing the trajectories (for example, an ID unique to each person).

[0092] The operation of the coordinate tracking unit 305 will be described with reference to Figure 10 The movement vector 1011 of time t2 is a movement vector obtained from the result of matching the person position at time tl (marker 1008) and the person position at time t2 (marker 1009). The movement vector 1012 of time t3 is a movement vector obtained from the result of matching the person position at time t2 (marker 1009) and the person position at time t3 (marker 1010). Since the end point of the movement vector 1011 of time t2 and the start point of the movement vector 1012 of time t3 are the same person position (marker 1009), a trajectory 1013 is generated from these movement vectors.

[0093] <Step S406>

[0094] The number of persons who have passed through the detection line (the number of passes) is counted for each piece (independently) acquired by the piece extraction unit 303, by the number-of-passes counting unit 306, based on the movement vectors of the person estimated by the estimation unit 304 or the trajectories of the person calculated by the coordinate tracking unit 305.

[0095] The counting of the number of passes can be obtained in substantially the same manner in the case of using the movement vectors or the trajectories. The counting of the number of passes can be obtained by determining whether or not there is an intersection between the trajectories or movement vectors of the person in the counting region in the piece region and the detection line. For example, in the case of using the movement vectors, the counting of the number of passes can be obtained by determining whether or not there is an intersection between the movement vectors of the person in the counting region and the detection line. Figure 6C Since there is an intersection between the arrow 612 and the detection line 621 in the counting region 615 in the piece region 613 shown in FIG. 12, it is determined that one person has passed through.

[0096] Thus, the number-of-passes counting unit 306 counts the number of "intersections between the movement vectors or trajectories and the detection line" in the counting region of each piece region. That is, the number-of-passes counting unit 306 counts only the number of "intersections between the movement vectors or trajectories and the detection line" in the counting region, and does not count the number of "intersections between the movement vectors or trajectories and the detection line" outside the counting region.

[0097] Note that the number-of-passing counting unit 306 can also count the number of passes for each passing direction by determining in which direction the trajectory or moving vector passes the detection line. For example, intersection determination between the trajectory or moving vector and the detection line can be performed by a geometric method such as line segment intersection determination.

[0098] In the case of counting the number of passes using a trajectory, it is necessary to impose a restriction so that the number of passes is not counted multiple times when the same person makes a movement of repeatedly crossing the detection line. To achieve this, the passing direction and the number of passes for each ID given to the trajectory can be stored.

[0099] <Step S407>

[0100] The number-of-passing counting unit 306 determines the sum of the number of passes counted for each tile region in step S406 as the "number of people passing the entire detection line". Note that the number-of-passing counting unit 306 can also count the number of passes for each direction in which the detection line is passed.

[0101] Note that each of the processes of steps S404 to S406 described above is a process that can be independently performed for each tile, and thus by performing the series of processes from steps S404 to S406 in parallel for each tile, the number of passes can be counted at high speed.

[0102] Thus, according to the present embodiment, the position of the tile region is dynamically determined based on the given detection line. The tile region is constituted by two regions, namely, a detection region in which estimation of a specific object is performed and a counting region in which counting is performed. By determining the position of the tile region so that all parts of the detection line are covered by the counting region of an arbitrary tile region, high-speed processing can be performed by parallelization while keeping the deviation of the number of passes for the entire detection line small.

[0103] Although the example in which the object to be counted for the number of passes is a person has been described in the present embodiment as described above, the object that can be used as this object is not limited to a person, but any object for which the number can be counted can be used. Examples of such an object include a riding object such as a bicycle or a motorcycle, a vehicle such as a car or a truck, an animal such as livestock, a cell, a microorganism, and the like.

[0104] [Second Embodiment]

[0105] In the first embodiment, the configuration for counting the number of passes on the entire detection line has been described. The number of passes counted in this way can be displayed on the display screen of the output device 25, or can be transmitted to an external device via the I / F 26. In the case where the number of passes is displayed on the display screen of the output device 25, the number of passes can be displayed as a character string (a number representing the number of passes), or can be displayed as an icon whose size, length, or color, etc. changes according to an increase or decrease in the number of passes. That is, the output destination and the output form of the counted number of passes are not limited to any particular output destination and output form.

[0106] The image processing device 200 can be a device separate from the source of the moving image, or can be a device integrated with the source. In the latter case, for example, the image processing device 200 can be incorporated in an imaging device for capturing an image of a monitoring target region, and can count the number of objects that have passed through the detection line with respect to a moving image captured by the imaging device.

[0107] Note that the numerical values, processing timings, processing sequences, and the like used in the above description are given as examples for providing concrete descriptions, and the present application is not intended to be limited to these numerical values, processing timings, processing sequences, and the like.

[0108] Some or all of the above-described embodiments can be appropriately combined. Some or all of the above-described embodiments can be selectively used.

[0109] [Other Embodiments]

[0110] Embodiments of the present application can also be realized by a computer of a system or apparatus that reads out and executes software (a program) stored in storage medium to function in accordance with the embodiments, or by a method, the like where the computer or CPU, MPU, or the like reads out a program and executes it. The software includes one or both of an object code form and a source code form. The software contains one or plural sets of the programs.

[0111] While the present application has been described with reference to exemplary embodiments, it is to be understood that the application is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be construed as encompassing all the means believed to fall within the scope of the application, equivalents thereof and modifications thereof that occur to the persons skilled in the art to which the present application pertains.

Claims

1. An image processing apparatus comprising: a setting unit configured to set a piece region in a display screen based on a detection line set in the display screen, the piece region including a detection region serving as a region for detecting a position of an object and a counting region surrounded by the detection region, wherein an entire detection line is covered by the counting region of a piece region; and a counting unit configured to count, from among intersection points between a moving vector of an object moving within the display screen in the detection region or a trajectory based on the moving vector and the detection line in the detection region, an intersection point located in the counting region.

2. The image processing device according to claim 1, wherein The setting unit determines a size of the counting region in the piece region in accordance with a speed of the object in the display screen.

3. The image processing device according to claim 1, wherein The setting unit determines positions of the respective piece regions in the display screen so that the counting regions in the respective piece regions do not overlap.

4. The image processing device according to claim 1, wherein The setting unit determines positions of the piece regions in the display screen so that a ratio between a size of the detection region and a size of the object appearing in the piece region is substantially constant.

5. The image processing device according to claim 1, wherein The counting unit performs the counting in parallel for the respective piece regions.

6. The image processing device according to claim 1, wherein The counting unit obtains a sum of the number of the intersection points counted for the respective piece regions as the number of objects that have passed through the detection line. 7.The image processing apparatus according to claim 6, further comprising: an output unit configured to output the sum obtained by the counting unit. 8.The image processing apparatus according to claim 1, further comprising: an acquisition unit configured to acquire a moving image captured by an image pickup apparatus for capturing an image of a monitoring target region, wherein images of respective frames in the moving image are displayed in the display screen.

9. The image processing device according to claim 8, wherein The image processing apparatus is integrated with the image pickup apparatus. 10.An image processing method performed by an image processing apparatus, the image processing method comprising: setting a piece region in a display screen based on a detection line set in the display screen, the piece region including a detection region serving as a region for detecting a position of an object and a counting region surrounded by the detection region, wherein an entire detection line is covered by the counting region of a piece region; and counting, from among intersection points between a moving vector of an object moving within the display screen in the detection region or a trajectory based on the moving vector and the detection line in the detection region, an intersection point located in the counting region. 11.A non-transitory computer-readable storage medium storing a computer program for causing a computer to function as: a setting unit configured to set a patch region in the display screen based on a detection line set in the display screen, the patch region including a detection region and a counting region, the detection region serving as a region for detecting a position of an object, the counting region being surrounded by the detection region, wherein an entire detection line is covered by the counting region of a piece region; and a counting unit configured to count, from among intersection points between a moving vector of an object moving within the display screen in the detection region or a trajectory based on the moving vector and the detection line in the detection region, an intersection point located in the counting region. ​

Images