Homography calculation method and program

Abstract

To provide a homography calculation method and a homography calculation program for calculating a homography matrix with high accuracy even when targeting an area in which an obstacle exists.SOLUTION: A homography calculation method specifies coordinates of each of grounding points of an object in an image coordinate system, from a photograph image obtained by photographing a state in which the object whose grounding point can be specified exists at each of positions in an area corresponding to four or more points designated on a plane in a map coordinate system, which is a bird's-eye view of the area to be photographed (S14), and calculates a homography matrix that associates each of the specified coordinates of the grounding points with each of the coordinates of the four or more points in the map coordinate system (S16).SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The disclosed technology relates to a homography calculation method and a homography calculation program. [Background technology]

[0002] In order to devise measures that will lead to an improvement in the experience of employees in offices and other workplaces, there is an increasing need to recognize the work of employees using sensors such as cameras. In particular, in order to understand the time people spend in indoor work areas and their movement trajectories, there is a demand for technology that can detect people from images captured by surveillance cameras, estimate the movement paths of people in the camera images, and visualize the movement paths by drawing them on a layout map of the work room.

[0003] For example, a people flow analysis system has been proposed that analyzes and visualizes the congestion status and movement flow of people. This people flow analysis system associates positions in an image captured by a network camera with positions on a map acquired in advance, detects people from the image captured by the camera, and converts the detected positions of people to corresponding positions on the map. Then, based on the converted positions, the system tallys up at least one result of the congestion status of people and the people flow indicating the movement of people, and displays the obtained result.

[0004] Also, for example, an image display device has been proposed that reads image data from a memory unit in which attribute information including an image and its acquisition location information is stored in association with each other, places the image on a background map specified by the user to create a map image, and displays the created map image.

[0005] Also, a calibration method has been proposed in which a projector projects a calibration image including an index indicating a coordinate position in a projected image, and a camera captures an image of the area onto which the calibration image is projected by the projector. In this method, a computer terminal divides a captured image captured by the camera into multiple images, and for each of the multiple divided images, calculates a projection transformation parameter for projecting the captured image onto a projected image based on the index included in the image. Then, the captured image captured by the camera is projected onto the projected image based on the calculated projection transformation parameter. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2018-148422 A [Patent Document 2] JP 2011-43871 A [Patent Document 3] JP 2010-273289 A Summary of the Invention [Problem to be solved by the invention]

[0007] In the conventional technology, when converting the position of a person in an image (image coordinate system) to the position of a person on a map such as a map or layout drawing (map coordinate system), a projective transformation matrix (homography matrix) that predefines the relationship between two coordinates is used. In order to convert the position from the image to the map with high accuracy, it is necessary to calculate this homography matrix with high accuracy. However, when a target area is photographed with a camera, such as in a work place such as an office, if there are many parts where the floor surface is not captured due to obstacles, there is a problem that the homography matrix cannot be calculated with high accuracy.

[0008] According to one aspect, the disclosed technology aims to calculate a homography matrix with high accuracy even when a region containing an obstacle is being targeted. [Means for solving the problem]

[0009] In one aspect, the disclosed technology acquires a captured image of a state in which an object with a identifiable ground point is present at each of positions in a region corresponding to four or more points specified on a plane of a map coordinate system obtained by looking down on the region of a subject from above.The disclosed technology then identifies the coordinates of each of the ground points of the object in the image coordinate system from the acquired captured image, and calculates a homography matrix that associates each of the identified coordinates of the ground points with each of the coordinates of the four or more points in the map coordinate system. Effect of the Invention

[0010] As one aspect, there is an effect that a homography matrix can be calculated with high accuracy even when a region containing an obstacle is targeted. [Brief description of the drawings]

[0011] [Figure 1] 1A and 1B are diagrams for explaining problems that may occur when an obstacle that blocks the floor surface is present in a target area. [Diagram 2] FIG. 2 is a functional block diagram of a homography calculation device. [Diagram 3] FIG. 13 is a diagram for explaining corresponding points. [Figure 4] FIG. 13 is a diagram for explaining detection of foot coordinates. [Diagram 5] 11 is a diagram for explaining detection of coordinates of a person's feet when the feet are not captured in a captured image. FIG. [Figure 6] FIG. 13 is a diagram for explaining visualization of an estimated flow line. [Figure 7] FIG. 1 is a block diagram showing a schematic configuration of a computer that functions as a homography calculation device. [Figure 8] 5 is a flowchart showing an example of a calculation process in the first embodiment. [Figure 9] 13 is a flowchart illustrating an example of a flow line estimation process. [Figure 10] FIG. 13 is a diagram for explaining a default flow line. [Figure 11] FIG. 13 is a diagram showing another example of a default traffic line. [Figure 12] 13 is a flowchart showing an example of a calculation process in the second embodiment. [Figure 13] 13 is a flowchart illustrating an example of a change point identification process. [Figure 14] FIG. 13 is a diagram for explaining an example of identifying a change point. [Figure 15] 13 is a diagram for explaining an error between the coordinates of feet and a predetermined flow line. FIG. [Figure 16] 13 is a flowchart illustrating an example of a homography matrix calculation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.

[0013] Before explaining the details of the embodiment, first, we will explain why it is not possible to accurately calculate the homography matrix when there are many areas where the floor surface is not visible due to obstacles when a target area is photographed with a camera.

[0014] As described above, a homography matrix is ​​used to convert coordinates in an image coordinate system in a captured image of a target area captured by a camera into coordinates on a plane of a map coordinate system of a layout diagram or the like that overlooks the target area. A homography matrix can be analytically calculated by specifying four or more corresponding points (hereinafter referred to as "corresponding points") in the image coordinate system and the map coordinate system. In order to calculate a homography matrix with high accuracy, it is necessary that none of the four corresponding points are on the same line, and that the corresponding points are not local but cover as wide an area as possible of the target area. A corresponding point that satisfies these conditions is specified on the layout diagram, and a marker is placed on the floor at a position corresponding to the corresponding point. Then, a marker is detected from the captured image of the target area, and a homography matrix is ​​calculated based on the correspondence between the coordinates in the image coordinate system of the corresponding point indicated by the marker and the coordinates in the map coordinate system.

[0015] However, as shown in the left diagram of Fig. 1, in a work place such as an office, desks, chairs, cabinets, etc. are arranged on the floor. Therefore, when corresponding points (black circles in Fig. 1) are specified in a layout drawing so as to cover a wide range of a target area as shown in the right diagram of Fig. 1, the positions of the floor surface in the photographed image showing some of the specified corresponding points are blocked by obstacles such as desks, and the markers are not captured (dotted circles in Fig. 1), as shown in the left diagram of Fig. 1. In this case, the coordinates of the corresponding points in the photographed image cannot be identified, and the homography matrix cannot be calculated.

[0016] One possible solution is to specify, as corresponding points, positions in the captured image where the floor surface is not blocked by obstacles, but in this case, it is not possible to specify four or more corresponding points at ideal positions, as it is not possible to cover a wide range of the target area, etc. As a result, the error generated in the homography matrix becomes large.

[0017] When the target area is a work place such as an office, the above-mentioned situation is highly likely to occur, and since the homography matrix cannot be calculated with high accuracy, it becomes impossible to accurately estimate the movement paths of people in the work place such as an office.

[0018] In the following embodiments, even when a region where an obstacle exists is targeted, a homography matrix is ​​calculated with high accuracy.

[0019] First Embodiment 2, homography calculation device 10 according to the first embodiment functionally includes a detection unit 12, an identification unit 14, a calculation unit 16, and a visualization unit 18. Furthermore, a map DB (database) 20 is stored in a predetermined storage area of ​​homography calculation device 10.

[0020] The map DB 20 stores a layout diagram (map coordinate system) of the target area and the coordinates of corresponding points specified on the layout diagram. In this embodiment, as shown in the right diagram of Fig. 3, a case will be described in which there are four corresponding points (black circles in Fig. 3). As described above, the corresponding points are specified so as to satisfy the conditions that none of the four points are on the same line, and that the corresponding points are not local but cover as wide an area as possible of the target area.

[0021] When calculating homography, the detection unit 12 acquires each of the captured images in which a person is standing at a position corresponding to the four corresponding points specified on the layout drawing. In this embodiment, the detection unit 12 acquires four captured images when calculating homography. When estimating a flow line, the detection unit 12 acquires each of the captured images of each frame constituting a video of a person moving within a target area. The detection unit 12 detects the coordinates of the person's contact point with the floor surface from each of the acquired captured images. In this embodiment, the coordinates of the person's feet are detected as the coordinates of the contact point.

[0022] Specifically, the detection unit 12 detects the coordinates of the feet based on the skeletal information of the person in the captured image. More specifically, the detection unit 12 detects a bounding box surrounding the person from each captured image. For example, a method called YOLO (You Only Look Once, Reference 1) can be used to detect the rectangle. Then, the detection unit 12 recognizes the skeletal shape of the person within the rectangle detected from each captured image using a machine-learned skeleton recognition model or the like. The detection unit 12 can recognize the skeletal shape by estimating the posture of the person within the rectangle using, for example, a method called CPN (Cascaded Pyramid Network, Reference 2).

[0023] Reference 1: J. Redmon et al., You Only Look Once: Unified, Real-Time Object Detection, In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.779-788, 2016. Reference 2: Y. Chen et al., Cascaded Pyramid Network for Multi-person Pose Estimation, In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.7103-7112, 2018.

[0024] When using CPN, the coordinate values ​​on the captured image of each joint or part, such as the nose, right shoulder, left shoulder, right hip, left hip, right knee, left knee, right ankle, left ankle, etc., are obtained as output, as shown in Fig. 4. The detection unit 12 detects, for example, the coordinate of the middle point between the right ankle and the left ankle (the position of the star in Fig. 4) as the coordinate of the feet.

[0025] Here, as shown in FIG. 5, there are cases where the feet of a person are blocked by an obstacle and are not captured in the captured image. In such a case, the detection unit 12 detects the coordinates of the feet based on the ratio of the upper body of the person in the captured image in which the whole body of the person is captured. For example, as shown in FIG. 4, the detection unit 12 obtains the distance A between the line connecting the right shoulder and the left shoulder and the line connecting the right ankle and the left ankle, and the distance B between the line connecting the right shoulder and the left shoulder and the line connecting the right hip and the left hip based on the skeletal information recognized from the captured image in which the whole body of the person is captured. Then, the detection unit 12 calculates the ratio B / A of B to A as the upper body ratio X. Also, as shown in FIG. 5, the detection unit 12 obtains the distance B' between the line connecting the right shoulder and the left shoulder and the line connecting the right hip and the left hip from the captured image in which the coordinates of the right ankle and the left ankle are not recognized in the same manner as described above. Then, the detection unit 12 detects the position (the position of the star in Figure 5) obtained by extending a straight line perpendicular to the line connecting the right shoulder and the left shoulder from the coordinate midway between the right shoulder and the left shoulder toward the feet by a distance B' / X as the coordinate of the feet.

[0026] When calculating the homography, the identification unit 14 identifies the coordinates of the feet detected by the detection unit 12 from each of the four captured images as the contact point with the floor surface of the person standing at the position indicating the corresponding point, i.e., the coordinates indicating the corresponding point.

[0027] The calculation unit 16 calculates a homography matrix that associates each of the coordinates in the image coordinate system indicating the identified corresponding points with each of the coordinates in the map coordinate system of the corresponding points specified in the layout drawing. Specifically, the calculation unit 16 calculates the homography matrix H using four pairs of the coordinates in the image coordinate system indicating the corresponding points identified by the identification unit 14 and the coordinates of the corresponding points in the map coordinate system stored in the map DB 20. For example, the DLT (Direct Linear Transform) method (Reference 3) can be used to calculate the homography matrix H.

[0028] Reference 3: R. Hartley and A. Zisserman. Multiple View Geometry In Computer Vision. Cambridge University Press, second edition, 2003.

[0029] When estimating a line of movement, the visualization unit 18 converts the coordinates of the feet detected by the detection unit 12 from each captured image of each frame constituting the video into coordinates in the map coordinate system using the homography matrix H calculated by the calculation unit 16. Specifically, the coordinates of the feet in the image coordinate system are converted into (u i ,v i ), the coordinates in the map coordinate system (u m ,v m ) is transformed into coordinates by the following equation (1). Note that the 3×3 matrix in equation (1) is the homography matrix H.

[0030]

number

[0031] The visualization unit 18 also generates and outputs a movement line estimation image in which a mark indicating the converted coordinates or a trajectory of the converted coordinates is superimposed on a layout diagram stored in the map DB 20. As a result, for example, the movement line estimated by converting the coordinates of feet detected from the captured images of each frame of a video of a person moving along the movement line indicated by the dashed arrow in the left diagram of Fig. 6 into the map coordinate system is superimposed on the layout diagram as shown in the right diagram of Fig. 6. As a result, the movement line of the person is visualized as indicated by the dashed arrow in the right diagram of Fig. 6.

[0032] The homography calculation device 10 can be realized by, for example, a computer 40 shown in Fig. 7. The computer 40 includes a CPU (Central Processing Unit) 41, a memory 42 as a temporary storage area, and a non-volatile storage unit 43. The computer 40 also includes an input / output device 44 such as an input unit and a display unit, and an R / W (Read / Write) unit 45 that controls reading and writing of data from and to a storage medium 49. The computer 40 also includes a communication I / F (Interface) 46 connected to a network such as the Internet. The CPU 41, the memory 42, the storage unit 43, the input / output device 44, the R / W unit 45, and the communication I / F 46 are connected to one another via a bus 47.

[0033] The storage unit 43 can be realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. A homography calculation program 50 for causing the computer 40 to function as the homography calculation device 10 is stored in the storage unit 43 as a storage medium. The homography calculation program 50 has a detection process 52, an identification process 54, a calculation process 56, and a visualization process 58. The storage unit 43 also has an information storage area 60 in which information constituting the map DB 20 is stored.

[0034] The CPU 41 reads out the homography calculation program 50 from the storage unit 43, expands it in the memory 42, and sequentially executes the processes of the homography calculation program 50. The CPU 41 operates as the detection unit 12 shown in FIG. 2 by executing the detection process 52. The CPU 41 operates as the identification unit 14 shown in FIG. 2 by executing the identification process 54. The CPU 41 operates as the calculation unit 16 shown in FIG. 2 by executing the calculation process 56. The CPU 41 operates as the visualization unit 18 shown in FIG. 2 by executing the visualization process 58. The CPU 41 reads out information from the information storage area 60 and expands the map DB 20 in the memory 42. As a result, the computer 40 that has executed the homography calculation program 50 functions as the homography calculation device 10. The CPU 41 that executes the program is hardware.

[0035] The functions realized by the homography calculation program 50 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit) or the like.

[0036] Next, the operation of the homography calculation device 10 according to the first embodiment will be described. When calculating homography, each captured image of a person standing at a position corresponding to four corresponding points specified on a layout drawing is input to the homography calculation device 10. In this case, the calculation process shown in FIG. 8 is executed in the homography calculation device 10. Also, when estimating a flow line, each captured image of each frame constituting a video of a person moving within a target area is input. In this case, the homography calculation device 10 executes a flow line estimation process shown in FIG. 9. The calculation process and the flow line estimation process are examples of a homography calculation method of the disclosed technology. Each of the calculation process and the flow line estimation process will be described below.

[0037] First, the calculation process shown in FIG. 8 will be described.

[0038] In step S10, the detection unit 12 acquires each of the captured images for each of the four corresponding points. Next, in step S12, the detection unit 12 detects a rectangle surrounding the person from each captured image, and recognizes the skeletal shape of the person within the rectangle detected from each captured image using a machine-learned skeleton recognition model or the like. From the recognized skeleton shape, the detection unit 12 acquires, as skeleton information, the coordinate values ​​on the captured image of each joint or part, such as the nose, right shoulder, left shoulder, right hip, left hip, right knee, left knee, right ankle, left ankle, etc.

[0039] Next, in step S14, the detection unit 12 detects, for example, the coordinates of the middle point between the right ankle and the left ankle as the coordinates of the feet based on the skeletal information. If the person's feet are blocked by an obstacle and are not captured in the captured image, the detection unit 12 detects the coordinates of the feet based on the ratio of the person's upper body in the captured image in which the person's entire body is captured.

[0040] Next, in step S16, the identification unit 14 identifies the coordinates of the feet detected from each of the four captured images as the contact points with the floor surface of the person standing at the positions indicating the corresponding points, i.e., the coordinates indicating the corresponding points. The calculation unit 16 then calculates a homography matrix H, for example, by the DLT method, using four pairs of coordinates in the image coordinate system indicating the identified corresponding points and coordinates of the corresponding points in the map coordinate system stored in the map DB 20, and the calculation process ends.

[0041] Next, the flow line estimation process shown in FIG. 9 will be described.

[0042] In step S20, the detection unit 12 acquires captured images of each frame constituting the video. Next, in steps S22 and S24, the coordinates of the person's feet are detected from each captured image, similar to steps S12 and S14 of the calculation process.

[0043] Next, in step S26, the visualization unit 18 converts the coordinates of the feet in the image coordinate system detected from each captured image into coordinates in the map coordinate system using the homography matrix H calculated in the calculation process.

[0044] Next, in step S28, the visualization unit 18 generates and outputs a flow line estimation image in which marks indicating each converted coordinate or the trajectory of the converted coordinates are superimposed on a layout diagram stored in the map DB 20, and the flow line estimation process is terminated.

[0045] As described above, the homography calculation device according to the first embodiment acquires captured images of a person standing at each of the positions in the area corresponding to the four corresponding points specified on a layout diagram in a map coordinate system that overlooks the area to be photographed from above. The homography calculation device detects the coordinates of the person's feet from each of the photographed images, and specifies these coordinates as the coordinates of the corresponding points in the image coordinate system. The homography calculation device then calculates a homography matrix that associates four pairs of the specified coordinates of the corresponding points in the image coordinate system with the coordinates of the corresponding points on the layout diagram in the map coordinate system. Even if there is a photographed image among the four photographed images in which the position of the floor corresponding to the corresponding point is blocked by an obstacle and is not captured in the photographed image, the coordinates of the feet can be detected using the upper body ratio or the like obtained from the photographed image in which the whole body of the person is captured. Therefore, even if the position corresponding to the corresponding point is not captured in the photographed image, the coordinates indicating the corresponding point in the image coordinate system can be specified. Therefore, even if the area in which an obstacle exists is the target, the homography matrix can be calculated with high accuracy.

[0046] In the above first embodiment, the coordinates of the feet of a person are detected from a captured image in which a person is standing at a position corresponding to the corresponding point, and are identified as coordinates indicating the corresponding point. However, the present invention is not limited to this. When calculating homography, it is sufficient to identify the coordinates of each of the ground points of an object in the image coordinate system from a captured image in which a person, not limited to a person, exists and an object whose ground point can be identified is present. For example, a captured image in which a rod-shaped object with a scale is placed on a ground point may be used. In this case, even if the ground point part of the object is blocked by an obstacle in the captured image, the ratio of the unblocked part of the object to the whole object may be calculated from a captured image in which the whole object is captured. This makes it possible to detect the coordinates of the ground point when the ground point part is blocked.

[0047] When the object is a person as in the first embodiment, it is not necessary to prepare a rod-shaped object as described above. In addition, when calculating the homography, the coordinates of the person's feet can be used as the coordinates of the object's grounding point by the same process as when estimating the flow line.

[0048] <Second embodiment> Next, a description will be given of a second embodiment. In the homography calculation device according to the second embodiment, the same components as those in the homography calculation device 10 according to the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0049] 2, homography calculation device 210 according to the second embodiment functionally includes a detection unit 212, an identification unit 214, a calculation unit 16, and a visualization unit 18. A map DB 220 is stored in a predetermined storage area of ​​homography calculation device 210.

[0050] The map DB 220 stores a layout diagram (map coordinate system) of the target area and coordinates of corresponding points specified on the layout diagram, similar to the map DB 20 in the first embodiment. In the second embodiment, as in the first embodiment, a case where the number of corresponding points is four will be described. The map DB 220 further stores a route (hereinafter, referred to as a "default flow line") that connects the four corresponding points with straight lines. For example, as shown in the right diagram of FIG. 10, the default flow line can be a route that goes around once along the four sides of a rectangle with the four corresponding points as vertices. Note that the default flow line is not limited to the above example, and may be, for example, a route that includes a diagonal line of a rectangle formed by the four corresponding points, as shown in FIG. 11.

[0051] When calculating homography, the detection unit 212 acquires each of the captured images of each frame constituting a video capturing a state in which a person moves along a flow line in a target area corresponding to a predetermined flow line on a layout drawing, as shown in the left diagram of FIG. 10. The detection unit 212 detects the coordinates of the feet from each of the acquired captured images, similar to the detection unit 12 in the first embodiment. In the second embodiment, since more captured images are acquired when calculating homography than in the first embodiment, the possibility that an image in which the whole body of a person is captured is included is higher than in the first embodiment. Therefore, the detection unit 212 can more reliably detect the coordinates of the feet from each captured image.

[0052] The identification unit 214 identifies the coordinates of the feet detected from each captured image by the detection unit 212 from the trajectory of the coordinates of the feet arranged in the chronological order of the images, as the coordinates indicating the corresponding point, the coordinates of the change point where the person's movement direction has changed. Specifically, the identification unit 214 identifies the intersection of a plurality of straight lines fitting the trajectory of the coordinates of the feet as the change point where the person's movement direction has changed. More specifically, the identification unit 214 clusters the coordinates of the feet detected from each captured image according to each line segment constituting the predetermined movement line, and estimates a straight line fitting the coordinates of the feet included in each class for each class. Therefore, when performing clustering according to the predetermined movement line shown in the right diagram of FIG. 10, the identification unit 214 estimates four straight lines and identifies four change points from the intersections of the straight lines. The intersections of the straight lines are points where the movement line of the person has changed, and correspond to points where the movement direction of the person has changed.

[0053] Homography calculation device 210 can be realized by, for example, computer 40 shown in Fig. 7. A homography calculation program 250 for causing computer 40 to function as homography calculation device 210 is stored in storage unit 43 of computer 40. Homography calculation program 250 has a detection process 252, an identification process 254, a calculation process 56, and a visualization process 58. Storage unit 43 also has an information storage area 260 in which information constituting map DB 220 is stored.

[0054] The CPU 41 reads out the homography calculation program 250 from the storage unit 43, loads it in the memory 42, and sequentially executes the processes of the homography calculation program 250. The CPU 41 operates as the detection unit 212 shown in FIG. 2 by executing the detection process 252. The CPU 41 also operates as the identification unit 214 shown in FIG. 2 by executing the identification process 254. The CPU 41 also reads out information from the information storage area 260, and loads the map DB 220 in the memory 42. The other processes are the same as those of the homography calculation program 50 according to the first embodiment. As a result, the computer 40 that has executed the homography calculation program 250 functions as the homography calculation device 210.

[0055] The functions realized by the homography calculation program 250 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC or the like.

[0056] Next, the operation of the homography calculation device 210 according to the second embodiment will be described. When calculating a homography, the homography calculation device 210 executes the calculation process shown in Fig. 12. When estimating a flow line, the homography calculation device 210 executes the flow line estimation process shown in Fig. 9, as in the first embodiment. The calculation process in the second embodiment will be described below.

[0057] In step S210, the detection unit 212 acquires each captured image of each frame constituting a video capturing a state in which a person moves along a flow line in a target area that corresponds to a predefined flow line on a layout drawing. Note that frame numbers t (t=0, 1, 2, . . .) are assigned in order from the start frame of the video, and the captured image of frame number 0 captures a state in which a person is present at a position corresponding to the first corresponding point.

[0058] Next, in steps S12 and S14, the coordinates of the person's feet are detected from each captured image in the same manner as in the first embodiment. Hereinafter, the coordinates of the person's feet detected from the captured image of frame number t are referred to as P tLet it be so.

[0059] Next, in step S30, the change point identification process is executed. Here, with reference to FIG. 13, the change point identification process will be described.

[0060] In step S32, the specifying unit 214 sets the foot coordinate P0 to the coordinate C0, sets 1 to the variable i for identifying the estimated straight line, and sets the frame number t to 1.

[0061] Next, in step S34, the slope d i-1 ~P t of the straight line obtained by fitting the coordinates included in t is calculated. The straight line to be fitted can be obtained, for example, such that the sum of the distances from each coordinate included in C i-1 ~P t is minimized.

[0062] Next, in step S36, the specifying unit 214 determines whether d t -d t-1 is greater than or equal to a predetermined threshold value dth. The threshold value dth is determined in advance as a value that can determine that each coordinate included in C i-1 ~P t-1 and the coordinate P t are not on the same straight line. If d t -d t-1 <dth, the process proceeds to step S38, the specifying unit 214 increments t by 1, and the process returns to step S34. On the other hand, if d t -d t-1 ≧dth, the process proceeds to step S40, and the specifying unit 214 sets the coordinate P t-1 to the coordinate C i . Note that at the stage of t = 1, since d t-1 is indeterminate, this step may be set to a negative determination.

[0063] Note that the processing of steps S32 to S40 above is an example of the above-described clustering processing. For example, as shown in FIG. 14, when t = 4, it is assumed that d4 - d3 < dth. Note that d3 is the slope of the straight line (the solid straight line in FIG. 14) obtained by fitting the coordinates included in C0(P0) to P3, and d4 is the slope of the straight line (the dotted straight line in FIG. 14) obtained by fitting the coordinates included in C0(P0) to P4. In this case, P3 is set to C1, and each coordinate included in C0 to C1 belongs to one class. That is, C i is the coordinate serving as the reference for coordinate clustering.

[0064] Next, in step S42, the specifying unit 214 estimates the equation of the straight line i obtained by fitting the coordinates included in the coordinates C i-1 to C i As described above, when t = 4 and d4 - d3 < dth, as shown in FIG. 14, the equation of the straight line obtained by fitting the coordinates included in C0(P0) to C1(P3) is estimated as the equation of the straight line 1.

[0065] Next, in step S44, the specifying unit 214 determines whether i = 3. If i ≠ 3, the process proceeds to step S46, the specifying unit 214 increments i by 1, and the process returns to step S34. On the other hand, when i = 3, as shown in FIG. 14, the coordinates C0 to C3 are specified, and the equations of the straight lines 1 to 3 are each estimated. Therefore, the process proceeds to step S48, and the specifying unit 214 estimates the equation of the straight line 4 obtained by fitting the coordinates included in the coordinates C3 to C0.

[0066] Next, in step S50, the specifying unit 214 obtains the intersection points of the straight line 1 and the straight line 2, the straight line 2 and the straight line 3, the straight line 3 and the straight line 4, and the straight line 4 and the straight line 1, specifies the coordinates of the four intersection points as the coordinates of the change points, and the process returns to the calculation process (FIG. 12).

[0067] Next, in step S16 of the calculation process, the calculation unit 16 uses the coordinates of the identified change points as the coordinates of corresponding points in the image coordinate system to calculate a homography matrix, and the calculation process ends.

[0068] As described above, the homography calculation device according to the second embodiment acquires each captured image of each frame constituting a video capturing a state in which a person moves along a predetermined flow line within a target area, which corresponds to a predetermined flow line that connects four corresponding points specified on a layout drawing with straight lines. Then, the coordinates of the person's feet are detected from each captured image, as in the first embodiment. This makes it possible to more reliably detect the coordinates of the person's feet from each captured image.

[0069] In addition, by identifying the coordinates of the change point where the person's direction of movement changes from the trajectory of the coordinates of their feet as coordinates indicating the corresponding point in the image coordinate system, it becomes unnecessary to manually specify a frame from the video in which a person is at the corresponding point.

[0070] In the second embodiment, a case has been described where a point where a person's moving direction changes is identified as a change point from the trajectory of the coordinates of the feet, but a point where a person's moving speed changes may also be identified as a change point. This utilizes the fact that a person's moving speed is slower when turning a corner than when moving in a straight line. Specifically, the identification unit identifies the coordinates of the feet whose distance from the coordinates of the feet detected from the immediately preceding captured image in chronological order is equal to or less than a threshold as a change point where the person's moving speed has changed. For example, the threshold is set to pth, and |P t -P t-1 If |≦pth, the specific part is P t can be identified as a change point.

[0071] Furthermore, by asking a person moving along a predetermined traffic path to slow down or stop moving when turning a corner, it is possible to more reliably and accurately identify the point at which the moving speed changes.

[0072] <Third embodiment> Next, a third embodiment will be described. In the homography calculation device according to the third embodiment, the same components as those in the homography calculation device 210 according to the second embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0073] 2, homography calculation device 310 according to the third embodiment functionally includes a detection unit 212, an identification unit 214, a calculation unit 316, and a visualization unit 18. Furthermore, a map DB 220 is stored in a predetermined storage area of ​​homography calculation device 310.

[0074] Similar to calculation unit 16 in the second embodiment, calculation unit 316 calculates a homography matrix using the change points identified by identification unit 214. Furthermore, calculation unit 316 converts the coordinates of the feet detected by detection unit 212 into coordinates in the map coordinate system using the calculated homography matrix, and repeats the calculation of the homography matrix so as to minimize the error between the converted coordinates and the predetermined movement line.

[0075] Specifically, as shown in Fig. 15, the calculation unit 316 calculates the sum of lengths of perpendicular lines (dashed lines in Fig. 15) drawn from the coordinates obtained by converting the coordinates of the feet into the map coordinate system to the corresponding sides of the predetermined movement line as the error in the map coordinate system. Note that the "corresponding sides" are sides corresponding to the straight lines fitted for each class to which each coordinate belongs when identifying the change point. Then, the calculation unit 316 recalculates the homography matrix so as to minimize the error, for example, by nonlinear optimization using the Levenberg-Marquardt method described in Reference 3.

[0076] Using the recalculated homography matrix, the calculation unit 316 converts the coordinates of the feet in the image coordinate system into coordinates in the map coordinate system, and calculates the error again. The calculation unit 316 repeats the calculation of the homography matrix and the calculation of the error until the error becomes equal to or smaller than a preset threshold value (e.g., 1 pixel).

[0077] Homography calculation device 310 can be realized by, for example, computer 40 shown in Fig. 7. A homography calculation program 350 for causing computer 40 to function as homography calculation device 310 is stored in storage unit 43 of computer 40. Homography calculation program 350 has detection process 252, identification process 254, calculation process 356, and visualization process 58. Storage unit 43 also has information storage area 260 in which information constituting map DB 220 is stored.

[0078] CPU 41 reads out homography calculation program 350 from storage unit 43, loads it in memory 42, and sequentially executes the processes of homography calculation program 350. CPU 41 operates as calculation unit 316 shown in FIG. 2 by executing calculation process 356. Other processes are similar to those of homography calculation program 250 according to the second embodiment. In this way, computer 40, which has executed homography calculation program 350, functions as homography calculation device 310.

[0079] The functions realized by the homography calculation program 350 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC or the like.

[0080] Next, the operation of homography calculation device 310 according to the third embodiment will be described. When calculating homography, homography calculation device 310 executes the calculation process shown in Fig. 12. In the third embodiment, in step S16 of the calculation process, homography matrix calculation process shown in Fig. 16 is executed. When estimating a flow line, similarly to the second embodiment, homography calculation device 310 executes the flow line estimation process shown in Fig. 9. Hereinafter, the homography matrix calculation process in the third embodiment will be described.

[0081] In step S62, the calculation unit 316 calculates the homography matrix H0 using the change points identified in step S30 of the calculation process (FIG. 12), and sets it as the initial value of the homography matrix H.

[0082] Next, in step S64, the calculation unit 316 uses the homography matrix H to convert the coordinates of the feet in the image coordinate system detected in step S14 of the calculation process (FIG. 12) into coordinates in the map coordinate system.

[0083] Next, in step S66, the calculation unit 316 calculates, as an error, the sum of lengths of perpendicular lines drawn to the corresponding sides of the predetermined flow line from the coordinates obtained by converting the coordinates of the feet into the map coordinate system.

[0084] Next, in step S68, the calculation unit 316 determines whether the error calculated in step S66 is equal to or smaller than a threshold value. If the error is greater than the threshold value, the process proceeds to step S70, where the calculation unit 316 recalculates the homography matrix by nonlinear optimization so as to minimize the error.

[0085] Next, in step S72, calculation unit 316 sets the homography matrix optimized in step S70 as a new homography matrix H, and the process returns to step S64. If it is determined in step S68 that the error is less than or equal to the threshold, the process returns to the calculation process (FIG. 12).

[0086] As described above, the homography calculation device according to the third embodiment converts the coordinates of the feet in the image coordinate system into coordinates in the map coordinate system using the calculated homography matrix. The homography calculation device then repeats the calculation of the homography matrix so as to minimize the error between the converted coordinates of the feet and the predetermined movement line. This allows points other than the corresponding points to be associated between the two coordinate systems, thereby further improving the calculation accuracy of the homography matrix.

[0087] In the above embodiments, the homography calculation program is stored (installed) in advance in the storage unit, but the present invention is not limited to this. The program according to the disclosed technology can also be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0088] The following supplementary notes are further disclosed regarding each of the above embodiments.

[0089] (Appendix 1) Identifying the coordinates of each of the grounding points of the object in an image coordinate system from a captured image in which an object with a identifiable grounding point is present at each of positions within the area corresponding to four or more points specified on a plane of a map coordinate system obtained by looking down on the area of ​​the object from above; Calculating a homography matrix that associates each of the coordinates of the identified grounding points with each of the coordinates of the four or more points in the map coordinate system. The homography calculation method includes a process performed by a computer.

[0090] (Appendix 2) When the object is a person, the coordinates of the feet of the person detected from the captured image are identified as the coordinates of the ground contact point; If the feet are not captured in the first captured image, the coordinates of the feet in the first captured image are detected based on the upper body ratio of the person in the second captured image in which the whole body of the person is captured. 2. A method for calculating a homography as described in appendix 1.

[0091] (Appendix 3) Detecting the coordinates of the feet from each of a plurality of captured images capturing a state in which the person moves along a line of movement within the area corresponding to a path connecting each of the four or more points with a straight line; From the trajectory of the coordinates of the feet, the coordinates of a change point at which the moving direction or moving speed of the person changes are identified as the coordinates of the ground contact point. A homography calculation method as described in Appendix 2.

[0092] (Appendix 4) The homography calculation method according to claim 3, wherein an intersection of a plurality of straight lines fitted to the trajectory of the coordinates of the feet is identified as a change point where the direction of movement of the person has changed.

[0093] (Appendix 5) The homography calculation method according to claim 3, wherein the coordinates of the feet whose distance from the coordinates of the feet detected in the immediately preceding captured image in chronological order is less than or equal to a threshold value are identified as a change point where the person's moving speed has changed.

[0094] (Appendix 6) 6. The homography calculation method according to any one of claims 3 to 5, further comprising: converting the detected coordinates of the feet into coordinates in a map coordinate system using the calculated homography matrix; and repeating calculation of the homography matrix so as to minimize an error between the converted coordinates and a path connecting each of the four or more points with straight lines.

[0095] (Appendix 7) 7. The homography calculation method according to claim 6, wherein the error is the sum of lengths of perpendicular lines drawn from the coordinates of the feet transformed into the map coordinate system to the path.

[0096] (Appendix 8) The homography calculation method according to any one of Supplementary Note 2 to Supplementary Note 7, wherein the coordinates of the feet are detected based on skeletal information of the person.

[0097] (Appendix 9) 9. The homography calculation method according to any one of claims 1 to 8, further comprising: converting coordinates of the grounding point into coordinates of a map coordinate system using the calculated homography matrix; and visualizing a mark indicating the converted coordinates or a trajectory of the converted coordinates by superimposing the mark on the plane.

[0098] (Appendix 10) Identifying the coordinates of each of the grounding points of the object in an image coordinate system from a captured image in which an object with a identifiable grounding point is present at each of positions within the area corresponding to four or more points specified on a plane of a map coordinate system obtained by looking down on the area of ​​the object from above; Calculating a homography matrix that associates each of the coordinates of the identified grounding points with each of the coordinates of the four or more points in the map coordinate system. A homography calculation program for causing a computer to execute a process including the steps of:

[0099] (Appendix 11) When the object is a person, the coordinates of the feet of the person detected from the captured image are identified as the coordinates of the ground contact point; If the feet are not captured in the first captured image, the coordinates of the feet in the first captured image are detected based on the upper body ratio of the person in the second captured image in which the whole body of the person is captured. A homography calculation program described in Appendix 10.

[0100] (Appendix 12) Detecting the coordinates of the feet from each of a plurality of captured images capturing a state in which the person moves along a line of movement within the area corresponding to a path connecting each of the four or more points with a straight line; From the trajectory of the coordinates of the feet, the coordinates of a change point at which the moving direction or moving speed of the person changes are identified as the coordinates of the ground contact point. A homography calculation program described in Appendix 11.

[0101] (Appendix 13) The homography calculation program according to claim 12, which identifies an intersection of a plurality of straight lines fitted to the trajectory of the coordinates of the feet as a change point where the direction of movement of the person has changed.

[0102] (Appendix 14) The homography calculation program according to claim 12, which identifies the coordinates of the feet whose distance from the coordinates of the feet detected in the immediately preceding captured image in chronological order is less than or equal to a threshold as a change point where the person's moving speed has changed.

[0103] (Appendix 15) 15. The homography calculation program according to any one of claims 12 to 14, further comprising: converting the detected coordinates of the feet into coordinates of a map coordinate system using the calculated homography matrix; and repeating calculation of the homography matrix so as to minimize an error between the converted coordinates and a path connecting each of the four or more points with straight lines.

[0104] (Appendix 16) 16. The homography calculation program according to claim 15, wherein the error is the sum of lengths of perpendicular lines drawn from the coordinates of the feet converted into the map coordinate system to the path.

[0105] (Appendix 17) 17. The homography calculation program according to any one of claims 11 to 16, which detects the coordinates of the feet based on skeletal information of the person.

[0106] (Appendix 18) 18. The homography calculation program according to any one of claims 10 to 17, which converts coordinates of the grounding points into coordinates of a map coordinate system using the calculated homography matrix, and visualizes a mark indicating the converted coordinates or a trajectory of the converted coordinates by superimposing it on the plane.

[0107] (Appendix 19) an identification unit that identifies the coordinates of each of the grounding points of the object in an image coordinate system from a captured image that captures a state in which an object with a grounding point that can be identified is present at each of positions within the area corresponding to four or more points specified on a plane of a map coordinate system that overlooks the area of ​​the object from above; a calculation unit that calculates a homography matrix that associates each of the coordinates of the identified ground points with each of the coordinates of the four or more points in the map coordinate system; A homography calculation device comprising:

[0108] (Appendix 20) Identifying the coordinates of each of the grounding points of the object in an image coordinate system from a captured image in which an object with a identifiable grounding point is present at each of positions within the area corresponding to four or more points specified on a plane of a map coordinate system obtained by looking down on the area of ​​the object from above; Calculating a homography matrix that associates each of the coordinates of the identified grounding points with each of the coordinates of the four or more points in the map coordinate system. A storage medium storing a homography calculation program for causing a computer to execute a process including the steps of: [Explanation of symbols]

[0109] 10, 210, 310 Homography calculation device 12, 212 Detector 14, 214 Specific part 16, 316 Calculation section 18 Visualization section 20, 220 Map DB 40 Computer 41 CPU 42 Memory 43 Storage section 49 Storage medium 50, 250, 350 Homography calculation program

Claims

1. A movement line within the area corresponding to a path connecting each of four or more points specified on a plane of a map coordinate system overlooking the area from above is detected from each of a plurality of captured images capturing a state in which an object capable of identifying a grounding point is moving, and from the trajectory of the coordinates of the object, coordinates of change points at which the moving direction or moving speed of the object has changed are identified as the coordinates of each of the grounding points of the object at the four or more points in the image coordinate system; Calculating a homography matrix that associates each of the coordinates of the identified grounding points with each of the coordinates of the four or more points in the map coordinate system. The homography calculation method includes a process performed by a computer.

2. When the object is a person, the coordinates of the feet of the person detected from the captured image are identified as the coordinates of the ground contact point; If the feet are not captured in the first captured image, the coordinates of the feet in the first captured image are detected based on the upper body ratio of the person in the second captured image in which the whole body of the person is captured. The homography calculation method according to claim 1 .

3. The homography calculation method according to claim 2 , further comprising identifying an intersection of a plurality of straight lines fitted to the trajectory of the coordinates of the feet as a change point where the person's moving direction has changed.

4. The homography calculation method according to claim 2 , wherein the coordinates of the feet whose distance from the coordinates of the feet detected in the immediately preceding captured image in chronological order is less than or equal to a threshold value are identified as a change point where the person's movement speed has changed.

5. The homography calculation method according to any one of claims 2 to 4, further comprising converting the detected coordinates of the feet into coordinates of a map coordinate system using the calculated homography matrix, and repeating the calculation of the homography matrix so as to minimize an error between the converted coordinates and a path connecting each of the four or more points with straight lines.

6. The homography calculation method according to claim 5 , wherein the error is a sum of lengths of perpendicular lines drawn from the coordinates of the feet converted into the map coordinate system to the route.

7. The homography calculation method according to any one of claims 2 to 6, wherein the coordinates of the feet are detected based on skeletal information of the person.

8. The homography calculation method according to any one of claims 1 to 7, further comprising: converting the coordinates of the grounding point into coordinates of a map coordinate system using the calculated homography matrix; and superimposing a mark indicating the converted coordinates or a trajectory of the converted coordinates on the plane to visualize the converted coordinates.

9. A movement line within the area corresponding to a path connecting each of four or more points specified on a plane of a map coordinate system overlooking the area from above is detected from each of a plurality of captured images capturing a state in which an object capable of identifying a grounding point is moving, and from the trajectory of the coordinates of the object, coordinates of change points at which the moving direction or moving speed of the object has changed are identified as the coordinates of each of the grounding points of the object at the four or more points in the image coordinate system; Calculating a homography matrix that associates each of the coordinates of the identified grounding points with each of the coordinates of the four or more points in the map coordinate system. A homography calculation program for causing a computer to execute a process including the steps of:

Images