Position estimation device, position estimation system, and program

Abstract

To provide a position estimation device, a position estimation system, and a program with which it is capable of estimating the position of an object with a high accuracy even when a part of the object is hidden.SOLUTION: Provided is a position estimation device comprising: an extraction unit which extracts the feature point of an object from one image among a plurality of images in which the object is imaged in time series; a first estimation unit which, when a predetermined feature point is extracted by the extraction unit, derives coordinates in real space of the predetermined feature point and estimates the position of the object on the basis of coordinates in one image and assumed height in real space of the predetermined feature point; and a second estimation unit which, when only other feature points are extracted by the extraction unit, estimates the position of the object using coordinates in one image of the other feature points and coordinates in height direction of the other feature points stored in a storage unit.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a position estimation device, a position estimation system, and a program. [Background technology]

[0002] There is a known method for detecting an object (e.g., a person) from an image captured by a camera whose orientation, position, and focal length in the world coordinate system are known, and estimating the position of the object in the world coordinate system by assuming that the bottom end of the detection frame is on the floor. For example, the system described in Non-Patent Document 1 calculates the world coordinates of the detected object in the following manner.

[0003] From the origin of the camera coordinates to any point on the image (X u ,Y u ) can be expressed as follows, where f is the focal length of the camera and R is the rotation matrix: JPEG2025058714000002.jpg2483 The point on the image is the floor surface of the world coordinate (z w =0), the position of the intersection with the ray vector in the world coordinate system is given by, where T is the camera translation vector. JPEG2025058714000003.jpg18114 In addition, the lower center point of the detection area on the image of the object (u b ,v b ) is in contact with the ground, and the world coordinate point (x wb ,y wb ,0), if the object is standing vertically on the ground, then y w -z w The upper center point of the detection area on the plane (u t ,v t ), the corresponding world coordinates can be obtained from JPEG2025058714000004.jpg3095 The obtained z wt indicates the height of the detected object from the floor surface. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] "Master-Slaving Tracking Camera System for Real Space" (Yuzo Ogawa, Hironobu Fujiyoshi, 9th Image Sensing Symposium, 2007) Summary of the Invention [Problem to be solved by the invention]

[0005] The method described in Non-Patent Document 1 above has a problem in that the error in the estimated position becomes very large when the feet are hidden by another object. When detecting a person using a detection frame, it is difficult to determine whether the feet are hidden or not, and when the feet are hidden, the above method must be applied as is, even though the assumption that the bottom edge of the detection area is on the floor (z=0) is incorrect. As a result, the calculated position is farther from the camera than the actual position.

[0006] The present invention has been made in consideration of the above-mentioned problems, and an object of the present invention is to provide a position estimation device, a position estimation system, and a program that can accurately estimate the position of an object even when part of the object is occluded. [Means for solving the problem]

[0007] A first aspect of the present invention includes an extraction unit that extracts a feature point of an object from one image of a plurality of images of the object photographed in time series, a first estimation unit that, when the extraction unit extracts a predetermined feature point, derives coordinates of the predetermined feature point in real space based on the coordinates of the predetermined feature point on the one image and an assumed height of the predetermined feature point in real space, and estimates a position of the object based on the derived coordinates, and a second estimation unit that, when the extraction unit extracts the predetermined feature point and a feature point other than the predetermined feature point, estimates a position of the object based on the derived coordinates. a derivation unit that derives height coordinates in real space of the other feature points using the coordinates in real space of the predetermined feature points derived by the first estimation unit and stores the derived height coordinates in a memory unit; and a second estimation unit that, when the extraction unit does not extract the predetermined feature points but extracts the other feature points, estimates a position of the object using the coordinates of the other feature points on the one image and the height coordinates stored in the memory unit that correspond to the extracted other feature points.

[0008] A second aspect of the present invention is a position estimation system including the position estimation device of the present invention and an imaging device that captures an image of the target object.

[0009] A third aspect of the present invention provides a computer including an extraction unit that extracts a feature point of an object from one of a plurality of images of the object photographed in time series, a first estimation unit that, when the extraction unit extracts a predetermined feature point, derives coordinates of the predetermined feature point in real space based on the coordinates of the predetermined feature point on the one image and an assumed height of the predetermined feature point in real space, and estimates a position of the object based on the derived coordinates, and a second estimation unit that, when the extraction unit extracts the predetermined feature point and a feature point other than the predetermined feature point, estimates a position of the object based on the derived coordinates. a derivation unit that derives height coordinates in real space of the other feature points by using the coordinates in real space of the predetermined feature points derived by the first estimation unit and stores the derived height coordinates in a memory unit; and a second estimation unit that, when the extraction unit does not extract the predetermined feature points but extracts the other feature points, estimates the position of the object by using the coordinates of the other feature points on the one image and the height coordinates stored in the memory unit that correspond to the extracted other feature points. Effect of the Invention

[0010] According to the present invention, the position of an object can be estimated with high accuracy even when a part of the object is occluded. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic diagram showing an example of a configuration of a position estimation system according to the present invention; [Diagram 2] 1 is a block diagram showing an example of a hardware configuration of a position estimation device according to the present invention; [Diagram 3] 1 is a functional block diagram showing an example of a functional configuration of a position estimation device according to the present invention; [Figure 4] 13 is a flowchart showing an example of the flow of a position estimation process according to a reference example. [Diagram 5] FIG. 2 is a schematic diagram showing a series of frames 1 to 4 used for position estimation. [Figure 6]13 is a graph showing a position estimation result (time transition of estimated positions of two pedestrians in the world coordinate system) by a position estimation process according to a reference example. [Figure 7] 13 is a graph showing a position estimation result (time transition of the estimated position interval between two pedestrians in the world coordinate system) by the position estimation process according to the reference example. [Figure 8] 4 is a flowchart showing an example of the flow of a position estimation process according to the present invention. [Figure 9] 5 is a schematic diagram for explaining a method of calculating an assumed height of each feature point in the position estimation process according to the present invention; FIG. [Figure 10] FIG. 13 is a schematic diagram for explaining an approximation method using a virtual plane. [Figure 11] 4 is a graph showing a position estimation result (time displacement of the position of each object in real space) by the position estimation process according to the present invention. [Figure 12] 4 is a graph showing a position estimation result (time change in the position interval between objects in real space) by the position estimation process according to the present invention. [Figure 13] FIG. 13 is a schematic diagram showing a transport cart, which is another example of an object. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] An embodiment of the present invention will now be described in detail with reference to the drawings.

[0013] [Location Estimation System] First, the position estimation system of the present invention will be described with reference to FIG. The position estimation system of the present invention includes a camera 10 and a position estimation device 12 communicably connected to the camera 10 via wired or wireless communication. The camera 10 is an example of an imaging device that captures an image of a predetermined area in real space. For example, the camera 10 may be a two-dimensional camera that can obtain two-dimensional information of the imaging area.

[0014] The position estimation device 12 acquires an image captured by the camera 10, detects an object O present in an imaging area from each frame image of the acquired image, and estimates the position of the object O in real space. The position estimation device 12 can also track the object O by associating the object O detected in the current frame with an object detected in the previous frame. By tracking the object O, the moving direction and speed of the object O can also be estimated.

[0015] In the following, a case will be described in which the object O to be the target of position estimation is a "person." However, the object O is not limited to a person, and an object having a plurality of feature points can be the object O.

[0016] (Position estimation device) Next, the hardware configuration of the position estimation device 12 will be described with reference to FIG. The position estimation device 12 is an information processing device such as a computer, and includes a CPU (Central Processing Unit) 14, a memory 16, a storage unit 18, and a communication unit 20. The CPU 14, the memory 16, the storage unit 18, and the communication unit 20 are connected via a bus 22, and can transmit and receive information between them.

[0017] The CPU 14 is a central processing unit. The memory 16 is composed of a RAM (Random Access Memory) and temporarily stores programs and data as a working area. The storage unit 18 is composed of a ROM (Read Only Memory), a HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. and stores various programs including an operating system and various data. In this embodiment, a position estimation program for performing position estimation processing is stored in the storage unit 18. In addition, as described later, data of the Z coordinate of each feature point is accumulated in the storage unit 18.

[0018] CPU 14 reads a program from storage unit 18 and executes the program using memory 16 as a working area. CPU 14 also controls the above-mentioned components and performs various types of arithmetic processing in accordance with the program stored in storage unit 18. Communication unit 20 is an interface for communicating with other devices. For example, CPU 14 communicates with camera 10 via communication unit 20 and acquires video captured by camera 10.

[0019] Furthermore, the functional configuration of the position estimation device 12 will be described with reference to FIG. The position estimation device 12 includes an extraction unit 30 that extracts a feature point of an object from one of a plurality of images of the object captured in time series, a first estimation unit 32 that, when the extraction unit 30 extracts a predetermined feature point, derives coordinates of the predetermined feature point in real space based on the coordinates of the predetermined feature point on one image and an assumed height of the predetermined feature point in real space, and estimates the position of the object based on the derived coordinates, and a second estimation unit 32 that, when the extraction unit 30 extracts the predetermined feature point and a feature point other than the predetermined feature point, estimates the position of the object based on the derived coordinates. and a derivation unit 34 that derives the height coordinate in real space of another feature point using the coordinate in real space of the predetermined feature point derived by the first estimation unit 32, and stores the derived height coordinate in the storage unit 18, and when the extraction unit 30 does not extract the predetermined feature point but extracts another feature point, a second estimation unit 36 ​​that estimates the position of the object using the coordinate on one image of the other feature point and the height coordinate stored in the storage unit 18 and corresponding to the other extracted feature point. The CPU 14 of the position estimation device 12 executes a position estimation program described later, so that the information processing device constituting the position estimation device 12 functions as each of these units.

[0020] Here, as a method for extracting feature points of an object from an image, a conventionally known method can be used, such as the method described in the non-patent document “Deep High-Resolution Representation Learning for Human Pose Estimation” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (2019), pp.5693-5703.

[0021] [Position estimation principle] Next, the principle of position estimation will be described with reference to FIG. As shown in Fig. 1, the real space in which a person exists is defined as a "three-dimensional space represented by the world coordinate system", the space on the image with one corner of the image as the origin is defined as a "two-dimensional space represented by the image coordinate system", and the real space with the optical center of the camera 10 as the origin is defined as a "three-dimensional space represented by the camera coordinate system". In addition, the feature points of the person that are used to estimate the position of the person are called "reference feature points", and the reference feature points are called "P O " should be written.

[0022] Put simply, the principle of position estimation is to detect a person from the obtained frame images, extract the feature points of the person, find a straight line (i.e., a ray vector) that passes through the optical center of the camera and the reference feature point, and output the point where this line intersects with a plane of known height as the position of the target object in real space. The ray vector is found from the coordinates of the reference feature point on the image and information on the position and orientation of the camera in three-dimensional space.

[0023] A specific procedure for calculating the coordinates of a reference feature point in a world coordinate system in a three-dimensional space from the coordinates of the reference feature point on an image will be described below in detail with reference to FIG.

[0024] The coordinates of the reference feature points in the image coordinate system after distortion (i.e., lens distortion) correction are (u O ,v O ), and the coordinates of the optical center of the camera 10 are (uC ,v C ), the coordinates of the reference feature point on the image in the camera coordinate system (X O ,Y O ) is expressed by the following formula (1).

[0025]

number

[0026] In this case, the focal length of the camera 10 is f, and the 3-row, 3-column matrix representing the orientation of the camera 10 in the world coordinate system is R. C Then, the light vector r is calculated by the following equation (2): O is required.

[0027]

number

[0028] Ray vector r O is a vector along a straight line connecting the optical center of the camera 10 and the reference feature point in the world coordinate system. Therefore, the position vector of the reference feature point in the world coordinate system is O , the position vector of the optical center of the camera 10 is P C Then, the relationship in equation (3) below can be obtained by vector addition.

[0029]

number

[0030] Here, α O is a coefficient. The coordinates of the reference feature point in the world coordinate system are (P OX ,P OY ,P OZ ) The light vector r O is calculated by the above formula (2), and the coordinates of the optical center of the camera 10 in the world coordinate system are (P CX ,P CY ,P CZ ) and is known.

[0031] Therefore, in the last equation of the above equation (3), P, which is the Z coordinate of the reference feature point in the world coordinate system, OZ Substituting the expected value into α O The value of α is obtained. O By substituting the value of into the other equations in equation (3), the X-coordinate of the reference feature point, P OX and P is the Y coordinate OY The following is obtained.

[0032] That is, the three-dimensional position of the reference feature point in the world coordinate system can be calculated from the coordinates of the reference feature point on the image, information on the position and orientation of the camera in the three-dimensional space, and an estimated value of the height of the reference feature point in the world coordinate system. Then, the obtained three-dimensional position of the reference feature point can be regarded as the estimated position of person O.

[0033] As described above, to estimate the position of a person using this procedure, information on the assumed height of reference feature points in the world coordinate system (hereinafter referred to as "assumed height") is required. Examples of feature points of a person include the neck, shoulders, waist, knees, ankles, elbows, wrists, buttocks, head, ears, eyes, etc. The height of feature points located at low positions such as the ankles is less affected by individual differences in height, but other feature points can be affected by individual differences in posture and height, so there is a large error between the assumed height and the actual height, and the error between the calculated three-dimensional position and the actual position is also large. On the other hand, feature points located at low positions are often hidden by other objects.

[0034] In an environment where workers are often standing or walking, such as in a factory, it is generally correct to assume that the ankle feature point is located at a height of about 0.1 m from the floor. Therefore, if the ankle feature point is extracted with a reliability equal to or higher than a threshold, the ankle feature point is set as the reference feature point. If the reliability of the ankle feature point is less than the threshold, a feature point other than the ankle (for example, the buttocks or ears) can be set as the reference feature point.

[0035] Here, the reliability of a feature point is an index indicating whether the feature point has been stably extracted. A feature point that has not been extracted due to occlusion or the like has a reliability equal to or lower than a threshold. For example, for a pixel extracted as a feature point of an ankle, the probability that the pixel is an ankle can be regarded as the reliability. In general, the reliability of a feature point is calculated when the feature point is extracted and is included in the extraction result.

[0036] [Location Estimation Process Related to Reference Example] Hereinafter, a flow of a position estimation process according to a reference example will be described with reference to Fig. 4. This process is executed by the CPU 14 of the position estimation device 12, similar to the position estimation process according to the present invention described later. For example, when a user (not shown) instructs the start of the position estimation process, the CPU 14 acquires video data from the camera 10 and starts the position estimation process.

[0037] In the position estimation process according to the reference example, if the ankle feature point is extracted with a reliability equal to or higher than a threshold, the ankle feature point is selected as the reference feature point, and if the reliability of the ankle feature point is less than the threshold, a feature point other than the ankle (in this example, the buttocks or ears) is selected as the reference feature point. If the ankle feature point is occluded, the buttocks feature point is used next, and if the ankle and buttocks feature points are occluded, the ear feature point is used next. In addition, the expected heights of the ankle, buttocks and ear feature points are set to preset fixed values ​​(Am, Bm and Cm, respectively).

[0038] First, in step S100, CPU 14 detects people from the frame image and assigns an ID to each person. A detection frame is added to the image in the area of ​​the detected person. Next, in step S102, CPU 14 selects one person from the detected people. Next, in step S104, CPU 14 extracts feature points within the detection frame of the selected person. The extraction result of the feature points includes the reliability of the feature points.

[0039] Next, in step S106, the CPU 14 determines whether the ankle feature points are extracted with a reliability equal to or greater than a threshold value. If the reliability of the ankle feature points is equal to or greater than the threshold value, the process proceeds to step S108. If the reliability of the ankle feature points is less than the threshold value (including the case where the ankle feature points are not extracted), the process proceeds to step S110.

[0040] If the reliability of the ankle feature point is equal to or greater than the threshold, then in step S108, CPU 14 selects the ankle feature point as a reference feature point and sets its assumed height as Am. On the other hand, if the reliability of the ankle feature point is less than the threshold, then in step S110, CPU 14 determines whether the buttock feature point has been extracted with reliability equal to or greater than the threshold.

[0041] If the reliability of the ankle feature point is less than the threshold but the reliability of the buttock feature point is equal to or greater than the threshold, the process proceeds to step S112. Next, in step S112, the CPU 14 selects the buttock feature point as the reference feature point and sets its expected height as Bm.

[0042] If the reliability of the ankle and buttock feature points is less than the threshold, the process proceeds to step S 114. Next, in step S114, the CPU 14 determines whether the ear feature points have been extracted with reliability equal to or greater than the threshold.

[0043] If the reliability of the ankle and buttock feature points is less than the threshold but the reliability of the ear feature point is equal to or greater than the threshold, the process proceeds to step S116. Next, in step S116, the CPU 14 selects the ear feature point as a reference feature point and sets its assumed height as Cm.

[0044] When any of the feature points has been selected as the reference feature point, then in step S118, the three-dimensional position of the reference feature point is calculated from the coordinates of the reference feature point on the image, information on the position and orientation of the camera in three-dimensional space, and an estimated value of the height of the reference feature point in the world coordinate system, and this is regarded as the estimated position of the selected person.

[0045] On the other hand, if the reliability of the ear feature points is also less than the threshold and no feature points are selected as reference feature points, the process proceeds to step S120 and the three-dimensional positions of the reference feature points are not calculated.

[0046] Next, in step S122, it is determined whether there is a person whose position has not yet been estimated. If there is a person whose position has not yet been estimated, the process returns to step S102, a new person is selected, and the processes from step S102 to step S120 are repeated. On the other hand, if the positions of all detected people have been estimated, the routine ends.

[0047] For the ankle feature points, the expected height may be set for each of the left and right feet, and the expected heights set for each foot may be the same or different. For example, when the reliability of only one ankle is equal to or higher than the threshold, the expected height of the one ankle may be used, and when the reliability of both ankles is equal to or higher than the threshold, the average of the expected heights of the right ankle and the left ankle may be used. The same applies to the buttocks and ears.

[0048] (Reference example position estimation result) FIG. 5 shows a series of frame images 1 to 4 used for position estimation. In this example, two pedestrians are moving with a roughly constant distance between them. The two pedestrians are person_0 (right side of frame image 1) and person_1 (left side of frame image 1). The two pedestrians walked while holding both ends of a tape measure, approximately 2 m long, near their abdomens, with their hands, so as not to let the tape measure sag. Frame images 1 to 4 are part of frames extracted at 10 Hz intervals from an image captured by a two-dimensional camera, and are arranged in chronological order from frame images 1 to 4.

[0049] The position estimation process shown in Fig. 4 was performed using a video including frame images 1 to 4. That is, people were detected in each frame image of the video, feature points were extracted within the detection frame for each detected person, and the estimated position of each person was output.

[0050] Figure 6 shows the estimated positions of two pedestrians plotted in time series on the XY plane of the world coordinate system. However, the estimated positions are not output at 10 Hz intervals precisely because the processing time is long compared to the interval between video clips, and some frames are skipped.

[0051] As indicated by the thick solid arrows or thick dashed arrows in Fig. 6, it can be seen that for both Person_0 and Person_1, positions were suddenly estimated to be 0.5 to 1 m off the trajectory of the estimated position. In most frames, the ankle feature point was selected as the reference feature point, whereas in the frame corresponding to the plot point indicated by the thick dashed arrow, the reliability of the ankle feature point was below the threshold due to the influence of bright areas on the floor, and the buttock feature point was selected as the reference feature point. Similarly, in the frame corresponding to the plot point indicated by the thick solid arrow, the ear feature point was selected as the reference feature point.

[0052] FIG. 7 plots the time transition of the estimated position interval of two pedestrians. In frames where the reference feature point was the ankle, the estimated position interval was between 2 and 2.5 m. In contrast, in frames where the reference feature point was the buttocks or ears, the estimated position interval was about 3 m. When the distance between the surfaces of the abdomen of two pedestrians is about 2 m, the estimated position interval of 2 to 2.5 m is a reasonable value considering the width of the torso. In other words, when the reference feature point was the ankle, the results were generally correct. In contrast, when the reference feature point was the buttocks or ears, the error was large. This is thought to be because, as mentioned above, the height of the buttocks and ears is easily affected by individual differences in height.

[0053] As described above, in the position estimation process according to the reference example, even when the ankle feature point is occluded, other feature points can be selected as reference feature points to perform position estimation. However, if a preset fixed value is used as the assumed height of the other reference point, the estimation error in the person's position will be larger than when the ankle feature point is used as the reference feature point.

[0054] [Position Estimation Processing According to the Present Invention] The position estimation process according to the present invention is similar to the position estimation process according to the reference example in that when an ankle feature point is extracted with a reliability equal to or greater than a threshold, the ankle feature point is used as the reference feature point, and when the reliability of the ankle feature point is less than the threshold, a feature point other than the ankle is selected as the reference feature point.

[0055] In addition, in the position estimation process of the present invention, even when the ankle feature point can be used as a reference feature point, the Z coordinate values ​​of other feature points are calculated and the obtained Z coordinate values ​​are stored in the storage unit. Then, when another feature point is selected as the reference feature point to perform position estimation, the estimated height calculated from the stored Z coordinate value for that feature point is used. However, until a desired amount of Z coordinate values ​​are stored in the storage unit, initial values ​​of the estimated heights of the ankle, buttocks, and ear feature points can be used. As such initial values, preset fixed values ​​(Am, Bm, and Cm, respectively) may be used.

[0056] Hereinafter, the flow of the position estimation process of the present invention will be described with reference to Fig. 8. This process is executed by the CPU 14 of the position estimation device 12. For example, when a user (not shown) instructs the start of the position estimation process, the CPU 14 acquires video data from the camera 10 and starts the position estimation process.

[0057] First, in step S200, the CPU 14 detects people from the image of the frame and assigns an ID to each person. Next, in step S202, the CPU 14 selects one person from the detected people. Next, in step S204, the CPU 14 extracts feature points within the detection frame of the selected person.

[0058] Next, in step S206, the CPU 14 determines whether the ankle feature points are extracted with a reliability equal to or greater than a threshold value. If the reliability of the ankle feature points is equal to or greater than the threshold value, the process proceeds to step S208. If the reliability of the ankle feature points is less than the threshold value (including the case where the ankle feature points are not extracted), the process proceeds to step S216.

[0059] If the reliability of the ankle feature point is equal to or greater than the threshold, then in step S208, CPU 14 selects the ankle feature point as a reference feature point and sets its assumed height as Am. Next, in step S210, CPU 14 calculates the three-dimensional position of the reference feature point from the coordinates of the reference feature point on the image, information on the position and orientation of the camera in three-dimensional space, and the assumed value of the height of the reference feature point in the world coordinate system, and sets this as the estimated position of the selected person.

[0060] Next, in step S211, CPU 14 determines whether or not there are other feature points whose reliability is equal to or greater than the threshold. If other feature points whose reliability is equal to or greater than the threshold have been extracted, then in step S212, CPU 14 calculates Z coordinate values ​​for the other feature points whose reliability is equal to or greater than the threshold. In the following step S214, CPU 14 accumulates the calculated Z coordinate values ​​in storage unit 18. On the other hand, if other feature points whose reliability is equal to or greater than the threshold have not been extracted, the process proceeds to step S226. The method of calculating the Z coordinates of the other feature points will be described later.

[0061] On the other hand, if the reliability of the ankle feature point is less than the threshold, then in step S216, the CPU 14 determines whether the buttock feature point, which has the second highest priority after the ankle, has been extracted with reliability equal to or greater than the threshold. If the reliability of the ankle feature point is less than the threshold but the reliability of the buttock feature point is equal to or greater than the threshold, the process proceeds to step S218.

[0062] Next, in step S218, CPU 14 selects the buttock feature point as the reference feature point. CPU 14 also reads out the Z coordinate values ​​of the accumulated buttock feature points to obtain their representative value, and uses the obtained representative value of the Z coordinate values ​​as the assumed height of the buttock feature point to calculate the three-dimensional position of the reference feature point, which is set as the estimated position of the selected person.

[0063] The representative value of the Z coordinate value may be the average value, median value, mode value, maximum value, or minimum value of the Z coordinate value, but here, the average value of the most recent N Z coordinate values ​​calculated for the same person is used as the representative value of the Z coordinate value. Here, if the accumulated Z coordinate values ​​are less than N, the average value for the shortage may be calculated using the initial value Bm of the assumed height of the feature point of the buttocks. N is preferably in the range of 1 to 20, and can be, for example, 5.

[0064] If the reliability of the ankle and buttock feature points is less than the threshold, the process proceeds to step S220. Next, in step S220, the CPU 14 determines whether the ear feature points, which have the second highest priority after the buttocks, have been extracted with reliability equal to or greater than the threshold. If the reliability of the ankle and buttock feature points is less than the threshold but the reliability of the ear feature points is equal to or greater than the threshold, the process proceeds to step S222.

[0065] Next, in step S222, CPU 14 selects the ear feature point as the reference feature point. CPU 14 also reads out the accumulated Z coordinate values ​​of the ear feature point to obtain a representative value thereof, and calculates the three-dimensional position of the reference feature point using the obtained representative value of the Z coordinate values ​​as the assumed height of the ear feature point, and sets this as the estimated position of the selected person. Note that when using the average value of the most recent N Z coordinate values ​​as the representative value of the Z coordinate value of the ear feature point, if the accumulated Z coordinate values ​​are less than N, the average value for the missing part may be obtained using the initial value Cm of the assumed height of the ear feature point, as in the case of the buttock feature point.

[0066] On the other hand, if the reliability of the ear feature points is also less than the threshold value and no feature points are selected as reference feature points, the process proceeds to step S224 and the three-dimensional positions of the reference feature points are not calculated.

[0067] Next, in step S226, it is determined whether there is a person whose position has not yet been estimated. If there is a person whose position has not yet been estimated, the process returns to step S202, a new person is selected, and the processes from step S202 to step S224 are repeated. On the other hand, if the positions of all detected people have been estimated, the routine ends.

[0068] (How to calculate the Z coordinate value of other feature points) Here, an example of a method for calculating the Z coordinate value of another feature point will be described with reference to Fig. 9 and Fig. 10. As shown in Fig. 9, another feature point P exists on a light ray vector r, so as long as a plane on which the feature point P exists is assumed, the intersection of the light ray vector r and the plane becomes the three-dimensional position of the feature point P.

[0069] Since a person is considered to be standing almost perpendicular to the floor, the plane on which the feature point P exists is the plane perpendicular to the floor and viewed from directly above, as shown in FIG. O Ray vector r heading towards O Assume a plane (hereinafter referred to as the "virtual plane") perpendicular to the

[0070] In other words, the virtual plane has a normal vector (r OX ,r OY , 0) and the reference feature point P O Such a virtual plane is a plane that passes through the coordinates of the reference feature point in the world coordinate system (P OX ,P OY ,P OZ ) can be expressed as the following equation (4).

[0071]

number

[0072] In addition, since the other feature point P is on the light vector r, the coordinates of feature point P in the world coordinate system are (P X ,P Y ,P Z), the following equation (5) holds, similar to the above equation (3), where α is a coefficient.

[0073]

number

[0074] Ray vector r O (r OX ,r OY ,r OZ ) and the coordinates of the reference feature points in the world coordinate system (P OX ,P OY ,P OZ ) has already been obtained, and the coordinates (P CX ,P CY ,P CZ ) is known.

[0075] If it can be assumed that the feature point P is on a virtual plane, the value of α can be obtained by substituting Px and Py in the above formula (5) for x and y in the above formula (4), and this value can be used to calculate Px, Py, and Pz. The value of Pz is the Z coordinate value.

[0076] In reality, the feature points of the elbow and wrist do not exist on a virtual plane. Assuming that the feature point P exists on a virtual plane, i.e., regarding the intersection of the light vector r and the virtual plane as the three-dimensional position of the feature point P, causes a position shift of the feature point, which in turn causes an error in the estimated position of the person. For example, the feature point of the wrist may be shifted by a maximum of the width of the person's body plus the length of the arm. In contrast, the feature points of the buttocks and ears will have a smaller position shift than feature points other than (the ankles, buttocks, and ears).

[0077] (Position estimation result of the present invention) The position estimation process shown in Fig. 8 was performed using a video including frame images 1 to 4 shown in Fig. 5. That is, people were detected in each frame image of the video, feature points were extracted within the detection frame for each detected person, and the estimated position of each person was output.

[0078] Fig. 11 shows the estimated positions output for two pedestrians plotted in time series on the XY plane of the world coordinate system. In most frames, the ankle feature point was selected as the reference feature point, whereas in the frame corresponding to the plot point indicated by the thick dashed arrow, the buttock feature point was selected as the reference feature point, and in the frame corresponding to the plot point indicated by the thick solid arrow, the ear feature point was selected as the reference feature point. Comparing with the result of the position estimation process of the reference example shown in Fig. 6, it can be seen that there are no sudden outliers in the estimated positions.

[0079] Fig. 12 plots the time transition of the estimated position interval of two pedestrians. Comparing with the result of the position estimation process of the reference example shown in Fig. 7, it can be seen that the interval between the estimated positions never exceeds 2.5 m, and the interval between the estimated positions is always within the range of 2 m to 2.5 m.

[0080] As described above, with the position estimation process according to the present invention, even when the ankle feature point is occluded, it is possible to select other feature points as reference feature points and perform position estimation, and the assumed height of the other feature point is found from the Z coordinate value of that feature point that was calculated in the previous frame and stored. This makes it possible to improve the accuracy of estimating the person's position compared to the case where the assumed height of the other feature point is set to a preset fixed value.

[0081] [Variations] It should be noted that the configurations of the position estimation device, the position estimation system, and the program described in the above embodiment are merely examples, and it goes without saying that the configurations may be changed without departing from the spirit of the present invention.

[0082] In the above-described embodiments, the functions realized by software when the CPU executes a program may be realized by hardware such as a semiconductor integrated circuit, or may be realized by a combination of hardware and software.

[0083] In the above embodiment, the various programs are pre-stored (installed) in the storage unit, but the present invention is not limited to this. The various programs may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The various programs may be downloaded from an external device via a network.

[0084] Also, although an example has been described in which the target object of position estimation is a "person," as mentioned above, the target object is not limited to a person. For example, the target object may be a moving body such as an autonomous mobile robot, an AGV (Automated Guided Vehicle), or a transport cart. FIG. 13 shows an example of a transport cart. The transport cart (part other than the cargo) generally has a rectangular parallelepiped shape, and eight points corresponding to the vertices of the rectangular parallelepiped can be extracted as feature points, as shown by the black circles in FIG. 13.

[0085] Although the height of the top plate of the trolley varies, the lower structure is the same because it is necessary to incorporate an AGV. Therefore, for the lower four points of the eight feature points, a fixed value is used as the assumed height, so that the position of the feature point in real space can be accurately estimated from the coordinates of the feature point on the image. Also, for example, if it is known that the upper right vertex and the lower right vertex are in a positional relationship that is almost perpendicular to the floor surface, the Z coordinate value of the upper right vertex can be calculated using the above formula (5). That is, the position estimation process according to the present invention can be applied to the position estimation of such a trolley as well.

[0086] Specifically, when the four lower feature points are extracted with a reliability equal to or higher than a threshold, the position is estimated using the information on the four lower feature points, and the Z coordinate values ​​of the four upper feature points are calculated and stored in a memory unit. When the reliability of the four lower feature points does not reach the threshold due to occlusion by another object, the average value of the Z coordinate values ​​of the four upper feature points stored in the memory unit is used as the assumed height, and the position of the trolley is estimated from the coordinates of the four upper feature points. This makes it possible to accurately estimate the position of the trolley even when the lower part of the trolley is occluded in an environment where trolleys of different heights are mixed.

[0087] [Note] In addition, the following supplementary notes are disclosed in relation to the above-described embodiment. <Appendix 1> An extraction unit that extracts feature points of an object from one of a plurality of images of the object captured in time series; a first estimation unit that, when the extraction unit extracts a predetermined feature point, derives coordinates of the predetermined feature point in real space based on coordinates of the predetermined feature point on the one image and an assumed height of the predetermined feature point in real space, and estimates a position of the object based on the derived coordinates; a derivation unit that, when the extraction unit extracts the predetermined feature point and another feature point other than the predetermined feature point, derives a height coordinate in real space of the other feature point by using the coordinate in real space of the predetermined feature point derived by the first estimation unit, and stores the derived height coordinate in a storage unit; a second estimation unit that, when the extraction unit does not extract the predetermined feature point but extracts the other feature point, estimates a position of the object using coordinates of the other feature point on the one image and coordinates in a height direction stored in the storage unit and corresponding to the extracted other feature point; A position estimation device comprising: <Appendix 2> 2. The position estimation device according to claim 1, wherein each of the predetermined feature points and the other feature points extracted by the extraction unit has a reliability equal to or higher than a threshold. <Appendix 3> the other feature points are a plurality of feature points to which priorities are assigned; the derivation unit derives a height coordinate in real space for each of the plurality of feature points and stores the coordinate in the storage unit; 3. The position estimation device according to claim 1, wherein the second estimation unit estimates the position of the object by using a height coordinate of a feature point to which the highest priority is set among the extracted feature points. <Appendix 4> 4. The position estimation device according to claim 1, wherein the predetermined feature point is located at a lower position than the other feature points. <Appendix 5> 5. The position estimation device according to claim 1, wherein when the object is a person, the predetermined feature point is at least one of a left ankle and a right ankle. <Appendix 6> 6. The position estimation device according to claim 5, wherein, when the object is a person, the other feature points are at least one of a buttock, a shoulder, an ear, and an eye. <Appendix 7> the storage unit stores coordinates of the other feature points in a height direction in real space, the coordinates being derived from other images captured prior to the one image among the plurality of images; and 7. The position estimation device according to claim 1, wherein the second estimation unit estimates the position of the object by using an average value of height coordinates of the other feature points in real space. <Appendix 8> The lead-out portion is The coordinates in real space of the predetermined feature point derived by the first estimation unit are defined as first coordinates; The coordinates in real space of the optical center of an imaging device that captures an image of the object are set as second coordinates; The real coordinates of the other feature points in the real space are defined as third coordinates, assuming a plane including the first coordinates and having a first straight line passing through the first coordinates and the second coordinates as a normal line, and assuming an intersection of the second straight line passing through the second coordinates and the third coordinates and the plane as an approximate coordinate in real space of the other feature point, deriving the approximate coordinate; deriving a height coordinate of the other feature point in real space from the derived approximate coordinate; 8. A position estimation device according to any one of claims 1 to 7. <Appendix 9> A position estimation device according to any one of Supplementary Note 1 to Supplementary Note 8, an imaging device that captures an image of the object; A position estimation system comprising: <Appendix 10> Computer, an extraction unit that extracts feature points of an object from one of a plurality of images of the object captured in time series; a first estimation unit that, when the extraction unit has extracted a predetermined feature point, derives coordinates of the predetermined feature point in real space based on coordinates of the predetermined feature point on the one image and an assumed height of the predetermined feature point in real space, and estimates a position of the object based on the derived coordinates; a derivation unit that, when the extraction unit extracts the predetermined feature point and another feature point other than the predetermined feature point, derives a height coordinate in real space of the other feature point by using the coordinate in real space of the predetermined feature point derived by the first estimation unit, and stores the derived height coordinate in a storage unit; and a second estimation unit that, when the extraction unit does not extract the predetermined feature point but extracts the other feature point, estimates a position of the object using coordinates of the other feature point on the one image and coordinates in a height direction stored in the storage unit and corresponding to the extracted other feature point; A program to function as a [Explanation of symbols]

[0088] 10. Camera 12 Position estimation device 14 CPU 16 Memory 18 Memory section 20 Communications Department 22 Bus 30 Image acquisition unit 32 Feature point extraction unit 34 Reference feature point selection unit 36 Position estimation part 38 Information Gathering Department O Object (person)

Claims

1. an extracting unit that extracts feature points of an object from one of a plurality of images of the object captured in time series; a first estimation unit that, when the extraction unit extracts a predetermined feature point, derives coordinates of the predetermined feature point in real space based on coordinates of the predetermined feature point on the one image and an assumed height of the predetermined feature point in real space, and estimates a position of the object based on the derived coordinates; a derivation unit that, when the extraction unit extracts the predetermined feature point and another feature point other than the predetermined feature point, derives a height coordinate in real space of the other feature point by using the coordinate in real space of the predetermined feature point derived by the first estimation unit, and stores the derived height coordinate in a storage unit; a second estimation unit that, when the extraction unit does not extract the predetermined feature point but extracts the other feature point, estimates a position of the object by using coordinates of the other feature point on the one image and coordinates in a height direction stored in the storage unit and corresponding to the extracted other feature point; A position estimation device comprising:

2. The position estimation device according to claim 1 , wherein each of the predetermined feature points and the other feature points extracted by the extraction unit has a reliability equal to or higher than a threshold value.

3. the other feature points are a plurality of feature points to which priorities are assigned; the derivation unit derives a height coordinate in real space for each of the plurality of feature points and stores the coordinate in the storage unit; The position estimation device according to claim 1 , wherein the second estimation unit estimates the position of the object by using a height coordinate of a feature point that is set to the highest priority among the extracted feature points.

4. The position estimation device according to claim 1 , wherein the predetermined feature point is located at a lower position than the other feature points.

5. The position estimation device according to claim 1 , wherein when the object is a human, the predetermined feature point is at least one of a left ankle and a right ankle.

6. The position estimation device according to claim 5 , wherein when the object is a human, the other feature points are at least one of a hip, a shoulder, an ear, and an eye.

7. the storage unit accumulates coordinates of the other feature points in a height direction in real space, the coordinates being derived from other images captured prior to the one image among the plurality of images; and The position estimation device according to claim 1 , wherein the second estimation unit estimates the position of the object by using an average value of height coordinates of the other feature points in real space.

8. The lead-out portion is The coordinates in real space of the predetermined feature point derived by the first estimation unit are defined as first coordinates; The coordinates in real space of the optical center of an imaging device that captures an image of the object are set as second coordinates; The real coordinates of the other feature points in the real space are defined as third coordinates, assuming a plane including the first coordinates and having a first straight line passing through the first coordinates and the second coordinates as a normal line, and assuming an intersection of the second straight line passing through the second coordinates and the third coordinates and the plane as an approximate coordinate in real space of the other feature point, deriving the approximate coordinate; deriving a height coordinate of the other feature point in real space from the derived approximate coordinate; The position estimation device according to claim 1 .

9. A position estimation device according to any one of claims 1 to 8, an imaging device that captures an image of the object; A position estimation system comprising:

10. Computer, an extraction unit that extracts feature points of an object from one of a plurality of images of the object captured in time series; a first estimation unit that, when the extraction unit extracts a predetermined feature point, derives coordinates of the predetermined feature point in real space based on coordinates of the predetermined feature point on the one image and an assumed height of the predetermined feature point in real space, and estimates a position of the object based on the derived coordinates; a derivation unit that, when the extraction unit extracts the predetermined feature point and another feature point other than the predetermined feature point, derives a height coordinate in real space of the other feature point by using the coordinate in real space of the predetermined feature point derived by the first estimation unit, and stores the derived height coordinate in a storage unit; and a second estimation unit that, when the extraction unit does not extract the predetermined feature point but extracts the other feature point, estimates a position of the object using coordinates of the other feature point on the one image and coordinates in a height direction stored in the storage unit and corresponding to the extracted other feature point; A program to function as a

Images