Camera parameter calculation device, camera parameter calculation method, and camera parameter calculation program

Abstract

A camera parameter calculation device (1) is provided with: an acquisition unit (21) that acquires an image captured by a camera (4); an estimation unit (22) that estimates time-series skeleton coordinates, which are image coordinates of skeleton points of the user, from the time-series images; a feature point calculation unit (23) that calculates a time-series feature point indicating a reference position of the body of the user on the basis of the time-series skeleton coordinates; and a camera parameter calculation unit (24) that calculates camera parameters for mutually converting the image coordinate system and the world coordinate system by minimizing the objective function. The objective function is an objective function based on a distance error between a walking straight line indicating a walking direction of the user and each of a plurality of camera line-of-sight straight lines passing through a plurality of line-of-sight vectors of the camera (4) corresponding to image coordinates of time-series feature points.

Description

Technical Field

[0001] The present disclosure relates to a technique for calculating camera parameters. Background Art

[0002] In order to perform camera calibration for sensing cameras, etc., it is necessary to establish a correspondence between the three-dimensional coordinate values ​​in the three-dimensional space and the pixel positions in the two-dimensional image in a geometry-based method. In the past, a repetitive pattern with a known shape was photographed, and the intersection or the center of the circle was detected from the obtained image, thereby corresponding the three-dimensional coordinates with the pixel positions in the two-dimensional image. The object with the repetitive pattern with a known shape is called a calibration index.

[0003] In addition, conventionally, a method of performing camera calibration based on the image coordinates of a person walking in a straight line shown in a moving image has been proposed. In addition, camera calibration refers to calculating camera parameters.

[0004] For example, in Non-Patent Document 1, camera parameters are calculated by a geometry-based method that uses a calibration index to associate three-dimensional coordinate values ​​in a three-dimensional space with pixel positions in a two-dimensional image.

[0005] Furthermore, for example, in Non-Patent Document 2, the coordinates of a head and feet of a person walking in a straight line shown in a moving image are extracted, and a horizontal line based on a vanishing point is estimated from the trajectories of the head and feet.

[0006] The method of non-patent document 1 requires a process of photographing a repetitive pattern of known shape, a process of detecting intersections or the center of a circle from the obtained image, and a process of matching three-dimensional coordinates with pixel positions in a two-dimensional image. Therefore, the calibration operation is complicated and the camera calibration may not be easily performed.

[0007] Furthermore, in the method of non-patent document 2, in a narrow space such as indoors, when the feet are not reflected, or when a sufficient distance for estimating the vanishing point is not captured, camera calibration may not be performed. Furthermore, in this method, when a lens with distortion such as a fisheye camera or a wide-angle camera is used, it is difficult to estimate the vanishing point, and camera calibration may not be performed.

[0008] In the above-mentioned conventional method, it is difficult to perform camera calibration when it is difficult to set calibration indicators or when it is difficult to ensure a sufficient walking distance required for camera calibration, as in the case of a sensing camera installed indoors.

[0009] Prior Art Literature

[0010] Non-patent literature

[0011] Non-patent literature 1: RYTsai, "A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras andLenses", IEEE Journal of Robotics and Automation, Volume 3, Number 4, pages 323-344, August 1987

[0012] Non-patent document 2: F. Lv, T. Zhao, R. Nevatia, "Camera Calibration from Video of a Walking Human", IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 28, Number 9, pages 1513-1518, September 2006 Summary of the invention

[0013] The present disclosure has been made to solve such a problem, and an object of the present disclosure is to provide a technology that can calculate camera parameters without requiring a calibration index even when the travel distance is short.

[0014] The camera parameter calculation device involved in the present disclosure comprises: an acquisition unit, which acquires an image captured by a camera; an estimation unit, which estimates the time series of skeletal coordinates as the image coordinates of the user's skeletal points based on the time series of images acquired by the acquisition unit; a feature point calculation unit, which calculates the time series of feature points representing the reference position of the user's body based on the time series of skeletal coordinates estimated by the estimation unit; and a camera parameter calculation unit, which calculates the camera parameters for converting the image coordinate system and the world coordinate system by minimizing an objective function, wherein the objective function is an objective function of the distance error between a walking straight line representing the walking direction of the user and each of a plurality of camera sight lines that pass through a plurality of sight vectors of the camera corresponding to the image coordinates of the feature points of the time series.

[0015] According to the present disclosure, the camera parameters can be calculated without calibration index even when the travel distance is short. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1This is a block diagram showing an example of the configuration of the camera parameter calculation system in Embodiment 1 of the present disclosure.

[0017] Figure 2 FIG. 1 is a diagram showing an example of skeleton information including skeleton points estimated by the estimating unit.

[0018] Figure 3 This is a flowchart showing an example of a camera parameter calculation process performed by the camera parameter calculation device according to Embodiment 1 of the present disclosure.

[0019] Figure 4 FIG. 1 is a diagram showing an example of feature points in an image obtained by photographing a walking user.

[0020] Figure 5 This is a diagram showing an example of a polynomial approximation curve for correcting characteristic points of a time series.

[0021] Figure 6 This is a schematic diagram for explaining the calculation of camera parameters by the camera parameter calculation unit.

[0022] Figure 7 This is a block diagram showing an example of the configuration of a camera parameter calculation system in Embodiment 2 of the present disclosure.

[0023] Figure 8 This is a flowchart showing an example of a camera parameter calculation process performed by the camera parameter calculation device according to the second embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] (Insights that form the basis of this disclosure)

[0025] In recent years, camera-based sensing has been implemented, but in order to perform image recognition with high accuracy, camera calibration is required. In the case where the camera is installed in a commercial facility or outdoors, camera calibration can be performed by the constructor. On the other hand, in places where calibration indicators cannot be set and the shooting space is narrow, there is a problem that the previous camera calibration method cannot be used. In particular, such a problem is more likely to occur in a residence where there are restrictions on the installation position of the camera. Therefore, in the previous camera calibration method, it is difficult to calculate the camera parameters of the camera installed indoors.

[0026] In order to solve the above problems, the following technology is disclosed.

[0027] (1) A camera parameter calculation device according to one embodiment of the present invention comprises: an acquisition unit for acquiring an image captured by a camera; an estimation unit for estimating a time series of skeletal coordinates as image coordinates of skeletal points of a user based on the time series of images acquired by the acquisition unit; a feature point calculation unit for calculating a time series of feature points representing a reference position of a body of the user based on the time series of skeletal coordinates estimated by the estimation unit; and a camera parameter calculation unit for calculating camera parameters for converting an image coordinate system into a world coordinate system by minimizing an objective function, wherein the objective function is an objective function of distance errors between a walking straight line representing a walking direction of the user and each of a plurality of camera sight lines passing through a plurality of sight line vectors of the camera corresponding to the image coordinates of the feature points of the time series.

[0028] According to this structure, a plurality of sight line vectors of a camera corresponding to the image coordinates of the feature points of the time series are represented using feature points of the time series representing the reference position of the user's body and camera parameters for converting the image coordinate system into the world coordinate system. Furthermore, the camera parameters are calculated by minimizing an objective function based on the distance error between a walking straight line representing the walking direction of the user and each of a plurality of camera sight lines passing through the plurality of sight line vectors. When there is an error in the camera parameters, the walking straight line does not intersect with the camera sight line, and a distance error occurs between the walking straight line and the camera sight line. The camera parameters are calculated by optimizing the camera parameters so that the distance error is minimized. At this time, if there are the same number of images of the time series as the calculated camera parameters, the camera parameters can be calculated. Therefore, no correction index is required, and the camera parameters can be calculated even when the walking distance is short.

[0029] (2) In the camera parameter calculation device described in (1) above, the plurality of sight line vectors may be calculated using the feature points of the time series calculated by the feature point calculation unit and the camera parameters so as to correspond to the image coordinates of the feature points of the time series.

[0030] According to this configuration, a plurality of line-of-sight vectors can be expressed using the time-series feature points calculated by the feature point calculation unit and the camera parameters.

[0031] (3) In the camera parameter calculation device described in (1) or (2) above, the camera parameter calculation device may further include an output unit that outputs the camera parameter calculated by the camera parameter calculation unit.

[0032] According to this configuration, by storing the output camera parameters, it is possible to perform image processing such as removing image distortion using the stored camera parameters at any time.

[0033] (4) In the camera parameter calculation device described in any one of (1) to (3) above, the camera parameter calculation unit may use the sum of distance errors between the travel line and each of the plurality of camera sight lines as the objective function.

[0034] According to this configuration, since the sum of distance errors is used as the objective function, it is possible to calculate the optimal camera parameters.

[0035] (5) In the camera parameter calculation device described in any one of (1) to (3) above, the camera parameter calculation unit may use the sum of squares of distance errors between the travel line and each of the plurality of camera sight lines as the objective function.

[0036] According to this configuration, since the sum of the squares of the distance errors is used as the objective function, it is possible to calculate the optimal camera parameters.

[0037] (6) In the camera parameter calculation device described in any one of (1) to (5) above, the camera parameter calculation device may also include: a determination unit that determines whether the user is walking straight based on the feature points of the time series calculated by the feature point calculation unit, and when it is determined that the user is walking straight, the camera parameter calculation unit calculates the camera parameters.

[0038] According to this configuration, the camera parameters are calculated when the user is walking straight, but the camera parameters are not calculated when the user is not walking straight. Therefore, the camera parameters can be calculated with high accuracy.

[0039] (7) In the camera parameter calculation device described in any one of (1) to (6) above, the feature point calculation unit may calculate the polynomial approximation curves of the x-coordinates and y-coordinates of the feature points of the time series based on the calculated image coordinates of the feature points of the time series, and use the polynomial approximation curves of the calculated x-coordinates and y-coordinates to correct the values ​​of the x-coordinates and y-coordinates of the feature points of the time series.

[0040] According to this structure, the estimated skeletal coordinates of the time series may contain errors, but the x-coordinate and y-coordinate values ​​of the feature points of the time series are corrected using the polynomial approximation curves of the x-coordinate and y-coordinate respectively. Therefore, the feature points of the time series become straight-line trajectories, and the camera parameters can be calculated with high precision.

[0041] (8) In the camera parameter calculation device described in any one of (1) to (7) above, the camera parameter calculation device may also include: a setting storage unit that pre-stores distortion parameters representing lens distortion of the camera, and the camera parameter calculation unit uses the distortion parameters stored in the setting storage unit as a part of the camera parameters to represent the multiple line of sight vectors.

[0042] According to this configuration, since it is not necessary to calculate distortion parameters indicating the lens distortion of the camera, it is possible to shorten the processing time required for calculating the camera parameters.

[0043] In addition, the present disclosure can be implemented not only as a camera parameter calculation device having the above-mentioned characteristic structure, but also as a camera parameter calculation method that executes characteristic processing corresponding to the characteristic structure of the camera parameter calculation device. In addition, it can also be implemented as a computer program that causes a computer to execute the characteristic processing included in such a camera parameter calculation method. Therefore, in the following another mode, the same effect as the above-mentioned camera parameter calculation device can be achieved.

[0044] (9) Another embodiment of the present invention relates to a camera parameter calculation method in a computer. In the camera parameter calculation method, an image captured by a camera is obtained, and a time series of skeletal coordinates as image coordinates of a skeletal point of a user are estimated based on the obtained time series of images. A time series of feature points representing a reference position of a body of the user is calculated based on the estimated time series of skeletal coordinates. The camera parameters for converting an image coordinate system into a world coordinate system are calculated by minimizing an objective function, wherein the objective function is an objective function based on the distance error between a walking straight line representing the walking direction of the user and each of a plurality of camera sight lines that pass through a plurality of sight line vectors of the camera corresponding to the image coordinates of the feature points of the time series.

[0045] (10) Another embodiment of the present invention relates to a camera parameter calculation program that enables a computer to perform the following functions: an acquisition unit that acquires an image captured by a camera; an estimation unit that estimates a time series of skeletal coordinates that are image coordinates of a skeletal point of a user based on the time series of images acquired by the acquisition unit; a feature point calculation unit that calculates a time series of feature points representing a reference position of a body of the user based on the time series of skeletal coordinates estimated by the estimation unit; and a camera parameter calculation unit that calculates camera parameters for converting an image coordinate system into a world coordinate system by minimizing an objective function, wherein the objective function is an objective function of distance errors between a walking straight line representing the walking direction of the user and each of a plurality of camera sight lines that pass through a plurality of sight line vectors of the camera corresponding to the image coordinates of the feature points of the time series.

[0046] Furthermore, the present disclosure can also be implemented as a camera parameter calculation system that operates by such a camera parameter calculation program. In addition, such a computer program can of course be distributed via a computer-readable non-transitory recording medium such as a CD-ROM or a communication network such as the Internet.

[0047] (11) Another embodiment of the present invention relates to a computer-readable non-temporary recording medium having a camera parameter calculation program recorded thereon, wherein the camera parameter calculation program enables a computer to perform the following functions: an acquisition unit that acquires an image captured by a camera; an estimation unit that estimates a time series of skeletal coordinates that are image coordinates of skeletal points of a user based on the time series of images acquired by the acquisition unit; a feature point calculation unit that calculates a time series of feature points representing a reference position of a body of the user based on the time series of skeletal coordinates estimated by the estimation unit; and a camera parameter calculation unit that calculates camera parameters for converting an image coordinate system into a world coordinate system by minimizing an objective function, wherein the objective function is an objective function of distance errors between a walking straight line representing the walking direction of the user and each of a plurality of camera sight lines that pass through a plurality of sight vectors of the camera corresponding to the image coordinates of the feature points of the time series.

[0048] In addition, the embodiments described below all represent a specific example of the present disclosure. The numerical values, shapes, structural elements, steps, the order of steps, etc. shown in the following embodiments are examples and are not intended to limit the present disclosure. In addition, among the structural elements in the following embodiments, the structural elements that are not recorded in the independent claims representing the highest concept are described as arbitrary structural elements. In addition, in all embodiments, the various contents can also be combined.

[0049] (Implementation method 1)

[0050] Hereinafter, Embodiment 1 of the present disclosure will be described with reference to the drawings.

[0051] Figure 1 This is a block diagram showing an example of the configuration of the camera parameter calculation system in Embodiment 1 of the present disclosure.

[0052] The camera parameter calculation system includes a camera parameter calculation device 1 and a camera 4 .

[0053] In the first embodiment, the camera 4 is a fixed camera installed in a house where a user who is to be sensed and recognized lives. The camera 4 captures the user at a given frame rate and outputs the captured image to the camera parameter calculation device 1 at a given frame rate.

[0054] The camera parameter calculation device 1 is composed of a computer including a processor 2, a memory 3, and an interface circuit (not shown). The processor 2 is, for example, a central processing unit. The memory 3 is, for example, a non-volatile rewritable storage device such as a flash memory, a hard disk drive, or a solid state drive. The interface circuit is, for example, a communication circuit.

[0055] The camera parameter calculation device 1 may be composed of an edge server installed in a residence, a smart speaker installed in a residence, or a cloud server. When the camera parameter calculation device 1 is composed of an edge server or a smart speaker, the camera 4 and the camera parameter calculation device 1 are connected via a local area network. In addition, when the camera parameter calculation device 1 is composed of a cloud server, the camera 4 and the camera parameter calculation device 1 are connected via a wide area communication network such as the Internet. In addition, part of the structure of the camera parameter calculation device 1 may be set on the edge side, and the remaining part may be set on the cloud side.

[0056] The processor 2 includes an acquisition unit 21, an estimation unit 22, a feature point calculation unit 23, a camera parameter calculation unit 24, and an output unit 25. The acquisition unit 21 to the output unit 25 may be implemented by a central processing unit executing a camera parameter calculation program, or may be configured by a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit).

[0057] The acquisition unit 21 acquires an image captured by the camera 4. The acquisition unit 21 stores the acquired image in the frame memory 31.

[0058] The estimation unit 22 estimates the image coordinates of the user's skeleton points, i.e., the skeleton coordinates of the time series, based on the time series images acquired by the acquisition unit 21. The estimation unit 22 estimates multiple skeleton points of the user and the reliability of each skeleton point based on the image read out from the frame memory 31. The estimation unit 22 estimates multiple skeleton points and the reliability by inputting the image into a learned model obtained by machine learning the relationship between the image and the skeleton points. An example of a learned model is a deep neural network. An example of a deep neural network is a convolutional neural network including a convolutional layer and a pooling layer. In addition, the estimation unit 22 can also be composed of a learning model other than a deep neural network.

[0059] Figure 2 1 is a diagram showing an example of skeleton information 201 including skeleton points P1 to P17 estimated by the estimating unit 22 .

[0060] The skeleton information 201 is information indicating skeleton points P1 to P17 of a person. The skeleton information 201 includes, for example, a skeleton point P1 of the left eye, a skeleton point P2 of the right eye, a skeleton point P3 of the left ear, a skeleton point P4 of the right ear, a skeleton point P5 of the nose, a skeleton point P6 of the left shoulder, a skeleton point P7 of the right shoulder, a skeleton point P8 of the left waist, a skeleton point P9 of the right waist, a skeleton point P10 of the left elbow, a skeleton point P11 of the right elbow, a skeleton point P12 of the left wrist, a skeleton point P13 of the right wrist, a skeleton point P14 of the left knee, a skeleton point P15 of the right knee, a skeleton point P16 of the left ankle, and a skeleton point P17 of the right ankle. The skeleton information 201 includes 17 skeleton points P1 to P17.

[0061] The estimation unit 22 estimates the 17 skeleton points P1 to P17. Furthermore, the skeleton information 201 includes links L1 to L12 indicating the connection between the skeleton points. For example, the skeleton information 201 includes a link L1 connecting the skeleton point P6 of the left shoulder and the skeleton point P7 of the right shoulder, a link L2 connecting the skeleton point P6 of the left shoulder and the skeleton point P8 of the left waist, a link L3 connecting the skeleton point P7 of the right shoulder and the skeleton point P9 of the right waist, a link L4 connecting the skeleton point P8 of the left waist and the skeleton point P9 of the right waist, a link L5 connecting the skeleton point P6 of the left shoulder and the skeleton point P10 of the left elbow, a link L6 connecting the skeleton point P7 of the right shoulder and the skeleton point P11 of the right elbow, and a link L7 connecting the skeleton point P7 of the left shoulder and the skeleton point P12 of the right elbow. Link L7 connecting the bone point P10 of the left wrist with the bone point P12, link L8 connecting the bone point P11 of the right elbow with the bone point P13 of the right wrist, link L9 connecting the bone point P8 of the left waist with the bone point P14 of the left knee, link L10 connecting the bone point P9 of the right waist with the bone point P15 of the right knee, link L11 connecting the bone point P14 of the left knee with the bone point P16 of the left ankle, and link L12 connecting the bone point P15 of the right knee with the bone point P17 of the right ankle.

[0062] exist Figure 2 In the figure, the dotted lines are auxiliary lines indicating the contour of the face and the position of the neck. The skeleton points P1 to P17 are represented by the X-coordinate and the Y-coordinate indicating the position on the image. The skeleton information 201 is represented by the parts key that uniquely determines the skeleton points P1 to P17, the coordinates of the skeleton points P1 to P17, and the reliability of the skeleton points P1 to P17. For example, the skeleton information 201 is represented in the form of a dictionary such as {parts key "right eye": [X coordinate, Y coordinate, reliability], part key point "left eye": [X coordinate, Y coordinate, reliability], ..., part key point "left ankle": [X coordinate, Y coordinate, reliability]}.

[0063] The reliability is the reliability estimated by the estimation unit 22 for each of the skeleton points P1 to P17. The reliability expresses the certainty of the estimated skeleton points P1 to P17 in terms of probability. As the value of the reliability increases, the certainty increases. The reliability is represented by a value greater than 0 and less than 1, for example. Figure 2 In the example, the skeleton information 201 is composed of 17 skeleton points P1 to P17, but this is only an example, and the number of skeleton points P1 to P17 can be less than 16 or more than 18. In this case, the learned model is constructed to estimate a given number of skeleton points less than 16 or more than 18. In addition, the skeleton information 201 may also include Figure 2 Skeletal points other than the illustrated skeletal points P1 to P17 (eg, skeletal points such as fingers and mouth).

[0064] The feature point calculation unit 23 calculates feature points based on the multiple skeleton points P1 to P17 estimated by the estimation unit 22. The feature point calculation unit 23 calculates the feature points of the time series representing the reference position of the user's body based on the skeleton coordinates of the time series estimated by the estimation unit 22. In addition, the feature point calculation unit 23 calculates the polynomial approximation curves of the x-coordinates and y-coordinates of the feature points of the time series based on the image coordinates of the calculated feature points of the time series. And, the feature point calculation unit 23 uses the polynomial approximation curves of the calculated x-coordinates and y-coordinates to correct the values ​​of the x-coordinates and y-coordinates of the feature points of the time series. In addition, the details of the feature point calculation unit 23 will be described later.

[0065] The camera parameter calculation unit 24 calculates the camera parameters based on the feature points calculated by the feature point calculation unit 23 and the settings stored in the setting storage unit 32. The camera parameter calculation unit 24 calculates the camera parameters for converting the image coordinate system and the world coordinate system by minimizing the objective function based on the distance error between the walking straight line indicating the walking direction of the user and each of the plurality of camera sight lines passing through the plurality of sight line vectors of the camera corresponding to the image coordinates of the feature points of the time series. Using the feature points of the time series calculated by the feature point calculation unit 23 and the camera parameters, the plurality of sight line vectors are calculated so as to correspond to the image coordinates of the feature points of the time series. In addition, the details of the camera parameter calculation unit 24 will be described later.

[0066] The output unit 25 outputs the camera parameters calculated by the camera parameter calculation unit 24 .

[0067] The memory 3 includes a frame memory 31 and a setting storage unit 32. The frame memory 31 stores the images acquired by the acquisition unit 21 from the camera 4. The frame memory 31 stores the time-series images acquired by the acquisition unit 21.

[0068] The setting storage unit 32 stores the settings of the installed camera 4. The setting storage unit 32 stores distortion parameters indicating lens distortion of the camera 4 in advance. The camera parameter calculation unit 24 uses the distortion parameters stored in the setting storage unit 32 as part of the camera parameters to indicate a plurality of sight line vectors. The setting storage unit 32 will be described in detail later.

[0069] The camera parameter calculation device 1 does not necessarily need to be realized by a separate computer device, and may be realized by a distributed processing system (not shown) including a terminal device and a server. In this case, the acquisition unit 21, the frame memory 31, and the estimation unit 22 may be provided in the terminal device, and the setting storage unit 32, the feature point calculation unit 23, the camera parameter calculation unit 24, and the output unit 25 may be provided in the server. In this case, data transmission between the components is performed via a wide area communication network.

[0070] The above is the configuration of the camera parameter calculation device 1. Next, the camera parameter calculation process of the camera parameter calculation device 1 will be described.

[0071] Figure 3 This is a flowchart showing an example of the camera parameter calculation process of the camera parameter calculation device 1 according to the first embodiment of the present disclosure. The camera parameter calculation process is performed when the camera 4 is installed, and then performed regularly, for example, every week or every month.

[0072] First, in step S1, the acquisition unit 21 acquires an image from the camera 4. The acquisition unit 21 stores the acquired image in the frame memory 31.

[0073] Next, in step S2, the estimation unit 22 obtains a plurality of time-series images from the frame memory 31, and estimates a plurality of skeleton points and the reliability of each skeleton point for each image by inputting the obtained plurality of time-series images into the learned model. Here, for the sake of convenience, only one user is shown in one image, and the case where an image obtained by shooting one walking user is used for calculating the camera parameters is described, but this is an example, and images obtained by shooting multiple walking users may also be used for calculating the camera parameters.

[0074] The estimation unit 22 tracks the user in time series during the estimation of the skeleton point and the reliability. Regarding the tracking of the user, between the images that are continuous in time series, the characters whose centers of gravity of the circumscribed rectangles of the plurality of skeleton coordinates are closest to each other can be set as the same character, or the Hungarian algorithm can be used to determine the combination of the distances between the centers of gravity of the circumscribed rectangles to be the minimum. In addition, the estimation unit 22 determines the user used for the calculation of the camera parameters. For example, the estimation unit 22 selects the user whose time series average of the area of ​​the circumscribed rectangle of the skeleton is the largest.

[0075] In addition, the estimating unit 22 may estimate only the skeleton points without estimating the reliability of the skeleton points.

[0076] Next, in step S3 , the feature point calculation unit 23 calculates feature points based on the skeleton coordinates estimated by the estimation unit 22 .

[0077] Figure 4 FIG. 4 is a diagram showing an example of feature points 401 in an image obtained by photographing a walking user.

[0078] The feature point represents the reference point of the upper body in the image coordinates, and is the center of gravity of the skeleton coordinates of the body. The feature point calculation unit 23 calculates the coordinates of the center of gravity of the four skeleton points P6 to P9 of the two shoulders and the two waists as feature points 401. Skeletal points that are not detected due to obstructions or the orientation of the body are excluded from the calculation of feature points. In addition, when no skeleton point required for calculating the center of gravity is detected at any point, the feature point calculation unit 23 does not calculate the feature point, but instead records the information indicating "no feature point" in correspondence with the image to replace the feature point, and ignores the image in the calculation of the camera parameters.

[0079] In addition, the feature point calculation unit 23 may also calculate the coordinates of the center of gravity of the body using the reliability of the skeleton points P6 to P9 as weights. In addition, the feature point calculation unit 23 may also calculate the coordinates of the center of gravity of the circumscribed rectangle including the body as the feature point 401 instead of the center of gravity of the skeleton points P6 to P9 of the shoulders and waist. In addition, the skeleton points used in the calculation of the center of gravity are not limited to the skeleton points P6 to P9 of the shoulders and waist, but may also include skeleton points of the knees or elbows.

[0080] Furthermore, the feature point calculation unit 23 may calculate feature points based on only the skeleton points whose reliability is greater than a threshold value.

[0081] Next, in step S4, the feature point calculation unit 23 extracts a plurality of time-series feature points of a given section. The given section is a section from the present time to a certain time in the past (for example, 10 seconds ago).

[0082] Next, in step S5, the feature point calculation unit 23 determines whether the given interval includes a walking interval in which the user walks. A walking interval is an interval in which the feature points of multiple time series are continuous and above a threshold value. The threshold value is, for example, 2 seconds. The feature point calculation unit 23 selects feature points of multiple time series of walking intervals. When there are multiple walking intervals in the given interval, the feature point calculation unit 23 selects feature points of multiple time series of the longest walking interval. Here, when it is determined that the given interval does not include a walking interval (No in step S5), the process returns to step S1.

[0083] In addition, when a walking user is photographed from the opposite side, feature points are calculated, but the time series feature points may not move. The feature point calculation unit 23 may determine that the given interval does not include a walking interval when the trajectories of the feature points of the time series do not move.

[0084] In addition, the user may also specify a given interval. For example, the terminal held by the user may also accept input of a shooting start instruction and a shooting end instruction by the user. The user may also start walking after inputting the shooting start instruction, and input the shooting end instruction after finishing walking. The terminal sends the shooting start instruction and the shooting end instruction to the camera parameter calculation device 1. The communication unit (not shown) of the camera parameter calculation device 1 receives the shooting start instruction and the shooting end instruction. The feature point calculation unit 23 may also set the time from the time when the shooting start instruction is input to the time when the shooting end instruction is input as a given interval, and extract feature points of multiple time series of the given interval.

[0085] Furthermore, the input unit (not shown) of the camera parameter calculation device 1 may receive the designation of a predetermined section by an operator from a moving image captured in advance.

[0086] On the other hand, when it is determined that the given interval includes a walking interval (Yes in step S5), in step S6, the feature point calculation unit 23 corrects the calculated feature points of the time series. The skeletal coordinates estimated by the estimation unit 22 contain estimation errors, and the trajectory of the feature points of the time series on the image is not smooth. Therefore, the feature point calculation unit 23 approximates the feature points of the time series with polynomials so that the trajectory of the feature points of the time series on the image moves smoothly.

[0087] Figure 5 is a diagram showing an example of a polynomial approximation curve for correcting characteristic points of a time series. Figure 5 In , the horizontal axis represents the frame, and the vertical axis represents the x-coordinate of the feature point.

[0088] First, the feature point calculation unit 23 calculates a polynomial approximation curve of the x-coordinates of the feature points of the time series based on the calculated image coordinates of the feature points of the time series. The feature point calculation unit 23 fits the polynomial g set to v=g(u) to the feature points of the time series when depicting the value (v) of the x-coordinate (horizontal direction of the image) of the feature points of the time series relative to the value (u) of the frame (time) of the feature points of the time series, and calculates the polynomial approximation curve. The degree N of the polynomial g is, for example, 4.

[0089] The feature point calculation unit 23 corrects the x-coordinate value of the feature point of the time series using the calculated polynomial approximation curve of the x-coordinate. The feature point calculation unit 23 calculates the corrected value v of the x-coordinate of the feature point by substituting the value u of the calculated value of the feature point before correction into the polynomial g.

[0090] Similarly, the feature point calculation unit 23 calculates the correction value of the y coordinate (vertical direction of the image) of the feature point. That is, the feature point calculation unit 23 calculates the polynomial approximation curve of the y coordinate of the feature point of the time series based on the calculated image coordinates of the feature point of the time series. And, the feature point calculation unit 23 corrects the value of the y coordinate of the feature point of the time series using the calculated polynomial approximation curve of the y coordinate.

[0091] Next, in step S7 , the camera parameter calculation unit 24 calculates camera parameters based on the time-series feature points calculated by the feature point calculation unit 23 and the setting values ​​of the user's home stored in the setting storage unit 32 .

[0092] Next, in step S8 , the output unit 25 outputs the camera parameters calculated by the camera parameter calculation unit 24 .

[0093] The above-described procedure enables calibration of the camera 4 installed indoors for sensing. In particular, the first embodiment is useful for calibration of a camera in a house where there are many restrictions on the installation position of the camera 4.

[0094] The following is an example of camera parameters in the present disclosure. The conversion formula from the world coordinate system to the image coordinate system is expressed by the following mathematical formulas (1) to (4). The camera parameters are parameters of the projection formula that projects the world coordinates to the image coordinates. Γ(η) in mathematical formula (3) is a projection function that represents lens distortion. In a pinhole camera model as an example, Γ(η)=ftan(η). In addition, f is the focal length and η is the angle of incidence.

[0095] [Mathematical formula 1]

[0096]

[0097] Here, (X, Y, Z) is the world coordinate value, and (x, y) is the image coordinate value. x , C y ) is the image coordinate of the principal point of camera 4, r 11 ~r 33 are the components of the 3x3 rotation matrix R representing the rotation relative to the world coordinate reference, (T X , T Y , T Z ) is the translation vector relative to the world coordinate system, d x and y is the horizontal and vertical pixel pitch of the image sensor of the camera 4. In the mathematical formulas (1) to (4), d x d y , C x , C y 、r 11 ~r 33 , T X , T Y , T Z are the camera parameters.

[0098] Mathematical formulas (1) to (4) represent the conversion from (X, Y, Z) to (x, y). When converting from (x, y) to (X, Y, Z) on the unit sphere, the inverse function or inverse matrix of mathematical formulas (1) to (4) is used to perform the conversion. In addition, the rotation matrix is ​​regular and the inverse matrix can always be calculated, and the 4x4 matrix containing the translation vector is also regular. Therefore, for example, when the inverse function of Γ can be calculated as in a pinhole camera, the conversion from (x, y) to (X, Y, Z) on the unit sphere can be performed.

[0099] The lens distortion Γ can be calculated by pre-calibrating the camera 4, or by using the design value of the lens, or by assuming a pinhole camera. The lens distortion Γ is represented by a function or a table equivalent to the function. The lens distortion Γ is stored in the setting storage unit 32. The camera parameter calculation unit 24 obtains the lens distortion Γ from the setting storage unit 32. For ease of explanation, the user's walking is assumed to be uniform linear motion. For uniform linear motion, the position of the feature point in the three-dimensional space (the center of gravity of the body) exists on a straight line. In addition, when the user's walking is non-uniform linear motion, the camera parameter calculation unit 24 expresses the user's walking speed through multiple parameters, and includes the multiple parameters representing the walking speed in the camera parameters, thereby enabling calculation.

[0100] In addition, the setting storage unit 32 may also store the principal point image coordinates (C x , C y ) and the horizontal and vertical pixel pitch d of the image sensor of the camera 4 x and y .

[0101] Furthermore, by setting the user's walking to uniform linear motion, the calculated center of gravity position (feature point) of the human body can be used as a correction index in non-patent document 1. Furthermore, since the calculated feature point is unstable compared to the correction index, the feature point is corrected so that the feature point in the time series image changes smoothly.

[0102] Next, calculation of camera parameters by the camera parameter calculation unit 24 will be described.

[0103] Figure 6 FIG. 2 is a schematic diagram for explaining the calculation of camera parameters by the camera parameter calculation unit 24. Figure 6 In the example, the user walks from the door toward the inner side of the corridor. The camera 4 is arranged above the inner side of the corridor.

[0104] Consider the image coordinate position p of the i-th feature point in N images obtained by photographing a user walking at a constant speed w. i (x i ,y i ), and the world coordinate position P corresponding to the feature point i (X i , Y i , Z i )=(w(i-1)+X0,Y0,Z0)。 In addition, i represents the frame index. In addition, (X0,Y0,Z0) is the initial position of the three-dimensional position corresponding to the feature point. When the camera parameter is Ω, p i (x i ,y i)=Ω(P i (X i , Y i , Z i )) holds, we can get N equations. Or, we can get P i (X i , Y i , Z i )=Ω -1 (p i (x i ,y i Here, since the walking straight line L representing the trajectory of the user's feature points walk The camera sight line L represents the optical axis in the world coordinate system of the camera 4. eye Since the N equations are not consistent, they are linearly independent. In addition, the special conditions such as the feature points of the time series not moving are removed. That is, the camera parameters can be calculated by defining the objective function using the camera parameters and performing nonlinear optimization on the objective function.

[0105] The definition of the objective function is explained below. When P is calculated using the inverse function of the camera parameter Ω i (X i , Y i , Z i )=Ω -1 (p i (x i ,y i )) In the case of P, since the scale is unknown, i It does not exist in the world coordinates of a single point, but exists on the camera's sight line L. eye That is, we can get the same as p i (x i ,y i ) corresponds to the camera sight line L eye，i When there is an error in the camera parameters, the trajectory of the body's center of gravity, i.e., the walking straight line L, which is the position of the feature point in the three-dimensional space walk The line of sight with the camera L eye，i Do not cross, walk in straight line L walk and the camera sight line L eye，i With distance error d i (The distance between two straight lines). Camera sight line L eye，i The direction vector is V e，i =P cam P i (In P cam P i A right-pointing arrow is attached to it, indicating a vector), and the parameter variable is s eye , and passes through the camera position P camCamera sight line L eye，i It is expressed by the following mathematical formula (5).

[0106] L eye，i =V e，i s eye +P cam … (5)

[0107] In addition, in mathematical formula (5), P cam is the camera position, V e，i is from the camera position P cam The world coordinate position P towards the feature point i (X i , Y i , Z i ) direction vector, s eye is a parameter variable. The camera position P cam With the translation vector T (T X , T Y , T Z ). Based on the image coordinate position p of the feature point i (x i ,y i ), through P i (X i , Y i , Z i )=Ω -1 (p i (x i ,y i )) to calculate P i (X i , Y i , Z i ).

[0108] In addition, the walking straight line L walk The walking direction vector is m (m X , m Y , m Z ), parameter variable is s walk , and a straight line passing through the walking start position P0. Walking straight line L walk It is expressed by the following mathematical formula (6).

[0109] L walk =ms walk +P0…(6)

[0110] Based on the walking straight line L walk The line of sight with the camera L eye，i The distance error d iThe camera parameter calculation unit 24 uses the time series feature points calculated by the feature point calculation unit 23 and the camera parameter Ω for converting the image coordinate system to the world coordinate system to represent the plurality of sight line vectors of the camera 4 corresponding to the image coordinates of the time series feature points. The camera parameter calculation unit 24 calculates the camera parameter Ω by minimizing the objective function, which is based on the walking straight line L indicating the walking direction of the user. walk and multiple camera sight lines L passing through multiple sight vectors eye，i The distance error d of each i In addition, the walking straight line L walk The line of sight with the camera L eye，i The distance error d i It can be calculated using a mathematical formula for calculating the distance between straight lines.

[0111] The camera parameter calculation unit 24 calculates the travel straight line L walk Line of sight L with multiple cameras eye，i The distance error d of each i The sum of is used as the objective function.

[0112] Alternatively, the camera parameter calculation unit 24 may calculate the travel straight line L walk Line of sight L with multiple cameras eye，i The distance error d of each i The sum of the squares of is used as the objective function.

[0113] The unknowns are the camera's rotation matrix R (3 degrees of freedom), translation vector T (T X , T Y , T Z ), walking speed w, walking starting position P0 (X0, Y0, Z0), walking direction vector m (m X , m Y , m Z ). The total number of degrees of freedom is 13, so when N is greater than or equal to 13, the camera parameter calculation unit 24 can calculate the camera parameters.

[0114] For nonlinear optimization of the objective function, for example, the Levenberg-Marquardt method is used. In addition, examples of the initial values ​​of the parameters are as follows. The tilt angle of the camera 4 is -20°, the pan angle is 0°, and the roll angle is 0°. The translation vector T of the camera 4 is X for Figure 6 The depth shown is 2.4m, T Y for Figure 6 The width shown is 0m, T Z for Figure 6 The height is 1.8m. The walking speed w is 3km / h. The walking start position P0 is (0, 0.5, 0.9) [m]. The walking direction vector m is (1, 0, 0) [m]. In addition, the camera position P cam (Translation vector T) T X , T Y and T Z Pre-measured values ​​can be used, or variables calculated from the camera parameters can be extracted.

[0115] In this way, the feature points of the time series representing the reference position of the user's body and the camera parameters for converting the image coordinate system and the world coordinate system are used to represent multiple sight line vectors of the camera corresponding to the image coordinates of the feature points of the time series. In addition, the camera parameters are calculated by minimizing the objective function, which is an objective function based on the distance error between the walking straight line representing the user's walking direction and each of the multiple camera sight lines passing through the multiple sight line vectors. When there is an error in the camera parameters, the walking straight line and the camera sight line do not intersect, and a distance error occurs between the walking straight line and the camera sight line. The camera parameters are calculated by optimizing the camera parameters so that the distance error is minimized. At this time, if there are the same number of time series images as the calculated camera parameters, the camera parameters can be calculated. Therefore, no correction index is required, and the camera parameters can be calculated even when the walking distance is short.

[0116] The above-mentioned non-patent document 2 estimates the vanishing point based on the trajectory of the head and feet of the person, so the camera parameters cannot be stably calculated unless the walking distance is long. On the other hand, in the present embodiment 1, the image coordinates of the center of gravity position of the body are reflected in the objective function without using the vanishing point, so the camera parameters can be stably calculated even for a camera with lens distortion.

[0117] (Implementation method 2)

[0118] In the first embodiment, whether the photographed user is walking straight is not considered. If the photographed user changes direction while walking, the calculation accuracy of the camera parameters may decrease. Therefore, in the second embodiment, it is determined whether the photographed user is walking straight.

[0119] In the following description, only the differences from the first embodiment are described.

[0120] Figure 7 This is a block diagram showing an example of the configuration of a camera parameter calculation system in Embodiment 2 of the present disclosure.

[0121] The camera parameter calculation system in the second embodiment includes a camera parameter calculation device 1A and a camera 4. In the second embodiment, the same configurations as those in the first embodiment are denoted by the same reference numerals, and description thereof will be omitted.

[0122] The camera parameter calculation device 1A is constituted by a computer including a processor 2A, a memory 3 and an interface circuit (not shown). The processor 2A includes an acquisition unit 21 , an estimation unit 22 , a feature point calculation unit 23 , a camera parameter calculation unit 24 , an output unit 25 and a determination unit 26 .

[0123] The determination unit 26 determines whether the user is walking straight based on the time-series feature points calculated by the feature point calculation unit 23. When the determination unit 26 determines that the user is walking straight, the camera parameter calculation unit 24 calculates the camera parameters.

[0124] Figure 8 This is a flowchart showing an example of a camera parameter calculation process performed by the camera parameter calculation device 1A according to the second embodiment of the present disclosure.

[0125] The processing of steps S11 to S15 is Figure 3 The processes in steps S1 to S5 shown are the same, and thus description thereof will be omitted.

[0126] Next, in step S16, the determination unit 26 determines whether the user is walking straight toward the camera 4. The determination unit 26 calculates a body index for determining whether the front of the user is facing the direction of the camera 4 for each frame based on the following mathematical formula (7).

[0127] Body index = (x-coordinate of left shoulder + x-coordinate of left waist) - (x-coordinate of right shoulder + x-coordinate of right waist) ... (7)

[0128] When the user walks straight toward the camera 4 and the front of the user is photographed, the body index is always a positive value. On the other hand, when the user bends or turns back while walking and the back of the user is photographed, the body index is a negative value. Therefore, by determining whether the sign of the body index is positive or negative, it is possible to determine whether the front of the user is facing the direction of the camera 4.

[0129] The determination unit 26 determines whether the ratio of the number of frames in which the body index is positive to the total number of frames in the walking interval is greater than a threshold value. The threshold value is, for example, 0.7. When the ratio of the number of frames in which the body index is positive to the total number of frames in the walking interval is greater than the threshold value, the determination unit 26 determines that the user is walking straight toward the camera 4. On the other hand, when the ratio of the number of frames in which the body index is positive to the total number of frames in the walking interval is less than the threshold value, the determination unit 26 determines that the user is not walking straight toward the camera 4.

[0130] When it is determined that the user is not walking straight toward the camera 4 (NO in step S16 ), the process returns to step S11 .

[0131] On the other hand, when it is determined that the user is walking straight toward the camera 4 (YES in step S16 ), the process proceeds to step S17 .

[0132] The processing of steps S17 to S19 is Figure 3 The processes of steps S6 to S8 shown are the same, and thus their description is omitted.

[0133] In the second embodiment, the camera parameters can be calculated with high accuracy by excluding feature points calculated based on walking in directions other than straight ahead, which have an adverse effect on the calculation of the camera parameters.

[0134] (Variation)

[0135] The camera parameter calculation device involved in one or more embodiments of the present disclosure is described above based on the implementation, but the present disclosure is not limited to the implementation. As long as it does not deviate from the main purpose of the present disclosure, various modifications that can be thought of by those skilled in the art are applied to the embodiments, and the methods constructed by combining the structural elements in different embodiments can also be included in the scope of one or more embodiments of the present disclosure.

[0136] In addition, in each of the above-mentioned embodiments, each structural element is composed of dedicated hardware, but it can also be implemented by executing a software program suitable for each structural element. Each structural element can also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory. In addition, the program can also be implemented by another independent computer system by recording the program in a recording medium and transferring it, or transferring the program via a network.

[0137] Part or all of the functions of the device involved in the embodiments of the present disclosure are typically implemented as LSI (Large Scale Integration) as an integrated circuit. They can be single-chip individually or in combination. In addition, integrated circuitization is not limited to LSI, and can also be implemented by a dedicated circuit or a general-purpose processor. It is also possible to use a programmable FPGA (Field Programmable Gate Array) after LSI manufacturing, or a reconfigurable processor that can reconfigure the connection and settings of the circuit monomer inside the LSI.

[0138] Furthermore, a part or all of the functions of the apparatus according to the embodiments of the present disclosure may be implemented by executing a program by a processor such as a CPU.

[0139] In addition, all the numbers used in the above description are exemplified for specifically describing the present disclosure, and the present disclosure is not limited to the exemplified numbers.

[0140] Industrial Applicability

[0141] The technology according to the present disclosure does not require a calibration index and can calculate camera parameters even when the travel distance is short, and is therefore useful as a technology for calculating camera parameters.

Claims

1. A camera parameter calculation device, comprising: An acquisition unit, which acquires an image captured by a camera; an estimating unit that estimates time-series bone coordinates as image coordinates of a user's bone points based on the time-series images acquired by the acquiring unit; a feature point calculation unit that calculates a time-series feature point representing a reference position of the user's body based on the time-series skeletal coordinates estimated by the estimation unit; and A camera parameter calculation unit calculates camera parameters for converting an image coordinate system into a world coordinate system by minimizing an objective function, wherein the objective function is an objective function based on distance errors between a walking straight line representing the walking direction of the user and a plurality of camera sight lines that pass through the image coordinates of the feature points of the time series and correspond to the plurality of sight vectors of the camera.

2. The camera parameter calculation device according to claim 1, wherein: The plurality of sight line vectors are calculated using the time-series feature points calculated by the feature point calculation unit and the camera parameters so as to correspond to the image coordinates of the time-series feature points, respectively.

3. The camera parameter calculation device according to claim 1 or 2, wherein: The camera parameter calculation device further includes an output unit that outputs the camera parameter calculated by the camera parameter calculation unit.

4. The camera parameter calculation device according to claim 1 or 2, wherein: The camera parameter calculation unit uses the sum of distance errors between the walking straight line and each of the plurality of camera sight lines as the objective function.

5. The camera parameter calculation device according to claim 1 or 2, wherein: The camera parameter calculation unit uses the sum of squares of distance errors between the walking straight line and each of the plurality of camera sight lines as the objective function.

6. The camera parameter calculation device according to claim 1 or 2, wherein: The camera parameter calculation device further includes: a determination unit that determines whether the user is walking straight based on the feature points of the time series calculated by the feature point calculation unit, The camera parameter calculation unit calculates the camera parameters when it is determined that the user is walking straight.

7. The camera parameter calculation device according to claim 1 or 2, wherein: The feature point calculation unit calculates the polynomial approximation curves of the x-coordinates and y-coordinates of the feature points of the time series based on the calculated image coordinates of the feature points of the time series, and uses the calculated polynomial approximation curves of the x-coordinates and y-coordinates to correct the values ​​of the x-coordinates and y-coordinates of the feature points of the time series.

8. The camera parameter calculation device according to claim 1 or 2, wherein: The camera parameter calculation device further comprises: a setting storage unit for pre-storing distortion parameters representing lens distortion of the camera; The camera parameter calculation unit uses the distortion parameter stored in the setting storage unit as a part of the camera parameters to express the plurality of line-of-sight vectors.

9. A camera parameter calculation method is a camera parameter calculation method in a computer, In the camera parameter calculation method, Get the image captured by the camera. Based on the acquired time series images, the time series bone coordinates, which are the image coordinates of the user's bone points, are estimated. Based on the estimated skeletal coordinates of the time series, a time series of feature points representing a reference position of the user's body is calculated, The camera parameters for converting the image coordinate system and the world coordinate system are calculated by minimizing the objective function, wherein the objective function is an objective function based on the distance error between a walking straight line representing the walking direction of the user and each of a plurality of camera sight lines that pass through the image coordinates of the feature points of the time series and correspond to the plurality of sight vectors of the camera.

10. A camera parameter calculation program, which enables a computer to perform the following functions: An acquisition unit, which acquires an image captured by a camera; an estimating unit that estimates time-series bone coordinates as image coordinates of a user's bone points based on the time-series images acquired by the acquiring unit; a feature point calculation unit that calculates a time-series feature point representing a reference position of the user's body based on the time-series skeletal coordinates estimated by the estimation unit; and A camera parameter calculation unit calculates camera parameters for converting an image coordinate system into a world coordinate system by minimizing an objective function, wherein the objective function is an objective function based on distance errors between a walking straight line representing the walking direction of the user and a plurality of camera sight lines that pass through the image coordinates of the feature points of the time series and correspond to the plurality of sight vectors of the camera.

Images