DETECTION APPARATUS, DETECTION METHOD, AND DETECTION PROGRAM

The detection device estimates skeletal information and calculates camera parameters to accurately detect social distance from a two-dimensional image, addressing the limitations of existing techniques and enhancing social distance measurement precision.

JP7585624B2Active Publication Date: 2024-11-19NEC CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2020096895
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-06-03
Publication Date
2024-11-19
Estimated Expiration
2040-06-03

AI Technical Summary

Technical Problem

Existing techniques struggle to accurately calculate social distance from a two-dimensional image, as they typically require a distance image input and do not account for position and distance estimation from a two-dimensional image.

Method used

A detection device and method that estimates skeletal information from a two-dimensional image, calculates camera parameters, and uses these parameters to detect social distance by analyzing joint points and bone line segments in the image.

Benefits of technology

Enables accurate detection of social distance from a two-dimensional image, improving the precision of social distance measurement and risk assessment in public spaces.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007585624000036
    Figure 0007585624000036
  • Figure 0007585624000037
    Figure 0007585624000037
  • Figure 0007585624000038
    Figure 0007585624000038
Patent Text Reader

Abstract

To provide a detection apparatus for detecting social distance from a person from a two-dimensional image, a detection method, and a detection program.SOLUTION: A detection apparatus 10 includes: an estimation unit 11 which estimates skeleton information of a person included in an image on the basis of the two-dimensional image captured by a camera; a calculation unit 12 which calculates a camera parameter on the basis of the estimated skeleton information of the person; and a detection unit 13 which detects a social distance from the person by using the calculated camera parameter.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a detection device, a detection method, and a detection program. [Background technology]

[0002] COVID-19 infections are spreading around the world, having a major impact on society and the economy. In order to prevent droplet infection, one of the routes of infection for COVID-19, it is necessary to maintain social distancing, which is the physical distance between people.

[0003] As a related technique, for example, Patent Document 1 is known. Patent Document 1 describes that in a passerby behavior analysis system, the position coordinates of passersby passing through a space are obtained, and the approach distance between passersby and the congestion degree are calculated. Other known techniques related to camera calibration include Patent Documents 2 and 3. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-346617 A [Patent Document 2] Patent No. 6011548 [Patent Document 3] International Publication No. 2015 / 045834 Summary of the Invention [Problem to be solved by the invention]

[0005] According to the technology described in Patent Document 1, it may be possible to calculate the distance between people from the position coordinates of the people in space. However, in related technologies such as Patent Document 1, it is assumed that a distance image is input, and calculation of the position and distance of a person from a two-dimensional image (image plane) is not taken into consideration. Therefore, in related technologies such as Patent Document 1, there is a problem that it is difficult to detect the social distance between people from a two-dimensional image.

[0006] In view of such problems, the present disclosure has an object to provide a detection device, a detection method, and a detection program capable of detecting the social distance of people from a two-dimensional image. [Means for solving the problem]

[0007] The detection device of the present disclosure includes an estimation unit that estimates skeletal information of a person included in an image based on a two-dimensional image captured by a camera, a calculation unit that calculates camera parameters of the camera based on the estimated skeletal information of the person, and a detection unit that detects the social distance of the person using the calculated camera parameters.

[0008] The detection method disclosed herein estimates skeletal information of a person included in a two-dimensional image captured by a camera based on the image, calculates camera parameters of the camera based on the estimated skeletal information of the person, and detects the social distance of the person using the calculated camera parameters.

[0009] The detection program of the present disclosure is a detection program for causing a computer to execute a process of estimating skeletal information of a person included in an image based on a two-dimensional image captured by a camera, calculating camera parameters of the camera based on the estimated skeletal information of the person, and detecting the social distance of the person using the calculated camera parameters. Effect of the Invention

[0010] According to the present disclosure, it is possible to provide a detection device, a detection method, and a detection program capable of detecting the social distance of a person from a two-dimensional image. [Brief description of the drawings]

[0011] [Figure 1] 1 is a configuration diagram showing an overview of a detection device according to an embodiment; [Diagram 2] FIG. 1 is a configuration diagram showing a configuration of a social distance analysis system according to a first embodiment. [Diagram 3] 4 is a flowchart showing the operation of the social distance analysis device according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of an image processed by the social distance analysis device according to the first embodiment. [Diagram 5] 6A to 6C are diagrams illustrating a detection example of the joint detection process according to the first embodiment. [Figure 6] 4 is a flowchart showing a risk determination process according to the first embodiment. [Figure 7] 5 is a flowchart showing a determination result display process according to the first embodiment. [Figure 8] FIG. 2 is a diagram showing a display example of the social distance analysis device according to the first embodiment. [Figure 9] FIG. 2 is a diagram showing a display example of the social distance analysis device according to the first embodiment. [Figure 10] FIG. 11 is a diagram showing the relationship between a camera coordinate system and a world coordinate system according to the second embodiment. [Figure 11] FIG. 11 is a configuration diagram showing a configuration of a camera calibration unit according to the second embodiment. [Figure 12] 10 is a flowchart showing an operation of a camera calibration unit according to the second embodiment. [Figure 13] 13 is a flowchart showing the operation of the social distance analysis device according to the third embodiment. [Figure 14] 13 is a flowchart showing the operation of the social distance analysis device according to the fourth embodiment. [Figure 15]FIG. 13 is a diagram showing the relationship between a camera coordinate system and a world coordinate system according to the fifth embodiment. [Figure 16] FIG. 13 is a configuration diagram showing the configuration of a camera calibration unit according to the fifth embodiment. [Figure 17] 13 is a flowchart showing the operation of a camera calibration unit according to the fifth embodiment. [Figure 18] 13 is a flowchart showing the operation of the social distance analysis device according to the sixth embodiment. [Figure 19] 13 is a flowchart showing the operation of the social distance analysis device according to the seventh embodiment. [Figure 20] FIG. 2 is a configuration diagram showing an overview of the hardware of a computer according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment will be described with reference to the drawings. In the drawings, the same elements are denoted by the same reference numerals, and repeated explanations will be omitted as necessary.

[0013] (Outline of the embodiment) 1 shows an overview of a detection device according to an embodiment. As shown in FIG 1, a detection device 10 according to an embodiment includes an estimation unit 11, a calculation unit 12, and a detection unit 13.

[0014] The estimation unit 11 estimates skeletal information of a person included in an image based on a two-dimensional image captured by a camera. The calculation unit 12 calculates camera parameters based on the estimated skeletal information of the person. The detection unit 13 detects the social distance of the person using the calculated camera parameters.

[0015] In this manner, in the embodiment, skeletal information of a person is estimated from a two-dimensional image, camera parameters are calculated based on the estimated skeletal information of the person, and the social distance of the person is detected using the camera parameters. This makes it possible to detect the social distance of a person from only the information on the two-dimensional image plane. In particular, by calculating the camera parameters based on the skeletal information of a person estimated from an image, the camera parameters can be calculated with high accuracy, and it becomes possible to improve the accuracy of detecting the social distance.

[0016] (Embodiment 1) Hereinafter, the first embodiment will be described with reference to the drawings. Fig. 2 shows an example of the configuration of a social distance analysis system 1 according to the present embodiment. The social distance analysis system 1 is a system for preventing the spread of infectious diseases in public spaces and the like by measuring whether social distance is being maintained between people who are walking or the like from camera footage and displaying the measurement results. As shown in Fig. 2, the social distance analysis system 1 includes a social distance analysis device 100 and a camera 200.

[0017] The camera 200 is an imaging unit such as a surveillance camera that is installed at a predetermined surveillance position and captures images of people in a surveillance space (imaging space). The camera 200 may be a camera with a fixed orientation and installation location, a camera whose orientation can be changed such as a PTZ (Pan Tilt Zoom) camera, or a mobile camera mounted on a mobile body such as a drone. The camera 200 and the social distance analysis device 100 are connected to each other via an arbitrary network so that they can communicate with each other. The camera 200 and the social distance analysis device 100 may be integrated into a single device.

[0018] The social distance analysis device 100 is an analysis device that analyzes the social distance of a person based on an image from the camera 200, and is also a display device that displays the analysis result. The social distance analysis device 100 can also be said to be a detection device that detects the social distance of a person based on an image. The social distance analysis device 100 includes an input unit 101, an acquisition unit 102, a camera calibration unit 103, a human joint detection unit 104, a coordinate calculation unit 105, a storage unit 106, a social distance determination unit 107, and a display unit 108. Note that the configuration of each unit (block) is an example, and may be configured with other units as long as the method (operation) described later is possible. In addition, the social distance analysis device 100 is realized by a computer device such as a personal computer or a server that executes a program, for example, but may be realized by one device or multiple devices. For example, the storage unit 106 and the display unit 108 may be external devices.

[0019] The input unit 101 is an input interface that acquires information input by a user operating the social distance analysis device 100. For example, the user is a monitor who monitors people who are not maintaining social distance from images of a surveillance camera. The input unit 101 is, for example, a GUI (Graphical User Interface), and information according to user operations is input from an input device such as a keyboard, a mouse, or a touch panel. For example, the input unit 101 accepts information necessary for the camera parameter calculation process of the camera calibration unit 103, and also accepts user operations for display processing of the display unit 108.

[0020] The acquisition unit 102 acquires a two-dimensional image including a person captured by the camera 200. The acquisition unit 102 acquires, for example, an image including a person captured by the camera 200 during a predetermined period (a video including a plurality of images in time series). Note that acquisition is not limited to from the camera 200, and an image including a person prepared in advance may be acquired from a database or the like.

[0021] The camera calibration unit (calculation unit) 103 calculates the camera parameters of the camera 200 based on the image acquired from the camera 200 (performs camera calibration). The camera parameters (imaging parameters) are parameters for converting a length in a three-dimensional real world space into a length in a two-dimensional image plane. The camera parameters include internal parameters such as the focal length of the camera, lens distortion, and optical center coordinates, and external parameters such as the position (translation vector) and angle (rotation matrix) of the camera. The camera calibration unit 103 stores the calculated camera parameters (position, angle, focal length, etc.) in the storage unit 106.

[0022] The camera calibration unit 103 calculates the camera parameters based on a normal vector (a line segment perpendicular to a reference plane (ground)) extracted from an image. As a calculation method using a normal vector, for example, the calibration method described in Patent Document 2 may be used, or other methods may be used. The camera calibration unit 103 calculates the camera parameters based on the length of the normal vector extracted from an image and a given reference length. For example, the normal vector is acquired from a bone line segment (skeleton information) between joint points of a person detected by the human joint detection unit 104, but may be specified by a user input. The bone line segment used for the normal vector may be a line segment of any bone included in the skeleton of the person. Although a line segment of a bone at the foot (a line segment between the ankle and the knee) may be used, a line segment that is stable in a nearly vertical state when a person is walking is preferable, for example, a line segment of the spine (a line segment between the waist and the neck) is used. In addition, the given reference length is a length corresponding to the line segment used for calculation, for example, the height of the person or the length of each bone (average value, etc.). The reference length used for the calculation may be input by the user or may be stored in advance in the storage unit 106. Note that the normal vector may be perpendicular to the ground, and may be a vertical line of a car, truck, or the like, instead of a human skeleton. Furthermore, the camera parameters may be calculated by capturing an image of a calibration object (an object whose length is known in advance) with the movement of a person or the like in the image stopped. Furthermore, if the camera 200 is a camera whose camera angle and zoom value can be changed, such as a PTZ camera, a difference value changed from the reference position may be acquired and reflected in the camera parameters.

[0023] The human joint detection unit (estimation unit) 104 detects (estimates) joint points of a person in a two-dimensional image based on an image acquired from the camera 200. For example, the human joint detection unit 104 uses a joint estimation technology (skeleton estimation technology) such as OpenPose that utilizes machine learning. The human joint detection unit 104 uses the joint estimation technology to detect joint points (key points) that are estimated to be the joints of a person based on the characteristics of each part of the person recognized in the image, and further extracts bone lines (bone links) connecting the detected joint points. The human joint detection unit 104 generates skeleton information including joint points and bone lines based on the image. Note that the detected joint points and bone lines are joint points and bone lines estimated by the joint estimation technology, and may not match the joints and bones of an actual person. It can also be said that the human joint detection unit 104 detects a skeleton structure including joint points and bone lines from the person's feet to the head. The human joint detection unit 104 recognizes people in a plurality of images (frames) in time series, and detects the joint points and bone segments of all people recognized in each image.

[0024] The coordinate calculation unit 105 calculates the coordinates (position) of the person in the three-dimensional real world space (world coordinate space) from the coordinates (skeleton information) of the detected joint points and bone segments by using the calculated camera parameters. The calculated coordinates are the coordinates (center coordinates) of the center point of the social distance range of the person. The center point is an arbitrary point included in the bone segment between the joint points detected by the human joint detection unit 104. The center point may be an arbitrary point of the joint point or bone segment from the person's feet to the head, or may be a point of a vertical line that can be extracted from the person. As with camera calibration, the center point may be a point included in the bone segment at the feet, but since a point of a line segment that is stable in a state close to vertical when the person is walking is preferable, for example, the center point is a point included in the spine segment. The coordinate calculation unit 105 calculates the center coordinates of all people in each image (frame) and stores the calculation results in the storage unit 106 in chronological order.

[0025] The storage unit 106 stores information (data) necessary for the operation (processing) of the social distance analysis device 100. For example, the storage unit 106 is a non-volatile memory such as a flash memory, a hard disk drive, or the like. For example, the storage unit 106 stores the camera parameters calculated by the camera calibration unit 103 and the time-series coordinate calculation results calculated by the coordinate calculation unit 105. The storage unit 106 is a camera parameter storage unit, and can also be said to be a coordinate calculation result storage unit. The storage unit 106 also stores, as necessary, the image (video) acquired by the acquisition unit 102, information on joint points and bone segments processed by the human joint detection unit 104, machine learning data for joint estimation, the average height of a person and the length of each bone, the length of social distance, and the like.

[0026] The social distance determination unit (determination unit) 107 determines the state of social distance (risk of infection) of a person based on the calculated position (center coordinates) of the person. The social distance determination unit 107 can also be said to be a detection unit that detects the social distance (range) of a person. For example, the social distance determination unit 107 determines a range of a predetermined length from the center coordinates (center point) of the person as the social distance range. The predetermined length of the social distance may be input by the user or may be stored in the storage unit 106 in advance. The social distance determination unit 107 determines the risk of infection based on the distance (social distance) between the positions (center coordinates) of the people. For example, the social distance determination unit 107 determines the risk of infection depending on whether the distance between the people is closer than a predetermined distance. In addition, the risk of infection may be determined depending on whether the social distance range of a person overlaps with the social distance range of another person, or the risk of infection may be determined depending on whether another person is included in the social distance range of a person. Furthermore, the social distance may be determined taking into account time in addition to distance. The social distance determination unit 107 may determine whether or not an alert is necessary in two stages based on one criterion, or may determine a multi-stage alert level based on multiple criteria.

[0027] The display unit 108 displays the results of the operation (processing) of the social distance analysis device 100 on the display screen. For example, the display unit 108 is a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 108 displays the image (video) captured by the camera 200, the judgment result of the social distance judgment unit 107, etc. on the display screen of the GUI. For example, the display unit 108 displays the social distance range of a person according to the judgment result (whether an alert is required or not and the alert level) using a two-dimensional top view (plan view) or a three-dimensional bird's-eye view. The display unit 108 changes the display mode of the social distance range of a person according to the judgment result of the risk of infection (status of securing social distance) by the social distance judgment unit 107. For example, as the display mode, the color, shape, size, etc. of the social distance range may be changed according to the risk of infection. Furthermore, the blinking interval may be changed when the social distance range is displayed in a blinking manner. In the present embodiment, as an example, the color of the social distance range is changed according to the distance between multiple people. For example, the social distance range is displayed in a different color or the like so that it is emphasized as the risk of infection increases (alert level). Furthermore, the display unit 108 displays an area of ​​a predetermined shape centered on the position of the person as the social distance range. Any area that allows the social distance to be specified may be displayed, for example, as a circular, semicircular, sector-shaped, elliptical, or other area when viewed from above. In this embodiment, as an example, a circular area of ​​a predetermined size (radius) is displayed. By changing the display mode of the social distance range according to the risk of infection, it becomes possible to appropriately grasp the social distance of a person, and also to clearly identify a person who is at high risk of infection (who is not maintaining social distance).

[0028] The display unit 108 is an example of an output unit that outputs the determination result of the state of social distancing. For example, in addition to or instead of the screen display, the determination result of the risk of infection (state of social distancing) may be output by sound or other notification method (such as email via a network). When sound is output, the output mode (volume, pitch, interval, type of voice message, etc. of the warning sound) may be changed according to the risk of infection, similar to the display mode.

[0029] FIG. 3 shows the operation of the social distance analysis device 100 according to the present embodiment. As shown in FIG. 3, the social distance analysis device 100 acquires an image (video) from the camera 200 (S101). The acquisition unit 102 acquires an image of a person (image capture space) for camera calibration, and also acquires an image of a person (image capture space) for social distance determination. The image for camera calibration and the image for social distance determination may be images captured by the same camera 200, and may be captured at the same or different timings. That is, camera calibration may be performed from a video captured in a first period (about 10 to 60 seconds), and then the social distance may be determined from a video captured in a second period, or camera calibration may be performed and social distance may be determined from a video captured in the same period. At least, it is sufficient that the camera calibration is performed before using the camera parameters in determining the social distance (calculating the coordinates). FIG. 4 shows an example of an image acquired from the camera 200. The example of FIG. 4 is an image in which six people are walking in the image capture space.

[0030] Next, a camera calibration process (S102 to S103) is performed, and a social distance determination process (S104 to S108) is performed. As an example of the camera calibration process, an example using bone segments between joint points of a person will be described. In the camera calibration process, first, the social distance analysis device 100 detects joint points of a person based on an image acquired from the camera 200 (S102). The human joint detection unit 104 detects the joint points of each person in the image plane using a joint estimation technique such as OpenPose, and extracts bone segments connecting the detected joint points.

[0031] Next, the social distance analysis device 100 calculates the camera parameters using the detected bone segments (S103). For example, the camera calibration unit 103 selects the line segments of the person's spine as the normal vector to be used for the calculation. The line segments of the spine of one person in the image may be selected, or the line segments of the spine of multiple people may be selected. When multiple spine segments are selected, the average of the line segments (vectors) may be calculated. The line segments of the spine may be used, or the height of the person (whole skeleton) may be estimated from the line segments of the spine and the line segment (center line segment) corresponding to the estimated height may be used. The camera calibration unit 103 calculates the camera parameters from the line segments of the person's spine and the average value of the length of the spine (or height).

[0032] For example, as in Patent Document 2, a normal vector (segment of the spine) may be projected onto a projection plane perpendicular to the ground (reference plane), and camera parameters may be calculated based on the perpendicularity of the projected normal vector to the ground. A rotation matrix (camera angle) can be calculated by evaluating the perpendicularity of the normal vector projected onto the projection plane. In addition, a translation vector (camera position) and focal length can be calculated from the difference between the length of the normal vector of the two-dimensional image projected onto three-dimensional space and the length in the real world (average length of bones) using the rotation matrix.

[0033] On the other hand, in the social distance determination process, the social distance analysis device 100 first detects the joint points of people based on the acquired image (S104). As in S102, the human joint detection unit 104 detects the joint points and bone segments of each person in the image acquired from the camera 200.

[0034] Next, the social distance analysis device 100 calculates the coordinates (center coordinates) of the center point of the person on the detected bone line segment using the calculated camera parameters (S105). For example, the coordinate calculation unit 105 selects a point included in the line segment of the person's spine (from the waist to the neck) as the center point of the person. In this example, the waist joint point (the lower end point of the spine line segment) is selected as the center point. The social distance analysis device 100 converts the coordinates of the spine line segment (waist joint point) in the two-dimensional image plane into coordinates (x, y, z) in the three-dimensional real world space (world coordinate space) using the camera parameters. The start point coordinates and end point coordinates of the line segment (vector) may be converted into three-dimensional coordinates to obtain the center coordinates, or the start point coordinates or end point coordinates of the line segment may be converted into three-dimensional coordinates to obtain the center coordinates.

[0035] FIG. 5 shows an example of detecting people's joint points and bone lines from the image in FIG. 4. As shown in FIG. 5, the joint points and bone lines of each of the six people in the image are detected. By detecting people's joint points and bone lines from multiple images (frames) in a time series, the joint points and bone lines of a person who is walking (moving) are detected. In the example in FIG. 5, the three-dimensional coordinates of the center point of the waist are calculated from the spinal lines for each of the six people in the image.

[0036] Next, the social distance analysis device 100 determines the social distance range of the person from the calculated central coordinates of the person (S106). The social distance determination unit 107 determines the range of a three-dimensional space of a predetermined length (for example, 1 m to 2 m) from the central coordinates of the person as the social distance range. In this example, the social distance range is a circular area in a top view with a radius of a predetermined length from the central point. In the example of FIG. 5, a circular area of ​​the social distance range is determined from the central coordinates for each of the six people in the image.

[0037] Next, the social distance analysis device 100 judges the risk of infection of the person based on the central coordinates (position) of the person (S107). FIG. 6 shows a specific example of the risk judgment process according to this embodiment. FIG. 6 shows an example of judging whether or not an alert is necessary in two stages based on one criterion. As shown in FIG. 6, the social distance judgment unit 107 acquires the distance between people (S201) and judges whether the acquired distance is shorter than a predetermined distance (S202). The social distance judgment unit 107 calculates the distance in the three-dimensional real world space (world coordinate space) between the central coordinates of the person and the central coordinates of another person.

[0038] If the distance between people is shorter than a predetermined distance, the social distance determination unit 107 determines that an alert is necessary (high risk of infection, social distance not being maintained) (S203), and if the distance between people is longer than the predetermined distance, it determines that an alert is unnecessary (low risk of infection, social distance being maintained) (S204). The predetermined distance that serves as the determination criterion may be the same as the distance used in the social distance range (e.g., 1m to 2m), or may be a longer distance. In the example of FIG. 5, the distance between each of the six people in the image is calculated, and whether or not an alert is necessary between each person is determined.

[0039] In addition, the necessity of an alert may be determined based on the determined social distance range. For example, if the social distance range of a person overlaps with the social distance range of another person, it may be determined that an alert is necessary, and if the social distance range of a person does not overlap with the social distance range of another person, it may be determined that an alert is unnecessary. In addition, if the social distance range of a person includes another person, it may be determined that an alert is necessary, and if the social distance range of a person does not include another person, it may be determined that an alert is unnecessary.

[0040] Furthermore, the social distance determination unit 107 may determine the risk of infection from the distance between people for each image (frame), or may analyze the positions of multiple images (frames) in a chronological order and determine the risk of infection by taking into account the movement of the analyzed people. For example, if the distance between people remains shorter than a predetermined distance for a certain period of time (if people are lingering), it may be determined that an alert is necessary. This makes it possible to exclude people who pass each other temporarily from the determination.

[0041] Next, the social distance analysis device 100 displays the risk determination result (S108). Fig. 7 shows a specific example of the determination result display process according to this embodiment. As shown in Fig. 7, the display unit 108 acquires the determination result of the social distance determination unit 107 (S301), and determines whether the acquired determination result requires an alert (S302). If the determination result indicates that an alert is required (if the risk of infection is high), the display unit 108 displays the social distance range of the person for whom an alert is required in red (S303), and if the determination result indicates that an alert is not required (if the risk of infection is low), the display unit 108 displays the social distance range of the person for whom an alert is not required in blue (S304).

[0042] 8 and 9 show display examples in the cases where an alert is unnecessary and an alert is necessary. As shown in FIG. 8 and FIG. 9, the display unit 108 displays a three-dimensional overhead view V1 and a two-dimensional top view V2 on a display screen (display window) W1. That is, the display screen W1 has an overhead view display area for displaying the overhead view V1 and a top view display area for displaying the top view V2, and the display unit 108 displays the social distance judgment results in each area. The overhead view V1 and the top view V2 can display multiple images (videos) and social distance ranges in a chronologically synchronized manner. The display unit 108 may display both the overhead view V1 and the top view V2, or may display either one of them.

[0043] The overhead view V1 is a three-dimensional display of the imaging space seen from the camera 200 (a predetermined viewpoint). The display unit 108 displays the input video (image) on the overhead view V1, and synthesizes the social distance range of the person with the video and displays it in a three-dimensional format. The viewpoint may be changed based on the video of multiple cameras. In the overhead view V1, for all people in the video, a center point P1 (e.g., a circle) of the person is displayed at a three-dimensional position corresponding to the calculated center coordinates, and a social distance range D1 of the person is displayed in a three-dimensional range corresponding to the social distance range determined from the center point P1. In this example, the social distance range D1 is a circular (disk-shaped) area of ​​a predetermined size centered on the center point P1 of the person, as in the above-mentioned social distance range determination process. As an example of displaying the social distance range, the outline of the circular area is displayed, but the inside of the circular area may be displayed in the same color as the outline or in another pattern. The predetermined distance range is not limited to being disk-shaped, and may be displayed three-dimensionally as a sphere. The overhead view V1 displays a social distance range D1, which is circular when viewed from above, in three dimensions with a size and shape according to its position. The overhead view V1 also displays a center point P1 and a social distance range D1 according to the position of a person moving within the video. The overhead view V1 may also display joint points and bone segments detected from each person.

[0044] The top view V2 is a two-dimensional plan view of the same imaging space as the overhead view V1, viewed from above. The display unit 108 displays the position of the person in the input video (image) and the image of the social distance range in a two-dimensional format on the top view V2. Note that in addition to the person, an image of another object may be displayed, or the video may be displayed in the same manner as the overhead view. As with the overhead view, the top view V2 displays the center point P2 (e.g., an x ​​mark) of every person in the video at a two-dimensional position corresponding to the calculated center coordinates, and displays a circular social distance range D2 in a two-dimensional range corresponding to the social distance range determined from the center point P2. As with the overhead view V1, the outline of the circular area is displayed as an example of the display of the social distance range, but the inside of the circular area may be displayed in the same color as the outline or in another pattern. Also, as with the overhead view V1, the top view V2 displays the center point P2 and the social distance range D2 according to the position of the person moving in the video.

[0045] For example, in FIG. 8 and FIG. 9, the display color (display mode) is changed according to the distance between people. In FIG. 8, the distance between all people A1 to A6 is greater than a predetermined distance. Therefore, in the bird's-eye view V1 and the top view V2, the social distance ranges D1 and D2 and the center points P1 and P2 of people A1 to A6 are displayed in blue. On the other hand, in FIG. 9, the distance between people A3 to A6 is greater than a predetermined distance, but the distance between person A1 and object A2 is closer than the predetermined distance (part of the social distance ranges overlap). Therefore, in the bird's-eye view V1 and the top view V2, the social distance ranges D1 and D2 and the center points P1 and P2 of people A3 to A6 are displayed in blue, and the social distance ranges D1 and D2 and the center points P1 and P2 of people A1 and A2 are displayed in red. In this example, red (highlighted color) / blue is used to clearly indicate high / low risk of infection, but other colors may be used as long as the risk of infection can be identified. Also, instead of being limited to two colors, a multi-stage gradation display may be used according to the overlap of distance and social distance ranges. The display mode is not limited to color, and the blinking interval, etc. may be changed. For example, the display may blink when an alarm is required, and the blinking interval may be shortened as the distance decreases (as the alert level increases). Furthermore, a mark such as an arrow or a character string indicating a high risk of infection may be displayed. A numerical value of the distance between people may be displayed. Also, the color, etc. of people in a social distance range with a high risk of infection may be changed.

[0046] As in the risk determination process described above, the color (display mode) may be changed according to the risk based on the social distance range. For example, if the social distance range of the first person overlaps with the social distance range of the second person, the social distance range of the first person and the second person may be displayed in red, and if the social distance range of the first person does not overlap with the social distance range of the second person, the social distance range of the first person and the second person may be displayed in blue. Furthermore, the color may be changed according to the amount of overlap of those social distance ranges. Also, if the social distance range of the first person includes the second person, the social distance range of the first person may be displayed in red, and if the social distance range of the first person does not include the second person, the social distance range of the first person may be displayed in blue. Furthermore, the color may be changed according to the density of multiple people. For example, the color may be changed according to the number of people included in a specified area, the number of overlaps (overlapping area) of the social distance ranges of the people, and the number of other people included in the social distance range of the person.

[0047] In the examples of FIG. 8 and FIG. 9, the center point P1 and the circular social distance range D1 are displayed at the position (height) of the person's waist in the overhead view V1. That is, the center point P1 and the social distance range D1 are displayed in a state of floating at a position higher than the person's feet. Since the possibility of droplet infection is higher the closer to the face, by displaying the social distance range at a higher position on the person, it is possible to display more suitable for the risk of infection. For example, the height of the waist at which the center point P1 and the social distance range D1 are displayed is the height of the joint point of the waist detected in the joint detection, and is the height at which the coordinates of the person are calculated. Since the line segment of the spine is used to calculate the coordinates, the height of the lower end of the spine (waist) may be used, or any height of the spine (from the waist to the neck) may be used. Since the positions of the spine and waist are stable when a person is walking, the social distance range can be displayed more easily by displaying the spine and waist as the center.

[0048] As described above, in this embodiment, camera calibration is performed from an image captured by a camera, and the position and social distance of a person are detected, and the detection results are displayed on a display screen. By using joint estimation technology, it is possible to automatically perform camera calibration and detect the distance (position) of a person from a captured image. In particular, by calculating camera parameters using the bone line segments of a person extracted from an image as normal vectors, it is possible to easily calculate camera parameters with high accuracy. Therefore, it is possible to improve the accuracy of coordinate conversion of three-dimensional coordinates, and to accurately detect social distance.

[0049] (Embodiment 2) Hereinafter, the second embodiment will be described with reference to the drawings. In this embodiment, an example of a camera calibration unit in the social distance analysis device of the first embodiment will be described.

[0050] <Relationship between camera coordinate system and world coordinate system> The relationship between the "camera coordinate system" of camera 200 and the "world coordinate system" in this embodiment will be described with reference to Fig. 10. In this example, camera 200 captures an image of a target area in the "world coordinate system" to form an image F1.

[0051] 10, the origin of the camera coordinate system is located on the z-axis of the world coordinate system and has coordinates [0,0,h] (=c) in the world coordinate system. That is, the position of the camera 200 can be expressed one-dimensionally by the z-coordinate of the world coordinate system, and the translation vector T for the camera 200 can also be expressed with a small number of parameters, that is, one dimension.

[0052] Also, z in the camera coordinate system c The z axis corresponds to the optical axis of the camera 200. c The projection of the axis coincides with the y-axis of the world coordinate system. That is, when viewed from the +z direction of the world coordinate system, the z axis of the camera coordinate system c The y-axis of the world coordinate system is set to overlap with the y-axis of the world coordinate system. cThis limits the roll rotation (z c rotation around the axis) and pitch rotation (x c Furthermore, the rotation matrix R of the camera 200 can also be expressed in two dimensions with fewer parameters.

[0053] In this embodiment, the three-dimensional coordinates of the world coordinate system are not explicitly given, so the world coordinate system and the camera coordinate system may be defined arbitrarily. Therefore, even if the world coordinate system and the camera coordinate system are set as shown in FIG. 10, generality is not lost.

[0054] <Example of camera calibration unit configuration> Fig. 11 shows an example of the configuration of the camera calibration unit according to this embodiment. As shown in Fig. 11, the camera calibration unit 103 according to this embodiment has a normal vector acquisition unit 111, a normalization coordinate conversion unit 112, and an external parameter calculation unit 113.

[0055] The normal vector acquisition unit 111 acquires a plurality of "image plane normal vectors" in the "image plane" of an image captured in the "world coordinate space" by the camera 200. The plurality of "image plane normal vectors" respectively correspond to a plurality of "world coordinate space normal vectors" with respect to a "reference plane" in the "world coordinate space". The "world coordinate space" is a space defined by a world coordinate system. The "reference plane" may be a horizontal plane such as the ground or a floor. In the example of FIG. 10, n world coordinate space normal vectors (M1 to M n ) corresponding to each of the n image plane normal vectors (m1 to m n ) to get the

[0056] For example, the normal vector acquisition unit 111 acquires bone line segments between joint points of a plurality of people in the image plane detected by the human joint detection unit 104 as in the first embodiment, as a plurality of "image plane normal vectors". As described above, it is preferable to use a line segment of a person's spine as a line segment nearly perpendicular to the ground. Note that the "world coordinate space normal vector" may be another line segment extending perpendicular to the above-mentioned horizontal plane. In addition, the lengths in the world coordinate space of the plurality of "world coordinate space normal vectors" corresponding to the acquired plurality of image plane normal vectors may be different from each other. A plurality of spine line segments of different people may be acquired from one or a plurality of images captured by the camera 200, or a plurality of spine line segments of the same person may be acquired from a plurality of images. For example, a plurality of spine line segments of a plurality of people are acquired from images captured within 10 to 60 seconds. As described above, the length (or height) of a person's bone is input as the reference length, so the length (height) of a person's spine line segment in real space may be used for calculation as being known. For example, the average value of the length of the spine may be the length of the line segments of the spine of multiple people in real space. Also, the start point and end point of the normal vector in the world coordinate space may or may not be on the reference plane. In the example shown in Fig. 10, the start point of the normal vector in the world coordinate space is on the reference plane. In the case of the line segments of the spine, the start point of the line segment of the spine is above the reference plane (ground).

[0057] The normalization coordinate conversion unit 112 converts the "image plane start coordinate" and "image plane end coordinate", which are the image plane coordinates for the start and end points of each image plane normal vector acquired by the normal vector acquisition unit 111, into "normalized start coordinate" and "normalized end coordinate", which are normalized coordinates, based on the "internal parameters" of the camera 200. For example, the "internal parameters" of the camera 200 are stored in advance in the storage unit 106. Here, the normalization coordinate conversion unit 112 homogenizes the "image plane start coordinate" and the "image plane end coordinate". Specifically, the "image plane start coordinate" and the "image plane end coordinate" are converted into the normalized coordinates m in FIG. 10. i and m iThat is, the "start point coordinates in the image plane" and the "end point coordinates in the image plane" are expressed as three-dimensional coordinates that add scale uncertainty to the two-dimensional image coordinates.

[0058] The external parameter calculation unit 113 calculates a translation vector and a rotation matrix using the normalized start coordinates and normalized end coordinates obtained for each image plane normal vector by the normalized coordinate conversion unit 112 and an "evaluation function." Here, the "evaluation function" is a function that overdetermines the rotation parameters around the normal to the "reference plane" in the world coordinate space.

[0059] For example, the external parameter calculation unit 113 calculates a cross product of the vector of normalized start point coordinates and the vector of normalized end point coordinates obtained for each image plane normal vector by the normalized coordinate conversion unit 112 and calculates z c The extrinsic parameter calculation unit 113 calculates a scalar triple product, which is an inner product with a rotation vector around the axis, for a plurality of image plane normal vectors. Then, the extrinsic parameter calculation unit 113 uses the sum of the calculated plurality of scalar triple products as the evaluation function described above to calculate a desired z c The external parameter calculation unit 113 calculates a rotation vector around the axis. c The external parameter calculation unit 113 calculates a rotation matrix based on the rotation vector around the axis. Then, the external parameter calculation unit 113 outputs the calculated rotation matrix and also calculates the desired z c The rotation vector around the axis is output as a translation vector for the camera 200.

[0060] Here, from one image plane normal vector to z c A single constraint on the rotation vector around the axis is obtained. Therefore, by using two or more image plane normal vectors, c The rotation vector around the axis can be calculated. Furthermore, as mentioned above, y c The rotation angle around the axis can be set to zero. Therefore, the calculated z c From the rotation vector around the axis, z cRotation angle around axis and x c Since the rotation angle around the axis can be estimated, the entire 3×3 rotation matrix can be restored. In addition, in the relationship between the camera coordinate system and the world coordinate system described with reference to FIG. 10, the translation vector of the camera 200 is z c It is equal to a constant multiple of the rotation vector around the axis. Furthermore, by utilizing the fact that the three-dimensional coordinates of the normal vector in the world coordinate space are unknown, the height (z coordinate) of the camera 200 may be set to 1. In other words, the translation vector may be calculated as the third column of the rotation matrix. Note that the calculation of the external parameters by the external parameter calculation unit 113 will be described in detail later.

[0061] In this manner, in this embodiment, the camera calibration unit calculates a rotation vector and a rotation matrix using the normalized start coordinates and normalized end coordinates obtained for each image plane normal vector corresponding to the line segments of a person's spine and the like, and an "evaluation function." The "evaluation function" is a function that overdetermines the rotation parameters around the normal to the "reference plane" in the world coordinate space. With this configuration, the external parameters can be calculated with high accuracy.

[0062] For example, in the related camera calibration method described in the above Patent Document 2, a line segment perpendicular to a horizontal plane is detected from an image, the detected multiple vertical line segments are projected onto a virtual plane, and a rotation matrix is ​​calculated using an evaluation function that minimizes the parallelism of each line segment on the virtual plane. However, in this method, there is a possibility that the number of unknowns increases as the number of observed line segments increases in minimization of the evaluation function, which is called an underdetermined problem. In the case of an underdetermined problem, even if the number of observed line segments is increased, the number of constraint equations is always insufficient, and since there are an infinite number of theoretical optimal solutions, even camera parameters that are not practically possible become optimal solutions. In other words, there is a possibility that the accuracy of the calculated external parameters will deteriorate. According to this embodiment, it is possible to avoid such an underdetermined problem and improve the calculation accuracy of the external parameters.

[0063] <Example of camera calibration section operation> FIG. 12 shows an example of the operation of the camera calibration unit according to the present embodiment.

[0064] As shown in FIG. 12, the normal vector acquisition unit 111 acquires a plurality of normal vectors within an image plane (for example, line segments of the spine) from an image captured by the camera 200 (S401).

[0065] The normalized coordinate conversion unit 112 converts the start point coordinates and end point coordinates of each image plane normal vector into normalized start point coordinates and normalized end point coordinates based on the internal parameters of the camera 200 (S402).

[0066] The external parameter calculation unit 113 calculates a translation vector and a rotation matrix using the normalized start coordinates and normalized end coordinates obtained for each image plane normal vector and the "evaluation function" (S403).

[0067] Here, the calculation of the external parameters by the external parameter calculation unit 113 will be described in detail using a concrete example. In the following description, the symbols shown in Fig. 10 will be used. That is, the camera 200 is located at c=[0,0,h] on the z-axis of the world coordinate system. T Also, z with respect to the xy plane of the world coordinate system c The projection of the axis overlaps with the y-axis of the world coordinate system. Also, the starting point M of the i-th normal vector in the world coordinate space i =[x i ,y i ,0] T and end point M i '=[x i ,y i ,z i ] T are respectively, m i =[u i ,v i ,1] T and m i '=[u i ',v i ',1] T It has been observed that, where m i and m i' is expressed in a homogeneous coordinate system normalized by the internal parameters. The superscript T denotes the transpose of a vector or matrix.

[0068] First, we define the problem to be solved.

[0069] The rotation matrix R of the camera 200 is expressed as y c Since the rotation angle around the axis can be set to zero, x c Rotation angle around the axis θ x and z c Rotation angle around the axis θ z Using this, it can be expressed by the following equation (1).

number

[0070] Furthermore, the translation vector t of the camera 200 can be expressed by the following equation (2) using the position c of the camera 200 and the rotation matrix R.

number

[0071] where r j represents the j-th column vector of the rotation matrix R. In other words, equation (2) expresses the translation vector t as z c It indicates that it is a constant multiple of the rotation vector around the axis.

[0072] Using the above formulas (1) and (2), the projective relationship between the i-th normal vector in the world coordinate space and the normal vector in the image plane is expressed by the following formula (3).

number

[0073] In this embodiment, since the world coordinate system can be set arbitrarily, the three-dimensional coordinates x of the start point and end point of the normal vector in the world coordinate space can be calculated. i ,y i ,z iThere is a constant multiple uncertainty between (2≦i≦n) and the camera height h. In other words, unless known information about the world coordinate space (for example, the length in meters of a normal vector in a certain world coordinate space) is given, the absolute positional relationship in the world coordinate space is unknown, and only the relative positional relationship can be found. In other words, the relative size can be determined arbitrarily. Therefore, if the camera height is set to h=1, equation (3) can be expressed as the following equation (4).

number

[0074] From equation (4), if n image plane normal vectors can be obtained, the number of unknowns is 2 + 3n (x c Rotation angle around the axis θ x and z c Rotation angle around the axis θ z , and the three-dimensional coordinates x of the start and end points of each normal vector in the world coordinate space. i ,y i ,z i On the other hand, the number of conditions obtained from one image plane normal vector is four (u i ,v i ,u i ',v i '), the condition number when n image plane normal vectors can be obtained is 4n. In other words, when n is 2 or more, the condition number exceeds the unknowns (4n>2+3n), and the problem can be solved.

[0075] Next, a method for calculating the rotation matrix R and the translation vector t by solving equation (4) will be described.

[0076] By using the projective depths λ and λ' to eliminate the uncertainty on both sides of equation (4), the following equation (5) is obtained.

number

[0077] The difference between the upper and lower equations in equation (5) is expressed by the following equation (6).

number

[0078] Equation (6) is a function of three vectors m i , m i This shows that r′ and r3 are linearly dependent on each other, i.e., the scalar triple product of these three vectors is zero, as shown in equation (7).

number

[0079] For each image plane normal vector, equation (7) is obtained.

[0080] Therefore, when two image plane normal vectors are obtained, r3 can be calculated using the cross product of the vectors, as shown in the following equation (8).

number

[0081] Furthermore, when n normal vectors in the image plane are obtained, r3 can be calculated by the least squares method shown in the following equation (9).

number

[0082] As described above, the third column r3 of the rotation matrix R can be calculated.

[0083] As shown in the above formula (1), the third column r3 of the rotation matrix R is θ x and θ z In other words, each component of r3 is expressed explicitly using inverse trigonometric functions, x and θ z The rotation matrix R can be calculated as x cRotation around the axis and z c Since it is expressed only by rotation around the axis, θ x and θ z If it is possible to calculate all components of the rotation matrix R,

[0084] Here, formulas (8) and (9) show that if two or more image plane normal vectors can be obtained, the rotation matrix can be estimated. Even if the number of image plane normal vectors obtained increases, the only unknown is the third column of the rotation matrix. Formula (9) is a numerically stable linear least squares method, and becomes more stable as n increases. Therefore, unlike the evaluation function used in Patent Document 2, the rotation matrix can be obtained with higher accuracy if the number of image plane normal vectors obtained is large.

[0085] The translation vector t is the third column r3 of the rotation matrix R itself, and is therefore already calculated at the same time as the calculation of the rotation matrix R.

[0086] As described above, in this embodiment, the camera calibration unit calculates the cross product of the vector of normalized start point coordinates and the vector of normalized end point coordinates obtained for each image plane normal vector corresponding to the line segments of the person's spine, and calculates the z c The camera calibration unit calculates a scalar triple product, which is an inner product with a rotation vector around the axis, for a plurality of image plane normal vectors. Then, the camera calibration unit uses the sum of the calculated plurality of scalar triple products as the evaluation function described above to find a desired z c The camera calibration unit calculates the rotation vector around the axis. c The camera calibration unit calculates a rotation matrix based on the rotation vector around the axis. The camera calibration unit then outputs the calculated rotation matrix and calculates the desired z c The rotation vector around the axis is output as a translation vector for the camera 200.

[0087] This camera calibration unit configuration makes it possible to calculate the extrinsic parameters while avoiding the underdetermined problem. This allows the extrinsic parameters to be calculated with high accuracy. In addition, it is possible to calibrate the monocular camera using only image information under conditions where the three-dimensional coordinates are unknown. Therefore, according to this embodiment, the camera parameters can be calculated with higher accuracy, and the social distance can be detected more effectively using the camera parameters.

[0088] <Modification> The social distance analysis device including the camera calibration unit in the second embodiment can be modified as follows.

[0089] <1> The settings of the world coordinate system and the camera coordinate system are not limited to the above. For example, the starting point of one normal vector in the world coordinate space may be set as the origin of the world coordinate system, and another normal vector in the world coordinate space may be set on the x-axis. In that case, M1=[0,0,0] T , M1'=[0,0,z1] T , M2=[0,y2,0] T , M1'=[0, y2, z2] T and the translation vector t is not the third column of the rotation matrix. However, since the number of unknowns does not change, if two or more image plane normal vectors can be obtained, the rotation matrix and translation vector can be found. Since different three-dimensional coordinate systems can be made to match by rigid body transformation, even if the unknowns to be solved change due to the setting of the origin, it is a mathematically equivalent problem.

[0090] <2> The reference plane is not limited to a horizontal plane. For example, a wall surface may be used as the reference plane, and a line segment parallel to the floor surface may be defined as the normal vector in the world coordinate space.

[0091] <3> The method of expressing the rotation matrix is ​​not limited to the above formula (1). For example, a quaternion with the component around the y-axis set to zero may be used.

[0092] <4> The method of calculating the third column of the rotation matrix is ​​not limited to formula (8) or formula (9). Formula (8) and formula (9) are equivalent to obtaining the intersection of image plane normal vectors, that is, the vanishing point in the vertical direction. Therefore, various methods for calculating the vanishing point in the vertical direction may be used instead. In addition, although formula (9) minimizes the so-called algebraic error, more advanced error analysis methods such as geometric error and Sampson error may be used. Furthermore, the well-known RANSAC (Random Sample Consensus) or weighted least squares method may be combined with formula (8) and formula (9) to calculate the vanishing point while removing image plane normal vectors with large errors.

[0093] <5> The z coordinate of the start point of the normal vector in the world coordinate space is not limited to zero. In other words, the normal vector in the world coordinate space does not necessarily need to be tangent to the reference plane. This is because, as shown in the derivation from equation (6) to equation (7), the value of the z coordinate is eliminated, and therefore does not affect equations (8) and (9).

[0094] (Embodiment 3) In the third embodiment, an example will be described in which three-dimensional coordinates are calculated using the camera parameters calculated by the camera calibration unit in the second embodiment.

[0095] In this embodiment, two three-dimensional coordinates in the world coordinate system are calculated for the normalized start coordinates and normalized end coordinates of each image plane normal vector based on the image plane normal vector, the rotation matrix, and the translation vector. The configuration of the social distance analysis device 100 including the camera calibration unit 103 is the same as in the first and second embodiments. That is, the camera calibration unit 103 calculates camera parameters by the camera calibration method according to the second embodiment, and the coordinate calculation unit 105 calculates three-dimensional coordinates of the spine segments in the image using the calculated camera parameters. Note that the camera calibration unit 103 may perform the camera calibration process and the three-dimensional coordinate calculation process.

[0096] The coordinate calculation unit 105 according to this embodiment calculates two three-dimensional coordinates in the world coordinate system for the normalized start coordinate and the normalized end coordinate of each image plane normal vector, based on the multiple image plane normal vectors acquired by the normal vector acquisition unit 111 and the rotation matrix and the translation vector calculated by the external parameter calculation unit 113.

[0097] The coordinate calculation unit 105 can calculate the above three-dimensional coordinates by using the following equation (10), which is obtained by modifying the above equation (4).

number

[0098] where vectors r1, r2, and r3 represent column vectors of the rotation matrix, respectively, and m i and m i ' represents the normalized start and end coordinates of the i-th image plane normal vector, and [x i ,y i ,z i ] represents the three-dimensional coordinates to be calculated.

[0099] Equation (10) is [x i ,y i ,z i ] T Since it is a linear simultaneous equation with unknowns, it can be easily solved.

[0100] 13 shows an example of the operation of a social distance analysis device including a camera calibration unit according to this embodiment. In FIG. 13, steps S401 to S403 are the same as in FIG.

[0101] Based on the image plane normal vector acquired in S401 and the rotation matrix and translation vector calculated in S403, the coordinate calculation unit 105 calculates two three-dimensional coordinates in the world coordinate system for the normalized start coordinate and normalized end coordinate of each image plane normal vector (e.g., a line segment of the spine) (S404).

[0102] <Modification> The social distance analysis device including the camera calibration unit in the third embodiment can be modified as follows.

[0103] <1> Although the formula (4) is used to calculate two three-dimensional coordinates in the world coordinate system for the normalized start coordinate and the normalized end coordinate of each image plane normal vector, the present invention is not limited to this. For example, the three-dimensional coordinates including the projective depth may be calculated by a formula obtained by modifying the formula (5).

[0104] <2> The coordinate calculation unit 105 may calculate absolute three-dimensional coordinates based on the multiple image plane normal vectors, the rotation matrix, and the translation vector, as well as the length of a world coordinate space normal vector corresponding to at least one of the multiple image plane normal vectors, or the coordinate on the z axis of the origin of the camera coordinate system. In other words, if the absolute length (e.g., in meters) of a world coordinate space normal vector corresponding to at least one of the multiple acquired image plane normal vectors is known, it can be used to convert the three-dimensional coordinates of the camera position and world coordinate space normal vectors, which are in a relative relationship, into absolute relationships. For example, if the z coordinate of the end point of a j-th world coordinate space normal vector is z j If it is known that =a, then for the three-dimensional coordinates of the normal vector in the world coordinate space and the camera height h, a / z j In this example, the length (height) of the line segment of the spine may be known, and the three-dimensional coordinates of the spine of the person may be calculated. Note that this process is performed by the coordinate calculation unit 105 using z j =a as an input and executed by the coordinate calculation unit 105 before outputting the three-dimensional coordinates, or may be executed after the coordinate calculation unit 105 outputs the three-dimensional coordinates.

[0105] (Embodiment 4) In the fourth embodiment, an example of optimization processing after calculation of three-dimensional coordinates by the coordinate calculation unit in the third embodiment will be described.

[0106] In this embodiment, optimization processing is performed on two three-dimensional coordinates for the rotation matrix, translation vector, and each image plane normal vector calculated by the coordinate calculation unit. The configurations of the camera calibration unit and the social distance analysis device are the same as those of the first to third embodiments. That is, the camera calibration unit 103 calculates camera parameters by the camera calibration method according to the second embodiment, and the coordinate calculation unit 105 calculates three-dimensional coordinates of spinal segments and the like in the image by the calculation method according to the third embodiment, and further performs optimization processing. Note that the camera calibration unit 103 may perform the camera calibration processing, the three-dimensional coordinate calculation processing, and the optimization processing.

[0107] The coordinate calculation unit (optimization unit) 105 according to this embodiment calculates two three-dimensional coordinates for each image plane normal vector, and executes an "optimization process" of a rotation matrix, a translation vector, and the calculated two three-dimensional coordinates, as in the third embodiment. Note that an optimization unit that executes the "optimization process" may be provided separately from the coordinate calculation unit 105.

[0108] For example, in the "optimization process", the coordinate calculation unit 105 calculates two "reprojected coordinates" by projecting the two three-dimensional coordinates for each calculated image plane normal vector onto the image plane using the rotation matrix and translation vector output from the external parameter calculation unit 113. Then, the coordinate calculation unit 105 obtains an adjusted rotation matrix, an adjusted translation vector, and the two three-dimensional coordinates after adjustment that are expected to minimize the "reprojection error" between the two "reprojected coordinates" for each calculated image plane normal vector and the normalized start coordinate and normalized end coordinate of each image plane normal vector. This improves the accuracy of the rotation matrix, the translation vector, and the two three-dimensional coordinates.

[0109] For example, the coordinate calculation unit 105 may perform nonlinear optimization based on the following equation (11) using the above two three-dimensional coordinates for each calculated image plane normal vector and the rotation matrix and translation vector output from the external parameter calculation unit 113 as initial values.

number

[0110] where Proj is a function that divides each of the first and second components by the third component of a three-dimensional vector to convert it into a homogenized coordinate system, R is a rotation matrix, r3 is a translation vector, and m i and m i ' represents the normalized start and end coordinates of the i-th image plane normal vector, and M i and M i ' represents the two 3D coordinates of the start and end points of the i-th image plane normal vector, and θ z represents the rotation angle around the first axis of the camera coordinate system, and θ x represents the rotation angle around the second axis other than the first axis of the camera coordinate system, and x i ,y i ,z i denote the two 3D coordinates for the start and end points of the i-th image plane normal vector, and / / / / denotes the norm.

[0111] Here, the above formulas (8) to (10) are based on a linear simultaneous equation that minimizes the so-called algebraic error. In addition, since the rotation matrix, translation vector, and 3D coordinates are calculated in order, there is a possibility that errors may accumulate in the later stages. Therefore, by minimizing the so-called reprojection error as expressed in formula (11), it is possible to homogenize the errors between parameters and calculate the statistically optimal parameters.

[0112] FIG. 14 shows an example of the operation of a social distance analysis device including a camera calibration unit according to this embodiment.

[0113] The coordinate calculation unit 105 executes an "optimization process" of the rotation matrix and translation vector calculated in S403, and two three-dimensional coordinates for each image plane normal vector (e.g., a line segment of the spine) calculated in step S404 (S405).

[0114] <Modification> The social distance analysis device including the camera calibration unit in the fourth embodiment can be modified as follows.

[0115] In the above formula (11), it is not necessary to optimize all the parameters. For example, in the modified example of the third embodiment, <2> As mentioned in, if there is a normal vector in the world coordinate space with a known length, the 3D coordinates of both ends of that normal vector (for example, the line segment of the spine) can be fixed values ​​rather than variables. This reduces the number of unknowns, making it possible to optimize equation (11) more stably.

[0116] (Embodiment 5) Hereinafter, a fifth embodiment will be described with reference to the drawings. In this embodiment, another example of the camera calibration unit in the social distance analysis device of the first embodiment will be described.

[0117] <Relationship between camera coordinate system and world coordinate system> The relationship between the "camera coordinate system" of camera 200 and the "world coordinate system" in this embodiment will be described with reference to Fig. 15. In this example, camera 200 captures an image of a target area in the "world coordinate system" to form image F2.

[0118] As shown in FIG. 15, the origin of the world coordinate system coincides with the start point (M1) of the normal vector in the first world coordinate space. The end point (M'1) of the normal vector in the first world coordinate space is located on the z-axis of the world coordinate system. The start point (M2) of the normal vector in the second world coordinate space is located on the x-axis of the world coordinate system. That is, the straight line connecting the start point (M1) of the normal vector in the first world coordinate space and the start point (M2) of the normal vector in the second world coordinate space coincides with the x-axis of the world coordinate system. The direction perpendicular to both the x-axis and the z-axis is the y-axis. That is, the above-mentioned "reference plane" coincides with the xy plane of the world coordinate system. Here, the lengths of the normal vector in the first world coordinate space and the normal vector in the second world coordinate space are the same, and the z coordinates of the end point (M'1) of the normal vector in the first world coordinate space and the end point (M'2) of the normal vector in the second world coordinate space are both "z".

[0119] In this embodiment, as in the second embodiment, the three-dimensional coordinates of the world coordinate system are not explicitly given, so the world coordinate system and the camera coordinate system may be defined arbitrarily as long as they are relatively equivalent. Therefore, even if the world coordinate system and the camera coordinate system are set as shown in FIG. 15, generality is not lost. In FIG. 15, R represents a rotation vector, and t represents a translation vector. That is, the position and orientation of the camera 200 with respect to the world coordinates are represented by the rotation matrix R and the translation vector t.

[0120] <Example of camera calibration unit configuration> Fig. 16 shows an example of the configuration of the camera calibration unit according to this embodiment. As shown in Fig. 16, the camera calibration unit 103 according to this embodiment has a normal vector acquisition unit 121, a projective depth calculation unit 122, and a camera parameter calculation unit 123.

[0121] The normal vector acquisition unit 121 acquires a plurality of "image plane normal vectors" in the "image plane" of the image captured by the camera 200 in the "world coordinate space". As in the second embodiment, the plurality of "image plane normal vectors" correspond to a plurality of "world coordinate space normal vectors" with respect to a "reference plane" in the "world coordinate space", and the "reference plane" may be a horizontal plane such as the ground or a floor. For example, the normal vector acquisition unit 121 acquires bone line segments between a plurality of human joint points detected by the human joint detection unit 104 as the plurality of "image plane normal vectors", as in the first and second embodiments. In this embodiment as well, it is preferable to use the line segments of the person's spine as the line segments nearly perpendicular to the ground. The "world coordinate space normal vectors" may be other line segments that extend perpendicular to the above-mentioned horizontal plane.

[0122] Here, the multiple "image plane normal vectors" acquired by the normal vector acquisition unit 121 include two image plane normal vectors corresponding to two world coordinate space normal vectors having the same length. In this embodiment, the two world coordinate space normal vectors having the same length may be called "first world coordinate space normal vector" and "second world coordinate space normal vector". In addition, the two image plane normal vectors corresponding to the "first world coordinate space normal vector" and "second world coordinate space normal vector", respectively, may be called "first image plane normal vector" and "second image plane normal vector". In the example of FIG. 15, in the image F2, the first image plane normal vector (m1) and the second image plane normal vector (m2) corresponding to the first world coordinate space normal vector (M1) and the second world coordinate space normal vector (M2) of the world coordinate system, respectively, are acquired. Two spinal line segments of two people may be acquired as having the same length (or height) from one or more images captured by camera 200, or two spinal line segments of the same person may be acquired from multiple images. For example, two spinal line segments of two people are acquired from a video captured within 10 to 60 seconds. As described above, since the length (or height) of the person's bones is input as the reference length, the length (height) of the person's spinal line segments in real space may be used in the calculation as being known. For example, the average value of the spinal length may be set as the length of the spinal line segments of the two people in real space.

[0123] The projective depth calculation unit 122 calculates a "projective depth vector" having four "projective depths" corresponding to the start point and end point of the first image plane normal vector and the start point and end point of the second image plane normal vector as vector elements. In this embodiment, the start point and end point of the first image plane normal vector may be referred to as the "first start point" and the "first end point". Also, the start point and end point of the second image plane normal vector may be referred to as the "second start point" and the "second end point".

[0124] That is, the projective depth calculation unit 122 calculates the projective depth from the camera 200 to the three-dimensional coordinates of the first start point. Similarly, the projective depth is calculated for the first end point, the second start point, and the second end point. That is, the "projective depth" means the depth to the three-dimensional coordinates as seen from the camera 200 in the projective space.

[0125] For example, the projective depth calculation unit 122 calculates the projective depth vector based on the relationship that the projective depth vector is the "null space" of a "first matrix" having the first image plane start vector, the first image plane end vector, the second image plane start vector, and the second image plane end vector as matrix elements. The calculation of the "projective depth vector" will be described in detail later.

[0126] The camera parameter calculation unit 123 calculates the internal parameters and external parameters of the camera 200. This calculation is performed based on the projective depth vector calculated by the projective depth calculation unit 122 and the "first image plane start point vector", "first image plane end point vector", "second image plane start point vector", and "second image plane end point vector". The "first image plane start point vector" and the "first image plane end point vector" correspond to the first start point and the first end point, respectively, and the "second image plane start point vector" and the "second image plane end point vector" correspond to the second start point and the second end point, respectively.

[0127] The camera parameter calculation section 123 has an internal parameter calculation section 123A and an external parameter calculation section 123B.

[0128] The internal parameter calculation unit 123A calculates the focal length f of the camera 200 based on the projective depth vector calculated by the projective depth calculation unit 122, and the first image plane start vector, the first image plane end vector, the second image plane start vector, and the second image plane end vector. The focal length f of the camera 200 is one of the internal parameters. In calculating the focal length f, for example, an orthogonal relationship between each two column vectors of the rotation matrix R is used. The calculation of the internal parameters will be described in detail later.

[0129] The external parameter calculation unit 123B calculates a rotation matrix R and a translation vector t based on the projective depth vector calculated by the projective depth calculation unit 122, the first image plane start vector, the first image plane end vector, the second image plane start vector, and the second image plane end vector, and the focal length f calculated by the internal parameter calculation unit 123A. The calculation of the external parameters will be described in detail later.

[0130] In this manner, in this embodiment, the camera calibration unit obtains the first image plane normal vector and the second image plane normal vector, which correspond to the first world coordinate space normal vector and the second world coordinate space normal vector, respectively, which are normal vectors to the reference plane in the world coordinate space and have the same length, from the line segments of the person's spine, and further calculates a projective depth vector having four projective depths as vector elements corresponding to the start point and end point of the first image plane normal vector and the start point and end point of the second image plane normal vector, respectively. With this configuration, it is possible to calculate the internal parameters and the external parameters in a simpler manner.

[0131] For example, the above-mentioned Patent Document 3 describes a method using a planar calibration board as a related camera calibration method. In this method, three-dimensional coordinates are predefined as points distributed on a plane, and the three-dimensional coordinates of each point are acquired by image processing to calculate the external parameters. In order to perform camera calibration with high accuracy, it is desirable that the three-dimensional coordinates are observed evenly over the entire screen, without corresponding points being unevenly distributed only in the center or a part of the screen. However, in this method, a huge planar calibration object is required to calibrate a camera that observes a wide-area environment, so that calibration is practically impossible, or even if possible, it may be inconvenient. According to this embodiment, since the projective depth vector can be calculated, the internal parameters and external parameters can be calculated without using a calibration board, and the internal parameters and external parameters can be calculated in a simpler manner.

[0132] <Example of camera calibration section operation> FIG. 17 shows an example of the operation of the camera calibration unit according to the present embodiment.

[0133] The normal vector acquisition unit 121 acquires a first image in-plane normal vector (for example, a first spinal line segment) and a second image in-plane normal vector (for example, a second spinal line segment) from an image captured by the camera 200 (S501).

[0134] The projective depth calculation unit 122 calculates a "projective depth vector" whose vector elements are four "projective depths" corresponding respectively to the first starting point and the first ending point of the first image plane normal vector and the second starting point and the second ending point of the second image plane normal vector (S502).

[0135] The internal parameter calculation unit 123A calculates the focal length f of the camera 200 based on the projective depth vector calculated in S502, the start vector in the first image plane, the end vector in the first image plane, the start vector in the second image plane, and the end vector in the second image plane (S503).

[0136] The external parameter calculation unit 123B calculates a rotation matrix R and a translation vector t based on the projective depth vector calculated by the projective depth calculation unit 122 in S502, the start vector in the first image plane, the end vector in the first image plane, the start vector in the second image plane, and the end vector in the second image plane, and the focal length f calculated in S503 (S504).

[0137] Here, the calculation of the projective depth vector, the internal parameters, and the external parameters will be described in detail using a concrete example. In the following, the symbols shown in FIG. 15 are used. That is, the origin of the world coordinate system is the starting point M1=[0,0,0] of the normal vector in the first world coordinate space. T The end point M'1 of the normal vector in the first world coordinate space is located on the z-axis of the world coordinate system, and M'1=[0,0,0] T The x-axis is the starting point M1 and the starting point M2 of the normal vector in the second world coordinate space (M2 = [x2, 0, 0] T ) coincides with the line connecting the i-th normal vector in the world coordinate space. The camera 200 is located at a position with a rotation matrix R and a translation vector t from the origin of the world coordinate system. i =[x i ,0,0] T and end point M i '=[x i ,0,z] T are respectively, m i =[u i ,v i ,1] T and m i '=[u i ',v i ',1] T It has been observed that m i and m i ' is expressed in a homogeneous coordinate system. The superscript T represents the transpose of a vector or a matrix. Here, when the camera 200 is a digital camera, there is no skew, and the optical center can be approximately approximated to the image center, so the unknown internal parameter is the focal length f.

[0138] First, we define the problem to be solved.

[0139] The start and end points of the normal vector in the i-th world coordinate space can be projected from the three-dimensional space to the image coordinates using equation (12).

number

[0140] Here, λ i represents the projective depth of the starting point of the i-th image plane normal vector, and λ' i represents the projective depth of the end point of the i-th image plane normal vector. K represents the intrinsic parameter matrix.

[0141] When equation (12) is written in terms of the first image plane normal vector, the second image plane normal vector, the first world coordinate space normal vector, and the second world coordinate space normal vector, it can be expressed as equation (13).

number

[0142] where r j represents the j-th column vector in the rotation matrix R.

[0143] By taking the difference between the first and second equations in equation (13), and the difference between the third and fourth equations, equation (14) is obtained.

number

[0144] Since equation (14) indicates that the first equation and the second equation are equal, by taking the difference between the first equation and the second equation, we obtain equation (15).

number

[0145] Equation (15) shows that the projective depth is the null space of the 3 × 4 matrix A. In other words, the projective depth vector [λ1, λ'1, λ2, λ'2] T is the null space of matrix A with matrix elements m1, m'1, m2, and m'2 in the first image plane. That is, the ratio of each projective depth can be calculated without scale uncertainty. Although there is sign uncertainty, the sign of the projective depth based on equation (12) is positive, so all λs should be selected to have positive signs.

[0146] Next, a method for calculating the focal length using the calculated projective depth will be described.

[0147] By substituting the third and fourth equations of (13) into equation (15) and rearranging, and also substituting the second and fourth equations of (13) into equation (15) and rearranging, we obtain equation (16).

number

[0148] Orthonormality of rotation matrix (r1 T Using r3 = 0), we obtain equation (17) for the focal length f.

number

[0149] Equation (17) is a quadratic equation for f, but it is sufficient to find the solution with a positive sign. In this way, the internal parameters can be calculated.

[0150] Next, a method for calculating the rotation matrix R and the translation vector t, which are external parameters, will be described.

[0151] Since the L2 norm of each column vector of the rotation matrix R is 1, equation (18) can be obtained from equation (16). / / / / represents the L2 norm.

number

[0152] Moreover, the height z of the normal vector in the first world coordinate space and the normal vector in the second world coordinate space, and the x-axis coordinate x2 of the normal vector in the second world coordinate space have positive signs according to the definition of the world coordinate system. Therefore, the rotation matrix R can be expressed by the formula (19).

number

[0153] Although the sign of the second column vector of the rotation matrix R is indefinite, it is sufficient to select a sign such that the determinant of the rotation matrix R is det(R) = 1. In this way, the rotation matrix R can be calculated by equations (18) and (19).

[0154] The translation vector t can be expressed by equation (20) using the first equation of equation (13).

number

[0155] Thus, the translation vector t can be calculated by equation (20).

[0156] As described above, in this embodiment, the camera calibration unit obtains the first image plane normal vector and the second image plane normal vector, which correspond to the first world coordinate space normal vector and the second world coordinate space normal vector, respectively, which are normal vectors to a reference plane in the world coordinate space and have the same length, from a line segment of a person's spine, etc. Then, the camera calibration unit calculates a projective depth vector whose vector elements are four projective depths corresponding to the start point and end point of the first image plane normal vector and the start point and end point of the second image plane normal vector, respectively.

[0157] This configuration of the camera calibration unit makes it possible to calculate the projective depth vector, and therefore the internal and external parameters can be calculated without using a calibration board. In other words, the internal and external parameters can be calculated by a simpler method. Also, it is possible to calibrate the monocular camera using only image information under conditions where the three-dimensional coordinates are unknown. The reason is as follows. First, according to equation (13), the number of unknowns is 13 (λ1, λ'1, λ2, λ'2, x2, z, f, t, and the three-axis rotation angles of the rotation matrix R). However, since there is scale uncertainty between the projective depth and the three-dimensional coordinates x2 and z, the actual degrees of freedom are 12. On the other hand, six unknowns (m i =[u i ,v i ,1] T , m i '=[u i ',v i ',1] T ) constraints can be observed, so there are 12 for the two image plane normal vectors. In this way, the number of unknown degrees of freedom matches the number of constraints, making it possible to solve the problem. Therefore, according to this embodiment, the camera parameters can be calculated by a simpler method, and the social distance can be detected more effectively using the camera parameters.

[0158] <Modification> The social distance analysis device including the camera calibration unit in the fifth embodiment can be modified as follows.

[0159] <1> The settings of the world coordinate system and the camera coordinate system are not limited to the above settings. For example, the following settings may be used. That is, the origin of the camera coordinate system is located on the z-axis of the world coordinate system and has coordinates [0,0,h] (=c) of the world coordinate system. c The z axis corresponds to the optical axis of the camera 200. cThe projection of the axis coincides with the y-axis of the world coordinate system. That is, when viewed from the +z direction of the world coordinate system, the z axis of the camera coordinate system c The y-axis and the y-axis of the world coordinate system are set to overlap. Even with this setting, the number of unknowns does not change, so it can be found if two or more normal vectors are obtained. Since three-dimensional coordinate systems with different scales can be matched by a similarity transformation, it is a mathematically equivalent problem.

[0160] <2> The reference plane is not limited to a horizontal plane. For example, a wall surface may be used as the reference plane, and a line segment parallel to the floor surface may be defined as the normal vector in the world coordinate space.

[0161] <3> When three or more image plane normal vectors are obtained, a well-known method such as RANSAC or weighted least squares may be used to calculate a plausible solution for the projective depth vector, intrinsic parameters, and extrinsic parameters.

[0162] (Embodiment 6) In the sixth embodiment, an example will be described in which three-dimensional coordinates are calculated using the camera parameters calculated by the camera calibration unit in the fifth embodiment.

[0163] In this embodiment, the three-dimensional coordinates of the start and end points of the first world coordinate space normal vector and the second world coordinate space normal vector are calculated based on the first image plane normal vector and the second image plane normal vector, the internal parameters and the external parameters. The configuration of the social distance analysis device including the camera calibration unit is the same as that of the first and fifth embodiments. That is, the camera calibration unit 103 calculates the camera parameters by the camera calibration method according to the fifth embodiment, and the coordinate calculation unit 105 calculates the three-dimensional coordinates of the spine segments in the image using the calculated camera parameters. Note that the camera calibration unit 103 may perform the camera calibration process and the three-dimensional coordinate calculation process.

[0164] The coordinate calculation unit 105 according to this embodiment calculates two three-dimensional coordinates in the world coordinate system for the start point coordinates and end point coordinates of each image plane normal vector, based on the first image plane normal vector and the second image plane normal vector acquired by the normal vector acquisition unit 121 and the internal parameters and external parameters calculated by the camera parameter calculation unit 123. That is, the coordinate calculation unit 105 calculates the three-dimensional coordinates of the start point and end point of each of the first world coordinate space normal vector and the second world coordinate space normal vector, based on the first image plane normal vector and the second image plane normal vector acquired by the normal vector acquisition unit 121 and the internal parameters and external parameters calculated by the camera parameter calculation unit 123.

[0165] By transforming equation (12), equation (21) or equation (22) is obtained. Equation (21) is an equation for finding three-dimensional coordinates when the length of the normal vector in the first world coordinate space is the same as the length of the normal vector in the second world coordinate space. On the other hand, equation (22) is an equation for finding three-dimensional coordinates when the length of the normal vector in the first world coordinate space is different from the length of the normal vector in the second world coordinate space. Note that the transformation of equation (12) here utilizes the fact that the vectors on both sides of the first equation of equation (12) are equal, and therefore the cross product of these is zero. The same is true for the second equation of equation (12).

number

number

[0166] where vectors r1, r2, and r3 represent column vectors of the rotation matrix R, respectively, and [x i ,y i ] and [x i ,y i ,z i ] represents the three-dimensional coordinates to be calculated. In the case of formula (21), z i is z.

[0167] Equations (21) and (22) are linear simultaneous equations and can be easily solved.

[0168] 18 shows an example of the operation of a social distance analysis device including a camera calibration unit according to this embodiment. In FIG. 18, steps S501 to S504 are the same as those in FIG.

[0169] Based on the first image plane normal vector and the second image plane normal vector acquired in S501 and the internal parameters and external parameters calculated in S503, S504, the coordinate calculation unit 105 calculates the three-dimensional coordinates of the respective start points and end points of the first world coordinate space normal vector and the second world coordinate space normal vector corresponding to the first image plane normal vector (e.g., the first spine line segment) and the second image plane normal vector (e.g., the second spine line segment) (S505).

[0170] <Modification> The social distance analysis device including the camera calibration unit in the sixth embodiment can be modified as follows.

[0171] <1> The equations for calculating the three-dimensional coordinates of the start point and end point of each of the normal vector in the first world coordinate space and the normal vector in the second world coordinate space are not limited to equations (21) and (22). For example, the three-dimensional coordinates may be calculated including the projective depth that is eliminated in equations (21) and (22).

[0172] <2> The coordinate calculation unit 105 may calculate absolute three-dimensional coordinates based on the multiple image plane normal vectors, internal parameters, and external parameters, as well as the length of a world coordinate space normal vector corresponding to at least one of the multiple image plane normal vectors, or the coordinate on the z axis of the origin of the camera coordinate system. In other words, if the absolute length (e.g., in meters) of a world coordinate space normal vector corresponding to at least one of the multiple acquired image plane normal vectors is known, it can be used to convert the three-dimensional coordinates of the camera position and world coordinate space normal vectors, which are in a relative relationship, into absolute relationships. For example, if the z coordinate of the end point of a certain j-th world coordinate space normal vector is z j = a, other 3D coordinates of normal vector in world coordinate space and camera position c = -R T About T A / Z j In this example, the length (height) of the line segment of the spine may be known, and the three-dimensional coordinates of the spine of the person may be calculated. Note that this process is performed by the coordinate calculation unit 105 using z j =a as an input and executed by the coordinate calculation unit 105 before outputting the three-dimensional coordinates, or may be executed after the coordinate calculation unit 105 outputs the three-dimensional coordinates.

[0173] (Embodiment 7) In the seventh embodiment, an example of optimization processing after calculation of three-dimensional coordinates by the coordinate calculation unit in the sixth embodiment will be described.

[0174] In this embodiment, optimization processing is performed on two three-dimensional coordinates for the internal parameters, external parameters, and normal vectors in each image plane calculated by the three-dimensional coordinate calculation unit. The configurations of the camera calibration unit and the social distance analysis device are the same as those of the first, fifth, and sixth embodiments. That is, the camera calibration unit 103 calculates the camera parameters by the camera calibration method according to the fifth embodiment, and the coordinate calculation unit 105 calculates the three-dimensional coordinates of the spine segments in the image by the calculation method according to the sixth embodiment, and further performs optimization processing. Note that the camera calibration unit 103 may perform the camera calibration processing, the three-dimensional coordinate calculation processing, and the optimization processing.

[0175] The coordinate calculation unit (optimization unit) 105 according to this embodiment calculates two three-dimensional coordinates for each image plane normal vector, as in embodiment 6, and performs an "optimization process" of the internal parameters, the external parameters, and the two three-dimensional coordinates for each calculated image plane normal vector. Note that an optimization unit that performs the "optimization process" may be provided separately from the coordinate calculation unit 105.

[0176] For example, the coordinate calculation unit 105 calculates two "reprojected coordinates" by projecting the three-dimensional coordinates of the start point and end point of each of the normal vector in the first world coordinate space and the normal vector in the second world coordinate space onto the image plane using the internal parameters and external parameters calculated by the camera parameter calculation unit 123. Then, the coordinate calculation unit 105 obtains the adjusted internal parameters, the adjusted external parameters, and the above two three-dimensional coordinates after adjustment that are expected to minimize the "reprojection error" between the two "reprojected coordinates" of each of the normal vector in the first world coordinate space and the normal vector in the second world coordinate space and the start point and end point of each of the normal vector in the first image plane and the normal vector in the second image plane. This makes it possible to optimize the internal parameters and external parameters of the camera 200 and the above three-dimensional coordinates.

[0177] For example, the coordinate calculation unit 105 may execute nonlinear optimization based on the formula (23) using the calculated three-dimensional coordinates and the internal parameters and external parameters calculated by the camera parameter calculation unit 123 as initial values. The coordinate calculation unit 105 may optimize each parameter by solving the formula (23) using a known method such as the Levenberg-Marquard method or the trust region method. Here, the coordinate calculation unit 105 may set at least one of the focal length, skew, optical center, and lens distortion of the camera 200 as the internal parameter to be optimized.

number

[0178] Here, Proj represents a function that converts the first and second components of a three-dimensional vector into a homogenized coordinate system by dividing the first and second components by the third component of the three-dimensional vector.

[0179] Since the method is homogenized, there is no need to optimize the projective depth. The constraints on the 3D coordinates in (23) are intended to eliminate relative uncertainty.

[0180] Here, Equation (15) and Equations (18) to (20) are based on a linear simultaneous equation that minimizes the so-called algebraic error. In addition, since the calculations are performed in the order of projective depth, focal length, rotation matrix, translation vector, and 3D coordinates, there is a possibility that errors may accumulate in the later stages. Therefore, by minimizing the so-called reprojection error as expressed in Equation (23), it is possible to homogenize the errors between parameters and calculate the statistically optimal parameters.

[0181] FIG. 19 shows an example of the operation of a social distance analysis device including a camera calibration unit according to this embodiment.

[0182] The coordinate calculation unit 105 executes an "optimization process" of the internal parameters and external parameters calculated in S503 and S504, and two three-dimensional coordinates for each image plane normal vector (e.g., a line segment of the spine) calculated in S505 (S506).

[0183] <Modification> The social distance analysis device including the camera calibration unit in the seventh embodiment can be modified as follows.

[0184] <1> In the calculation of the above formula (23), it is not necessary to optimize all parameters. For example, in the modified example of the sixth embodiment, <2> As mentioned in, if there is a normal vector in the world coordinate space with a known length, the 3D coordinates of both ends of that normal vector (for example, the line segment of the spine) can be fixed values ​​rather than variables. This reduces the number of unknowns, making it possible to optimize equation (23) more stably.

[0185] <2> Although the internal parameter calculation unit 123A does not calculate lens distortion, the lens distortion may be included in the parameters to be optimized in equation (23). That is, even if the image plane normal vector becomes a curve on the image due to lens distortion, the final solution can be obtained by using the image plane normal vector at the center of the image where the lens distortion is small, or by performing the processing up to S505 by assuming that the lens distortion is small, and then including the lens distortion as an unknown in the calculation of equation (23). This allows the coordinate calculation unit 105 to calculate the lens distortion.

[0186] Note that the present disclosure is not limited to the above-mentioned embodiments, and can be appropriately modified within the scope of the present disclosure. For example, only one of the first camera calibration method according to the second embodiment and the second camera calibration method according to the fifth embodiment may be implemented, or both may be implemented. For example, when calculating external parameters, either the method according to the second embodiment or the method according to the fifth embodiment may be used, and when calculating internal parameters and external parameters, the method according to the fifth embodiment may be used.

[0187] Each configuration in the above-described embodiment may be configured by hardware or software, or both, and may be configured by one piece of hardware or software, or may be configured by multiple pieces of hardware or software. Each device and each function (processing) may be realized by a computer 20 having a processor 21 such as a CPU (Central Processing Unit) and a memory 22 which is a storage device, as shown in FIG. 20. For example, a program for performing a method in the embodiment (e.g., processing of a social distance analysis device) may be stored in the memory 22, and each function may be realized by executing the program stored in the memory 22 by the processor 21.

[0188] These programs can be stored and provided to a computer using various types of non-transitory computer readable media. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory)). The programs may also be provided to a computer by various types of transitory computer readable media. Examples of transitory computer readable media include electric signals, optical signals, and electromagnetic waves. The transitory computer readable media can provide the programs to a computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

[0189] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) an estimation unit that estimates skeletal information of a person included in a two-dimensional image captured by a camera based on the image; a calculation unit that calculates camera parameters of the camera based on the estimated skeletal information of the person; a detection unit that detects a social distance of the person using the calculated camera parameters; A detection device comprising: (Appendix 2) the calculation unit calculates the camera parameters based on bone segments corresponding to bones of a person included in the skeletal information; 2. A detection device as described in claim 1. (Appendix 3) the calculation unit acquires a normal vector with respect to a reference plane in the image from a bone segment included in the skeletal information, and calculates the camera parameters based on the acquired normal vector. 3. A detection device as described in appended claim 2. (Appendix 4) The calculation unit obtains the normal vector from a line segment corresponding to the person's spine. 4. A detection device as described in appended claim 3. (Appendix 5) The calculation unit is an acquisition unit that acquires, based on the skeleton information, a plurality of image plane normal vectors in an image plane of the image, the image normal vectors corresponding to a plurality of world coordinate space normal vectors with respect to a reference plane in the world coordinate space in which the image is captured; a normalization coordinate conversion unit that converts image plane start point coordinates and image plane end point coordinates, which are image plane coordinates for the start point and end point of each of the acquired image plane normal vectors, into normalized start point coordinates and normalized end point coordinates, which are normalized coordinates, based on internal parameters of the camera; an external parameter calculation unit that calculates a translation vector and a rotation matrix for the camera using normalized start point coordinates and normalized end point coordinates of each image plane normal vector obtained by the transformation and an evaluation function that is overdetermined for a rotation parameter around the normal to a reference plane in the world coordinate space; 5. The detection device according to claim 1, further comprising: (Appendix 6) The external parameter calculation unit Calculating a scalar triple product for each of the image plane normal vectors obtained by the transformation, the scalar triple product being an inner product of a cross product of the vector of the normalized start coordinate and the vector of the normalized end coordinate of each of the image plane normal vectors obtained by the transformation and a rotation vector about an axis of a first axis of a camera coordinate system in a rotation matrix for the camera; using the calculated sum of the plurality of scalar triple products as the evaluation function, calculating a desired rotation vector that is expected to minimize the sum, and calculating the rotation matrix based on the calculated desired rotation vector; the calculated desired rotation vector is a translation vector for the camera; 6. The detection device according to claim 5. (Appendix 7) the plurality of image plane normal vectors are a first image plane normal vector and a second image plane normal vector, The external parameter calculation unit calculates the rotation vector based on the following formula: 7. A detection device according to claim 6.

number

number

number

number

number

number

number

number

number

number

number

number

[0190] 1. Social Distance Analysis System 10. Detection Device 11 Estimation part 12 Calculation section 13 Detection section 20. Computers 21 Processors 22 Memory 100 Social Distance Analyzer 101 Input section 102 Acquisition Department 103 Camera Calibration Section 104 Human joint detection unit 105 Coordinate calculation unit 106 Storage section 107 Social Distance Judgment Department 108 Display section 111 Normal vector acquisition section 112 Normalization coordinate transformation unit 113 External parameter calculation unit 121 Normal vector acquisition section 122 Projective Depth Calculation Unit 123 Camera Parameter Calculation Unit 123A Internal parameter calculation section 123B External parameter calculation section 200 Cameras

Claims

1. an estimation unit that estimates skeletal information of a person included in a two-dimensional image captured by a camera based on the image; a calculation unit that calculates camera parameters for converting coordinates of a two-dimensional image plane captured by the camera into coordinates of a three-dimensional world coordinate space based on the estimated skeletal information of the person; a detection unit that converts coordinates of a center point of the person in the image plane into coordinates of the world coordinate space using the calculated camera parameters, and detects a social distance range of the person, which is a predetermined range from the converted center point; Equipped with the calculation unit calculates the camera parameters based on bone segments connecting joints of the person included in the skeletal information; the calculation unit selects the bone line segment included in the skeletal information as a normal vector with respect to a reference plane corresponding to the ground in the image, calculates a rotation matrix as the camera parameter from perpendicularity of the selected normal vector with respect to the reference plane, and calculates a translation vector and a focal length as the camera parameters from a difference between a length of the normal vector of the image plane projected onto the world coordinate space and a reference length using the rotation matrix, the reference length being a length in the world coordinate space of the bone corresponding to the bone line segment selected as the normal vector; Detection device.

2. the calculation unit selects, as the normal vector, the bone line segment of the person's spine included in the skeleton information; The detection device according to claim 1 .

3. Estimating skeletal information of a person included in a two-dimensional image captured by a camera based on the image; calculating camera parameters for converting coordinates of a two-dimensional image plane captured by the camera into coordinates of a three-dimensional world coordinate space based on the estimated skeletal information of the person; Using the calculated camera parameters, coordinates of a center point of the person in the image plane are transformed into coordinates of the world coordinate space, and a social distance range of the person is detected, which is a predetermined range from the transformed center point; The calculation of the camera parameters includes calculating the camera parameters based on bone segments connecting joints of the person included in the skeletal information; In calculating the camera parameters, the bone line segment included in the skeletal information is selected as a normal vector with respect to a reference plane corresponding to the ground in the image, a rotation matrix is ​​calculated as the camera parameters from the perpendicularity of the selected normal vector with respect to the reference plane, and a translation vector and a focal length are calculated as the camera parameters using the rotation matrix from a difference between a length of the normal vector of the image plane projected onto the world coordinate space and a reference length, the reference length being a length in the world coordinate space of the bone corresponding to the bone line segment selected as the normal vector. Detection methods.

4. Estimating skeletal information of a person included in a two-dimensional image captured by a camera based on the image; calculating camera parameters for converting coordinates of a two-dimensional image plane captured by the camera into coordinates of a three-dimensional world coordinate space based on the estimated skeletal information of the person; Using the calculated camera parameters, coordinates of a center point of the person in the image plane are transformed into coordinates of the world coordinate space, and a social distance range of the person is detected, which is a predetermined range from the transformed center point; The calculation of the camera parameters includes calculating the camera parameters based on bone segments connecting joints of the person included in the skeletal information; In calculating the camera parameters, the bone line segment included in the skeletal information is selected as a normal vector with respect to a reference plane corresponding to the ground in the image, a rotation matrix is ​​calculated as the camera parameters from the perpendicularity of the selected normal vector with respect to the reference plane, and a translation vector and a focal length are calculated as the camera parameters using the rotation matrix from a difference between a length of the normal vector of the image plane projected onto the world coordinate space and a reference length, the reference length being a length in the world coordinate space of the bone corresponding to the bone line segment selected as the normal vector. A detection program for causing a computer to execute a process.

Citation Information

Patent Citations

  • Low smoking polyvinyl chloride composition

    JP1985011548A

  • Passer-by behavior analysis system

    JP2005346617A

  • Image monitoring device

    JP2019106631A

  • Marker image processing system

    WO2015045834A1