Display device, display method, and display program

Abstract

To provide a display device, a display method, and a display program capable of appropriately grasping a social distance to prevent infection of an infectious disease such as a new coronavirus infection.SOLUTION: A display device 10 includes: a detection unit 11 which detects a social distance range to prevent infection of infectious diseases of a person included in an image acquired from a camera; and a display unit 12 which displays the social distance range detected by the detection unit 11 in a display format corresponding to the risk of infection of infectious diseases.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a display device, a display method, and a display program. [Background technology]

[0002] COVID-19 infections are spreading around the world, having a major impact on society and the economy. In order to prevent droplet infection, one of the routes of infection for COVID-19, it is necessary to maintain social distancing, which is the physical distance between people.

[0003] Related techniques include, for example, Patent Document 1. Patent Document 1 discloses a technique for controlling the reach of droplets emitted from a person by adjusting the environment in a space. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2020 / 044826 Summary of the Invention [Problem to be solved by the invention]

[0005] According to the technology described in Patent Document 1, it is possible to suppress droplet infection by adjusting the environment in a space. However, related technologies such as Patent Document 1 do not take into consideration a method for making social distance visible, so there is a problem that it is difficult to properly grasp social distance.

[0006] In view of such problems, the present disclosure aims to provide a display device, a display method, and a display program that are capable of properly grasping social distancing. [Means for solving the problem]

[0007] The display device of the present disclosure includes a detection unit that detects a social distance range for preventing infection of an infectious disease of a person included in an acquired image, and a display unit that displays the detected social distance range in a display mode corresponding to the risk of infection of the infectious disease.

[0008] The display device of the present disclosure includes a detection unit that detects a social distance range for preventing infection of a person included in an acquired image, and a display unit that displays an area of ​​a predetermined distance range centered at a position higher than the person's feet as the detected social distance range.

[0009] The display method of the present disclosure detects a social distance range for preventing infection of a person included in an acquired image, and displays the detected social distance range in a display manner according to the risk of infection of the infectious disease.

[0010] The display method of the present disclosure detects a social distance range for preventing the spread of infectious diseases to a person included in an acquired image, and displays an area of ​​a predetermined distance range centered at a position higher than the person's feet as the detected social distance range.

[0011] The display program of the present disclosure is a display program for causing a computer to execute a process of detecting a social distance range for preventing infection of an infectious disease of a person included in an acquired image, and displaying the detected social distance range in a display manner according to the risk of infection of the infectious disease.

[0012] The display program of the present disclosure is a display program for causing a computer to execute a process of detecting a social distance range for preventing infection of a person included in an acquired image, and displaying an area of ​​a predetermined distance range centered at a position higher than the person's feet as the detected social distance range. Effect of the Invention

[0013] According to the present disclosure, it is possible to provide a display device, a display method, and a display program that are capable of properly grasping social distance. [Brief description of the drawings]

[0014] [Figure 1] 1 is a configuration diagram showing an overview of a display device according to a first embodiment. [Diagram 2] FIG. 11 is a configuration diagram showing the configuration of a social distance analysis system according to a third embodiment. [Diagram 3] 13 is a flowchart showing the operation of the social distance analysis device according to the third embodiment. [Figure 4] FIG. 11 is a diagram showing an example of an image processed by a social distance analysis device according to the third embodiment. [Diagram 5] 13A to 13C are diagrams illustrating a detection example of the joint detection process according to the third embodiment. [Figure 6] 13 is a flowchart showing a risk determination process according to the third embodiment. [Figure 7] 13 is a flowchart showing a determination result display process according to the third embodiment. [Figure 8] FIG. 11 is a diagram showing a display example of the social distance analysis device according to the third embodiment. [Figure 9] FIG. 11 is a diagram showing a display example of the social distance analysis device according to the third embodiment. [Figure 10] FIG. 11 is a diagram showing a display example of the social distance analysis device according to the third embodiment. [Figure 11] FIG. 11 is a diagram showing a display example of the social distance analysis device according to the third embodiment. [Figure 12] FIG. 13 is a configuration diagram showing the configuration of a social distance analysis system according to a fourth embodiment. [Figure 13] FIG. 13 is a diagram showing a display example of a social distance analysis device according to the fourth embodiment. [Figure 14] FIG. 13 is a configuration diagram showing the configuration of a social distance analysis system according to a fifth embodiment. [Figure 15] FIG. 13 is a diagram showing a display example of a social distance analysis device according to the fifth embodiment. [Figure 16]FIG. 13 is a diagram showing a display example of a social distance analysis device according to the fifth embodiment. [Figure 17] FIG. 13 is a diagram showing a display example of a social distance analysis device according to the fifth embodiment. [Figure 18] FIG. 2 is a configuration diagram showing an overview of the hardware of a computer according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment will be described with reference to the drawings. In the drawings, the same elements are denoted by the same reference numerals, and repeated explanations will be omitted as necessary. (Embodiment 1) 1 shows a configuration of a display device according to embodiment 1. As shown in FIG. 1, a display device 10 according to the present embodiment includes a detection unit 11 and a display unit 12.

[0016] The detection unit 11 detects a social distance range for preventing infection of a person included in an acquired image. The social distance range is a range (area) that specifies the social distance required to prevent infection of a person. The display unit 12 displays the detected social distance range in a display mode according to the risk of infection of the infectious disease. For example, the display mode may be changed to any one of the color, shape, size, etc. of the social distance range.

[0017] In this way, in this embodiment, when making the social distance necessary to prevent the spread of infectious diseases visible, the display mode of the social distance range is changed depending on the risk of infection, making it possible to properly grasp a person's social distance.

[0018] (Embodiment 2) The display device according to the second embodiment has the same configuration as that shown in Fig. 1. As in the first embodiment, the display device 10 according to the present embodiment includes a detection unit 11 and a display unit 12.

[0019] As in the first embodiment, the detection unit 11 detects a social distance range for preventing infection of a person included in an acquired image. Meanwhile, in this embodiment, the display unit 12 displays an area of ​​a predetermined distance range centered at a position higher than the person's feet as the detected social distance range. For example, the color, shape, size, etc. of the social distance range may be fixed in advance, and a circular area of ​​a predetermined size in a top view may be displayed centered at waist height of the person.

[0020] In this way, in this embodiment, when making the social distance necessary to prevent the spread of an infectious disease visible, by displaying an area of ​​a predetermined distance range, such as a circular shape, at a position higher than a person's feet as the social distance range, the social distance can be displayed more easily and can be tailored to the actual risk.

[0021] (Embodiment 3) Hereinafter, the third embodiment will be described with reference to the drawings. FIG. 2 shows an example of the configuration of a social distance analysis system 1 according to the present embodiment. The social distance analysis system 1 is a system for preventing the spread of infectious diseases in public spaces and the like by measuring whether social distance is being maintained between people who are walking or the like from camera footage and displaying the measurement results. As shown in FIG. 2, the social distance analysis system 1 includes a social distance analysis device 100 and a camera 200.

[0022] The camera 200 is an imaging unit such as a surveillance camera that is installed at a predetermined surveillance position and captures images of people in a surveillance space (imaging space). The camera 200 may be a camera with a fixed orientation and installation location, a camera whose orientation can be changed such as a PTZ (Pan Tilt Zoom) camera, or a mobile camera mounted on a mobile body such as a drone. The camera 200 and the social distance analysis device 100 are connected to each other via an arbitrary network so that they can communicate with each other. The camera 200 and the social distance analysis device 100 may be integrated into a single device.

[0023] The social distance analysis device 100 is an analysis device that analyzes (detects) the social distance of a person based on an image of the camera 200, and is also a display device that displays the analysis result. The social distance analysis device 100 includes an input unit 101, an acquisition unit 102, a camera calibration unit 103, a human joint detection unit 104, a coordinate calculation unit 105, a storage unit 106, a social distance determination unit 107, and a display unit 108. Note that the configuration of each unit (block) is an example, and may be configured with other units as long as the method (operation) described later is possible. In addition, the social distance analysis device 100 is realized by a computer device such as a personal computer or a server that executes a program, for example, but may be realized by one device or multiple devices. For example, the storage unit 106 and the display unit 108 may be external devices.

[0024] The input unit 101 is an input interface that acquires information input by a user operating the social distance analysis device 100. For example, the user is a monitor who monitors people who are not maintaining social distance from images of a surveillance camera. The input unit 101 is, for example, a GUI (Graphical User Interface), and information according to user operations is input from an input device such as a keyboard, a mouse, or a touch panel. For example, the input unit 101 accepts information necessary for the camera parameter calculation process of the camera calibration unit 103, and also accepts user operations for display processing of the display unit 108.

[0025] The acquisition unit 102 acquires a two-dimensional image including a person captured by the camera 200. The acquisition unit 102 acquires, for example, an image including a person captured by the camera 200 during a predetermined period (a video including a plurality of images in time series). Note that acquisition is not limited to from the camera 200, and an image including a person prepared in advance may be acquired from a database or the like.

[0026] The camera calibration unit 103 calculates the camera parameters of the camera 200 based on the image acquired from the camera 200 (performs camera calibration). The camera parameters (imaging parameters) are parameters for converting a length in a three-dimensional real world space into a length in a two-dimensional image plane. The camera parameters include internal parameters such as the focal length of the camera, lens distortion, and optical center coordinates, and external parameters such as the position (translation vector) and angle (rotation matrix) of the camera. The camera calibration unit 103 stores the calculated camera parameters (position, angle, focal length, etc.) in the storage unit 106.

[0027] The camera parameters may be calculated from the acquired image, and the calculation method is not limited. For example, the camera calibration unit 103 calculates the camera parameters based on the length of a normal vector (a line segment perpendicular to the ground) extracted from the image and a given reference length. The normal vector may be acquired from a bone segment between the joint points of a person detected by the human joint detection unit 104, or may be specified by a user's input. The bone segment used for the normal vector may be a line segment of any bone included in the skeleton of the person. Although a line segment of a bone at the foot (a line segment between the ankle and the knee) may be used, a line segment that is stable in a nearly vertical state when the person is walking is preferable, and therefore, for example, a line segment of the spine (a line segment between the waist and the neck) is used. In addition, the given reference length is a length corresponding to the line segment used for the calculation, and is, for example, the height of the person or the length of each bone (average value, etc.). The reference length used for the calculation may be input by the user, or may be stored in advance in the storage unit 106. Furthermore, since the normal vector only needs to be perpendicular to the ground, it is not limited to the skeleton of a person, and the vertical lines of a car, truck, etc. may be used. Furthermore, the camera parameters may be calculated by capturing an image of a calibration object (an object whose length is known in advance) with the movement of a person, etc. in the image stopped. Furthermore, if the camera 200 is a camera whose camera angle and zoom value can be changed, such as a PTZ camera, a difference value changed from the reference position may be acquired and reflected in the camera parameters.

[0028] The human joint detection unit 104 detects the joint points of a person in a two-dimensional image based on an image acquired from the camera 200. For example, the human joint detection unit 104 uses a joint estimation technology (skeleton estimation technology) such as OpenPose that utilizes machine learning. The human joint detection unit 104 detects joint points (key points) that are estimated to be the joints of a person based on the characteristics of each part of the person recognized in the image using the joint estimation technology, and further extracts bone lines (bone links) connecting the detected joint points. Note that the detected joint points and bone lines are joint points and bone lines estimated by the joint estimation technology, and may not match the joints and bones of an actual person. It can also be said that the human joint detection unit 104 detects a skeletal structure including joint points and bone lines from the person's feet to the head. The human joint detection unit 104 recognizes people in multiple images (frames) in a time series, and detects the joint points and bone lines of all people recognized in each image.

[0029] The coordinate calculation unit 105 calculates the coordinates (position) of the person in the three-dimensional real world space from the coordinates of the detected joint points and bone segments by using the calculated camera parameters. The calculated coordinates are the coordinates (center coordinates) of the center point of the social distance range of the person. The center point is an arbitrary point included in the bone segment between the joint points detected by the human joint detection unit 104. The center point may be an arbitrary point of the joint point or bone segment from the person's feet to the head, or may be a point of a vertical line that can be extracted from the person. As with camera calibration, the center point may be a point included in the bone segment at the feet, but since a point of a line segment that is stable in a state close to vertical when the person is walking is preferable, for example, the center point is a point included in the spine segment. The coordinate calculation unit 105 calculates the center coordinates of all people in each image (frame) and stores the calculation results in the storage unit 106 in chronological order.

[0030] The storage unit 106 stores information (data) necessary for the operation (processing) of the social distance analysis device 100. For example, the storage unit 106 is a non-volatile memory such as a flash memory, a hard disk drive, or the like. For example, the storage unit 106 stores the camera parameters calculated by the camera calibration unit 103 and the time-series coordinate calculation results calculated by the coordinate calculation unit 105. The storage unit 106 is a camera parameter storage unit, and can also be said to be a coordinate calculation result storage unit. The storage unit 106 also stores, as necessary, the image (video) acquired by the acquisition unit 102, information on joint points and bone segments processed by the human joint detection unit 104, machine learning data for joint estimation, the average height of a person and the length of each bone, the length of social distance, and the like.

[0031] The social distance determination unit (determination unit) 107 determines the state of social distance (risk of infection) of the person based on the calculated position (center coordinates) of the person. For example, the social distance determination unit 107 determines a range of a predetermined length from the center coordinates (center point) of the person as the social distance range. The predetermined length of the social distance may be input by the user or may be stored in advance in the storage unit 106. The social distance determination unit 107 determines the risk of infection based on the distance (social distance) between the positions (center coordinates) of the people. For example, the social distance determination unit 107 determines the risk of infection depending on whether the distance between the people is closer than a predetermined distance. In addition, the social distance determination unit 107 may determine the risk of infection depending on whether the social distance range of the person overlaps with the social distance range of another person, or may determine the risk of infection depending on whether the social distance range of the person includes another person. Furthermore, the social distance may be determined taking into account time in addition to distance. The social distance determination unit 107 may determine whether or not a two-stage alert is necessary based on one criterion, or may determine a multi-stage alert level based on multiple criteria.

[0032] The display unit 108 displays the results of the operation (processing) of the social distance analysis device 100 on the display screen. For example, the display unit 108 is a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 108 displays the image (video) captured by the camera 200, the judgment result of the social distance judgment unit 107, etc. on the display screen of the GUI. For example, the display unit 108 displays the social distance range of a person according to the judgment result (whether an alert is required or not and the alert level) using a two-dimensional top view (plan view) or a three-dimensional bird's-eye view. The display unit 108 changes the display mode of the social distance range of a person according to the judgment result of the risk of infection (status of securing social distance) by the social distance judgment unit 107. For example, as the display mode, the color, shape, size, etc. of the social distance range may be changed according to the risk of infection. Furthermore, the blinking interval may be changed when the social distance range is displayed in a blinking manner. In the present embodiment, as an example, the color of the social distance range is changed according to the distance between multiple people. For example, the social distance range is displayed in a different color or the like so that it is emphasized as the risk of infection increases (alert level). Furthermore, the display unit 108 displays an area of ​​a predetermined shape centered on the position of the person as the social distance range. Any area that allows the social distance to be specified may be displayed, for example, as a circular, semicircular, sector-shaped, elliptical, or other area when viewed from above. In this embodiment, as an example, a circular area of ​​a predetermined size (radius) is displayed.

[0033] The display unit 108 is an example of an output unit that outputs the determination result of the state of social distancing. For example, in addition to or instead of the screen display, the determination result of the risk of infection (state of social distancing) may be output by sound or other notification method (such as email via a network). When sound is output, the output mode (volume, pitch, interval, type of voice message, etc. of the warning sound) may be changed according to the risk of infection, similar to the display mode.

[0034] FIG. 3 shows the operation of the social distance analysis device 100 according to the present embodiment. As shown in FIG. 3, the social distance analysis device 100 acquires an image (video) from the camera 200 (S101). The acquisition unit 102 acquires an image of a person (image capture space) for camera calibration, and also acquires an image of a person (image capture space) for social distance determination. The image for camera calibration and the image for social distance determination may be images captured by the same camera 200, and may be captured at the same or different timings. That is, camera calibration may be performed from a video captured in a first period (about 10 to 60 seconds), and then the social distance may be determined from a video captured in a second period, or camera calibration may be performed and social distance may be determined from a video captured in the same period. At least, it is sufficient that the camera calibration is performed before using the camera parameters in determining the social distance (calculating the coordinates). FIG. 4 shows an example of an image acquired from the camera 200. The example of FIG. 4 is an image in which six people are walking in the image capture space.

[0035] Next, a camera calibration process (S102 to S103) is performed, and a social distance determination process (S104 to S108) is performed. As an example of the camera calibration process, an example using bone segments between joint points of a person will be described. In the camera calibration process, first, the social distance analysis device 100 detects joint points of a person based on an image acquired from the camera 200 (S102). The human joint detection unit 104 detects the joint points of each person in the image plane using a joint estimation technique such as OpenPose, and extracts bone segments connecting the detected joint points.

[0036] Next, the social distance analysis device 100 calculates the camera parameters using the detected bone segments (S103). For example, the camera calibration unit 103 selects the line segments of the person's spine as the normal vector to be used for the calculation. The line segments of the spine of one person in the image may be selected, or the line segments of the spine of multiple people may be selected. When multiple spine segments are selected, the average of the line segments (vectors) may be calculated. The line segments of the spine may be used, or the height of the person (whole skeleton) may be estimated from the line segments of the spine and the line segment (center line segment) corresponding to the estimated height may be used. The camera calibration unit 103 calculates the camera parameters from the line segments of the person's spine and the average value of the length of the spine (or height).

[0037] For example, the normal vector (segment of the spine) may be projected onto a projection plane perpendicular to the ground (reference plane), and the camera parameters may be calculated based on the perpendicularity of the projected normal vector to the ground. The rotation matrix (camera angle) can be calculated by evaluating the perpendicularity of the normal vector projected onto the projection plane. In addition, the translation vector (camera position) and focal length can be calculated from the difference between the length of the normal vector of the two-dimensional image projected onto three-dimensional space and the length in the real world (average length of the bones) using the rotation matrix.

[0038] On the other hand, in the social distance determination process, the social distance analysis device 100 first detects the joint points of people based on the acquired image (S104). As in S102, the human joint detection unit 104 detects the joint points and bone segments of each person in the image acquired from the camera 200.

[0039] Next, the social distance analysis device 100 calculates the coordinates (center coordinates) of the center point of the person on the detected bone line segment using the calculated camera parameters (S105). For example, the coordinate calculation unit 105 selects a point included in the line segment of the person's spine (from the waist to the neck) as the center point of the person. In this example, the waist joint point (the lower end point of the spine line segment) is selected as the center point. The social distance analysis device 100 uses the camera parameters to convert the coordinates of the spine line segment (waist joint point) in the two-dimensional image plane into coordinates (x, y, z) in the three-dimensional real world space. The start point coordinates and end point coordinates of the line segment (vector) may be converted into three-dimensional coordinates to obtain the center coordinates, or the start point coordinates or end point coordinates of the line segment may be converted into three-dimensional coordinates to obtain the center coordinates.

[0040] FIG. 5 shows an example of detecting people's joint points and bone lines from the image in FIG. 4. As shown in FIG. 5, the joint points and bone lines of each of the six people in the image are detected. By detecting people's joint points and bone lines from multiple images (frames) in a time series, the joint points and bone lines of a person who is walking (moving) are detected. In the example in FIG. 5, the three-dimensional coordinates of the center point of the waist are calculated from the spinal lines for each of the six people in the image.

[0041] Next, the social distance analysis device 100 determines the social distance range of the person from the calculated central coordinates of the person (S106). The social distance determination unit 107 determines the range of a three-dimensional space of a predetermined length (for example, 1 m to 2 m) from the central coordinates of the person as the social distance range. In this example, the social distance range is a circular area in a top view with a radius of a predetermined length from the central point. In the example of FIG. 5, a circular area of ​​the social distance range is determined from the central coordinates for each of the six people in the image.

[0042] Next, the social distance analysis device 100 judges the risk of infection of the person based on the central coordinates (position) of the person (S107). FIG. 6 shows a specific example of the risk judgment process according to this embodiment. FIG. 6 shows an example of judging whether or not an alert is necessary in two stages based on one criterion. As shown in FIG. 6, the social distance judgment unit 107 acquires the distance between people (S201), and judges whether or not the acquired distance is shorter than a predetermined distance (S202). The social distance judgment unit 107 calculates the distance in the three-dimensional real world space between the central coordinates of the person and the central coordinates of another person.

[0043] If the distance between people is shorter than a predetermined distance, the social distance determination unit 107 determines that an alert is necessary (high risk of infection, social distance not being maintained) (S203), and if the distance between people is longer than the predetermined distance, it determines that an alert is unnecessary (low risk of infection, social distance being maintained) (S204). The predetermined distance that serves as the determination criterion may be the same as the distance used in the social distance range (e.g., 1m to 2m), or may be a longer distance. In the example of FIG. 5, the distance between each of the six people in the image is calculated, and whether or not an alert is necessary between each person is determined.

[0044] In addition, the necessity of an alert may be determined based on the determined social distance range. For example, if the social distance range of a person overlaps with the social distance range of another person, it may be determined that an alert is necessary, and if the social distance range of a person does not overlap with the social distance range of another person, it may be determined that an alert is unnecessary. In addition, if the social distance range of a person includes another person, it may be determined that an alert is necessary, and if the social distance range of a person does not include another person, it may be determined that an alert is unnecessary.

[0045] Furthermore, the social distance determination unit 107 may determine the risk of infection from the distance between people for each image (frame), or may analyze the positions of multiple images (frames) in a chronological order and determine the risk of infection by taking into account the movement of the analyzed people. For example, if the distance between people remains shorter than a predetermined distance for a certain period of time (if people are lingering), it may be determined that an alert is necessary. This makes it possible to exclude people who pass each other temporarily from the determination.

[0046] Next, the social distance analysis device 100 displays the risk determination result (S108). Fig. 7 shows a specific example of the determination result display process according to this embodiment. As shown in Fig. 7, the display unit 108 acquires the determination result of the social distance determination unit 107 (S301), and determines whether the acquired determination result requires an alert (S302). If the determination result indicates that an alert is required (if the risk of infection is high), the display unit 108 displays the social distance range of the person for whom an alert is required in red (S303), and if the determination result indicates that an alert is not required (if the risk of infection is low), the display unit 108 displays the social distance range of the person for whom an alert is not required in blue (S304).

[0047] 8 and 9 show display examples in the cases where an alert is unnecessary and an alert is necessary. As shown in FIG. 8 and FIG. 9, the display unit 108 displays a three-dimensional overhead view V1 and a two-dimensional top view V2 on a display screen (display window) W1. That is, the display screen W1 has an overhead view display area for displaying the overhead view V1 and a top view display area for displaying the top view V2, and the display unit 108 displays the social distance judgment results in each area. The overhead view V1 and the top view V2 can display multiple images (videos) and social distance ranges in a chronologically synchronized manner. The display unit 108 may display both the overhead view V1 and the top view V2, or may display either one of them.

[0048] The overhead view V1 is a three-dimensional display of the imaging space seen from the camera 200 (a predetermined viewpoint). The display unit 108 displays the input video (image) on the overhead view V1, and synthesizes the social distance range of the person with the video and displays it in a three-dimensional format. The viewpoint may be changed based on the video of multiple cameras. In the overhead view V1, for all people in the video, a center point P1 (e.g., a circle) of the person is displayed at a three-dimensional position corresponding to the calculated center coordinates, and a social distance range D1 of the person is displayed in a three-dimensional range corresponding to the social distance range determined from the center point P1. In this example, the social distance range D1 is a circular (disk-shaped) area of ​​a predetermined size centered on the center point P1 of the person, as in the above-mentioned social distance range determination process. As an example of displaying the social distance range, the outline of the circular area is displayed, but the inside of the circular area may be displayed in the same color as the outline or in another pattern. The predetermined distance range is not limited to being disk-shaped, and may be displayed three-dimensionally as a sphere. The overhead view V1 displays a social distance range D1, which is circular when viewed from above, in three dimensions with a size and shape according to its position. The overhead view V1 also displays a center point P1 and a social distance range D1 according to the position of a person moving within the video. The overhead view V1 may also display joint points and bone segments detected from each person.

[0049] The top view V2 is a two-dimensional plan view of the same imaging space as the overhead view V1, viewed from above. The display unit 108 displays the position of the person in the input video (image) and the image of the social distance range in a two-dimensional format on the top view V2. Note that in addition to the person, an image of another object may be displayed, or the video may be displayed in the same manner as the overhead view. As with the overhead view, the top view V2 displays the center point P2 (e.g., an x ​​mark) of every person in the video at a two-dimensional position corresponding to the calculated center coordinates, and displays a circular social distance range D2 in a two-dimensional range corresponding to the social distance range determined from the center point P2. As with the overhead view V1, the outline of the circular area is displayed as an example of the display of the social distance range, but the inside of the circular area may be displayed in the same color as the outline or in another pattern. Also, as with the overhead view V1, the top view V2 displays the center point P2 and the social distance range D2 according to the position of the person moving in the video.

[0050] For example, in FIG. 8 and FIG. 9, the display color (display mode) is changed according to the distance between people. In FIG. 8, the distance between all people A1 to A6 is greater than a predetermined distance. Therefore, in the bird's-eye view V1 and the top view V2, the social distance ranges D1 and D2 and the center points P1 and P2 of people A1 to A6 are displayed in blue. On the other hand, in FIG. 9, the distance between people A3 to A6 is greater than a predetermined distance, but the distance between person A1 and object A2 is closer than the predetermined distance (part of the social distance ranges overlap). Therefore, in the bird's-eye view V1 and the top view V2, the social distance ranges D1 and D2 and the center points P1 and P2 of people A3 to A6 are displayed in blue, and the social distance ranges D1 and D2 and the center points P1 and P2 of people A1 and A2 are displayed in red. In this example, red (highlighted color) / blue is used to clearly indicate high / low risk of infection, but other colors may be used as long as the risk of infection can be identified. Also, instead of being limited to two colors, a multi-stage gradation display may be used according to the overlap of distance and social distance ranges. The display mode is not limited to color, and the blinking interval, etc. may be changed. For example, the display may blink when an alarm is required, and the blinking interval may be shortened as the distance decreases (as the alert level increases). Furthermore, a mark such as an arrow or a character string indicating a high risk of infection may be displayed. A numerical value of the distance between people may be displayed. Also, the color, etc. of people in a social distance range with a high risk of infection may be changed.

[0051] As in the risk determination process described above, the color (display mode) may be changed according to the risk based on the social distance range. For example, if the social distance range of the first person overlaps with the social distance range of the second person, the social distance range of the first person and the second person may be displayed in red, and if the social distance range of the first person does not overlap with the social distance range of the second person, the social distance range of the first person and the second person may be displayed in blue. Furthermore, the color may be changed according to the amount of overlap of those social distance ranges. Also, if the social distance range of the first person includes the second person, the social distance range of the first person may be displayed in red, and if the social distance range of the first person does not include the second person, the social distance range of the first person may be displayed in blue. Furthermore, the color may be changed according to the density of multiple people. For example, the color may be changed according to the number of people included in a specified area, the number of overlaps (overlapping area) of the social distance ranges of the people, and the number of other people included in the social distance range of the person.

[0052] In the examples of FIG. 8 and FIG. 9, the center point P1 and the circular social distance range D1 are displayed at the position (height) of the person's waist in the overhead view V1. That is, the center point P1 and the social distance range D1 are displayed in a state of floating at a position higher than the person's feet. Since the possibility of droplet infection is higher the closer to the face, by displaying the social distance range at a higher position on the person, it is possible to display more suitable for the risk of infection. For example, the height of the waist at which the center point P1 and the social distance range D1 are displayed is the height of the joint point of the waist detected in the joint detection, and is the height at which the coordinates of the person are calculated. Since the line segment of the spine is used to calculate the coordinates, the height of the lower end of the spine (waist) may be used, or any height of the spine (from the waist to the neck) may be used. Since the positions of the spine and waist are stable when a person is walking, the social distance range can be displayed more easily by displaying the spine and waist as the center.

[0053] For example, the center point P1 and the social distance range D1 may be displayed at any position (height) higher than the person's feet. They may be displayed at any height between the feet and the waist, or at any height from the waist to the head. FIG. 10 is an example of displaying the center point P1 and the social distance range D1 at the position (height) of the person's knees. For example, they may be displayed at a height between the waist and the feet on an extension line of the spine extending from the waist to the feet. Also, when the coordinates of a person are calculated using a bone line from the knee to the feet, or a bone line from the knee to the waist, they may be displayed at the height of the knee included in the line used for the calculation. FIG. 11 is an example of displaying the center point P1 and the social distance range D1 at the position (height) of the person's face. For example, they may be displayed at a height near the head on an extension line of the spine extending from the neck to the head. Also, when the coordinates of a person are calculated using a bone line from the neck to the head, they may be displayed at the height of the face included in the line used for the calculation.

[0054] As described above, in this embodiment, camera calibration is performed from the video captured by the camera, the position and social distance of a person are calculated, and a social distance range of a predetermined shape indicating the calculated position and social distance of the person is displayed on the display screen. In particular, on this display screen, the display mode of the social distance range, such as the color, is changed according to the risk of infection. This allows the social distance of people in the video to be grasped at a glance, and people who are at high risk of infection (people not maintaining social distance) to be clearly identified.

[0055] (Embodiment 4) Hereinafter, the fourth embodiment will be described with reference to the drawings. In this embodiment, an example will be described in which the display mode of the social distance range is changed depending on the direction of a person's face in the social distance analysis device in the third embodiment.

[0056] Fig. 12 shows an example of the configuration of a social distance analysis system 1 (social distance analysis device 100) according to this embodiment. As shown in Fig. 12, the social distance analysis device 100 according to this embodiment further includes a face direction detection unit 109 in addition to the configuration of Fig. 2 of the third embodiment.

[0057] The face direction detection unit 109 detects the face direction of a person included in an image acquired from the camera 200. For example, the face direction detection unit 109 recognizes the face of a person in an image using face recognition technology that uses machine learning, and detects the face direction from the position of the eyes and nose of the recognized face. For example, the risk of droplet infection differs in front of and behind a person's face, with a high risk of infection in front of the face and a low risk of infection behind the face. For this reason, in this embodiment, the risk determination criteria and the display of the social distance range are changed depending on the face direction.

[0058] When determining a social distance range from the center point of a person, the social distance determination unit 107 determines the social distance range so that the range is wider (longer) in front of the person's face than behind it. For example, the social distance range is set to a semicircular or fan-shaped area spreading in front of the face. For example, the social distance determination unit 107 determines the risk of infection based on the determined social distance range. In addition, the social distance determination unit 107 may change the determination criterion according to the direction of the person's face, and determine the risk of infection using a longer threshold value in front of the person's face (longer threshold value than behind), and determine the risk of infection using a shorter threshold value behind the person's face.

[0059] The display unit 108 changes the display mode of the social distance range according to the direction of the person's face. That is, when displaying the social distance range of a person, the display unit 108 displays a semicircular or fan-shaped area in a top view centered on the position of the person so that the area is wider in front of the person's face than behind it, as in the determination of the social distance range described above. FIG. 13 shows a display example in this embodiment. In the example of FIG. 13, in the overhead view V1 and the top view V2, semicircular areas in a top view that extend only in front of the face excluding the back of the face are displayed as the social distance ranges D1 and D2. Note that the social distance range D1 similar to that in the third embodiment may be displayed, and the area that is the social distance range in this embodiment may be displayed as an area that is susceptible to infection.

[0060] As described above, in this embodiment, the display mode of the social distance range is changed according to the direction of a person's face as a display mode according to the risk of infection of an infectious disease. This makes it possible to display the social distance range more in line with the risk of infection. In addition, by changing the criteria for determining social distance according to the direction of a person's face, the risk of infection can be more appropriately determined.

[0061] (Embodiment 5) Hereinafter, a fifth embodiment will be described with reference to the drawings. In this embodiment, an example will be described in which the display mode of the social distance range is changed according to the moving direction and speed of a person in the social distance analysis device in the third or fourth embodiment.

[0062] Fig. 14 shows an example of the configuration of a social distance analysis system 1 (social distance analysis device 100) according to this embodiment. As shown in Fig. 14, the social distance analysis device 100 according to this embodiment further includes a movement detection unit 110 in addition to the configuration of Fig. 2 of the third embodiment.

[0063] The movement detection unit 110 detects the movement of a person included in an image acquired from the camera 200. The movement detection unit 110, for example, analyzes the coordinates of the person stored in the storage unit 106 in time series and detects the direction and speed of movement of the person. For example, when an infected person is running, droplets of the infected person may fly over a wider range than when the infected person is walking, and there is a particular risk that droplets may fly over a long range behind the person. For this reason, in this embodiment, the risk determination criteria and the display of the social distance range are changed according to the direction and speed of movement of the person.

[0064] When determining a social distance range from the center point of a person, the social distance determination unit 107 determines the range that extends in the direction of movement of the person (the distance becomes longer) as the social distance range. For example, the social distance range is set to an elliptical shape centered on the center point of the person, and the range behind the person is made longer (wider) depending on the moving speed. For example, the social distance determination unit 107 determines the risk of infection based on the determined social distance range. In addition, the social distance determination unit 107 may change the determination criteria according to the movement of the person, and determine the risk of infection using a longer threshold value in the forward / backward direction of the person's movement (a longer threshold value than in the left / right direction), and determine the risk of infection using a shorter threshold value in the left / right direction of the person's movement.

[0065] The display unit 108 changes the display mode of the social distance range according to the moving direction or moving speed of the person. That is, when displaying the social distance range of the person, the display unit 108 displays an elliptical area in a top view centered on the position of the person that extends in the moving direction of the person (so that the front-back direction is wider than the left-right direction) as in the determination of the social distance range, and also displays the range behind as long (wide) according to the moving speed. Figures 15 to 17 show display examples in this embodiment. In the example of Figure 15, the social distance ranges D1 and D2 are displayed in an elliptical shape that extends in the front-back direction of the movement, for example, in the overhead view V1 and the top view V2. In the example of Figure 16, the length of the long axis direction of the ellipse of the social distance ranges D1 and D2 is changed according to the speed of the person. For example, if the moving speed of the person A1 is faster than other people, the social distance ranges D1 and D2 of the person A1 are displayed longer in the moving direction than other people. The display is made so that the long axis of the ellipse becomes longer as the moving speed becomes faster. Furthermore, the display may be such that the area behind the person is longer than the area in front of the person depending on the moving speed. Also, as shown in FIG. 17, the social distance ranges D1 and D2 may be displayed as circles, and the size of the circles may be changed depending on the moving speed. For example, if the moving speed of person A1 is faster than that of other people, the circles of the social distance ranges D1 and D2 of person A1 are displayed larger than those of other people. The circles are displayed so that they become larger as the moving speed becomes faster. Note that the same social distance range D1 as in embodiment 3 may be displayed, and the area that is the social distance range in this embodiment may be displayed as an area susceptible to infection.

[0066] As described above, in this embodiment, the display mode of the social distance range is changed according to the direction and speed of a person's movement as a display mode according to the risk of infection of an infectious disease. This makes it possible to display the social distance range more in line with the risk of infection. In addition, by changing the criteria for determining social distance according to the direction and speed of a person's movement, the risk of infection can be more appropriately determined.

[0067] (Other embodiments) Not limited to the above embodiment, in determining social distance (risk of infection), pairs of people presumed to be parent and child may be excluded from the determination. For example, if a tall person and a short person are close to each other for a certain period of time, the two people are presumed to be parent and child. In this case, even if the people are close to each other, it may be determined that an alert is unnecessary, and an alert display may not be performed.

[0068] The display mode of the alert may be changed depending on whether or not the person is wearing a mask or face shield. If a person is wearing a mask, etc., the risk of droplet infection is low, so the display mode is set to indicate a lower risk than if the person is not wearing a mask, etc. For example, if a person is detected as wearing a mask, etc. using face recognition technology, even if the people are close to each other, it may be determined that an alert is unnecessary, and an alert may not be displayed, or a display may be made indicating that the risk is low due to the mask, etc. (e.g., a neutral color display). Similarly, if there is an object between people that prevents droplets, such as a vinyl wall, the display may indicate that the risk of infection is low.

[0069] The present disclosure is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present disclosure.

[0070] Each configuration in the above-described embodiment may be configured by hardware or software, or both, and may be configured by one piece of hardware or software, or may be configured by multiple pieces of hardware or software. Each device and each function (processing) may be realized by a computer 20 having a processor 21 such as a CPU (Central Processing Unit) and a memory 22 which is a storage device, as shown in Fig. 18. For example, a program for performing a method in the embodiment (e.g., a matching method in a cyber-physical person attribute matching device) may be stored in the memory 22, and each function may be realized by having the processor 21 execute the program stored in the memory 22.

[0071] These programs can be stored and provided to a computer using various types of non-transitory computer readable media. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory)). The programs may also be provided to a computer by various types of transitory computer readable media. Examples of transitory computer readable media include electric signals, optical signals, and electromagnetic waves. The transitory computer readable media can provide the programs to a computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

[0072] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) A detection unit that detects a social distance range for preventing infection of a person included in an acquired image; A display unit that displays the detected social distance range in a display mode according to the risk of infection of the infectious disease; A display device comprising: (Appendix 2) The display unit changes any one of the color, shape, and size of the social distance range as the display mode. 2. The display device according to claim 1. (Appendix 3) The display unit displays the social distance range in such a manner that the higher the risk of infection, the more emphasized the range is. 3. The display device according to claim 1 or 2. (Appendix 4) The display unit displays an area of ​​a predetermined shape centered on the position of the person as the social distance range. 4. The display device according to claim 1 , (Appendix 5) The predetermined shape is any one of a circle, a semicircle, a sector, and an ellipse when viewed from above. 5. The display device according to claim 4. (Appendix 6) The display unit changes a display mode of the social distance range according to a distance between a plurality of people included in the image. 6. The display device according to claim 1, (Appendix 7) The display unit changes a display manner of a social distance range between a first person and a second person included in the image depending on whether a distance between the first person and the second person is closer than a predetermined distance. 7. The display device according to claim 6. (Appendix 8) The display unit changes a display manner of the social distance ranges of the first person and the second person according to whether or not the social distance ranges of the first person and the second person included in the image overlap. 7. The display device according to claim 6. (Appendix 9) The display unit changes a display manner of the social distance ranges of the first person and the second person according to an overlap amount between the social distance range of the first person and the social distance range of the second person. 9. The display device according to claim 8. (Appendix 10) The display unit changes a display manner of a social distance range of a first person included in the image depending on whether a second person is included in the social distance range of the first person. 7. The display device according to claim 6. (Appendix 11) The display unit changes a display mode of the social distance range according to a density of a plurality of people included in the image. 11. The display device according to claim 1. (Appendix 12) The display unit changes a display mode of the social distance range according to the number of overlaps of the social distance ranges of the multiple people included in the image. 12. The display device according to claim 11. (Appendix 13) The display unit changes a display mode of the social distance range depending on the number of other people included in the social distance range of the person included in the image. 12. The display device according to claim 11. (Appendix 14) The display unit changes a display mode of the social distance range depending on a direction of the face of the person. 14. The display device according to claim 1 . (Appendix 15) The display unit displays the social distance range so that the front of the person's face is wider than the back of the face. 15. The display device according to claim 14. (Appendix 16) The display unit changes a display mode of the social distance range depending on a moving direction of the person. 16. The display device according to claim 1 . (Appendix 17) The display unit displays the social distance range so that the front and rear are wider than the left and right with respect to the direction of movement of the person. 17. The display device according to claim 16. (Appendix 18) The display unit changes a display mode of the social distance range according to a moving speed of the person. 18. The display device according to claim 1. (Appendix 19) The display unit displays the social distance range so that it becomes larger as the moving speed of the person becomes faster. 19. The display device according to claim 18. (Appendix 20) The display unit displays the social distance range in a three-dimensional format seen from a predetermined viewpoint in an overhead view display area of ​​a display screen. 20. A display device according to any one of claims 1 to 19. (Appendix 21) The display unit displays the social distance range together with the acquired image in the overhead view display area. 21. The display device according to claim 20. (Appendix 22) The display unit displays the social distance range in a top view display area of ​​a display screen in a two-dimensional format viewed from above. 22. The display device according to claim 1 . (Appendix 23) The display unit displays the social distance ranges in the bird's-eye view display area and the top view display area in a time-series synchronized manner. 23. The display device according to any one of claims 20 to 22. (Appendix 24) A detection unit that detects a social distance range for preventing infection of a person included in an acquired image; a display unit that displays an area of ​​a predetermined distance range centered at a position higher than the feet of the person as the detected social distance range; A display device comprising: (Appendix 25) The height of the center is the height of the person's waist. 25. The display device according to claim 24. (Appendix 26) The region of the predetermined distance range is a circular region in a top view. 26. The display device according to claim 24 or 25. (Appendix 27) Detecting a social distance range for preventing infection of people included in the acquired image; The detected social distance range is displayed in a display mode according to the risk of infection of the infectious disease. Display method. (Appendix 28) In the display, any one of the color, shape, and size of the social distance range is changed as the display mode. Display method as described in Appendix 27. (Appendix 29) Detecting a social distance range for preventing infection of people included in the acquired image; and displaying an area of ​​a predetermined distance range centered at a position higher than the feet of the person as the detected social distance range. Display method. (Appendix 30) The height of the center is the height of the person's waist. Display method as described in Appendix 29. (Appendix 31) Detecting a social distance range for preventing infection of people included in the acquired image; The detected social distance range is displayed in a display mode according to the risk of infection of the infectious disease. A display program for causing a computer to execute processing. (Appendix 32) In the display, any one of the color, shape, and size of the social distance range is changed as the display mode. 32. The display program according to claim 31. (Appendix 33) Detecting a social distance range for preventing infection of people included in the acquired image; and displaying an area of ​​a predetermined distance range centered at a position higher than the feet of the person as the detected social distance range. A display program for causing a computer to execute processing. (Appendix 34) The height of the center is the height of the person's waist. 34. The display program according to claim 33. [Explanation of symbols]

[0073] 1. Social Distance Analysis System 10 Display device 11 Detection section 12 Display section 20. Computers 21 Processors 22 Memory 100 Social Distance Analyzer 101 Input section 102 Acquisition Department 103 Camera Calibration Section 104 Human joint detection unit 105 Coordinate calculation unit 106 Storage section 107 Social Distance Judgment Department 108 Display section 109 Face direction detection unit 110 Movement detection unit 200 Cameras

Claims

1. a person-to-person distance acquisition unit that acquires a distance between a first person and a second person other than the first person included in an image acquired by a camera, A display mode that allows a social distance range for preventing infection of an infectious disease to be visually recognized is superimposed on the image near the first person; changing the display mode based on the distance; changing the display mode according to a face direction of the first person; Display device.

2. displaying the display mode in the vicinity of the first person based on a position of the first person who is moving; The display device according to claim 1 .

3. An output unit that outputs a sound based on the distance is further provided. The display device according to claim 1 .

4. Changing any one of the color, shape, and size of the social distance range as the display mode; The display device according to claim 1 .

5. a person-to-person distance acquisition unit that acquires a distance between a first person and a second person other than the first person included in an image acquired by a camera, A display mode that allows a social distance range for preventing infection of an infectious disease to be visually recognized is superimposed on the image near the first person; changing the display mode based on the distance; changing the display mode in accordance with a moving direction or a moving speed of the first person; Display device.

6. displaying an area of ​​a predetermined distance range centered at a position higher than the feet of the first person as the social distance range; The display device according to claim 1 .

7. Obtaining a distance between a first person and a second person other than the first person included in an image obtained from a camera; A display mode that allows a social distance range for preventing infection of an infectious disease to be visually recognized is superimposed on the image near the first person; changing the display mode based on the distance; changing the display mode according to a face direction of the first person; Display method.

8. Obtaining a distance between a first person and a second person other than the first person included in an image obtained from a camera; A display mode that allows a social distance range for preventing infection of an infectious disease to be visually recognized is superimposed on the image near the first person; changing the display mode based on the distance; changing the display mode in accordance with a moving direction or a moving speed of the first person; Display method.

9. Obtaining a distance between a first person and a second person other than the first person included in an image obtained from a camera; A display mode that allows a social distance range for preventing infection of an infectious disease to be visually recognized is superimposed on the image near the first person; changing the display mode based on the distance; changing the display mode according to a face direction of the first person; A display program for causing a computer to execute processing.

10. Obtaining a distance between a first person and a second person other than the first person included in an image obtained from a camera; A display mode that allows a social distance range for preventing infection of an infectious disease to be visually recognized is superimposed on the image near the first person; changing the display mode based on the distance; changing the display mode in accordance with a moving direction or a moving speed of the first person; A display program for causing a computer to execute processing.

Images