Real-world monitoring system

Abstract

The objective is to provide a real-space monitoring system that allows for easy understanding of the status of dynamic and static point cloud data in real space through human visual inspection, while respecting the privacy of dynamic objects. [Solution] The system comprises an image data acquisition device 1 that acquires static 3D image data in real space, a point cloud data acquisition device 2 that sequentially acquires real-time point cloud data in real space, a server device 3 connected to the image data acquisition device 1 and the point cloud data acquisition device 2 in a communication-enabled manner, and an information terminal device 4 connected to the server device 3 in a communication-enabled manner. The server device 3 integrates the static 3D image data in real space transmitted from the image data acquisition device 1 and the real-time point cloud data in real space transmitted from the point cloud data acquisition device 2 and transmits it to the information terminal device 4.

Description

Technical Field

[0001] The present invention relates to a real-space monitoring system used for monitoring the situation of a real space based on image data and point cloud data in the real space.

Background Art

[0002] In recent years, autonomous driving such as micromobility, mobile robots, and autonomous vehicles has become widespread. For this autonomous driving, remote monitoring of visual inspections by humans may be required due to regulations and the like.

[0003] Generally, in micromobility, mobile robots, autonomous vehicles, etc., various sensors for acquiring various data in the real space are provided at low positions. Therefore, by providing various sensors on the environmental side such as utility poles and columns in the real space, it is possible to acquire various data of the entire real space while covering the dead zones of micromobility, mobile robots, autonomous vehicles, etc.

[0004] And recently, in addition to cameras and millimeter-wave radars, a sensor called LiDAR (light detection and ranging) has attracted attention. This LiDAR is a type of sensor that uses laser light, and has a higher radiation beam density than radio waves. By irradiating a target object while scanning it with laser light of a short wavelength, in addition to the distance to the target object, the position and shape of the target object can be accurately detected. Therefore, when using LiDAR in a real space where static objects such as walls and dynamic objects such as automobiles exist, three-dimensional information of the real space can be acquired as image sensor data consisting of a point cloud of laser light. Moreover, by using LiDARs arranged in the surrounding environment such as roadside signal lights and street corners in addition to the LiDARs mounted on micromobility, mobile robots, and autonomous vehicles, these can transmit image sensor data to automobiles to inform about the situation and danger of road dead zones.

[0005] However, since point cloud data acquired by LiDAR consists only of multiple point clouds of a single color, it has been difficult to grasp the state of static and dynamic objects in real space through human visual inspection. Therefore, a technique has been proposed to combine point cloud data acquired by LiDAR with image data acquired by a camera (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] K. Kwak et al., "Extrinsic calibration of a single line scanning lidar and a camera," in 2011 IEEE / RSJ International Conference on Intelligent Robots and Systems, 2011, pp. 3283-3289 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, the images acquired by the camera displayed detailed information about moving objects (such as people's faces and vehicle license plates), which presented a problem in terms of protecting the privacy of these moving objects. Furthermore, at night, the image data acquired by the camera was dark and difficult to see, making it difficult for humans to visually inspect the point cloud data and understand the state of moving and static objects in real space.

[0008] This invention has been made in view of the above-mentioned problems, and aims to provide a real-space monitoring system that allows for easy understanding of the status of dynamic and static objects in point cloud data in real space through human visual inspection, while giving consideration to the privacy of dynamic objects, and thereby enabling easy and reliable monitoring of real space. [Means for solving the problem]

[0009] To achieve the above objective, the present invention provides a real-space monitoring system used for monitoring the conditions in real space, comprising: an image data acquisition device for acquiring static image data in real space; a point cloud data acquisition device for sequentially acquiring real-time point cloud data in real space; and a server device connected to the image data acquisition device and the point cloud data acquisition device in a communicative manner, wherein the server device includes an image data storage unit for storing static image data in real space acquired at predetermined timings by the image data acquisition device; a selection unit for selecting point cloud data of a predetermined frame from each frame of real-time point cloud data in real space sequentially acquired by the point cloud data acquisition device; and the image data storage unit. The system is characterized by comprising: a calibrator unit that performs calibration between static image data of a predetermined timing in real space stored in the image data storage unit and point cloud data of a predetermined frame in real space selected by the selection unit; an integration unit that generates integrated image data by sequentially integrating real-time point cloud data acquired sequentially by the point cloud data acquisition device with the static image data of a predetermined timing in real space stored in the image data storage unit; and a transmission unit that transmits the integrated image data generated by the integration unit to another device.

[0010] Furthermore, the calibrator unit may use static image data acquired at a time when no moving objects exist in real space for calibration.

[0011] Furthermore, the calibrator unit may use static image data acquired at a time when the real space is bright enough to be visually observed for calibration.

[0012] Furthermore, the selection unit may select point cloud data from each frame of real-time point cloud data acquired sequentially in real space by the point cloud data acquisition device for a predetermined frame in which no dynamic objects exist in real space.

[0013] Furthermore, the calibrator unit may periodically perform calibration between static image data of a predetermined timing in real space stored in the image data storage unit and point cloud data of a predetermined frame in real space selected by the point cloud data selection unit.

[0014] Furthermore, when the calibrator unit periodically performs calibration between static image data at a predetermined timing in real space and point cloud data of a predetermined frame, the selection unit may compare the initial frame of the point cloud data selected when the calibration between static image data in real space and real-time point cloud data was first performed by the calibrator unit with each frame of the real-time point cloud data, and select the point cloud data of a predetermined frame in which the position of each point cloud is closest to the point cloud data of the initial frame in real space.

[0015] Furthermore, when the calibrator unit periodically performs calibration between static image data at a predetermined timing in real space and point cloud data of a predetermined frame, the selection unit may compare the first frame of the point cloud data P selected when the calibration between static image data in real space and real-time point cloud data was first performed with each frame of the real-time point cloud data Q, and select the point cloud data Q of a predetermined frame in which the position of each point cloud is closest to the point cloud data P of the first frame in real space. In this case, the selection unit may select the point cloud data Q of a predetermined frame in which the DM value is smallest in the [1] below.

[0016] [Formula 1] JPEG2026081985000002.jpg1766P={p1, p2, p3,..., pn} Q={q1, q2, q3,..., qk} C(q, P): The point in the point cloud P that is closest to q. n: Number of points in the point cloud data P k: Number of points in the point cloud data Q

[0017] Further, when the calibration unit periodically performs calibration between static image data at a predetermined timing in the real space and point cloud data of a predetermined frame by the calibrator unit, when the calibration between the static image data in the real space and the real-time point cloud data is performed for the first time, the first frame of the point cloud data P selected at that time is compared with each frame of the real-time point cloud data Q, and when selecting the point cloud data Q of a predetermined frame whose position of the point cloud data P in the first frame in the real space is closest to each point cloud, the point cloud data Q of a predetermined frame in which the DM value becomes the smallest value in [Equation 2][Equation 3] may be selected.

[0018] [Equation 2] JPEG2026081985000003.jpg1767[Equation 3] JPEG2026081985000004.jpg1960P = {p1, p2, p3, ···, pn} Q = {q1, q2, q3, ···, qk} C(q, P): The point in point cloud P closest to q n: The number of points in point cloud data P k: The number of points in point cloud data Q

[0019] Further, the image data acquisition unit may include a camera unit that captures a two-dimensional image of the real space, a scanner unit that acquires three-dimensional data of the real space, and an image generation unit that generates three-dimensional image data of the real space based on the two-dimensional image of the real space captured by the camera unit and the three-dimensional data of the real space acquired by the scanner unit.

[0020] Further, the integration unit may switch the viewpoint in the integrated image data in the real space.

[0021] Further, the integration unit may detect a moving object of the point cloud data in the integrated image data in the real space and continuously or sequentially switch to a viewpoint that tracks the moving object.

[0022] Further, the transmitting unit may preferentially transmit integrated image data related to the moving object in the point cloud data and the space in the vicinity of the moving object to other devices in the integrated image data in the real space.

[0023] Further, the integrating unit may extract point cloud data limited to a predetermined viewpoint and integrate the point cloud data with the corresponding three-dimensional image data.

[0024] Further, the integrating unit may extract point cloud data within a predetermined range of the Euclidean distance from the position coordinates of a predetermined viewpoint and / or the cosine similarity of the direction vector of the predetermined viewpoint.

[0025] Further, the present invention is a computer program used in a real space monitoring system including an image data acquisition device that acquires static image data in the real space, a point cloud data acquisition device that sequentially acquires real-time point cloud data in the real space, and a server device connected to the image data acquisition device and the point cloud data acquisition device in a communicable state. The server device includes an image data storage unit that stores static image data in the real space acquired at a predetermined timing by the image data acquisition device, a selection unit that selects point cloud data of a predetermined frame from each frame of the real-time point cloud data in the real space sequentially acquired by the point cloud data acquisition device, a calibration unit that performs calibration between the static image data at a predetermined timing in the real space stored in the image data storage unit and the point cloud data of a predetermined frame in the real space selected by the selection unit, and after the calibration of the static image at a predetermined timing in the real space and the point cloud data of a predetermined frame is performed by the calibration unit, an integration unit that generates integrated image data obtained by sequentially integrating the real-time point cloud data in the real space sequentially acquired by the point cloud data acquisition device with the static image data at a predetermined timing in the real space stored in the image data storage unit, and is characterized by causing the server device to function as a transmission unit that transmits the integrated image data generated by the integration unit to other devices.

Effect of the Invention

[0026] According to the present invention, while respecting the privacy of dynamic objects, the status of dynamic and static objects in point cloud data in real space can be easily grasped by human visual inspection, and consequently, real space can be monitored easily and reliably. [Brief explanation of the drawing]

[0027] [Figure 1] This figure shows the overall configuration of a real-space monitoring system according to an embodiment of the present invention. [Figure 2] This figure shows the configuration of the 3D data acquisition device shown in Figure 1. [Figure 3] This diagram shows the configuration of the point cloud data acquisition device shown in Figure 1. [Figure 4] This diagram shows the configuration of the server device shown in Figure 1. [Figure 5] This diagram shows the configuration of the information terminal device shown in Figure 1. [Figure 6] This figure shows an example of three-dimensional image data according to the first embodiment. [Figure 7] This figure shows an example of point cloud data according to the first embodiment. [Figure 8] This figure shows a first example of integrated image data according to the first embodiment. [Figure 9] This figure shows a second example of integrated image data according to the first embodiment. [Figure 10] This figure shows an example of 3D image data related to the embodiment. [Figure 11] This figure shows an example of point cloud data related to the embodiment. [Figure 12] This figure shows the correlation between DM values, EM values, and calibration error. [Figure 13] This figure shows the frame-by-frame errors for DM value, EM value, and calibration error. [Figure 14] This figure shows an example of integrated image data according to the second embodiment. [Figure 15] This figure shows the relationship between point cloud data and viewpoint according to the third embodiment. [Figure 16] This figure shows an example of integrated image data from a predetermined viewpoint according to the third embodiment. [Modes for carrying out the invention]

[0028] <First Embodiment> Next, a first embodiment of the real-space monitoring system according to the present invention (hereinafter referred to as "this system") will be described with reference to Figures 1 to 9. In this embodiment, "real space" refers to a predetermined space where indoor and outdoor monitoring is desired, such as an outdoor area like an intersection or road, or the interior of a building or facility.

[0029] [Overall structure] As shown in Figure 1, this system comprises an image data acquisition device 1 that acquires static 3D image data in real space, a point cloud data acquisition device 2 that sequentially acquires real-time point cloud data in real space, a server device 3 connected to the image data acquisition device 1 and the point cloud data acquisition device 2 in a communication-enabled manner, and an information terminal device 4 connected to the server device 3 in a communication-enabled manner. The server device 3 integrates the static 3D image data in real space transmitted from the image data acquisition device 1 and the real-time point cloud data in real space transmitted from the point cloud data acquisition device 2 and transmits it to the information terminal device 4.

[0030] The configurations and functions of the image data acquisition device 1, point cloud data acquisition device 2, server device 3, and information terminal device 4 will be described in detail below.

[0031] [Configuration of Image Data Acquisition Device 1] The image data acquisition device 1 includes a camera unit 11 that captures a two-dimensional image of real space, a scanner unit 12 that acquires three-dimensional data of real space, an image generation unit 13 that generates static three-dimensional image data of real space based on the two-dimensional image and three-dimensional data of real space, and a transmission unit 14 that transmits the static three-dimensional image data generated by the image generation unit 13 to the server device 3.

[0032] The camera unit 11 is a general-purpose optical machine for imaging a predetermined real space, and it captures a two-dimensional color or monochrome image of the real space. This two-dimensional image may be a static image captured at a predetermined timing, or it may be a static image extracted at a predetermined timing from a continuously captured video.

[0033] The scanner unit 12 is a so-called 3D scanner that acquires three-dimensional data of a predetermined real space, and acquires point cloud data relating to the position and distance of static objects such as walls, corridors, pillars, and furniture that exist in the real space using a laser or the like. This scanner unit 12 may use LiDAR, which will be described later, or other devices.

[0034] The image generation unit 13 generates static 3D image data in real space by combining a 2D image of real space captured by the camera unit 11 with 3D data of real space acquired by the scanner unit 12.

[0035] In this embodiment, as shown in Figure 6, when the image generation unit 13 generates static 3D image data of real space, it generates static 3D image data of real space based on a 2D image of real space captured by the camera unit 11 and 3D data of real space acquired by the scanner unit 12, at a time when no dynamic objects such as vehicles or people are present. As a result, since the static 3D image data of real space does not include dynamic objects such as vehicles or people, the privacy of dynamic objects can be considered.

[0036] Furthermore, when generating static 3D image data of real space, the image generation unit 13 generates static 3D image data of real space based on a 2D image of real space captured by the camera unit 11 and 3D data of real space acquired by the scanner unit 12, at a time when the real space is visible to the human eye (for example, on a sunny day). As a result, even at dark times such as at night, as will be described later, the status of dynamic and static objects in point cloud data in real space can be easily grasped by human visual inspection.

[0037] [Configuration of Point Cloud Data Acquisition Device 2] The point cloud data acquisition device 2 includes a sensor unit 21 that acquires point cloud data in real space sequentially in real time, a data processing unit 22 that performs predetermined processing on the point cloud data in real space, and a transmission unit 23 that transmits the point cloud data to the server device 3 sequentially or periodically.

[0038] The aforementioned sensor unit 21 is a sensor known as LiDAR (light detection and ranging). This LiDAR is a type of sensor that uses laser light, and compared to radio waves, it has a higher density of radiant flux. By scanning and irradiating an object with short-wavelength laser light, it acquires image sensor data consisting of point clouds in real space, and accurately detects not only the distance to the object but also the position and shape of the object.

[0039] Examples of LiDAR systems include those that acquire 360-degree omnidirectional point cloud data by rotating multiple laser emitters, and those that acquire image sensor data by directly irradiating with laser light within a predetermined light irradiation angle range. Furthermore, while increasing the number of laser emitters improves the accuracy of the point cloud data, it also increases the cost, so inexpensive LiDAR systems with fewer emitters may be used.

[0040] The data processing unit 22 performs aggregation and smoothing processing on the point cloud data in real space acquired by the sensor unit 21 at predetermined frame intervals as needed, thereby constructing one frame of point cloud data in real time for each predetermined scan by the sensor unit 21.

[0041] Figure 7(a) shows an example of point cloud data for a frame without dynamic objects, while Figure 7(b) shows an example of point cloud data for a frame with dynamic objects (pedestrians).

[0042] [Configuration of Server Device 3] The server device 3 includes an image data storage unit 31 that stores static 3D image data in real space, a receiving unit 32 that receives real-time point cloud data in real space, a selection unit 33 that selects point cloud data for a predetermined frame, a calibrator unit 34 that performs calibration of static 3D image data and point cloud data in real space, an integration unit 35 that sequentially integrates real-time point cloud data with static 3D image data in real space, and a transmission unit 36 ​​that transmits integrated image data, which is an integration of static 3D image data and real-time point cloud data in real space, to the information terminal device 4.

[0043] The image data storage unit 31 stores static 3D image data transmitted from the image data acquisition device 1 at a predetermined timing in real space. In this embodiment, this 3D image data includes at least static 3D image data acquired at a time when no dynamic objects such as vehicles or people exist in real space, and when the real space is bright enough to be visually observed (for example, on a sunny day).

[0044] The receiving unit 32 receives real-time point cloud data in real space that has been transmitted sequentially or periodically from the point cloud data acquisition device 2.

[0045] The selection unit 33 selects point cloud data for a predetermined frame from each frame of real-time point cloud data in real space received by the receiving unit 32. Specifically, when the calibrator unit 34 performs calibration between static 3D image data at a predetermined timing in real space and point cloud data for a predetermined frame, the selection unit 33 selects point cloud data for a frame in which no dynamic objects such as people or vehicles exist in real space. As a result, when the calibrator unit 34 performs calibration between 3D image data and point cloud data in real space, as will be described later, the calibration can be performed with high accuracy under the same or similar conditions in which no dynamic objects exist between the 3D image data and point cloud data in real space.

[0046] Furthermore, the selection unit 33 compares the initial frame of the point cloud data P selected when the calibration of static 3D image data in real space and real-time point cloud data was first performed by the calibrator unit 34 with each frame of the real-time point cloud data Q, and selects the point cloud data Q of a predetermined frame in which the position of each point cloud is closest to the point cloud data P of the initial frame. As a result, since the real-time point cloud data Q closest to the point cloud data P selected when the calibration was first performed is selected, it is possible to easily select the point cloud data Q of a frame that is optimal for calibration where there are no dynamic objects, which are the same or similar conditions as the initial point cloud data P.

[0047] In this case, the selection unit 33 selects the real-time point cloud data Q whose DM value is the smallest in the following equation [1].

[0048] [Formula 1] JPEG2026081985000005.jpg1766P={p1, p2, p3,..., pn} Q={q1, q2, q3,..., qk} C(q, P): The point in the point cloud P that is closest to q. n: Number of points in the point cloud data P k: Number of points in the point cloud data Q Alternatively, the selection unit 33 may select the real-time point cloud data Q that has the smallest EM value in the following equations [2][3].

[0049] [Formula 2] JPEG2026081985000006.jpg1767[Formula 3] JPEG2026081985000007.jpg1960

[0050] To explain DM and EM in more detail, when the calibrator unit 34 performs the initial calibration of static 3D image data in real space with real-time point cloud data, the frame of point cloud data selected is represented as P={p1, p2, p3, ..., pn}, where pi(i=1,2,3…) represents an individual point and n is the number of points. In periodic calibration, a predetermined frame of real-time point cloud data is appropriately selected, and this frame is represented as Q={q1, q2, q3, ..., qk}, where qi(i=1,2,3…) represents an individual point and k is the number of points.

[0051] The Distance Metric (DM) is based on the hypothesis that elements shared between two point cloud datasets shorten the distance between nearest neighbors. A smaller DM value indicates more shared elements and more similar frames, which is expressed by equation [1] above, where C is a function for finding the nearest point in the point cloud.

[0052] On the other hand, EM (Effective Metric) is based on the hypothesis that the more inappropriate points there are in the point cloud, the lower the accuracy of the calibration. A smaller EM value indicates fewer inappropriate points in the frame and is expressed by equation [2] above. Here, D is expressed by equation [3] above.

[0053] Furthermore, during periods when calibration is not being performed (between calibration timings), the selection unit 33 sequentially transmits point cloud data of multiple real-time frames in real space, received by the receiving unit 32, to the integration unit 35.

[0054] The calibrator unit 34 performs calibration to align the spatial coordinate axes of static 3D image data of a predetermined timing in real space stored in the image data storage unit 31 and point cloud data of a predetermined frame in real space selected by the selection unit 33.

[0055] Specifically, the calibrator unit 34 first performs calibration in the initial calibration using static 3D image data in real space stored in the image data storage unit 31 at a predetermined timing in real space (a timing when no moving objects exist in real space and the real space is bright enough to be visually observed), and point cloud data of a predetermined frame in real space (a frame in which no moving objects exist) selected by the selection unit 33. At this time, the 3D image data and point cloud data in real space may be from the same timing or from different timings, but it is preferable that both are from a timing when no moving objects exist.

[0056] Furthermore, the calibrator unit 34 periodically performs calibration between static 3D image data at predetermined timings in real space, stored in the image data storage unit 31, and point cloud data of predetermined frames in real space selected by the selection unit 33. This is because, when monitoring real space for a long period of time, the coordinates of the 3D image data and point cloud data in real space may shift due to, for example, a shift in the direction or position of the camera unit 11 or scanner unit 12 of the image data acquisition device 1, or the sensor unit 21 of the point cloud data acquisition unit. By periodically performing this calibration between the 3D image data and point cloud data in real space, the integration unit 35 can accurately superimpose and display the point cloud data on the 3D image data in real space, as will be described later. Note that the period in this invention is not necessarily limited to a constant period, and the selected period may be changed midway through the process.

[0057] The integration unit 35 generates integrated image data by sequentially integrating real-time point cloud data acquired sequentially by the point cloud data acquisition device 2 with the static 3D image data at predetermined timings stored in the image data storage unit 31, after the calibration of static 3D image data at predetermined timings and point cloud data for predetermined frames has been performed by the calibrator unit 34. This integrated image data in real space is transmitted sequentially or periodically to the information terminal device 4 by the transmission unit 36.

[0058] For example, Figure 8 shows integrated image data of a dynamic object (pedestrian h) walking in a straight line from the back of the corridor to the front in a real indoor space where the corridor is located at the back and the elevator hall is located at the front. Figure 9 shows integrated image data of a dynamic object (pedestrian h) walking in the elevator hall from the back to the front and then back again.

[0059] Specifically, Figure 8(a) shows integrated image data where no dynamic object exists in real space yet, Figure 8(b) shows integrated image data where a dynamic object (pedestrian h) exists at the far end of the corridor, and Figure 8(c) shows integrated image data where a dynamic object (pedestrian h) exists at the near end of the corridor.

[0060] Furthermore, Figure 9(a) shows integrated image data where a dynamic object (pedestrian h) is present at the back of the elevator hall, Figure 9(b) shows integrated image data where a dynamic object (pedestrian h) is present at the front of the elevator hall, and Figure 9(c) shows integrated image data where a dynamic object (pedestrian h) is again present at the back of the elevator hall.

[0061] According to this method, real-time point cloud data is sequentially superimposed on static 3D image data in a given real space, making it possible to understand the situation of dynamic objects walking in corridors and elevator lobbies within that given space. Moreover, since dynamic objects do not exist in static 3D image data, the status of real-time point cloud data (for example, the movement of point clouds of dynamic objects) can be easily understood on the 3D image in real space. Furthermore, since dynamic objects in point cloud data are simply point clouds and details of people or vehicles cannot be identified, the privacy of dynamic objects can be respected.

[0062] [Configuration of Information Terminal Device 4] The information terminal device 4 is a general-purpose personal computer, tablet terminal, or smartphone, and includes a receiving unit 41 that receives an integrated image which is a combination of static 3D images and point cloud data of the real space transmitted from the server device 3, and a display unit 42 that displays the integrated image of the real space received by the receiving unit 41. This allows the monitor to monitor the real space by viewing the integrated image data of the real space displayed on the display unit 42. [Examples]

[0063] Next, an embodiment of the first embodiment of this system will be described with reference to Figures 10 to 13.

[0064] Figure 10 visualizes 3D image data of a real-world room. An iPad Pro (iPad is a registered trademark) was used for image data acquisition device 1, and a Livox Avia was used for the LiDAR unit of point cloud data acquisition device 2. The LiDAR unit operated at 10 fps. 71 frames of data were used for evaluation. These frames consist of 32 frames containing information about dynamic objects (pedestrians) and 39 frames containing only information about static objects. Figure 11 shows the point cloud data frames where pedestrians are present.

[0065] A Point Cloud Library (PCL) was used for calibration. First, downsampling was performed on both the 3D image data and the real-time point cloud data. Next, Fast Point Feature Histograms (FPFH) features were calculated for both sets of data. Then, Sample Consensus Initial Alignment (SAC-IA) was performed for initial rough alignment. Finally, Iterative Closest Point (ICP) was used for precise alignment. The correlation between DM, EM values ​​and calibration error was investigated. The real-time data was split into two frame sets: point cloud data P for the initial frames and point cloud data Q for the remaining frames. The calibration error is based on the Mean Squared Error (MSE) of the nearest-neighbor point distance between the two point cloud sets. A smaller calibration error indicates higher calibration accuracy.

[0066] Figure 12(a) is a scatter plot of DM and calibration error values, showing a strong correlation with each other, with a correlation coefficient of 0.9934. On the other hand, Figure 12(b) is a scatter plot of EM and calibration error, also showing a strong correlation, with a correlation coefficient of 0.949. Figure 13 shows the values ​​of DM, EM, and calibration error for each frame number, and it can be seen that all three values ​​increase. During the period of these increasing frames, a dynamic object (pedestrian) was walking in the experiment. From the above results, it can be seen that DM is a better metric than EM for frame selection to maximize the accuracy of periodic calibration in the proposed system.

[0067] <Second Embodiment> Next, a second embodiment of this system will be described with reference to Figure 14. In the following description, only configurations that differ from the above embodiment will be described, and identical configurations will be omitted from the description and given the same reference numerals.

[0068] In this embodiment, the integration unit 35 detects dynamic objects in the point cloud data within the integrated image data in real space and continuously or sequentially switches to a viewpoint that tracks the dynamic objects.

[0069] For example, consider a case where the integration unit 35 generates integrated image data, as shown in Figure 14, in which real-time point cloud data is superimposed onto a static 3D image of a real space (corridor) at a predetermined timing. First, as shown in Figure 14(a), integrated image data is displayed in which the point cloud data of a dynamic object (pedestrian h) is located on the near side of the corridor. Then, as shown in Figure 14(b), integrated image data is successively generated in which the point cloud data of the dynamic object moves towards the far side of the corridor as the dynamic object (pedestrian h) walks further away. At this time, the integration unit 35 detects the dynamic object (pedestrian h) and automatically switches to a viewpoint that tracks the dynamic object (pedestrian h) (a viewpoint behind the dynamic object (pedestrian h)), as shown in Figure 14(c). This makes it possible to monitor the real space while tracking the dynamic object (pedestrian h) in the real space.

[0070] Furthermore, when the transmission unit 36 ​​of the server device 3 transmits integrated image data to the information terminal device 4, it is preferable to prioritize the transmission of integrated image data from the space where the dynamic object exists (the space around the dynamic object). For example, by shortening the transmission period (increasing the transmission frequency) of integrated image data from the space where the dynamic object exists (the space around the dynamic object), and lengthening the transmission period (decreasing the transmission frequency) of integrated image data from other spaces where the dynamic object does not exist, the total amount of integrated image data transmitted can be reduced, and the information terminal device 4 can display the integrated image data sequentially without stress.

[0071] In this embodiment, the viewpoint switching of the integrated image is performed automatically, but it may also be performed manually by a monitor viewing the information terminal device 4.

[0072] <Third Embodiment> Next, a third embodiment of this system will be described with reference to Figures 15 and 16. In the following, only configurations different from the above embodiments will be described, and identical configurations will be omitted from the description and given the same reference numerals.

[0073] In this embodiment, when the integration unit 35 generates integrated image data in real space, it extracts point cloud data limited to a predetermined viewpoint and integrates it with the 3D image data corresponding to the point cloud data.

[0074] Specifically, as shown in Figure 15, in three-dimensional real space, the viewpoint is uniquely determined by the position and direction from which it is looking. By automatically or manually defining the viewpoint (position and direction) to be monitored on the screen of the information terminal device 4, the viewpoint is uniquely defined in three-dimensional real space as shown by the arrow in Figure 15, and the integrated image data with the defined viewpoint will look like Figure 16. In this embodiment, the integrated image data is displayed side by side, as shown in Figure 16, with static three-dimensional image data in real space (left figure) and real-time point cloud data (right figure).

[0075] For example, let the viewpoint coordinates be the origin (0,0,0) and the direction of the viewpoint be the vector (X,Y,Z). The direction in which a point (x,y,z) in the point cloud data is visible from the viewpoint is the vector (x,y,z) from the origin. The difference in angle between vector (X,Y,Z) and vector (x,y,z) is generally calculated using cosine similarity. The smaller this cosine similarity, the closer the point is to the viewpoint. Therefore, by extracting points from the point cloud within a predetermined range of cosine similarity, the range of the viewpoint's direction can be defined. Also, the closer a point is to the origin, the closer it is to the viewpoint, and the magnitude of the vector (X,Y,Z), i.e., the Euclidean distance, is X. 2 +Y 2 + Z 2Since a smaller square root value indicates a point being closer to the viewpoint, the range of distances from the viewpoint can be defined by extracting a point cloud within a predetermined range from a given coordinate position of the viewpoint using Euclidean distance. Note that points that are far from the viewpoint direction (vector), such as those outside the screen, are far from the viewpoint, so cosine similarity takes precedence over Euclidean distance.

[0076] Because the viewpoint's position (coordinate position) and direction (vector) are fixed in this way, points that are in the direction (vector) of the viewpoint and are close to the viewpoint's position (coordinate position) are more likely to be important points for the observer, while other points (for example, points on the opposite side of the vector from the viewpoint's coordinate position that are invisible to the observer) are less important. Therefore, it is possible to monitor real space while focusing on important points.

[0077] Although embodiments of the present invention have been described above with reference to the drawings, the present invention is not limited to the illustrated embodiments. Various modifications and variations can be made to the illustrated embodiments within the same scope as the present invention, or within the equivalent scope. [Explanation of Symbols]

[0078] 1…Image data acquisition device 11…Camera Club 12…Scanner section 13…Image generation unit 14…Transmitter 2…Point cloud data acquisition device 21...Sensor section 22...Data Processing Unit 23...Transmitter 3…Server device 31…Image data storage unit 32... Receiver 33…Selection department 34...Calibreter section 35…Integration Department 36...Transmitter 4… Information terminal device 41... Receiver 42...Display section

Claims

1. A real-space monitoring system used to monitor the situation in real space, The system comprises an image data acquisition device that acquires static image data in real space, a point cloud data acquisition device that sequentially acquires real-time point cloud data in real space, and a server device connected to the image data acquisition device and the point cloud data acquisition device in a communication-enabled state. The server device is An image data storage unit that stores static image data in real space acquired at a predetermined timing by the image data acquisition device, A selection unit that selects point cloud data for a predetermined frame from each frame of real-time point cloud data in real space acquired sequentially by the point cloud data acquisition device, A calibrator unit that performs calibration between static image data of a predetermined timing in real space stored in the image data storage unit and point cloud data of a predetermined frame in real space selected by the selection unit, After the calibrator unit performs calibration between a static image in real space at a predetermined timing and point cloud data for a predetermined frame, the integration unit generates integrated image data by sequentially integrating real-time point cloud data acquired sequentially by the point cloud data acquisition device with the static image data in real space at a predetermined timing stored in the image data storage unit. A real-space monitoring system characterized by comprising a transmission unit that transmits integrated image data generated by the aforementioned integration unit to another device.

2. The real-space monitoring system according to claim 1, wherein the calibrator unit uses static image data acquired at a time when no moving objects exist in real space for calibration.

3. The real-space monitoring system according to claim 1, wherein the calibrator unit uses static image data acquired at a bright time when the real space is visible for calibration.

4. The real-space monitoring system according to claim 1, wherein the selection unit selects point cloud data of a predetermined frame in which no dynamic objects exist in real space from each frame of real-time point cloud data acquired sequentially in real space by the point cloud data acquisition device.

5. The real-space monitoring system according to claim 1, wherein the calibrator unit periodically performs calibration between static image data of a predetermined timing in real space stored in the image data storage unit and point cloud data of a predetermined frame in real space selected by the point cloud data selection unit.

6. The real-space monitoring system according to claim 5, wherein the selection unit periodically performs calibration between static image data at a predetermined timing in real space and point cloud data of a predetermined frame by the calibrator unit, and compares the first frame of the point cloud data selected when the calibration between static image data in real space and real-time point cloud data was first performed by the calibrator unit with each frame of the real-time point cloud data, and selects the point cloud data of a predetermined frame in which the position of each point cloud is closest to the point cloud data of the first frame in real space.

7. The real-space monitoring system according to claim 6, wherein the selection unit periodically performs calibration between static image data at a predetermined timing in real space and point cloud data of a predetermined frame using the calibrator unit, compares the first frame of the point cloud data P selected when the calibration between static image data in real space and real-time point cloud data was first performed with each frame of the real-time point cloud data Q, and selects the point cloud data Q of a predetermined frame in which the position of each point cloud is closest to the point cloud data P of the first frame in real space, and selects the point cloud data Q of a predetermined frame in which the DM value is smallest in the lower [1]. [Formula 1] P={p1, p2, p3,..., pn} Q={q1, q2, q3, ..., qk} C(q, P): The point in the point cloud P that is closest to q. n: Number of points in the point cloud data P k: Number of points in the point cloud data Q

8. The real-space monitoring system according to claim 6, wherein the selection unit periodically performs calibration between static image data at a predetermined timing in real space and point cloud data of a predetermined frame using the calibrator unit, and compares the first frame of the point cloud data P selected when the calibration between static image data in real space and real-time point cloud data was first performed with each frame of the real-time point cloud data Q, and selects the point cloud data Q of a predetermined frame in which the position of each point cloud is closest to the point cloud data P of the first frame in real space, and selects the point cloud data Q of a predetermined frame in which the DM value is smallest in [Equation 2] [Equation 3]. [Formula 2] [Formula 3] P={p1, p2, p3,..., pn} Q={q1, q2, q3, ..., qk} C(q, P): The point in the point cloud P that is closest to q. n: Number of points in the point cloud data P k: Number of points in the point cloud data Q

9. The real-space monitoring system according to claim 1, wherein the image data acquisition unit comprises a camera unit that captures a two-dimensional image of real space, a scanner unit that acquires three-dimensional data of real space, and an image generation unit that generates three-dimensional image data of real space based on the two-dimensional image of real space captured by the camera unit and the three-dimensional data of real space acquired by the scanner unit.

10. The integration unit switches the viewpoint in integrated image data in real space, as described in claim 1 of the real-space monitoring system.

11. The real-space monitoring system according to claim 10, wherein the integration unit detects dynamic objects in point cloud data in integrated image data in real space and continuously or sequentially switches to a viewpoint that tracks the dynamic objects.

12. The real-space monitoring system according to claim 11, wherein the transmitting unit preferentially transmits to other devices integrated image data relating to a dynamic object in point cloud data and the space surrounding the dynamic object in the integrated image data in real space.

13. The real-space monitoring system according to claim 1, wherein the integration unit extracts point cloud data limited to a predetermined viewpoint and integrates the point cloud data with corresponding three-dimensional image data.

14. The real-space monitoring system according to claim 13, wherein the integration unit extracts point cloud data in which the cosine similarity of the Euclidean distance from the position coordinates of a predetermined viewpoint and / or the direction vector of the predetermined viewpoint is within a predetermined range.

15. A computer program used in a real-space monitoring system comprising: an image data acquisition device for acquiring static image data in real space; a point cloud data acquisition device for sequentially acquiring real-time point cloud data in real space; and a server device connected to the image data acquisition device and the point cloud data acquisition device in a communication-enabled manner. The server device An image data storage unit that stores static image data in real space acquired at a predetermined timing by the image data acquisition device, A selection unit that selects point cloud data for a predetermined frame from each frame of real-time point cloud data in real space acquired sequentially by the point cloud data acquisition device, A calibrator unit that performs calibration between static image data of a predetermined timing in real space stored in the image data storage unit and point cloud data of a predetermined frame in real space selected by the selection unit, After the calibrator unit performs calibration between a static image in real space at a predetermined timing and point cloud data for a predetermined frame, the integration unit generates integrated image data by sequentially integrating real-time point cloud data acquired sequentially by the point cloud data acquisition device with the static image data in real space at a predetermined timing stored in the image data storage unit. A computer program characterized by functioning as a transmission unit that transmits integrated image data generated by the integration unit to another device.

Images