Image Sensor Data Control System

Abstract

In order to provide an image sensor data control system capable of controlling priority levels of image sensor data of each spatial region pertaining to static objects and dynamic objects in real space, in this image sensor data control system, a server device comprises: a learning unit for learning, on the basis of the image sensor data comprising a point cloud in a consolidated real space, a movement feature index of the image sensor data for each spatial region of the real space, as an indicator of a feature relating to movement of static objects or dynamic objects in each spatial region of the real space; a movement feature index information storage unit for storing information relating to the movement feature index of the image sensor data of each spatial region of the real space learned by the learning unit; and a control unit for setting a priority level of the image sensor data of each spatial region of the real space on the basis of the information relating to the movement feature index of the image sensor data of each spatial region of the real space stored in the movement feature index information storage unit.

Description

Technical Field

[0001] The present invention relates to an image sensor data control system for controlling image sensor data acquired in real space.

Background Art

[0002] In recent years, smart cities have attracted attention as urban development aimed at promoting economic circulation through improving the quality of life and creating new value by utilizing ICT (Information and Communication Technology), and solving social issues. This smart city is not just digitalization, but can also be said to be a transformation similar to the innovation from mobile phones to smartphones. And as a cyber-physical system that supports smart cities, real machines such as robots, drones, and autonomous vehicles perform tasks by utilizing virtual information such as 3D images.

[0003] For example, the applicant's Tokyo Institute of Technology is also promoting the concept of "Next Use-Type Future City" as part of smart cities. In this concept, there are various stakeholders such as residents, workers, and visitors. In the Toyosu area of Tokyo, which is in the process of growth, service solutions are provided in various fields by utilizing advanced technologies and urban OS, aiming to improve individual satisfaction and solve street problems, as well as to achieve coexistence and co-prosperity of various facilities and individuals.

[0004] In realizing such a smart city, cameras and millimeter-wave radars are already installed in autonomous vehicles to obtain virtual information such as the above-mentioned 3D images. For example, a camera can detect most of the information necessary for driving, such as white lines, signs, other vehicles, and the surrounding environment, but it is difficult to detect the distance to an object. Also, a millimeter-wave radar is a device that emits radio waves with a frequency of 30 GHz to 300 GHz, called millimeter waves, and measures their reflection. It can easily detect the relative speed with other vehicles running around, but its azimuth resolution is not very high, and it is difficult to detect objects with a low radio wave reflectivity, such as trees.

[0005] Therefore, recently, in addition to cameras and millimeter-wave radars, a sensor called LiDAR (light detection and ranging) has attracted attention. This LiDAR is a type of sensor that uses laser light and has a higher radiation beam density compared to radio waves. By irradiating a target object while scanning it with a laser beam of short wavelength, it can accurately detect not only the distance to the object but also the position and shape of the object. Therefore, when LiDAR is used in the real space where there are static objects such as walls and dynamic objects such as automobiles, three-dimensional information of the real space can be obtained as image sensor data consisting of a point cloud of laser light. Moreover, in addition to the LiDAR installed in an autonomous vehicle, by using LiDARs arranged in the surrounding environment such as roadside traffic signals and street corners, these can transmit image sensor data to the autonomous vehicle to inform about the situation and danger of blind spots on the road (see, for example, Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0006]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] However, when acquiring three-dimensional information of the real space in real time by LiDAR, the data volume becomes huge. Therefore, it is effective in data control to reduce the image sensor data of static objects without temporal changes such as walls, floors, and ceilings and prioritize the image sensor data of dynamic objects with temporal changes such as automobiles and pedestrians. In particular, by using a plurality of LiDARs, it is possible to improve the quality of three-dimensional information in the real space by eliminating dead angles and increasing the point density of the data. However, since the data volume further increases, it becomes more effective to reduce the information of static objects. Such a problem is not limited to LiDAR, but also occurs in the same way in other sensors that acquire image sensor data composed of point clouds.

[0008] The present applicant has made this invention in view of the above technical background, and an object thereof is to provide an image sensor data control system capable of controlling the priority of image sensor data of each spatial region related to static objects and dynamic objects in the real space.

Means for Solving the Problems

[0009] In order to achieve the above object, the present invention provides an image sensor data control system in which one or more terminal devices and a server device are connected in a communicable state, and the server device controls image sensor data acquired by each terminal device. The terminal device includes a sensor unit that acquires image sensor data consisting of a point cloud in the real space, and a terminal-side transmission unit that transmits the image sensor data consisting of the point cloud in the real space acquired by the sensor unit to the server device. The server device includes a reception unit that receives the image sensor data consisting of the point cloud in the real space transmitted from each terminal device, an aggregation unit that aggregates the image sensor data consisting of the point cloud in the real space received by the reception unit, and based on the image sensor data consisting of the point cloud in the real space aggregated by the aggregation unit, a learning unit that learns a movement feature index of the image sensor data of each space region in the real space as an indication of the characteristics related to the movement of static or dynamic objects in each space region in the real space, a movement feature index information storage unit that stores information related to the movement feature index of the image sensor data of each space region in the real space learned by the learning unit, and a control unit that sets the priority of the image sensor data of each space region in the real space based on the information related to the movement feature index of the image sensor data of each space region in the real space stored in the movement feature index information storage unit.

[0010] Further, a plurality of the terminal devices may be provided to acquire image sensor data consisting of a point cloud from different directions with respect to the same real space, and the aggregation unit may aggregate by synthesizing the image sensor data consisting of the point cloud acquired by each terminal device in time series.

[0011] Further, the learning unit may use the number of point clouds of the image sensor data of each space region in the real space as a movement feature index.

[0012] Further, the learning unit may determine a space region related to a static object or a dynamic object by learning the temporal change of the movement feature index of the image sensor data of each space region in the real space.

[0013] Further, when the deviation of the movement feature index of the image sensor data in a predetermined spatial region in the real space is within a predetermined range, the learning unit determines that it is a spatial region related to a static object, while when the deviation of the movement feature index of the image sensor data in the predetermined spatial region is outside the predetermined range, it may be determined that it is a spatial region related to a dynamic object.

[0014] Further, when the average of the movement feature indices of the image sensor data in a predetermined spatial region in the real space exceeds a predetermined threshold, the learning unit determines that there are many dynamic objects moving in the spatial region, while when the average of the movement feature indices of the image sensor data is equal to or less than the predetermined threshold, it may be determined that there are few dynamic objects moving in the spatial region.

[0015] Further, the control unit may set a high priority for the image sensor data in the spatial region determined to be a dynamic object by the learning unit, while setting a low priority for the image sensor data in the spatial region determined to be a static object by the learning unit.

[0016] Further, the control unit may set a high priority for the image sensor data in the spatial region determined to have many dynamic objects moving by the learning unit, while setting a low priority for the image sensor data in the spatial region determined to have few dynamic objects moving by the learning unit.

[0017] Further, for the image sensor data acquired in real time by the terminal device, when the change rate of the movement feature index of the image sensor data in a predetermined spatial region in the real space exceeds a predetermined threshold within a predetermined time, it is determined that the movement speed of the dynamic object in the spatial region is fast, and the priority of the image sensor data in the spatial region is set high, while when the change rate of the movement feature index of the image sensor data in the predetermined spatial region in the real space is equal to or less than the predetermined threshold within a predetermined time, it is determined that the movement speed of the dynamic object in the spatial region is slow, and the priority of the image sensor data in the spatial region may be set low.

[0018] Also, a server-side transmission unit may be provided on the output side of the aggregation unit, and the control unit may control the server-side transmission unit to preferentially transmit the image sensor data of a spatial region with a high priority to a predetermined moving body for the image sensor data acquired in real time in the terminal device.

[0019] Also, the control unit may control the terminal-side transmission unit to preferentially transmit the image sensor data of a spatial region with a high priority to a predetermined server device for the image sensor data acquired in real time within a predetermined time in the terminal device.

[0020] Also, the terminal-side transmission unit may receive in advance the priority of the image sensor data of each spatial region from the server device, and preferentially transmit the image sensor data of a spatial region with a high priority to the server device.

[0021] Also, the real space may be composed of a plurality of cells arranged in a grid as spatial regions, and the control unit may set the priority of the image sensor data for each cell.

Effect of the Invention

[0022] According to the present invention, by learning the movement characteristic index of image sensor data composed of a point cloud acquired in the real space, the priority of the image sensor data of each space region related to a static object or a dynamic object in the real space can be set. Therefore, for a space region where the priority of the image sensor data is set high (mainly a space region related to a dynamic object), real-time image sensor data is preferentially transmitted to a moving body or the like, while for a space region where the priority of the image sensor data is set low (mainly a space region related to a static object), real-time image sensor data may not be transmitted to the moving body or may be transmitted with a delay. Thus, according to the priority of the image sensor data of each space region in the real space, the image sensor data acquired in real time at the terminal device can be transmitted to a moving body or the like for each space region, and it becomes possible to quickly notify the moving body or the like of the situation and danger of the blind spot of the road.

Brief Description of the Drawings

[0023]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

[0024] Next, an image sensor data control system according to an embodiment of the present invention (hereinafter referred to as this system) will be described with reference to FIGS. 1 to 10.

[0025] [Overall Configuration] As shown in FIG. 1, this system includes a plurality of terminal devices 1 and a server device 2 connected to each terminal device 1 via a predetermined communication path. The server device 2 controls the image sensor data in the real space acquired by each terminal device 1, and transmits the real-time image sensor data in each space region of the real space to a moving body 3 (for example, an autonomous vehicle, etc.) via a network or the like.

[0026] Note that the real space refers to a mainly three-dimensional space that actually exists related to social life or the environment, such as roads, streets, buildings, indoors, rivers, mountains, etc. Also, the space region in the real space refers to a region obtained by dividing the real space into a plurality of regions with an arbitrary shape such as a cube.

[0027] [Configuration of Terminal Device 1] The terminal device 1 is installed in traffic signal devices on roads, in the streets, in security camera devices, traffic camera devices, etc. As shown in FIG. 1, it includes a sensor unit 11 that acquires image sensor data composed of a point cloud in real space, and a terminal-side transmission unit 12 that transmits the image sensor data composed of the point cloud in real space acquired by the sensor unit 11 to the server device 2.

[0028] The sensor unit 11 is a sensor called so-called LiDAR (light detection and ranging). This LiDAR is a type of sensor that uses laser light. Compared with radio waves, it has a higher radiation beam density. By irradiating the target object while scanning the laser light with a short wavelength, it acquires image sensor data composed of a point cloud in real space, and accurately detects not only the distance to the target object but also the position and shape of the target object.

[0029] As this LiDAR, there are methods such as a method of acquiring 360-degree omnidirectional image sensor data by rotating a light emitter that emits a plurality of laser lights, and a method of directly irradiating the laser light within a range of a predetermined light irradiation angle to acquire image sensor data. Also, the more the number of light emitters that emit laser light, the higher the accuracy of the image sensor data, but it also becomes more expensive. Therefore, an inexpensive LiDAR with a small number of light emitters may be used.

[0030] Here, an example of acquiring image sensor data consisting of a point cloud in real space by LiDAR will be described. As shown in Fig. 3, taking the interior of a certain room surrounded by walls on all four sides, with a predetermined desk, chair, and bookshelf arranged at various locations and a person waving their hand in the center as a three-dimensional real space, LiDAR (a) is installed on the left-center wall of Fig. 3, and LiDAR (b) is installed on the lower-right wall of Fig. 3. Then, when irradiating the interior of the room with laser light while scanning at a predetermined irradiation angle from LiDAR (a), LiDAR (a) acquires image sensor data consisting of a point cloud as shown in Fig. 4, recognizes the person in the center, and also recognizes the surrounding walls, desk, chair, bookshelf, and can even recognize the shadow of a person on the right side. On the other hand, when irradiating the interior of the room with laser light while scanning at a predetermined irradiation angle from LiDAR (b), LiDAR (b) acquires image sensor data consisting of a point cloud as shown in Fig. 5, recognizes the person in the center on the right side, and also recognizes the surrounding walls, desk, chair, bookshelf, and can even recognize the shadow of a person in the center. The reason why such a person's shadow is formed is that due to the high directivity of the irradiated laser light, the laser light irradiated on the person does not reach the wall behind the person.

[0031] In such a three-dimensional real space, the space region where there are things that do not move or hardly move, such as walls, desks, chairs, and bookshelves, becomes a space region related to static objects, while the space region where there are things that move, such as a person waving their hand, becomes a space region related to dynamic objects. And in the image sensor data in real space, as will be described later, the image sensor data of the space region related to static objects has a lower priority, while the image sensor data of the space region related to dynamic objects has a higher priority. Therefore, it is important to distinguish and judge the space region related to static objects and the space region related to dynamic objects in real space.

[0032] When image sensor data is acquired by a single LiDAR, as described above, shadows of moving objects are generated, and the spatial region where such shadows are generated becomes blank, making it difficult to distinguish and determine the spatial region related to static objects and the spatial region related to moving objects in some cases. Therefore, for example, by synthesizing the image sensor data composed of the point cloud shown in FIG. 4 obtained by LiDAR (a) and the image sensor data composed of the point cloud shown in FIG. 5 obtained by LiDAR (b), image sensor data composed of the point cloud as shown in FIG. 6 can be obtained, and the influence of the shadows of moving objects can be reduced. For this reason, it is preferable that a plurality of terminal devices 1 are provided so as to acquire image sensor data composed of point clouds from different directions with respect to the same real space.

[0033] In the present embodiment, LiDAR is used as the sensor unit 11. However, any other sensor may be used as long as it can acquire image sensor data composed of point clouds.

[0034] [Configuration of Server Device 2] The server device 2 is arranged near the terminal device 1 side. As shown in FIG. 2, it includes a receiving unit 21 that receives image sensor data composed of point clouds in the real space transmitted from each terminal device 1, an aggregating unit 22 that aggregates the image sensor data composed of point clouds in the real space, a learning unit 23 that learns the movement feature index of the image sensor data in each spatial region in the real space, a movement feature index information storage unit 24 that stores information regarding the movement feature index of the image sensor data in each spatial region in the real space, a control unit 25 that controls the priority of the image sensor data in each spatial region in the real space, and a server-side transmission unit 26 that transmits the image sensor data of the spatial region with a high priority in the real space to the moving body 3. In the present embodiment, the number of point clouds of the image sensor data in each spatial region in the real space is described as the movement feature index.

[0035] The aggregating unit 22 aggregates image sensor data composed of a point cloud in the real space received by the receiving unit 21. In the present embodiment, the aggregating unit 22 aggregates the image sensor data composed of the point cloud in the real space transmitted from each terminal device 1 by synthesizing it in time series.

[0036] Specifically, FIG. 7 is a graph showing the temporal change in the number of point clouds (movement feature index) of image sensor data with the number of point clouds (movement feature index) of image sensor data obtained by LiDAR (a) on the vertical axis and time (number of shooting frames) on the horizontal axis for two spatial regions (wall / person) in the three-dimensional real space shown in FIG. 3. Among them, in the upper graph, since the number of image sensor data hardly changes over time, it can be determined that it is a spatial region related to a static object. On the other hand, in the lower graph, since the number of image sensor data changes somewhat over time, it can be determined that it is a spatial region related to a dynamic object.

[0037] When comparing the temporal changes in the number of point clouds of the image sensor data for the spatial region related to the static object and the number of point clouds of the image sensor data for the spatial region related to the dynamic object, there is a difference in the number of point clouds in the range where the frame number is around 500 to 1800. However, since the difference in the number of point clouds is relatively small, it may be difficult to distinguish between the spatial region related to the static object and the spatial region related to the dynamic object.

[0038] On the other hand, as shown in FIG. 8, when the number of point clouds of image sensor data acquired by two LiDARs (a) and (b) is synthesized in time series, the number of point clouds of image sensor data in the spatial region related to static objects (walls (×2)) does not change much over time, while the number of point clouds of image sensor data in the spatial region related to dynamic objects (persons (×2)) changes significantly over time. Therefore, a large temporal difference occurs in the number of point clouds between the spatial region related to static objects and the spatial region related to dynamic objects, making it easier to clearly distinguish the spatial region related to static objects from the spatial region related to dynamic objects. For this reason, it is preferable that the aggregation unit 22 aggregates by synthesizing in time series the image sensor data composed of point clouds in the real space acquired by each terminal device 1.

[0039] Note that when aggregating the image sensor data composed of point clouds in the real space received by the receiving unit 21, the aggregation unit 22 may perform smoothing processing on the point clouds of the image sensor data.

[0040] Based on the image sensor data of the point clouds in the real space aggregated by the aggregation unit 22, the learning unit 23 learns the temporal change of the number of point clouds of the image sensor data (movement feature index) in each spatial region in the real space as an indication of the characteristics related to the movement of static or dynamic objects in each spatial region in the real space.

[0041] Specifically, as described above, while the number of point clouds of image sensor data in the spatial region related to dynamic objects changes significantly over time, the number of point clouds of image sensor data in the spatial region related to static objects does not change much over time. Therefore, the learning unit 23 learns a large amount while using machine learning (for example, Random Forest, XGBoost) the temporal change of the number of point clouds of image sensor data for each spatial region in the real space to determine whether each spatial region is a spatial region related to static objects or a spatial region related to dynamic objects.

[0042] At this time, when the deviation of the number of point clouds of the image sensor data in a predetermined spatial region in the real space (for example, the difference between the maximum value P1 and the minimum value P2 of the image sensor data in the spatial region of the static object (wall) in FIG. 8) is within a predetermined range, it is determined that it is a spatial region related to a static object. On the other hand, when the deviation of the movement feature index of the image sensor data in the predetermined spatial region (for example, the difference between the maximum value P3 and the minimum value P4 of the image sensor data in the spatial region of the dynamic object (person) in FIG. 8) is outside the predetermined range, it may be determined that it is a spatial region related to a dynamic object.

[0043] Further, the learning unit 23 calculates the temporal average of the number of point clouds of the image sensor data in a predetermined spatial region in the real space. When the temporal average of the number of point clouds of the image sensor data exceeds a predetermined threshold, it is determined that there are many dynamic objects moving in the spatial region. On the other hand, when the temporal average of the number of point clouds of the image sensor data is equal to or less than the predetermined threshold, it may be determined that there are few dynamic objects moving in the spatial region.

[0044] Further, the learning unit 23 may determine the spatial region related to the static object or the dynamic object by combining the deviation and the average of the movement feature index of the image sensor data in each spatial region in the real space.

[0045] The movement feature index information storage unit 24 stores information regarding the number of point clouds (movement feature index) of the image sensor data in each spatial region in the real space learned by the learning unit 23. At this time, as the information regarding the number of point clouds (movement feature index) of the image sensor data in each spatial region, in addition to the number of point clouds of the image sensor data itself, information regarding the determination of the spatial region related to the dynamic object or the static object as described above, information regarding the number of dynamic objects moving in the spatial region, and the like are also included.

[0046] The control unit 25 sets the priority of the image sensor data for each spatial region in the real space based on the information regarding the number of point clouds of the image sensor data for each spatial region in the real space stored in the movement feature index information storage unit 24 (movement feature index).

[0047] Specifically described, the control unit 25 sets the priority of the image sensor data for the spatial region determined to be a spatial region related to a static object due to the small temporal change in the number of point clouds of the image sensor data by the learning unit 23 to be low, while setting the priority of the image sensor data for the spatial region determined to be a spatial region related to a dynamic object due to the large temporal change in the number of point clouds of the image sensor data by the learning unit 23 to be high.

[0048] Further, the control unit 25 sets the priority of the image sensor data for the spatial region learned to have more moving dynamic objects among the spatial regions learned to be related to dynamic objects by the learning unit 23 to be even higher, while the priority of the image sensor data for the spatial region learned to have fewer moving dynamic objects may be set slightly lower.

[0049] Also, for the image sensor data acquired in real time within a predetermined time in the terminal device 1, when the change speed of the number of point clouds of the image sensor data for a predetermined spatial region in the real space (especially a spatial region related to a dynamic object) within the predetermined time exceeds a predetermined threshold, it is determined that the moving speed of the dynamic object in the spatial region is fast, and the priority of the image sensor data for the spatial region is set high, while when the change speed of the number of point clouds of the image sensor data within the predetermined time is equal to or less than the predetermined threshold, it is determined that the moving speed of the dynamic object in the spatial region is slow, and the priority of the image sensor data for the spatial region may be set low.

[0050] When the aggregating unit 22 aggregates image sensor data within a predetermined time in real time, the control unit 25 controls the server-side transmission unit 26 to preferentially transmit real-time image sensor data to the moving body 3 for a spatial region (a spatial region mainly related to a moving object) in which the priority of the image sensor data is set high. On the other hand, for a spatial region (a spatial region mainly related to a static object) in which the priority of the image sensor data is set low, the server-side transmission unit 26 is controlled not to transmit real-time image sensor data to the moving body 3 or to transmit it with a delay. By setting the priority of the image sensor data for each spatial region in the real space in this way, it is possible to preferentially transmit real-time image sensor data for a spatial region (especially a spatial region related to a moving object) in which the priority is set high to the moving body 3.

[0051] [Flow during learning of image sensor data by this system] First, the flow when learning about the movement characteristic index of the image sensor data for each spatial region in the real space by this system will be described with reference to FIG. 9. In the following description, "step" will be abbreviated as "S".

[0052] First, in each terminal device 1, the sensor unit 11 irradiates the real space while scanning a laser beam, thereby acquiring image sensor data composed of a point cloud in the real space (S1).

[0053] Then, the terminal-side transmission unit 12 transmits the image sensor data composed of the point cloud in the real space acquired by the sensor unit 11 to the server device 2 (S2).

[0054] Next, in the server device 2, the receiving unit 21 receives the image sensor data composed of the point cloud in the real space transmitted from each terminal device 1 (S3).

[0055] Then, the aggregation unit 22 aggregates the image sensor data composed of a point cloud in the real space received by the reception unit 21 (S4). In the present embodiment, the image sensor data composed of a point cloud in the real space transmitted from each terminal device 1 is aggregated by being synthesized in time series.

[0056] Then, the learning unit 23 learns the temporal change in the number of point clouds (movement feature index) of the image sensor data of a plurality of spatial regions related to static or dynamic objects in the real space based on the image sensor data of the point cloud in the real space aggregated by the aggregation unit 22 (S5).

[0057] Then, the movement feature index storage unit 24 stores information regarding the number of point clouds (movement feature index) of the image sensor data of each spatial region in the real space learned by the learning unit 23 (S6).

[0058] Thus, in this system, by repeating the processes of S1 to S6 above, since a large amount of temporal changes in the number of point clouds (movement feature index) of the image sensor data of each spatial region in the real space are learned, information regarding the number of point clouds (movement feature index) of the image sensor data of each spatial region in the real space can be accumulated in the movement feature index information storage unit 24.

[0059] [Flow at the time of transmitting image sensor data by this system] Next, the flow when transmitting the image sensor data of each spatial region in the real space from this system to a predetermined moving object will be described with reference to FIG. 10.

[0060] First, in each terminal device 1, the sensor unit 11 irradiates while scanning a laser beam onto the real space, thereby acquiring image sensor data composed of a real-time point cloud in the real space (S11).

[0061] Then, the terminal-side transmission unit 12 transmits image sensor data composed of real-time point clouds in the real space acquired by the sensor unit 11 to the server device 2 (S12).

[0062] Next, in the server device 2, the reception unit 21 receives image sensor data composed of real-time point clouds in the real space transmitted from each terminal device 1 (S13).

[0063] Then, the aggregation unit 22 aggregates the image sensor data composed of real-time point clouds in the real space received by the reception unit 21 (S14).

[0064] Then, based on the information regarding the number of point clouds of the image sensor data (movement feature index) in each space region in the real space stored in the movement feature index information storage unit 24, the control unit 25 sets the priority of the image sensor data in each space region in the real space (S15). Note that the control unit may set in advance the priority of the image sensor data in each space region in the real space during the above-described learning or the like.

[0065] Then, for the space region where the priority of the image sensor data is set high (the space region mainly related to moving objects), the control unit 25 controls the server-side transmission unit 26 to preferentially transmit the real-time image sensor data to the moving body 3, while for the space region where the priority of the image sensor data is set low (the space region mainly related to static objects), the control unit 25 controls the server-side transmission unit 26 not to transmit the real-time image sensor data to the moving body 3 or to transmit it with a delay (S16).

[0066] Thus, according to the priority of the image sensor data in each space region in the real space, the image sensor data acquired in real time in the terminal device 1 can be transmitted to the moving body 3 or the like for each space region, and it becomes possible to quickly notify the moving body 3 or the like of the situation and danger of the blind spots on the road.

[0067] In addition, in the present embodiment, the number of point clouds is used as a movement feature index of image sensor data in the spatial region in the real space. However, other indexes related to the point cloud, such as the density and distribution of the point cloud, may also be used.

[0068] Further, the control unit 25 controls the server-side transmission unit 26 of the server device 2 so as to preferentially transmit real-time image sensor data to the moving body 3 for the spatial region (mainly the spatial region related to the dynamic object) in which the priority of the image sensor data is set high. On the other hand, for the spatial region (mainly the spatial region related to the static object) in which the priority of the image sensor data is set low, the control unit 25 controls the server-side transmission unit 26 of the server device 2 so as not to transmit the real-time image sensor data to the moving body 3 or to transmit it with a delay. However, the present invention is not limited to this. For example, the control unit 25 controls the terminal-side transmission unit 12 of the terminal device 1 so as to transmit real-time image sensor data to the server device 2 for the spatial region (mainly the spatial region related to the dynamic object) in which the priority of the image sensor data is set high. On the other hand, for the spatial region (mainly the spatial region related to the static object) in which the priority of the image sensor data is set low, the control unit 25 controls the terminal-side transmission unit 12 of the terminal device 1 so as not to transmit the real-time image sensor data to the server device 2 or to transmit it with a delay.

[0069] Further, the terminal-side transmission unit 12 receives in advance the priority of the image sensor data of each spatial region from the server device 2. For the spatial region (mainly the spatial region related to the dynamic object) in which the priority of the image sensor data is set high, the terminal-side transmission unit 12 transmits the image sensor data to the server device 2. On the other hand, for the spatial region (mainly the spatial region related to the static object) in which the priority of the image sensor data is set low, the terminal-side transmission unit 12 may not transmit the image sensor data to the server device 2 or may transmit it with a delay.

[0070] Further, the terminal device 1 integrally configures the sensor 11 and the terminal-side transmission unit 12, but they may be configured separately.

Example

[0071] Next, Examples 1 to 3 of the present invention will be described with reference to FIGS. 11 to 13.

[0072] In any of Examples 1 to 3, a plurality of terminal devices 1 are installed at predetermined positions in a virtual real space, and image sensor data composed of a point group in the same virtual real space is acquired from a plurality of directions.

[0073] Further, after aggregating the image sensor data composed of the point group in the entire virtual real space transmitted from each terminal device 1, the server device 2 learns the temporal change of the number of point groups (movement feature index) in each space region (a plurality of cells arranged in a two-dimensional lattice in this embodiment) and stores it in a database, and based on the information regarding the number of point groups (movement feature index) of the image sensor data in each space region, sets the priority of the image sensor data in each space region in the real space. Note that although a two-dimensional cell is described in this embodiment, a three-dimensional cell or the like may be used.

[0074] <Example 1> In Example 1, the case of avoiding a head-on collision accident of the moving body 3 (automated vehicle) by the present system will be described.

[0075] As shown in FIG. 11, in a virtual real space composed of 4 cells in the vertical direction and 5 cells in the horizontal direction, two cells surrounded by a thick frame on the left are the space region A related to the moving object that the moving body 3 travels toward the right, and two cells surrounded by a thick frame on the right are the space region B related to the moving object that the moving body 3 travels toward the left. Each of the four colored cells in the lower left and lower right is the space region C related to a static object such as a wall. Assume a case where real-time image sensor data in the virtual real space is transmitted to the moving body 3 traveling between the space regions C. Note that the average and change speed of the number of point groups (movement feature index) of the image sensor data in each of the space regions A, B, and C are as shown in each table of FIG. 11.

[0076] In this virtual physical space, as shown in FIG. 11, since the deviation "10" of the number of point clouds (moving feature index) of the image sensor data in the space region C is small, the server device 2 determines that the space region C is a space region related to a static object. Further, since the deviations "100" and "500" of the number of point clouds (moving feature index) of the image sensor data in the space regions A and B are high, the server device 2 determines that the space regions A and B are space regions related to dynamic objects. Therefore, the server device 2 sets a high priority for the image sensor data of the space regions A and B related to dynamic objects, while setting a low priority for the image sensor data of the space region C related to static objects. Thus, it is possible to preferentially transmit the real-time image sensor data of the space regions A and B with a high priority to the moving body 3.

[0077] Also, when comparing the temporal average of the number of point clouds (moving feature index) of the image sensor data for the space region A and the space region B, which are determined to be space regions related to dynamic objects, since the temporal average of the number of point clouds of the image sensor data in the space region A is larger than that in the space region B (A: 500 > B: 100), it is determined that the traffic volume of the moving body 3 in the space region A is larger than that in the space region B. Therefore, the server device 2, during normal times, sets a particularly high priority for the image sensor data of the space region A among the space regions A and B related to dynamic objects, and can preferentially transmit the real-time image sensor data of the space region A with a particularly high priority to the moving body 3.

[0078] Also, regarding the spatial regions determined to be the spatial regions A and B related to the moving object, when comparing the change rates of the number of point clouds (moving feature indicators) of the real-time image sensor data, the change rate of the number of point clouds of the image sensor data in the spatial region B is greater than that in the spatial region A (B: 500 > A: 50). Therefore, it can be determined that the moving speed of the moving body 3 currently traveling in the spatial region B is faster than that in the spatial region A. For this reason, in the event of an emergency, the server device 2 can set the priority of the image sensor data in the spatial region B, which is particularly related to the moving object, to be higher among the spatial regions A and B related to the moving object, and can preferentially transmit the real-time image sensor data of the spatial region B with a particularly high priority setting to the moving body 3.

[0079] Thus, according to the priority of the image sensor data of each spatial region in the real space acquired by the terminal device 1, by preferentially transmitting the real-time image sensor data of the spatial regions A and B, which are mainly set with high priorities, to the moving body 3, it becomes possible to avoid the head-on collision between the moving body 3 traveling between the spatial regions C and the moving body 3 traveling in the spatial regions A and B.

[0080] <Example 2> In Example 2, the case of avoiding the contact accident between the moving body 3 (autonomous senior car) and the parked motorcycle by this system will be described.

[0081] As shown in FIG. 12, in a virtual real space composed of 4 cells vertically and 5 cells horizontally, assume a case where the real-time image sensor data in the virtual real space is transmitted to the moving body 3 moving in the virtual real space, and the 4 cells surrounded by the thick frame in the upper right are regarded as the spatial region A related to the moving object where the motorcycle is parked.

[0082] In this embodiment, as shown in FIG. 12, since the temporal average "10" of the number of point clouds (movement feature index) of the image sensor data in the spatial region A is small, the server device 2 determines that the spatial region A is normally a spatial region related to static objects and not a parking lot. However, since the deviation "1000" of the number of point clouds (movement feature index) of the image sensor data in the spatial region A is large, it is determined that the moving body 3 (motorcycle) is parked in the spatial region A with a low frequency. Therefore, the server device 2 can set a high priority for the image sensor data in the spatial region A and preferentially transmit the real-time image sensor data in the spatial region A to the moving body 3, thereby avoiding contact accidents between the moving body 3 and parked motorcycles.

[0083] <Example 3> In Example 3, the case where the moving body 3 (a mobile robot that performs cleaning and patrolling within the facility) moves while avoiding people and obstacles by this system will be described.

[0084] As shown in FIG. 13, in a virtual real space composed of 4 cells vertically and 5 cells horizontally, there are spatial regions (cells) where people and obstacles exist everywhere in each cell. Assume a case where real-time image sensor data in the virtual real space is transmitted to the moving body 3 moving in the virtual real space.

[0085] For example, as shown in FIG. 14, in a spatial region where a pedestrian jumps out, a spatial region where a vehicle jumps out, a spatial region where a parked vehicle suddenly accelerates, a spatial region where a parked motorcycle falls over, a crowded spatial region, and a spatial region where high-speed vehicles frequently pass, the average, deviation, and change speed trend of the number of point clouds (movement feature index) of the image sensor data in each spatial region are displayed. The server device 2 sets the priority of the image sensor data based on the trend of the number of point clouds (movement feature index) of these image sensor data. Therefore, the server device 2 can preferentially transmit the real-time image sensor data in the spatial region with a high priority to the moving body 3 (robot) according to the priority of the image sensor data in each spatial region, enabling the moving body 3 (robot) to perform cleaning and patrolling while avoiding people and obstacles.

[0086] Note that, as shown in FIG. 14, when the server device 2 learns the movement characteristics of the image sensor data in the space region, elements such as the magnitude of the occupancy rate of the space region and the presence or absence of periodicity of the movement characteristic index of the image sensor data in the space region may be considered. Further, when the server device 2 learns the movement characteristics of the image sensor data in the space region, anomaly prediction by machine learning without a teacher may be performed, or anomaly prediction by machine learning with a teacher may be performed.

[0087] As described above, the embodiments of the present invention have been described with reference to the drawings, but the present invention is not limited to the illustrated embodiments. Various modifications and variations can be made to the illustrated embodiments within the same scope or an equivalent scope of the present invention.

Description of Reference Numerals

[0088] 1... Terminal device 11... Sensor unit 12... Terminal-side transmission unit 2... Server device 21... Reception unit 22... Aggregation unit 23... Learning unit 24... Movement characteristic index information storage unit 25... Control unit 26... Server-side transmission unit 3... Moving body

Claims

1. An image sensor data control system in which one or more terminal devices and a server device are connected in a communicable state, and the server device controls image sensor data acquired by each terminal device, wherein the terminal device includes: a sensor unit that acquires image sensor data composed of a point cloud in a real space; and a terminal-side transmission unit that transmits the image sensor data composed of the point cloud in the real space acquired by the sensor unit to the server device, wherein the server device includes: a reception unit that receives the image sensor data composed of the point cloud in the real space transmitted from each terminal device; an aggregation unit that aggregates the image sensor data composed of the point cloud in the real space received by the reception unit; a learning unit that learns a movement feature index of the image sensor data in each space region in the real space as indicating a feature related to the movement of a static object or a dynamic object in each space region in the real space, based on the image sensor data composed of the point cloud in the real space aggregated by the aggregation unit; a movement feature index information storage unit that stores information related to the movement feature index of the image sensor data in each space region in the real space learned by the learning unit; and a control unit that sets the priority of the image sensor data in each space region in the real space based on the information related to the movement feature index of the image sensor data in each space region in the real space stored in the movement feature index information storage unit, wherein the movement feature index is a temporal change in any of the number of point clouds, the density of point clouds, or the distribution of point clouds of the image sensor data. The image sensor data control system is characterized by this.

2. A plurality of the terminal devices are provided to acquire image sensor data composed of point clouds from different directions with respect to the same real space, and the aggregation unit aggregates the image sensor data composed of the point clouds acquired by each terminal device by synthesizing them in time series. The image sensor data control system according to claim 1.

3. The image sensor data control system according to claim 1, wherein the learning unit uses the number of point clouds of the image sensor data in each space region in the real space as the movement feature index.

4. The image sensor data control system according to claim 1, wherein the learning unit learns the movement feature index of the image sensor data in each space region in the real space to determine a space region related to a static object or a dynamic object.

5. When the deviation of the movement characteristic index of the image sensor data in a predetermined spatial region in the real space is within a predetermined range, the learning unit determines that it is a spatial region related to a static object. On the other hand, when the deviation of the movement characteristic index of the image sensor data in the predetermined spatial region is outside the predetermined range, the learning unit determines that it is a spatial region related to a dynamic object. The image sensor data control system according to claim 4.

6. When the average of the movement characteristic indexes of the image sensor data in a predetermined spatial region in the real space exceeds a predetermined threshold, the learning unit determines that there are many dynamic objects moving in the spatial region. On the other hand, when the average of the movement characteristic indexes of the image sensor data is equal to or less than the predetermined threshold, the learning unit determines that there are few dynamic objects moving in the spatial region. The image sensor data control system according to claim 4.

7. The control unit sets a high priority for the image sensor data in the spatial region determined to be a dynamic object by the learning unit, while setting a low priority for the image sensor data in the spatial region determined to be a static object by the learning unit. The image sensor data control system according to claim 4 or claim 5.

8. The control unit sets a high priority for the image sensor data in the spatial region determined to have many dynamic objects moving by the learning unit, while setting a low priority for the image sensor data in the spatial region determined to have few dynamic objects moving by the learning unit. The image sensor data control system according to claim 6.

9. Regarding the image sensor data acquired in real time by the terminal device, when the change speed of the movement characteristic index of the image sensor data in a predetermined spatial region in the real space exceeds a predetermined threshold within a predetermined time, the control unit determines that the movement speed of the dynamic object in the spatial region is fast and sets a high priority for the image sensor data in the spatial region. On the other hand, when the change speed of the movement characteristic index of the image sensor data in the predetermined spatial region in the real space is equal to or less than the predetermined threshold within a predetermined time, the control unit determines that the movement speed of the dynamic object in the spatial region is slow and sets a low priority for the image sensor data in the spatial region. The image sensor data control system according to claim 1.

10. A server-side transmission unit is provided on the output side of the aggregation unit. The control unit controls the server-side transmission unit to preferentially transmit the image sensor data of a spatial region with a high priority setting to a predetermined moving body with respect to the image sensor data acquired in real time in the terminal device. The image sensor data control system according to claim 1.

11. The control unit controls the terminal-side transmission unit to preferentially transmit the image sensor data of a spatial region with a high priority setting to the server device with respect to the image sensor data acquired in real time in the terminal device. The image sensor data control system according to claim 1.

12. The terminal-side transmission unit receives in advance the priority of the image sensor data of each spatial region from the server device, and preferentially transmits the image sensor data of the spatial region with a high priority setting to the server device. The image sensor data control system according to claim 1.

13. The real space is composed of a plurality of cells arranged in a grid as spatial regions, The control unit sets the priority of the image sensor data for each cell. The image sensor data control system according to claim 1.

Images