Three-dimensional sensor controller, system, method, and program

Abstract

To provide a three-dimensional sensor controller, etc. with which, when acquiring point cloud data by mobile imaging and creating synthesized data, it is possible to secure the real-time capability of a three-dimensional sensor and contribute to heightening and equalizing the point cloud density of the synthesized data.SOLUTION: Provided is a three-dimensional sensor controller attached to a camera operator, comprising: a movement measurement unit for measuring the movement state of a camera operator carrying a three-dimensional sensor; a position identification unit for identifying the position of the camera operator; and a three-dimensional sensor control unit for controlling the behavior of the three-dimensional sensor on the basis of the movement state and / or the position of the camera operator. The three-dimensional sensor captures an image of the object to be captured while its behavior is controlled by the three-dimensional sensor control unit and generates point cloud data. The point cloud data is used for generation of synthesized data.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a three-dimensional sensor control device, a system, a method, and a program. [Background technology]

[0002] There is a method in which 3D sensors are attached to moving objects (heavy machinery, vehicles, drones, etc.) or people (e.g. workers) to take pictures while moving, and multiple point cloud data (3D point cloud data) acquired by each 3D sensor are synthesized to create one huge synthesized map (3D map) (see Patent Document 1). This method makes it possible to flexibly respond to areas that change from moment to moment in places where conditions change rapidly. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2023-022867 Summary of the Invention [Problem to be solved by the invention]

[0004] The following analysis is provided by the present inventors.

[0005] However, there are several problems with creating a 3D map based on multiple point cloud data acquired by moving around and capturing images with multiple 3D sensors.

[0006] The first problem is the sparseness of point cloud data when shooting while moving. This is due to the fact that in the process of a 3D sensor acquiring point cloud data, there is a trade-off between the number of acquired point clouds and the frame rate. When acquiring point cloud data by shooting while moving, the faster the moving speed, the more real-time (immediate) performance is required of the 3D sensor, so it is necessary to reduce the number of acquired point clouds. Synthetic data created based on multiple point cloud data acquired by shooting while moving at a high speed becomes sparse data overall, which may affect the accuracy of data analysis.

[0007] The second problem is the bias or imbalance in the density of the point cloud in the created synthetic data. This is due to the fact that in the process of moving around and taking pictures using multiple 3D sensors, the point cloud density obtained varies depending on the moving speed and the frequency of patrols. For example, the point cloud density will be high in places where the patrol frequency is high, and conversely, the point cloud density will be low in places where the patrol frequency is low. In addition, when a 3D sensor is attached to a worker performing work at a work site and point cloud data of the work site is acquired at the same time as the work, the point cloud density will be high in places where the worker stops and works, and the point cloud density will be low in places where the worker finishes the work in a short time and leaves. In this way, even if there is a bias in the point cloud density of the synthetic data, it may affect the accuracy of data analysis.

[0008] The main objective of the present invention is to provide a 3D sensor control device, system, method, and program that can contribute to increasing and uniforming the point cloud density of synthetic data while ensuring the real-time capability of the 3D sensor when acquiring point cloud data by mobile shooting and generating synthetic data. [Means for solving the problem]

[0009] A three-dimensional sensor control device relating to a first perspective is a three-dimensional sensor control device attached to a photographer or a moving body, and comprises a motion measurement unit configured to measure the motion state of the photographer or the moving body attached to a three-dimensional sensor, a position identification unit configured to identify the position of the photographer or the moving body, and a three-dimensional sensor control unit configured to control the behavior of the three-dimensional sensor based on at least one of the motion state and the position of the photographer or the moving body, wherein the three-dimensional sensor photographs the object to be photographed with the behavior controlled by the three-dimensional sensor control unit to generate point cloud data, and the point cloud data is used to generate synthetic data.

[0010] The three-dimensional sensor control system relating to the second viewpoint includes a three-dimensional sensor control device relating to the first viewpoint, a three-dimensional sensor attached to the photographer or the moving object and configured to photograph the object to be photographed and generate the point cloud data while the behavior is controlled by the three-dimensional sensor control device, and a synthetic data generation device that generates the synthetic data based on the point cloud data.

[0011] A three-dimensional sensor control method according to a third aspect includes the steps of a three-dimensional sensor control device attached to a photographer or a moving body measuring the motion state of the photographer or the moving body to which a three-dimensional sensor is attached, a step of the three-dimensional sensor control device identifying the position of the photographer or the moving body, a step of the three-dimensional sensor control device controlling the behavior of the three-dimensional sensor based on at least one of the motion state and the position of the photographer or the moving body, a step of the three-dimensional sensor photographing an object to be photographed with the behavior controlled and generating point cloud data, and a step of a synthetic data generation device generating synthetic data based on the point cloud data.

[0012] The program relating to the fourth viewpoint is a program to be executed by a three-dimensional sensor control device attached to a photographer or a moving object, and causes the three-dimensional sensor control device to execute a process of measuring the motion state of the photographer or the moving object equipped with a three-dimensional sensor, a process of identifying the position of the photographer or the moving object, and a process of controlling the behavior of the three-dimensional sensor based on at least one of the motion state and the position of the photographer or the moving object, the three-dimensional sensor photographing the object to be photographed with the behavior controlled to generate point cloud data, and the point cloud data is used to generate synthetic data.

[0013] The program can be recorded in a computer-readable storage medium. The storage medium can be a non-transitory medium such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. In the present disclosure, the program can also be embodied as a computer program product. The program is input to the computer device from an input device or an external device via a communication interface, stored in a storage device, drives the processor according to a predetermined step or process, and can display the processing result, including an intermediate state, at each stage via a display device as necessary, or can communicate with the outside via the communication interface. For example, a computer device for this purpose typically includes a processor, a storage device, an input device, a communication interface, and a display device as necessary, which are connectable to each other via a bus. Effect of the Invention

[0014] According to the first to fourth aspects, when multiple point cloud data are acquired by mobile shooting to generate synthetic data, it is possible to contribute to increasing and uniforming the point cloud density of the synthetic data while ensuring the real-time capability of the 3D sensor. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram illustrating a schematic example of a configuration of a three-dimensional sensor control system according to the present disclosure. [Diagram 2] FIG. 1 is an image diagram showing a schematic representation of one mode of use when a three-dimensional sensor and a three-dimensional sensor control device of the three-dimensional sensor control system according to the present disclosure are carried by an operator. [Diagram 3] FIG. 1 is an image diagram showing a schematic example of a three-dimensional sensor control system according to the present disclosure applied to a work site. [Figure 4] 11 is a sequence chart illustrating an example of an operation of the three-dimensional sensor control system according to the present disclosure. [Diagram 5] 1 is a block diagram illustrating a schematic example of a configuration of a three-dimensional sensor control device according to the present disclosure. [Figure 6] FIG. 2 is a block diagram illustrating a schematic configuration of hardware resources. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] The embodiments will be described below with reference to the drawings. In this application, when reference symbols are given to the drawings, they are intended to aid in understanding and are not intended to limit the embodiments shown in the drawings. The embodiments below are merely examples and do not limit the present invention. The connection lines between blocks in the drawings and the like referred to in the following description include both bidirectional and unidirectional. One-way arrows are used to diagrammatically indicate the flow of a main signal (data) and do not exclude bidirectionality. Furthermore, although not explicitly shown in the circuit diagrams, block diagrams, internal configuration diagrams, connection diagrams, and the like shown in the present disclosure, input ports and output ports exist at the input and output ends of each connection line. The same is true for input / output interfaces. The program is executed via a computer device, which includes, for example, a processor, a storage device, an input device, a communication interface, and a display device as necessary, and the computer device is configured to be able to communicate with devices (including computers) inside or outside the device via the communication interface, regardless of whether it is wired or wireless.

[0017] [Form 1] The three-dimensional sensor control system according to the first embodiment will be described with reference to the drawings. FIG. 1 is a block diagram showing a schematic example of the configuration of the three-dimensional sensor control system according to the present disclosure. FIG. 2 is an image diagram showing a schematic example of a usage mode when a three-dimensional sensor and a three-dimensional sensor control device of the three-dimensional sensor control system according to the present disclosure are attached to a worker. FIG. 3 is an image diagram showing a schematic example of an application of the three-dimensional sensor control system according to the present disclosure to a work site.

[0018] The three-dimensional sensor control system 1 is a system that controls a three-dimensional sensor 10 used for mobile photography to acquire point cloud data and generate composite data (see FIG. 1). The three-dimensional sensor control system 1 can be used, for example, for progress management of work in the construction industry. The three-dimensional sensor control system 1 has a function of suppressing bias in point cloud density for the composite data by controlling the behavior of the three-dimensional sensor 10 according to the movement and position of the photographer 3 in mobile photography using the three-dimensional sensor 10 and synthesizing point cloud data acquired by multiple three-dimensional sensors 10. Here, the photographer 3 can be, for example, a worker, supervisor, security guard, or other person moving around the site, or may be a moving object such as a heavy machine, a vehicle, a drone, or a robot. The behavior of the three-dimensional sensor 10 can be an increase or decrease (adjustment) in point cloud density, for example, an increase or decrease (adjustment) in the number of point clouds, frame rate (fps; frame per seconds), number of samples, sampling period, resolution, and the like. The three-dimensional sensor control system 1 includes a mobile photography set 2, a composite data generation device 30, and a network 50.

[0019] The mobile photography set 2 is a set including a three-dimensional sensor 10 used for mobile photography and a three-dimensional sensor control device 20 (see Figs. 1 and 2). The mobile photography set 2 can be used in association with a photographer 3 working at a work site. At least one mobile photography set 2 is required.

[0020] The three-dimensional sensor 10 is a sensor that senses and photographs the surface of a photographing object (for example, a work site) in real space (see FIG. 1 and FIG. 2). The three-dimensional sensor 10 is connected to the three-dimensional sensor control device 20 so as to be able to communicate (wired communication or wireless communication). The three-dimensional sensor 10 photographs the photographing object to generate point cloud data, and outputs the generated point cloud data to the three-dimensional sensor control device 20. The generated point cloud data may be transmitted directly from the three-dimensional sensor 10 to the composite data generating device 30 without going through the three-dimensional sensor control device 20, but is not limited to this. In addition, the timing of transmitting the point cloud data may be, for example, periodically, when a certain amount of data is reached, or when an inquiry is received from the composite data generating device 30, but may be other timing and is not limited to this. The three-dimensional sensor 10 may be, for example, a ToF (Time of Flight) camera, a stereo camera, a 3D-LIDAR (Laser Imaging Detection And Ranging), a depth sensor, a distance measuring sensor, a distance camera, or the like. The three-dimensional sensor 10 can be used accompanying the photographer 3 (worn on the head), and may also be used accompanying a moving object such as a drone or a robot. Here, the point cloud data is data generated by the three-dimensional sensor 10, and is data depicted as a point cloud (a large number of points having three-dimensional coordinates). The three-dimensional sensor 10 can be changed to a sensor with various output formats according to customer requests. The behavior of the three-dimensional sensor 10 (for example, the number of point clouds, frame rate, etc.) is controlled by a three-dimensional sensor control unit 23 of the three-dimensional sensor control device 20. The point cloud data obtained by photographing with the three-dimensional sensor 10 is transmitted to the three-dimensional sensor control device 20.

[0021] The three-dimensional sensor control device 20 is a device that controls the behavior of the three-dimensional sensor 10 (see FIG. 1 and FIG. 2). The three-dimensional sensor control device 20 is connected to the three-dimensional sensor 10 so as to be able to communicate (wired communication or wireless communication). The three-dimensional sensor control device 20 is connected to the composite data generation device 30 via a network 50 so as to be able to communicate (wired communication or wireless communication). The three-dimensional sensor control device 20 has a function of acquiring point cloud data from the three-dimensional sensor 10 and transmitting it to the composite data generation device 30. The three-dimensional sensor control device 20 has a function of acquiring and displaying composite data from the composite data generation device 30. As the three-dimensional sensor control device 20, a device (computer device) having functional units (e.g., a processor, a storage device, an input device, a communication interface, and a display device) that constitute a computer can be used, and for example, a smartphone, a tablet terminal, or the like can be used. The three-dimensional sensor control device 20 can be used accompanying the photographer 3, and may be used accompanying a moving object such as a drone or a robot. The three-dimensional sensor control device 20 may be configured to be communicably connected to an eyeglass-type display device such as smart glasses or AR glasses that are worn (attached) to cover the eyes of the head of the photographer 3. The three-dimensional sensor control device 20 can be virtually configured to include a motion measurement unit 21, a position identification unit 22, a three-dimensional sensor control unit 23, a point cloud data acquisition unit 24, a point cloud data transmission unit 25, and a composite data display unit 26 by executing a predetermined program.

[0022] The motion measuring unit 21 is a functional unit that measures the motion state of the photographer 3 equipped with the three-dimensional sensor 10 (see FIG. 1). Here, the motion state may be, for example, a moving speed or the like, but is not limited to this and may be other motion states. In addition, the motion state measuring method may be, for example, a moving speed measuring method using an acceleration sensor or a GNSS (Global Navigation Satellite System), but is not limited to this and may be other measuring methods.

[0023] The position identification unit 22 is a functional unit that identifies the position of the photographer 3 equipped with the three-dimensional sensor 10 (see FIG. 1). Here, examples of methods for identifying the position include a method for identifying the position using an image of point cloud data acquired from the three-dimensional sensor 10 (for example, an image including position reference objects such as an AR (Augmented Reality) marker, a two-dimensional marker, and a reference sphere), and a method for identifying the position using a GNSS or an altitude sensor, but other methods may also be used and are not limited to these. The identified position may be not only a two-dimensional position but also a three-dimensional position. The position identification unit 22 may transmit data related to the identified position of the photographer 3 to the synthetic data generation device 30.

[0024] The three-dimensional sensor control unit 23 is a functional unit that controls the behavior of the three-dimensional sensor 10 based on the results of the motion measurement unit 21 and the position identification unit 22 (motion state and position of the photographer 3 (three-dimensional sensor 10)) (see FIG. 1). An example of a control method is a control method using a table in which the number of point clouds, frame rate, and power supply state are set according to the motion state and position, but other control methods may also be used and are not limited to this. Examples of the content of the control include an increase or decrease in the number of point clouds in the point cloud data, an increase or decrease in the frame rate of the point cloud data, and power ON / OFF, but other control contents may also be used and are not limited to this.

[0025] Furthermore, the three-dimensional sensor control unit 23 may share not only the position of the photographer 3 himself but also the positions of other photographers 3, and increase or decrease the number of point clouds and the frame rate of the point cloud data output by the three-dimensional sensor 10 according to the relative positions of the other photographers. For example, in an area where there are no other photographers 3 in the vicinity (within a predetermined range from the photographer himself) and only the photographer himself is present, the number of point clouds output by the three-dimensional sensor 10 can be increased. Also, in an area where multiple photographers 3 are close to each other, the number of point clouds output by the three-dimensional sensor 10 can be decreased.

[0026] Furthermore, the three-dimensional sensor control unit 23 may hold the composite data 40 and increase or decrease the number of point clouds and the frame rate of the point cloud data output by the three-dimensional sensor 10 according to the position of the photographer 3 and the point cloud density on the composite data 40. For example, if the photographer 3 is in an area on the composite data 40 where the point cloud density is low (sparse), the number of point clouds output by the three-dimensional sensor 10 can be increased. Also, if the photographer 3 is in an area on the composite data 40 where sufficient point cloud density has already been acquired, the number of point clouds output by the three-dimensional sensor 10 can be decreased.

[0027] Furthermore, the three-dimensional sensor control unit 23 may hold the composite data 40 and increase or decrease the number of point clouds and the frame rate of the point cloud data output by the three-dimensional sensor 10 according to the position of the photographer 3 and the update history of the point cloud on the composite data 40. For example, if the photographer 3 is in an area on the composite data 40 where the point cloud has not been updated for a certain period of time, the number of point clouds output by the three-dimensional sensor 10 can be increased. Also, if the photographer 3 is in an area on the composite data 40 where the point cloud has been updated within a preset period of time, the number of point clouds output by the three-dimensional sensor 10 can be decreased.

[0028] Furthermore, the 3D sensor control unit 23 may increase or decrease the number of point clouds and the frame rate of the point cloud data output by the 3D sensor 10 depending on the height of the position of the photographer 3. For example, if the work site has multiple floors, control may be performed to change the number of point clouds and the frame rate of the 3D sensor 10 for each floor.

[0029] Furthermore, when the position of the photographer 3 is in a specific area set in advance, the 3D sensor control unit 23 may increase or decrease the number of point clouds or the frame rate of the point cloud data output by the 3D sensor 10. For example, control may be performed to increase the number of point clouds of the 3D sensor 10 for a specific area where multiple pieces of equipment are crowded together.

[0030] In addition, the three-dimensional sensor control unit 23 may increase or decrease the number of points in the point cloud data output by the three-dimensional sensor 10 or the frame rate when a specific object set in advance is detected in the point cloud data acquired by the three-dimensional sensor 10. A plane detection process may be performed on the collected point cloud data, and if a planar object of low importance such as a floor or wall is detected, the behavior of the 3D sensor 10 may be controlled to reduce the number of point clouds output by the 3D sensor 10.

[0031] In addition, the 3D sensor control unit 23 may allow the photographer 3 (who may be a user of the synthetic data generation device 30 in a remote location) to manually select a location where he or she wants to increase or decrease the number of point clouds while referring to the synthetic data 40, and increase or decrease the number of point clouds or the frame rate of the point cloud data output by the 3D sensor 10 of the photographer 3 located at that location.

[0032] The point cloud data acquisition unit 24 is a functional unit that acquires point cloud data from the three-dimensional sensor 10 (see FIG. 1). The point cloud data acquired by the point cloud data acquisition unit 24 is point cloud data from the three-dimensional sensor 10 in a state in which the behavior is controlled by the three-dimensional sensor control unit 23.

[0033] The point cloud data transmission unit 25 is a functional unit that transmits the acquired point cloud data to the composite data generation device 30 via the network 50 (see FIG. 1). When transmitting the point cloud data, the point cloud data transmission unit 25 may include data related to the position of the photographer 3 identified by the position identification unit 22 in the point cloud data.

[0034] The composite data display unit 26 is a functional unit that displays (visualizes) the composite data from the composite data generation device 30 (see FIG. 1). Examples of the display method include display using a Web browser on a display device in the three-dimensional sensor control device 20, and display adapted to the real space on a glasses-type display device (smart glasses, AR glasses, etc.) connected to the three-dimensional sensor control device 20, but other display methods may also be used and are not limited to these.

[0035] The synthetic data generating device 30 is a device that generates synthetic data 40 based on at least one point cloud data from the three-dimensional sensor control device 20 (see Figs. 1 and 2). The synthetic data generating device 30 can align the acquired point cloud data with existing synthetic data by registration and synthesize the data to generate new synthetic data 40. The synthetic data generating device 30 can store the generated synthetic data 40 and transmit it to the three-dimensional sensor control device 20. The synthetic data generating device 30 is connected to the three-dimensional sensor control device 20 via a network 50 so as to be able to communicate (wired communication or wireless communication). As the synthetic data generating device 30, a device (computer device) having functional units (e.g., a processor, a storage device, an input device, a communication interface, and a display device) constituting a computer can be used, and for example, a server, a personal computer, etc. can be used. The synthetic data generating device 30 can be configured to include a synthetic data generating unit 31, a synthetic data storage unit 32, and a synthetic data transmitting unit 33 by executing a predetermined program.

[0036] The synthetic data generating unit 31 is a functional unit that generates synthetic data 40 based on the point cloud data from the three-dimensional sensor control device 20 (see FIG. 1). The synthetic data generating unit 31 includes a point cloud data acquiring unit 31a, a position matching unit 31b, and a point cloud data synthesizing unit 31c.

[0037] Point cloud data acquiring unit 31a is a functional unit that acquires point cloud data from point cloud data transmitting unit 25 of three-dimensional sensor control device 20 (see FIG. 1).

[0038] The positioning unit 31b is a functional unit that aligns the acquired point cloud data with existing composite data by registration (see FIG. 1). Examples of the positioning method include a method of aligning based on an image of the acquired point cloud data (e.g., an image including a position reference such as an AR marker, a two-dimensional marker, or a reference sphere), a method of aligning based on data related to the position of the photographer 3 acquired from the three-dimensional sensor control device 20 (which may be the position of the photographer 3 included in the point cloud data), and the like, but other positioning methods may also be used and are not limited to these. The positioning unit 31b can omit the positioning if there is no existing composite data.

[0039] The point cloud data synthesis unit 31c is a functional unit that synthesizes (adds) the aligned point cloud data to the existing synthetic data to generate new synthetic data 40 (see FIG. 1). The point cloud data synthesis unit 31c stores the generated synthetic data 40 in the synthetic data storage unit 32 and transmits it to the three-dimensional sensor control device 20. When there is no existing synthetic data, the point cloud synthesis unit 31c stores the acquired point cloud data as synthetic data 40 in the synthetic data storage unit 32 and transmits it to the three-dimensional sensor control device 20. When synthesizing the synthetic data 40, the point cloud synthesis unit 31c may weight the time at which the point cloud data was acquired. For example, the older the photographing time of the point cloud in the synthetic data 40, the sparser or erased it. Also, the reliability of the point cloud with the newer photographing time may be increased, and the reliability of the point cloud with the older photographing time may be decreased.

[0040] The composite data storage unit 32 is a functional unit that stores the generated composite data 40 (see FIG. 1). The composite data storage unit 32 may store not only the latest composite data 40 but also past composite data 40 so that they can be referenced in chronological order. For example, the composite data 40 at a specific time may be stored so that they can be referenced.

[0041] The composite data transmission unit 33 is a functional unit that transmits the generated composite data 40 to the composite data display unit 26 of the three-dimensional sensor control device 20 (see FIG. 1).

[0042] When the above-described three-dimensional sensor control system 1 is applied to a work site as shown in Fig. 3, the behavior of the three-dimensional sensor 10 is controlled as follows. The behavior of the three-dimensional sensor 10 is controlled according to the motion state and position of the photographers 3a-3h who are accompanied by the three-dimensional sensor 10.

[0043] For the 3D sensor 10 of the photographer 3a who moves quickly, the frame rate is controlled to be increased. This is done for the purpose of preventing the range where point clouds cannot be acquired from increasing in proportion to the speed of the photographer 3a's movement. In this case, when using a 3D sensor 10 that can acquire high-density point clouds but takes time to acquire one-shot point cloud data, control may be performed to reduce the number of point clouds to be output in order to improve real-time performance.

[0044] For the 3D sensors 10 of the photographers 3b, 3c, 3d, and 3e who are close to each other, control is performed to reduce the number of point clouds to be output. This is done to prevent the density of the point cloud data from becoming locally high in the composite data, since the point cloud data acquired by the photographers 3b, 3c, 3d, and 3e contains many overlapping areas when creating the composite data.

[0045] For the 3D sensor 10 of the photographer 3f who is in an area where the composite data already contains sufficient point cloud data, control is performed to reduce the number of point clouds to be output. This is done to prevent the density of point cloud data from becoming locally high when creating the composite data.

[0046] Contrary to the fast moving photographer 3a, the 3D sensor 10 for the slow moving photographer 3g is controlled to lower the frame rate or to increase the number of point clouds output since real-time performance is not required.

[0047] When the photographer 3h leaves the work site, the three-dimensional sensor 10 may be controlled to be turned off.

[0048] Next, the operation of the three-dimensional sensor control system according to the first embodiment will be described with reference to the drawings. Fig. 4 is a sequence chart showing an example of the operation of the three-dimensional sensor control system according to the present disclosure. Please refer to Figs. 1 and 2 for the configuration of the three-dimensional sensor control system.

[0049] First, the motion measuring unit 21 of the three-dimensional sensor control device 20 measures the motion state of the photographer 3 equipped with the three-dimensional sensor 10 (step A1).

[0050] Next, the position identifying unit 22 of the three-dimensional sensor control device 20 identifies the position of the photographer 3 carrying the three-dimensional sensor 10 (step A2).

[0051] Next, the three-dimensional sensor control unit 23 of the three-dimensional sensor control device 20 controls the behavior of the three-dimensional sensor 10 based on the results of the motion measurement unit 21 and the position identification unit 22 (motion state and position of the photographer 3 (three-dimensional sensor 10)) (step A3).

[0052] Next, the 3D sensor 10 photographs the object to be photographed while its behavior is controlled by the 3D sensor control unit 23 of the 3D sensor control device 20, generates point cloud data, and outputs the generated point cloud data to the 3D sensor control device 20 (step A4).

[0053] Next, the point cloud data acquisition unit 24 of the 3D sensor control device 20 acquires point cloud data from the 3D sensor 10, and transmits the acquired point cloud data from the point cloud data transmission unit 25 of the 3D sensor control device 20 to the point cloud data acquisition unit 31a of the synthetic data generation unit 31 of the synthetic data generation device 30 (step A5).

[0054] Next, point cloud data acquisition unit 31a of synthetic data generation unit 31 of synthetic data generation device 30 acquires the point cloud data from point cloud data transmission unit 25 of three-dimensional sensor control device 20 (step A6).

[0055] Next, the position adjustment unit 31b of the synthetic data generating unit 31 of the synthetic data generating device 30 adjusts the position of the acquired point cloud data to the existing synthetic data by registration (step A7).

[0056] Next, the point cloud data synthesis unit 31c of the synthetic data generation unit 31 of the synthetic data generation device 30 synthesizes (adds) the aligned point cloud data to the existing synthetic data to generate new synthetic data 40, and stores the generated synthetic data 40 in the synthetic data storage unit 32 of the synthetic data generation device 30 (step A8).

[0057] Next, the synthetic data transmission unit 33 of the synthetic data generation device 30 transmits the generated synthetic data 40 to the synthetic data display unit 26 of the three-dimensional sensor control device 20 (step A9).

[0058] Next, composite data display unit 26 of three-dimensional sensor control device 20 acquires composite data 40 from composite data transmission unit 33 of composite data generation device 30, displays the acquired composite data 40 (step A10), and then ends and returns to the start. This allows photographer 3 to perform work while referring to composite data 40 displayed on three-dimensional sensor control device 20.

[0059] According to form 1, the behavior of the 3D sensor 10 attached to the photographer 3 is controlled based on the motion state and position of the photographer 3. Therefore, when point cloud data is acquired by mobile shooting to create composite data 40, it is possible to contribute to increasing and uniforming the point cloud density of the composite data 40 (suppressing bias in the point cloud density) while ensuring the real-time capability of the 3D sensor 10.

[0060] Furthermore, according to the first aspect, by reducing the number of point clouds and the frame rate in accordance with the motion state and position of the photographer 3, the amount of data transfer can be reduced.

[0061] Furthermore, according to the first aspect, the power supply of the three-dimensional sensor is automatically turned on or off when the photographer 3 enters or leaves the work site, thereby making it possible to save power.

[0062] Furthermore, according to form 1, areas of low point cloud density in the composite data 40 are automatically detected, and the number of point clouds output by the 3D sensor 10 attached to the photographer 3 in the area of ​​low point cloud density is increased, thereby making it possible to homogenize the point cloud density of the composite data 40.

[0063] [Form 2] A three-dimensional sensor control device 20 according to the second embodiment will be described with reference to the drawings. Fig. 5 is a block diagram showing a schematic example of a configuration of a three-dimensional sensor control device according to the present disclosure.

[0064] The three-dimensional sensor control device 20 is a device that controls the three-dimensional sensor 10. The three-dimensional sensor control device 20 is attached to a photographer or a moving object. The three-dimensional sensor control device 20 includes a motion measurement unit 21, a position identification unit 22, and a three-dimensional sensor control unit 23.

[0065] The motion measuring unit 21 is configured to measure the motion state of the photographer or moving object to which the three-dimensional sensor 10 is attached. The position identifying unit 22 is configured to identify the position of the photographer or moving object. The three-dimensional sensor control unit 23 is configured to control the behavior of the three-dimensional sensor 10 based on at least one of the motion state and the position of the photographer or moving object. The three-dimensional sensor 10 captures an image of a subject to be photographed while its behavior is controlled by the three-dimensional sensor control unit 23, and generates point cloud data. The point cloud data is used to generate synthetic data.

[0066] According to form 1, the behavior of the 3D sensor 10 attached to the photographer 3 is controlled based on the motion state and position of the photographer or the moving object. Therefore, when point cloud data is acquired by mobile shooting to create composite data 40, it is possible to contribute to increasing and uniforming the point cloud density of the composite data 40 (suppressing bias in the point cloud density) while ensuring the real-time capability of the 3D sensor 10.

[0067] The three-dimensional sensor control device according to the first and second aspects can be configured by so-called hardware resources (information processing device, computer), and can use one having the configuration shown in Fig. 6. For example, the hardware resource 100 includes a processor 101, a memory 102, a network interface 103, etc., which are connected to each other by an internal bus 104.

[0068] Note that the configuration shown in Fig. 6 is not intended to limit the hardware configuration of the hardware resource 100. The hardware resource 100 may include hardware (e.g., an input / output interface) that is not shown. Furthermore, the number of units such as the processor 101 included in the device is not intended to be limited to the example shown in Fig. 6, and for example, multiple processors 101 may be included in the hardware resource 100. For example, a CPU (Central Processing Unit), an MPU (Micro Processor Unit), a GPU (Graphics Processing Unit), etc. can be used as the processor 101.

[0069] The memory 102 may be, for example, a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), a solid state drive (SSD), or the like.

[0070] The network interface 103 may be, for example, a LAN (Local Area Network) card, a network adapter, a network interface card, or the like.

[0071] The functions of the hardware resource 100 are realized by the above-mentioned processing module. The processing module is realized, for example, by the processor 101 executing a program stored in the memory 102. The program can be updated by downloading it via a network or by using a storage medium storing the program. Furthermore, the processing module may be realized by a semiconductor chip. That is, it is sufficient that the functions performed by the processing module are realized by executing software on some kind of hardware.

[0072] Some or all of the above aspects may be described as follows, but are not limited to the following:

[0073] [Appendix 1] A three-dimensional sensor control device attached to a photographer or a moving object, A motion measurement unit having a three-dimensional sensor and configured to measure a motion state of the photographer or the moving object; a position identification unit configured to identify a position of the photographer or the moving object; a three-dimensional sensor control unit configured to control a behavior of the three-dimensional sensor based on at least one of the motion state and the position of the photographer or the moving object; Equipped with The three-dimensional sensor captures an image of a target object while the behavior of the three-dimensional sensor is controlled by the three-dimensional sensor control unit, and generates point cloud data; The point cloud data is used to generate synthetic data. 3D sensor control device. [Appendix 2] The control of the behavior is an increase or decrease in at least one of the number of points and the frame rate of the point cloud data output by the three-dimensional sensor. 2. The three-dimensional sensor control device according to claim 1. [Appendix 3] The three-dimensional sensor control unit is configured to share the position of the photographer or the moving object and the positions of other photographers or moving objects, and increase or decrease at least one of the number of point clouds and the frame rate of the point cloud data output by the three-dimensional sensor according to the mutual positional relationship. 3. The three-dimensional sensor control device according to claim 2. [Appendix 4] The three-dimensional sensor control unit is configured to increase or decrease at least one of the number of points and the frame rate of the point cloud data output by the three-dimensional sensor according to the position of the photographer or the moving object and the point cloud density on the composite data. 4. The three-dimensional sensor control device according to claim 2 or 3. [Appendix 5] The three-dimensional sensor control unit is configured to increase or decrease at least one of the number of points and the frame rate of the point cloud data output by the three-dimensional sensor according to the position of the photographer or the moving object and an update history of the point cloud on the composite data. 5. A three-dimensional sensor control device according to any one of claims 2 to 4. [Appendix 6] The three-dimensional sensor control unit is configured to increase or decrease at least one of the number of point clouds and the frame rate of the point cloud data output by the three-dimensional sensor depending on the height of the position of the photographer or the moving object. 6. A three-dimensional sensor control device according to any one of claims 2 to 5. [Appendix 7] The three-dimensional sensor control unit is configured to increase or decrease at least one of the number of points and the frame rate of the point cloud data output by the three-dimensional sensor when the position of the photographer or the moving object is in a predetermined specific area. 7. A three-dimensional sensor control device according to any one of claims 2 to 6. [Appendix 8] The three-dimensional sensor control unit is configured to increase or decrease at least one of the number of points and the frame rate of the point cloud data output by the three-dimensional sensor when a predetermined specific object is detected in the point cloud data acquired by the three-dimensional sensor. 8. A three-dimensional sensor control device according to any one of claims 2 to 7. [Appendix 9] The three-dimensional sensor control unit is configured to manually select a location where the number of points in the point cloud is to be increased or decreased, and to increase or decrease at least one of the number of points in the point cloud data and the frame rate of the point cloud data output by the three-dimensional sensor attached to the photographer or the moving object present at the location. 9. A three-dimensional sensor control device according to any one of claims 2 to 8. [Appendix 10] The behavior control is to turn on / off the power of the three-dimensional sensor. 10. A three-dimensional sensor control device according to any one of claims 1 to 9. [Appendix 11] The motion state is a moving speed of the photographer or the moving object. 11. A three-dimensional sensor control device according to any one of claims 1 to 10. [Appendix 12] The motion measurement unit is configured to measure the moving speed using an acceleration sensor or a GNSS. 12. The three-dimensional sensor control device according to claim 11. [Appendix 13] a point cloud data acquisition unit configured to acquire the point cloud data from the three-dimensional sensor in a state in which the behavior is controlled by the three-dimensional sensor control unit; a point cloud data transmission unit configured to transmit the acquired point cloud data to a synthetic data generation device that generates the synthetic data; Equipped with 13. A three-dimensional sensor control device according to any one of claims 1 to 12. [Appendix 14] the point cloud data transmission unit is configured to transmit the point cloud data to the synthetic data generation device while including data related to the position of the photographer or the moving object identified by the position identification unit in the point cloud data. 14. The three-dimensional sensor control device according to claim 13. [Appendix 15] The three-dimensional sensor transmits the generated point cloud data to a synthetic data generation device that generates the synthetic data. 13. A three-dimensional sensor control device according to any one of claims 1 to 12. [Appendix 16] The position specifying unit is configured to transmit data relating to the specified position of the photographer or the moving object to the synthetic data generating device. 16. The three-dimensional sensor control device according to claim 15. [Appendix 17] a synthetic data display unit configured to display the synthetic data from the synthetic data generation device; 17. A three-dimensional sensor control device according to any one of appendixes 13 to 16. [Appendix 18] The composite data display unit is configured to display the composite data on a web browser using a display device in the three-dimensional sensor control device, or to display the composite data in accordance with real space on a glasses-type display device connected to the three-dimensional sensor control device. 18. The three-dimensional sensor control device according to claim 17. [Appendix 19] The position identification unit is configured to identify the position using an image of the point cloud data, or to identify the position using a GNSS or an altitude sensor. 19. A three-dimensional sensor control device according to any one of claims 1 to 18. [Appendix 20] A three-dimensional sensor control device according to any one of appendix 1 to 19, The three-dimensional sensor is attached to the photographer or the moving object, and is configured to capture an image of the object to be photographed and generate the point cloud data while the behavior of the object is controlled by the three-dimensional sensor control device; a synthetic data generating device for generating the synthetic data based on the point cloud data; A three-dimensional sensor control system comprising: [Appendix 21] A three-dimensional sensor control device attached to a photographer or a moving object measures a motion state of the photographer or the moving object to which a three-dimensional sensor is attached; A step in which the three-dimensional sensor control device specifies a position of the photographer or the moving object; a step of the three-dimensional sensor control device controlling a behavior of the three-dimensional sensor based on at least one of the motion state and the position of the photographer or the moving object; A step in which the three-dimensional sensor captures an image of a target object while controlling the behavior, thereby generating point cloud data; A synthetic data generating device generates synthetic data based on the point cloud data; A three-dimensional sensor control method comprising: [Appendix 22] A program to be executed by a three-dimensional sensor control device attached to a photographer or a moving object, A process of measuring a motion state of the photographer or the moving object equipped with a three-dimensional sensor; A process of identifying a position of the photographer or the moving object; A process of controlling a behavior of the three-dimensional sensor based on at least one of the motion state and the position of the photographer or the moving object; The three-dimensional sensor control device executes the above-mentioned The three-dimensional sensor captures an image of the object to be photographed while the behavior is controlled to generate point cloud data; The point cloud data is used to generate synthetic data. program.

[0074] The disclosures of the above patent documents are incorporated herein by reference and may be used as the basis or part of the present invention as necessary. Within the framework of the entire disclosure of the present invention (including the claims and drawings), modifications and adjustments of the forms and embodiments are possible based on the basic technical ideas. Furthermore, within the framework of the entire disclosure of the present invention, various combinations or selections (or non-selection as necessary) of various disclosed elements (including each element of each claim, each element of each form or embodiment, each element of each drawing, etc.) are possible. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art would be able to make in accordance with the entire disclosure, including the claims and drawings, and the technical ideas. Furthermore, with regard to the numerical values ​​and numerical ranges described in this application, any intermediate value, lower numerical value, and small range are considered to be described even if not specified. Furthermore, the disclosures of the above cited documents may be used in part or in whole in combination with the descriptions in this document as part of the disclosure of the present invention in accordance with the spirit of the present invention as necessary, and are considered to be included (belong) to the disclosures of this application. [Explanation of symbols]

[0075] 1. 3D sensor control system 2 Mobile shooting set 3, 3a~3h Photographer 10 3D Sensor 20 3D sensor control device 21 Motion Measurement Section 22 Location identification part 23 3D sensor control unit 24 Point cloud data acquisition section 25 Point cloud data transmission unit 26 Composite data display section 30 Synthetic data generator 31 Synthetic Data Generation Unit 31a Point cloud data acquisition section 31b Alignment section 31c Point cloud data synthesis section 32 Synthetic Data Storage Unit 33 Synthetic data transmission unit 40 Synthetic Data 50 Network 100 Hardware Resources 101 Processor 102 Memory 103 Network Interface 104 Internal Bus

Claims

1. A three-dimensional sensor control device attached to a photographer or a moving object, A motion measurement unit having a three-dimensional sensor and configured to measure a motion state of the photographer or the moving object; a position identification unit configured to identify a position of the photographer or the moving object; a three-dimensional sensor control unit configured to control a behavior of the three-dimensional sensor based on at least one of the motion state and the position of the photographer or the moving object; Equipped with The three-dimensional sensor captures an image of a target object while the behavior of the target object is controlled by the three-dimensional sensor control unit to generate point cloud data; The point cloud data is used to generate synthetic data. 3D sensor control device.

2. The control of the behavior is an increase or decrease in at least one of the number of points and the frame rate of the point cloud data output by the three-dimensional sensor. The three-dimensional sensor control device according to claim 1.

3. The behavior control is to turn on / off the power supply of the three-dimensional sensor. The three-dimensional sensor control device according to claim 1.

4. The motion state is a moving speed of the photographer or the moving object. The three-dimensional sensor control device according to claim 1.

5. a point cloud data acquisition unit configured to acquire the point cloud data from the three-dimensional sensor in a state in which the behavior is controlled by the three-dimensional sensor control unit; a point cloud data transmission unit configured to transmit the acquired point cloud data to a synthetic data generation device that generates the synthetic data; Equipped with The three-dimensional sensor control device according to claim 1.

6. a synthetic data display unit configured to display the synthetic data from the synthetic data generation device; 6. The three-dimensional sensor control device according to claim 5.

7. The composite data display unit is configured to display the composite data on a web browser using a display device in the three-dimensional sensor control device, or to display the composite data in accordance with a real space on a glasses-type display device connected to the three-dimensional sensor control device.

7. The three-dimensional sensor control device according to claim 6.

8. A three-dimensional sensor control device according to any one of claims 1 to 7, The three-dimensional sensor is attached to the photographer or the moving body, and is configured to capture an image of the object to be photographed and generate the point cloud data while the behavior of the object is controlled by the three-dimensional sensor control device; a synthetic data generating device for generating the synthetic data based on the point cloud data; A three-dimensional sensor control system comprising:

9. A three-dimensional sensor control device attached to a photographer or a moving object measures a motion state of the photographer or the moving object to which a three-dimensional sensor is attached; A step in which the three-dimensional sensor control device specifies a position of the photographer or the moving object; a step of the three-dimensional sensor control device controlling a behavior of the three-dimensional sensor based on at least one of the motion state and the position of the photographer or the moving object; A step of capturing an image of a target object with the three-dimensional sensor while the behavior of the target object is controlled to generate point cloud data; A synthetic data generating device generates synthetic data based on the point cloud data; A three-dimensional sensor control method comprising:

10. A program to be executed by a three-dimensional sensor control device attached to a photographer or a moving object, A process of measuring a motion state of the photographer or the moving object equipped with a three-dimensional sensor; A process of identifying a position of the photographer or the moving object; A process of controlling a behavior of the three-dimensional sensor based on at least one of the motion state and the position of the photographer or the moving object; The three-dimensional sensor control device executes the above-mentioned The three-dimensional sensor captures an image of the object to be photographed while the behavior is controlled, and generates point cloud data; The point cloud data is used to generate synthetic data. program.

Images