Data generation processing device, data generation processing method and program

Abstract

To quickly provide three-dimensional space information around a recognized object or three-dimensional space information using a mutual physical relationship between recognized objects.SOLUTION: A data generation processing device includes: an RGB camera 10 that acquires image data; a depth camera 20 that acquires point group data; a pixel position information detection unit 30 that detects pixel position information on an object from the acquired image data; a three-dimensional coordinate data acquisition unit 110 that acquires three-dimensional coordinate data of the object on the basis of the detected pixel position information on the object and the acquired point group data; and a data generation unit 120 that generates three-dimensional space information for a peripheral space of the object on the basis of the acquired three-dimensional coordinate data.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a data generation processing device, a data generation processing method, and a program. [Background technology]

[0002] A technique has been known in the past in which an image camera and a depth camera are used to recognize a target object from image data obtained by the image camera, and the position information of the recognized target object is measured by the depth camera to obtain three-dimensional coordinate data of the target object.

[0003] The above-mentioned technologies are often attracting attention in terms of obtaining highly accurate positional information of an object in three-dimensional space, which is required for machining, when performing machining on a recognized object, or reducing the time required to process the spatial coordinates.

[0004] As an example of this type of technology, there is disclosed an image processing device that has a 3D point cloud acquisition unit, a 2D label acquisition unit, and a label conversion unit, in which the 3D point cloud acquisition unit acquires 3D point cloud data indicating the 3D point cloud of an object, the 2D label acquisition unit acquires 2D label data corresponding to a 2D image of the object, and the label conversion unit associates the 2D label data with the 3D point cloud data and converts the 2D label data into 3D point cloud label data indicating the labels of the 3D point cloud (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 7205613 Summary of the Invention [Problem to be solved by the invention]

[0006] On the other hand, there are cases where the required information is not the recognized object itself, but the three-dimensional spatial information around it, or the three-dimensional spatial information that utilizes the mutual positional relationships between the recognized objects. Specifically, this is the case when the intention is to obtain three-dimensional spatial information such as license plates installed at the front or rear of a car, a stroller being pushed by a person, a hat or shoes worn by a person, or a pet running alongside a person.

[0007] However, the technology described in Patent Document 1 relates to generating label data used in an object recognition technology that efficiently uses three-dimensional point cloud data for image data and point cloud data. Therefore, the technology described in Patent Document 1 had the problem that it was not possible to quickly provide three-dimensional spatial information around a recognized object or three-dimensional spatial information utilizing the mutual positional relationships between recognized objects.

[0008] Therefore, the present invention has been made in consideration of the above-mentioned problems, and aims to quickly provide three-dimensional spatial information around a recognized object, or three-dimensional spatial information utilizing the mutual positional relationships between recognized objects. [Means for solving the problem]

[0009] Form 1: One or more embodiments of the present invention propose a data generation processing device including an RGB camera that acquires image data, a depth camera that acquires point cloud data, a pixel position information detection unit that detects pixel position information of an object from the acquired image data, a three-dimensional coordinate data acquisition unit that acquires three-dimensional coordinate data of the object based on the detected pixel position information of the object and the acquired point cloud data, and a data generation unit that generates three-dimensional spatial information of the space surrounding the object based on the acquired three-dimensional coordinate data.

[0010] Form 2: One or more embodiments of the present invention propose a data generation processing device, wherein the three-dimensional coordinate data acquisition unit includes a three-dimensional coordinate data generation unit that associates pixel position information of the detected object with the acquired point cloud data to generate three-dimensional coordinate data of the object and the space surrounding the object, and a three-dimensional coordinate data extraction unit that extracts three-dimensional coordinate data of a specified object or the three-dimensional space surrounding the object based on extraction conditions from the generated three-dimensional coordinate data of the object and the space surrounding the object.

[0011] Mode 3: One or more embodiments of the present invention propose a data generating and processing device in which the extraction conditions are variable based on the actual dimensions of the object.

[0012] Form 4: One or more embodiments of the present invention propose a data generation processing device in which the three-dimensional coordinate data extraction unit extracts three-dimensional coordinate data of the three-dimensional space around the object based on the positional relationship between the multiple objects when there are multiple objects.

[0013] Mode 5: One or more embodiments of the present invention propose a data generation processing device in which the positional relationship is distance information between the predetermined object and a plurality of the objects.

[0014] Form 6: One or more embodiments of the present invention propose a data generation processing device including: a field of view detection unit that detects the field of view of the RGB camera and the depth camera; and a surrounding three-dimensional space setting unit that changes the setting of the three-dimensional space around the object based on the detected field of view information of the RGB camera and the depth camera.

[0015] Form 7: One or more embodiments of the present invention propose a data generation processing method in a data generation processing device including an RGB camera that acquires image data, a depth camera that acquires point cloud data, a pixel position information detection unit, a three-dimensional coordinate data acquisition unit, and a data generation unit, the data generation processing method including: a first step in which the pixel position information detection unit detects pixel position information of an object from the image data acquired by the RGB camera; a second step in which the three-dimensional coordinate data acquisition unit acquires three-dimensional coordinate data of the object based on the pixel position information of the object detected in the first step and the point cloud data acquired by the depth camera; and a third step in which the data generation unit generates three-dimensional spatial information of the space surrounding the object based on the three-dimensional coordinate data acquired in the second step.

[0016] Form 8: One or more embodiments of the present invention propose a data generation processing method including a fourth step in which the three-dimensional coordinate data acquisition unit associates pixel position information of the object detected in the second step with the acquired point cloud data to generate three-dimensional coordinate data of the object and the space surrounding the object, and a fifth step in which three-dimensional coordinate data of a specified object or the three-dimensional space surrounding the object is extracted from the generated three-dimensional coordinate data of the object and the space surrounding the object based on extraction conditions.

[0017] Form 9: One or more embodiments of the present invention propose a program for causing a computer to execute a data generation processing method in a data generation processing device including an RGB camera that acquires image data, a depth camera that acquires point cloud data, a pixel position information detection unit, a three-dimensional coordinate data acquisition unit, and a data generation unit, wherein the program causes a computer to execute the following steps: a first step in which the pixel position information detection unit detects pixel position information of an object from image data acquired by the RGB camera; a second step in which the three-dimensional coordinate data acquisition unit acquires three-dimensional coordinate data of the object based on the pixel position information of the object detected in the first step and the point cloud data acquired by the depth camera; and a third step in which the data generation unit generates three-dimensional spatial information of the space surrounding the object based on the three-dimensional coordinate data acquired in the second step.

[0018] Form 10: One or more embodiments of the present invention propose a program including: a fourth step in which the three-dimensional coordinate data acquisition unit associates pixel position information of the object detected in the second step with the acquired point cloud data to generate three-dimensional coordinate data of the object and the space surrounding the object; and a fifth step in which three-dimensional coordinate data of a specified object or the three-dimensional space surrounding the object is extracted from the generated three-dimensional coordinate data of the object and the space surrounding the object based on extraction conditions. [Effects of the Invention]

[0019] According to one or more embodiments of the present invention, it is possible to quickly provide three-dimensional spatial information around a recognized object, or three-dimensional spatial information utilizing the mutual positional relationships between recognized objects. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a diagram illustrating a configuration of a data generation processing device according to a first embodiment of the present invention. [Figure 2]1 is a diagram illustrating a configuration of a processor in a data generation processing device according to a first embodiment of the present invention. [Figure 3] FIG. 2 is a process flow diagram of the data generation processing device according to the first embodiment of the present invention. [Figure 4] FIG. 2 is a process flow diagram of the data generation processing device according to the first embodiment of the present invention. [Figure 5] 3A to 3C are diagrams for explaining the principle of determining actual dimensions obtained in the data generation processing device according to the first embodiment of the present invention. [Figure 6] FIG. 10 is a diagram showing a modified example of three-dimensional coordinate data obtained in the data generation processing device according to the first embodiment of the present invention. [Figure 7] FIG. 10 is a diagram showing a modified example of three-dimensional coordinate data obtained in the data generation processing device according to the first embodiment of the present invention. [Figure 8] FIG. 10 is a diagram illustrating a configuration of a data generation processing device according to a second embodiment of the present invention. [Figure 9] FIG. 10 is a diagram illustrating a configuration of a processor in a data generation processing device according to a second embodiment of the present invention. [Figure 10] FIG. 10 is a process flow diagram of a data generation processing device according to a second embodiment of the present invention. [Figure 11] 10A and 10B are diagrams for explaining the principle of determining actual dimensions obtained in a data generation processing device according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0021] The data generation processing device will be described with reference to FIGS.

[0022] First Embodiment A data generation processing device 1 according to this embodiment will be described with reference to FIGS.

[0023] <Configuration of data generation processing device 1> As shown in FIG. 1, the data generation processing device 1 according to this embodiment includes an RGB camera 10, a depth camera 20, a pixel position information detection unit 30, a processor 100, an operation unit 50, a display unit 60, and a memory unit 200.

[0024] The RGB camera 10 uses visible light with a wavelength of, for example, 400 nm to 700 nm, captures light with red, green, and blue wavelengths, and captures a color image of an object in real time. The color image data of an object captured by the RGB camera 10 includes pixel-by-pixel information about the object (pixel position information of the object). The color image data of the object acquired by the RGB camera 10 is detected by a pixel position information detection unit 30, which will be described later. In the present embodiment, the RGB camera 10 is used as an example in the following description, but an image capture device or the like may also be used.

[0025] The depth camera 20 acquires point cloud data as distance information to the target object. The point cloud data acquired by the depth camera 20 is output to the processor 100, which will be described later. In the present embodiment, the depth camera 20 will be exemplified in the following description, but for example, 3D LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) or the like may also be used. Moreover, instead of the combination of the RGB camera 10 and the depth camera 20, an RGBD camera or the like may be used. It is assumed that the observation target in the visible image (image data) captured by the RGB camera 10 corresponds to the observation target in the point cloud acquired by the depth camera 20. In other words, unless otherwise specified, the image data output by the RGB camera 10 and the point cloud data output by the depth camera 20 are assumed to be acquired at the same time in the same environment under conditions where the relative positions of the RGB camera 10 and the depth camera 20 are known.

[0026] The pixel position information detection unit 30 detects pixel position information of an object from the image data acquired by the RGB camera 10 . The method of detecting pixel position information in the pixel position information detection unit 30 may be detection by image recognition using machine learning, or detection by image processing using hue, shape, or the like. The pixel position information of the object detected by the pixel position information detection unit 30 is output to the processor 100, which will be described later. In this embodiment, the pixel position information detection unit 30 is provided separately from the processor, but the processing of the pixel position information detection unit 30 may be performed by the processor.

[0027] The processor 100 inputs pixel position information of the object from the pixel position information detection unit 30 and point cloud data from the depth camera 20, generates three-dimensional coordinate data according to a control program, and outputs it to the display unit 60 or memory unit 200 described later.

[0028] The operation unit 50 functions as an input device that accepts information input from the user. Specifically, the operation unit 50 can be exemplified by a touch sensor or the like provided on a panel of a display unit, which will be described later.

[0029] The display unit 60 is configured, for example, by a liquid crystal display, and has a touch sensor and the like disposed on the bottom surface. The display unit 60 displays image information, text information, and the like.

[0030] The storage unit 200 is configured from a ROM (Read Only Memory) or a RAM (Random Access Memory), and stores programs, data, and the like.

[0031] <Configuration of the processor 100> As shown in FIG. 2, the processor 100 according to this embodiment includes a three-dimensional coordinate data acquisition unit 110, a data generation unit 120, and a control unit .

[0032] The three-dimensional coordinate data acquisition unit 110 acquires three-dimensional coordinate data of an object based on pixel position information of the object detected by the RGB camera 10 and point cloud data acquired by the depth camera 20. The three-dimensional coordinate data acquired by the three-dimensional coordinate data acquisition unit 110 is sent to the control unit 130 via the bus line BL. The three-dimensional coordinate data acquisition unit 110 includes a three-dimensional coordinate data generation unit that associates pixel position information of an object detected by the RGB camera 10 with point cloud data acquired by the depth camera 20 to generate three-dimensional coordinate data of the object and the space surrounding the object, and a three-dimensional coordinate data extraction unit that extracts three-dimensional coordinate data of a specified object or the three-dimensional space surrounding the object based on extraction conditions from the three-dimensional coordinate data of the object and the space surrounding the object generated by the three-dimensional coordinate data generation unit. Here, examples of extraction conditions include an object whose height information is larger than the unit dimension, or an object whose horizontal size is smaller than a certain dimension.

[0033] In the data generation processing device 1 according to this embodiment, as shown in FIGS. 5(A) and 5(B), the actual dimensions of an object can be obtained by applying the imaging principle of a pinhole camera to an image sensor. In other words, if the size h' of the image sensor and the focal length d' are known, and the object distance d is known, then h':d' = h:d, and the actual dimensions of the camera image can be calculated using the following equation 1. Therefore, it is possible to set the extraction conditions as described above.

[0034]

number

[0035] When there are multiple objects, the three-dimensional coordinate data extraction unit extracts three-dimensional coordinate data of the three-dimensional space around the objects based on the positional relationships between the multiple objects. Here, the positional relationship may be, for example, distance information between a predetermined object and a plurality of objects.

[0036] The data generating unit 120 generates three-dimensional space information about the space surrounding the object based on the acquired three-dimensional coordinate data. Here, examples of three-dimensional spatial information of the space surrounding an object include three-dimensional spatial information in which the recognized object exists, three-dimensional spatial information surrounding the recognized object by a unit length, three-dimensional spatial information about the area above or below the recognized object, and three-dimensional spatial information in a unit space above and to the left of the front of the recognized object.

[0037] In the data generation processing device 1 of this embodiment, as shown in Figures 6(A), 6(B), 6(C), 7(A), 7(B), and 7(C), it is possible to obtain three-dimensional spatial information formed by a rectangular parallelepiped when a recognized object is surrounded by a rectangular parallelepiped in Figure 6(A), three-dimensional spatial information formed by a rectangular parallelepiped of unit length when a recognized object is surrounded by a rectangular parallelepiped of unit length from below in Figure 6(B), and three-dimensional spatial information formed by a rectangular parallelepiped of unit length when the intermediate position of the recognized object is surrounded by a rectangular parallelepiped of unit length in Figure 6(C). In addition, in Figure 7(A), the recognized object can be shown as a two-dimensional plane, in Figure 7(B), the recognized object can be shown as a one-dimensional line, and in Figure 7(C), the recognized object can be shown as a zero-dimensional point. Therefore, as described above, three-dimensional spatial information with a degree of freedom can be obtained.

[0038] The control unit 130 inputs the pixel position information of the object from the pixel position information detection unit 30 and the point cloud data from the depth camera 20, generates three-dimensional coordinate data according to a control program, and outputs it to the display unit 60 or memory unit 200 described later.

[0039] <Processing of data generation processing device 1> The processing of the data generation processing device 1 according to this embodiment will be described with reference to FIG.

[0040] As shown in FIG. 3, the pixel position information detection unit 30 detects pixel position information of an object from image data acquired by the RGB camera 10 (step S110). The pixel position information of the object detected by the pixel position information detection unit 30 is output to a processor (three-dimensional coordinate data acquisition unit, data generation unit) 100, which will be described later.

[0041] The processor 100 acquires three-dimensional coordinate data of the object based on the pixel position information of the object detected in step S110 and the point cloud data acquired by the depth camera 20 (step S120). The three-dimensional coordinate data acquired by the processor 100 is sent to the control unit 130, which will be described later, via the bus line BL.

[0042] The processor 100 generates three-dimensional space information of the space around the object based on the three-dimensional coordinate data acquired in step S120 (step S130). Then, the processor 100 sends the generated three-dimensional space information of the space around the object to the display unit 60 and the storage unit 200, and ends the process.

[0043] <Processing of the three-dimensional coordinate data acquisition unit 110> The processing of the three-dimensional coordinate data acquisition unit 110 in the data generation processing device 1 according to this embodiment will be described with reference to FIG.

[0044] As shown in FIG. 4, the three-dimensional coordinate data acquisition unit (three-dimensional coordinate data generation unit) 110 associates pixel position information of the object detected by the RGB camera 10 with point cloud data acquired by the depth camera 20, and generates three-dimensional coordinate data of the object and the space surrounding the object (step S121).

[0045] The three-dimensional coordinate data acquisition unit (three-dimensional coordinate data extraction unit) 110 extracts three-dimensional coordinate data of a specified object or the three-dimensional space around the object based on the extraction conditions from the three-dimensional coordinate data of the object and the space around the object generated by the three-dimensional coordinate data acquisition unit (three-dimensional coordinate data generation unit) 110 (step S122). Then, the control unit 130 sends the three-dimensional coordinate data of the extracted predetermined object or the three-dimensional space around the object to the display unit 60 and the storage unit 200, and ends the process.

[0046] <Actions and Effects> As described above, the data generation processing device 1 according to this embodiment includes an RGB camera 10 that acquires image data, a depth camera 20 that acquires point cloud data, a pixel position information detection unit 30 that detects pixel position information of an object from the acquired image data, a three-dimensional coordinate data acquisition unit 110 that acquires three-dimensional coordinate data of the object based on the pixel position information of the detected object and the acquired point cloud data, and a data generation unit 120 that generates three-dimensional spatial information of the space surrounding the object based on the acquired three-dimensional coordinate data. That is, the three-dimensional coordinate data acquisition unit 110 acquires three-dimensional coordinate data of the object based on the pixel position information of the detected object and the acquired point cloud data, and the data generation unit 120 generates three-dimensional spatial information of the space surrounding the object based on the acquired three-dimensional coordinate data. The data generation unit 120 generates three-dimensional spatial information of the surrounding space of the object based on the acquired three-dimensional coordinate data, and by applying the pinhole camera mapping principle to the three-dimensional spatial information of the surrounding space of the object, it is possible to generate three-dimensional spatial information of the surrounding space of the object, including the size of the object, the center of gravity of the object, etc. Therefore, it is possible to quickly provide three-dimensional spatial information around a recognized object, or three-dimensional spatial information utilizing the mutual positional relationships between recognized objects.

[0047] The three-dimensional coordinate data acquisition unit 110 in the data generation processing device 1 of this embodiment is configured to include a three-dimensional coordinate data generation unit that associates pixel position information of a detected object with the acquired point cloud data to generate three-dimensional coordinate data of the object and the space surrounding the object, and a three-dimensional coordinate data extraction unit that extracts three-dimensional coordinate data of a specified object or the three-dimensional space surrounding the object based on extraction conditions from the generated three-dimensional coordinate data of the object and the space surrounding the object. In other words, the three-dimensional coordinate data generation unit associates pixel position information of the detected object with the acquired point cloud data to generate three-dimensional coordinate data of the object and the space surrounding the object, and the three-dimensional coordinate data extraction unit extracts three-dimensional coordinate data of a specified object or the three-dimensional space surrounding the object based on extraction conditions from the generated three-dimensional coordinate data of the object and the space surrounding the object. The three-dimensional coordinate data generation unit associates the pixel position information of the detected object with the acquired point cloud data to generate three-dimensional coordinate data of the object and the space surrounding the object. Therefore, by applying the pinhole camera mapping principle to the three-dimensional spatial information of the space surrounding the object, it is possible to generate three-dimensional spatial information of the space surrounding the object, including the size of the object, the center of gravity of the object, etc. The three-dimensional coordinate data extraction unit extracts three-dimensional coordinate data of a specified object or the three-dimensional space surrounding the object based on extraction conditions from the generated three-dimensional coordinate data of the object and the space surrounding the object.Therefore, by providing information including the size of the object and the center of gravity of the object as extraction conditions to the three-dimensional coordinate data extraction unit, it can quickly extract three-dimensional coordinate data of a specified object or the three-dimensional space surrounding the object that meets the extraction conditions. Therefore, it is possible to quickly provide three-dimensional spatial information around a recognized object, or three-dimensional spatial information utilizing the mutual positional relationships between recognized objects.

[0048] The extraction conditions in the data generation processing device 1 according to this embodiment are variable based on the actual dimensions of the object. In other words, if information including the size (actual dimensions) of an object and the center of gravity of the object is provided to the three-dimensional coordinate data extraction unit as extraction conditions, three-dimensional coordinate data of a specified object or the three-dimensional space surrounding the object that meets the extraction conditions can be quickly extracted. Therefore, it is possible to quickly provide three-dimensional spatial information around a recognized object, or three-dimensional spatial information utilizing the mutual positional relationships between recognized objects.

[0049] When there are multiple objects, the three-dimensional coordinate data extraction unit in the data generation processing device 1 according to this embodiment extracts three-dimensional coordinate data of the three-dimensional space around the objects based on the positional relationships between the multiple objects. In other words, when there are multiple objects, the three-dimensional coordinate data extraction unit can extract three-dimensional coordinate data of the three-dimensional space around the objects based on the positional relationship of the second object, for example, using the first object as a reference. Therefore, it is possible to quickly provide three-dimensional spatial information around a recognized object, or three-dimensional spatial information utilizing the mutual positional relationships between recognized objects.

[0050] The positional relationship in the data generation processing device 1 according to this embodiment is distance information between a predetermined object and multiple objects. In other words, the positional relationship is distance information between a predetermined object and multiple objects. Therefore, it is possible to quickly provide three-dimensional spatial information around a recognized object, or three-dimensional spatial information utilizing the mutual positional relationships between recognized objects.

[0051] <Second embodiment> A data generation processing device 1A according to this embodiment will be described with reference to FIGS.

[0052] <Configuration of the data generation processing device 1A> As shown in FIG. 8, the data generation processing device 1A according to this embodiment includes an RGB camera 10, a depth camera 20, a pixel position information detection unit 30, a processor 100A, an operation unit 50, a display unit 60, a field of view detection unit 70, a surrounding three-dimensional space setting unit 80, and a memory unit 200. Note that components with the same reference numerals as those in the first embodiment have the same functions, and therefore detailed descriptions thereof will be omitted.

[0053] The processor 100 inputs pixel position information of the object from the pixel position information detection unit 30 and point cloud data from the depth camera 20, generates three-dimensional coordinate data according to a control program, and outputs it to the display unit 60 or memory unit 200 described later. In this embodiment, for example, the same processing as in the first embodiment is executed in a surrounding three-dimensional space set by a surrounding three-dimensional space setting unit 80, which will be described later.

[0054] The angle-of-view detection unit 70 detects the angles of view of the RGB camera 10 and the depth camera 20 . The information on the angles of view of the RGB camera 10 and the depth camera 20 detected by the angle-of-view detection unit 70 is sent to a surrounding three-dimensional space setting unit 80, which will be described later.

[0055] The surrounding three-dimensional space setting unit 80 changes the setting of the three-dimensional space around the object based on the detected information on the angle of view of the RGB camera 10 and the depth camera 20. Specifically, as shown in FIG. 11, a data generation processing device according to this embodiment is installed. For example, a change in the vertical direction of a robot is detected as a change in the angle of view detected by the angle of view detection unit 70, and if there is a change in the angle of view, the surrounding three-dimensional space is changed and a new surrounding three-dimensional space is set. The three-dimensional space information around the object set in the surrounding three-dimensional space setting unit 80 is sent to the processor 100A.

[0056] <Configuration of Processor 100A> As shown in FIG. 9, the processor 100A according to this embodiment includes a three-dimensional coordinate data acquisition unit 110, a data generation unit 120, and a control unit 130A. Note that components with the same reference numerals as those in the first embodiment have the same functions, and therefore detailed descriptions thereof will be omitted.

[0057] The control unit 130A inputs pixel position information of the object from the pixel position information detection unit 30 and point cloud data from the depth camera 20, generates three-dimensional coordinate data according to a control program, and outputs it to the display unit 60 or memory unit 200 described later. In this embodiment, the control unit 130A inputs pixel position information of the object from the pixel position information detection unit 30, point cloud data from the depth camera 20, and three-dimensional spatial information around the object, and generates three-dimensional coordinate data taking into account the three-dimensional spatial information around the object in accordance with a control program, and outputs the data to the display unit 60 or memory unit 200, which will be described later.

[0058] <Processing of the data generation processing device 1A> The processing of the data generation processing device 1A according to this embodiment will be described with reference to FIG.

[0059] As shown in FIG. 10, the angle-of-view detection unit 70 detects the angles of view of the RGB camera 10 and the depth camera 20, and the surrounding three-dimensional space setting unit 80 changes the setting of the three-dimensional space around the object based on the detected angle-of-view information of the RGB camera 10 and the depth camera 20, and sends the information to the processor 100A (step S200).

[0060] The pixel position information detection unit 30 detects pixel position information of the object from the image data acquired by the RGB camera 10 (step S110). The pixel position information of the object detected by pixel position information detection unit 30 is output to a processor (three-dimensional coordinate data acquisition unit, data generation unit) 100A, which will be described later.

[0061] In step S110, the processor 100A acquires three-dimensional coordinate data of the object based on the pixel position information of the detected object, the point cloud data acquired by the depth camera 20, and three-dimensional spatial information around the object (step S120). The three-dimensional coordinate data acquired by the processor 100A is sent to a control unit 130A, which will be described later, via a bus line BL.

[0062] The processor 100A generates three-dimensional space information of the space around the object based on the three-dimensional coordinate data acquired in step S120 (step S130). Then, processor 100A sends the generated three-dimensional space information of the space around the object to display unit 60 and storage unit 200, and ends the process.

[0063] <Actions and Effects> As described above, the angle-of-view detection unit 70 of the data generation processing device 1A according to this embodiment detects the angles of view of the RGB camera 10 and the depth camera 20, and the surrounding three-dimensional space setting unit 80 changes the settings of the three-dimensional space around the object based on the detected angle-of-view information of the RGB camera 10 and the depth camera 20, and sends the information to the processor 100A. That is, the surrounding three-dimensional space setting unit 80 changes the setting of the three-dimensional space around the object based on the detected information on the angles of view of the RGB camera 10 and the depth camera 20. Therefore, even if the posture of a robot on which the data generation processing device 1A according to this embodiment is mounted changes, for example, it is possible to set an appropriate three-dimensional space around an object. Therefore, during operation of the data generation processing device 1A of this embodiment, three-dimensional spatial information around a recognized object, or three-dimensional spatial information utilizing the mutual positional relationships between recognized objects, can be provided at high speed.

[0064] The data generation processing devices 1, 1A of the present invention can be realized by recording the processing of the processors 100, 100A on a recording medium readable by a computer system, and having the processors 100, 100A read and execute the program recorded on this recording medium. The computer system here includes hardware such as an OS and peripheral devices.

[0065] Furthermore, if a WWW (World Wide Web) system is used, the "computer system" also includes the homepage providing environment (or display environment). The program may also be transmitted from a computer system that stores the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

[0066] The program may also be for realizing part of the above-mentioned functions. Furthermore, the above-mentioned functions may be realized in combination with a program already recorded in the computer system, that is, a so-called differential file (differential program).

[0067] The above has described in detail an embodiment of the present invention with reference to the drawings, but all data generation processing devices that can be implemented by a person skilled in the art by making appropriate design modifications based on the data generation processing devices 1, 1A described above as embodiments of the present invention also fall within the technical scope of the present invention as long as they include the gist of the present invention. Within the scope of the concept of the present invention, a person skilled in the art may conceive of various modifications and alterations, and it is understood that these modifications and alterations also fall within the technical scope of the present invention. For example, any embodiment in which a person skilled in the art has appropriately added or deleted components or modified the design, or added or omitted steps or modified conditions, is included within the technical scope of the present invention as long as it contains the gist of the present invention.

[0068] Furthermore, other effects and advantages brought about by the aspects described in this embodiment that are clear from the description in this specification or that can be appropriately thought of by a person skilled in the art are naturally understood to be brought about by the present invention. Various inventions can be formed by appropriately combining the multiple components disclosed in the above embodiments. For example, some components may be omitted from all the components shown in the embodiment. Furthermore, components from different embodiments may be combined as appropriate. [Explanation of symbols]

[0069] 1. Data generation processing device 1A: Data generation processing device 10;RGB camera 20;Depth camera 30: Pixel position information detection unit 50;Operation section 60;Display section 70: View angle detection unit 80: Surrounding 3D space setting section 100;processor 100A; processor 110: 3D coordinate data acquisition unit 120;Data generation section 130;Control unit 130A; Control section 200;Memory 210;Memory part

Claims

1. an RGB camera for acquiring image data; a depth camera that acquires point cloud data; a pixel position information detection unit that detects pixel position information of an object from the acquired image data; a three-dimensional coordinate data acquisition unit that acquires three-dimensional coordinate data of the object based on pixel position information of the detected object and the acquired point cloud data; a data generating unit that generates three-dimensional space information of a space surrounding the object based on the acquired three-dimensional coordinate data; A data generating processing device comprising:

2. The three-dimensional coordinate data acquisition unit a three-dimensional coordinate data generation unit that associates pixel position information of the detected object with the acquired point cloud data to generate three-dimensional coordinate data of the object and a space surrounding the object; a three-dimensional coordinate data extraction unit that extracts three-dimensional coordinate data of a predetermined object or the three-dimensional space surrounding the object based on extraction conditions from the generated three-dimensional coordinate data of the object and the space surrounding the object; The data generating and processing device according to claim 1 , comprising:

3. The data generation processing device according to claim 2 , wherein the extraction conditions are variable based on the actual dimensions of the object.

4. 4. The data generation processing device according to claim 2, wherein when there are a plurality of the objects, the three-dimensional coordinate data extraction unit extracts three-dimensional coordinate data of a three-dimensional space around the object based on a positional relationship between the plurality of the objects.

5. The data generation processing device according to claim 4 , wherein the positional relationship is distance information between the predetermined object and a plurality of the objects.

6. a field of view detection unit that detects the field of view of the RGB camera and the depth camera; a surrounding three-dimensional space setting unit that changes the setting of the three-dimensional space around the object based on the detected angle of view information of the RGB camera and the depth camera; The data generating and processing device according to claim 2 , comprising:

7. A data generation processing method in a data generation processing device including an RGB camera that acquires image data, a depth camera that acquires point cloud data, a pixel position information detection unit, a three-dimensional coordinate data acquisition unit, and a data generation unit, a first step in which the pixel position information detection unit detects pixel position information of an object from image data acquired by the RGB camera; a second step in which the three-dimensional coordinate data acquisition unit acquires three-dimensional coordinate data of the object based on pixel position information of the object detected in the first step and the point cloud data acquired by the depth camera; a third step in which the data generation unit generates three-dimensional space information of a peripheral space of the object based on the three-dimensional coordinate data acquired in the second step; A data generation processing method including:

8. a fourth step in which the three-dimensional coordinate data acquisition unit associates pixel position information of the object detected in the second step with the acquired point cloud data to generate three-dimensional coordinate data of the object and a space surrounding the object; a fifth step of extracting three-dimensional coordinate data of a predetermined object or a three-dimensional space surrounding the object based on extraction conditions from the generated three-dimensional coordinate data of the object and the space surrounding the object; The data generation and processing method according to claim 7, comprising:

9. A program for causing a computer to execute a data generation processing method in a data generation processing device including an RGB camera that acquires image data, a depth camera that acquires point cloud data, a pixel position information detection unit, a three-dimensional coordinate data acquisition unit, and a data generation unit, a first step in which the pixel position information detection unit detects pixel position information of an object from image data acquired by the RGB camera; a second step in which the three-dimensional coordinate data acquisition unit acquires three-dimensional coordinate data of the object based on pixel position information of the object detected in the first step and the point cloud data acquired by the depth camera; a third step in which the data generation unit generates three-dimensional space information of a peripheral space of the object based on the three-dimensional coordinate data acquired in the second step; A program that causes a computer to execute the following.

10. a fourth step in which the three-dimensional coordinate data acquisition unit associates pixel position information of the object detected in the second step with the acquired point cloud data to generate three-dimensional coordinate data of the object and a space surrounding the object; a fifth step of extracting three-dimensional coordinate data of a predetermined object or a three-dimensional space surrounding the object based on extraction conditions from the generated three-dimensional coordinate data of the object and the space surrounding the object; of, The program according to claim 9,

Images