Object recognition device, object recognition method, and object recognition system

Abstract

The apparatus includes a data acquisition unit (31) that acquires point cloud data, a three-dimensional voxel grid data generation unit (32a) that generates three-dimensional voxel grid data based on the point cloud data acquired by the data acquisition unit (31), a recognition target area detection unit (32b) that detects, as a recognition target area, an area where an object exists among areas consisting of a plurality of grids represented by the three-dimensional voxel grid data, based on the three-dimensional voxel grid data generated by the three-dimensional voxel grid data generation unit (32a), a pseudo three-dimensional data generation unit (32c) that generates pseudo three-dimensional data based on the data of the recognition target area detected by the recognition target area detection unit (32b), and an object recognition unit (33) that inputs the pseudo three-dimensional data generated by the pseudo three-dimensional data generation unit (32c) into a two-dimensional convolutional neural network to obtain data related to the object, thereby recognizing the object.

Description

[Technical Field]

[0001] The present disclosure relates to an object recognition device, an object recognition method, and an object recognition system that use a three-dimensional sensor. [Background technology]

[0002] Object recognition technology using a three-dimensional sensor is known. One example of an object recognition method using a 3D sensor is to generate 3D information by mapping 3D point cloud data obtained from the 3D sensor onto a 3D space of a predetermined size, and then input the 3D information into a 3D convolutional neural network to recognize the object. However, object recognition using a 3D convolutional neural network that inputs 3D information requires a huge amount of calculation. For example, Patent Document 1 discloses a technology for recognizing an object by generating multiple pieces of two-dimensional information showing two-dimensional drawings of a solid object represented by the three-dimensional positions as viewed from multiple directions based on three-dimensional information including the three-dimensional positions along the outer shape of at least a portion of the object, and then performing image processing using a convolutional neural network based on the multiple pieces of two-dimensional information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2020 / 008684 Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional technology disclosed in Patent Document 1, even though the amount of calculation can be reduced compared to object recognition using a 3D convolutional neural network that uses 3D information as input, there is a problem that the object recognition accuracy may be lower than that of object recognition using the above-mentioned 3D convolutional neural network, because a 2D drawing cannot acquire information about space other than the surface, in other words, information about space in the depth direction.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an object recognition device that reduces the amount of calculation compared to object recognition using a 3D convolutional neural network that uses 3D information as input, and that can perform object recognition with higher accuracy compared to conventional object recognition techniques such as those disclosed in Patent Document 1. [Means for solving the problem]

[0006] The object recognition device according to the present disclosure includes a data acquisition unit that acquires point cloud data, a three-dimensional voxel grid data generation unit that generates three-dimensional voxel grid data based on the point cloud data acquired by the data acquisition unit, a recognition target area detection unit that detects, as a recognition target area, an area consisting of a plurality of grids represented by the three-dimensional voxel grid data, in which an object exists, based on the three-dimensional voxel grid data generated by the three-dimensional voxel grid data generation unit, a pseudo three-dimensional data generation unit that generates pseudo three-dimensional data that can be two-dimensionally convolved based on the data of the recognition target area detected by the recognition target area detection unit, and an object recognition unit that recognizes the object by inputting the pseudo three-dimensional data generated by the pseudo three-dimensional data generation unit into a two-dimensional convolutional neural network to obtain data related to the object. [Effects of the Invention]

[0007] According to the present disclosure, the object recognition device reduces the amount of calculation compared to object recognition using a 3D convolutional neural network that uses 3D information as input, and can perform object recognition with higher accuracy compared to conventional object recognition techniques such as those disclosed in Patent Document 1. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of an object recognition system according to a first embodiment. [Figure 2] 1 is a diagram illustrating an example of the configuration of an object recognition device according to a first embodiment. [Figure 3] FIG. 2 is a diagram schematically showing an example of three-dimensional voxel grid data generated by a three-dimensional voxel grid data generating unit in the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a process in which the pseudo three-dimensional data generating unit generates pseudo three-dimensional data from recognition target area data in the first embodiment. [Figure 5] FIG. 2 is a diagram for explaining an example of a development image generated by a development generating unit based on recognition target area data in the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of object recognition processing performed by an object recognition unit using a two-dimensional convolutional neural network in the first embodiment. [Figure 7] 4 is a flowchart for explaining the operation of the object recognition device according to the first embodiment. [Figure 8] 8A and 8B are diagrams illustrating an example of a hardware configuration of the object recognition device according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] In the present disclosure, an object recognition device acquires data (hereinafter referred to as "sensing data") obtained by sensing an area targeted by the three-dimensional sensor (hereinafter referred to as "target area") from the three-dimensional sensor, and recognizes objects present in the target area based on the acquired sensing data. In the present disclosure, the three-dimensional sensor is assumed to be, for example, a LiDAR (Light Detection And Ranging) or millimeter wave sensor. The object recognition device outputs the results of recognizing an object (hereinafter referred to as "object recognition results") to various devices. For example, the target area is the area ahead of a traveling autonomous vehicle, and the equipment is an autonomous driving control equipment that controls the autonomous driving of the autonomous vehicle. In this case, the object recognition device recognizes objects present in the area ahead of the autonomous vehicle based on sensing data acquired from a three-dimensional sensor, and outputs the object recognition results to the autonomous driving control equipment. Based on the object recognition results, the autonomous driving control equipment performs autonomous driving control of the autonomous vehicle, such as avoiding objects present ahead. This is just one example, and the object recognition device can be applied to any situation where an object is recognized based on sensing data acquired from a three-dimensional sensor. Furthermore, the object recognition results obtained by the object recognition device can be used in an appropriate manner in various devices.

[0010] Embodiment 1 FIG. 1 is a diagram illustrating an example of the configuration of an object recognition system 1 according to the first embodiment. The object recognition device 3 is connected to the three-dimensional sensor 2 and the device 4 via a network, and the object recognition device 3, the three-dimensional sensor 2 and the device 4 constitute an object recognition system 1. For example, if the target area is the area ahead of an autonomous vehicle (not shown) and the device 4 is an autonomous driving control device, the object recognition device 3, the three-dimensional sensor 2, and the device 4 are mounted on, for example, the autonomous driving vehicle.

[0011] The three-dimensional sensor 2 senses the target area and outputs point cloud data within the target area. More specifically, the 3D sensor 2 acquires reflected light or waves that are the result of light or radio waves emitted toward the target area being reflected by objects within the target area. As a result, the 3D sensor 2 outputs point cloud data. Specifically, the 3D sensor 2 measures the distance to an object, its position, shape, or direction based on the time difference between the reflected light or reflected waves hitting an object within the target area and bouncing back. The three-dimensional sensor 2 generates sensing data based on the measured distance and the like. The sensing data is point cloud data, where each point contains distance data expressed as three-dimensional coordinates (x, y, z) and intensity data that represents the reflectance of the object's surface.

[0012] The method by which the three-dimensional sensor 2 generates point cloud data is known, but here, an example of the method by which the three-dimensional sensor 2 generates point cloud data will be described. Here, as an example, the three-dimensional sensor 2 will be described as being a LiDAR. The three-dimensional sensor 2 sequentially irradiates the target area with light in the x and y directions. For example, the light is irradiated at 6 points in the x direction and 2 points in the y direction of the target area, for a total of 12 points. Note that, as viewed from the three-dimensional sensor 2, the x direction is horizontal to the target area, and the y direction is upward to the target area. Light emitted sequentially from the 3D sensor 2 in the x and y directions of the target area is irradiated and reflected by objects within the target area to form reflected light. In the example above, the 3D sensor 2 has a resolution of 6 pixels in the x direction and 2 pixels in the y direction, for a total of 12 pixels. The three-dimensional sensor 2 receives the reflected light and calculates the distance to the object based on the time difference between light emission and reception, and obtains the distance data. In the above example, the three-dimensional sensor 2 calculates the distance individually for all of the irradiation positions distributed across a total of 12 points, six points in the x direction of the target area and two points in the y direction of the target area, and obtains the distance data. Furthermore, the three-dimensional sensor 2 calculates the reflectance at each point of the reflective object based on the ratio between the amount of light emitted and the amount of light received for all of the irradiation positions, and obtains the intensity data. The distance data and intensity data calculated by the three-dimensional sensor 2 are called point cloud data. The point cloud data is output to the object recognition device 3. The point cloud data is output to the object recognition device 3 on a frame-by-frame basis. The three-dimensional sensor 2 outputs the point cloud data obtained by sensing the entire target area once, that is, in the above example, the point cloud data obtained by sensing a total of 12 points in the target area (6 points in the x direction and 2 points in the y direction), as one frame's worth of point cloud data, in other words, as sensing data, to the object recognition device 3. The three-dimensional sensor 2 irradiates light in a dispersed manner in the horizontal and vertical directions at the same angle at all times. In this way, the three-dimensional sensor 2 obtains the following three-dimensional information of x, y, and z: The three-dimensional sensor 2 also obtains intensity data (i) at each point. x;horizontal coordinate y: vertical coordinate z; distance data i;Intensity data The horizontal coordinate x indicates the horizontal position of each pixel, and the vertical coordinate y indicates the vertical position of each pixel. The distance data z is depth information in the z-axis direction obtained at each pixel's position. Distance data z is obtained by measuring each of the pixels identified by all combinations of the horizontal coordinate x and the vertical coordinate y. Therefore, in the three-dimensional information, each of a plurality of combinations of horizontal coordinate x, vertical coordinate y, and distance data z indicates a specific spatial position.

[0013] The object recognition device 3 recognizes objects by processing using a two-dimensional convolutional neural network based on sensing data acquired from the three-dimensional sensor 2, in other words, point cloud data. More specifically, the object recognition device 3 recognizes objects by inputting data generated based on the point cloud data (hereinafter referred to as "object recognition data") into the two-dimensional convolutional neural network and obtaining data related to the object output from the two-dimensional convolutional neural network. The object recognition device 3 recognizes the type of object by processing using the two-dimensional convolutional neural network. The object recognition device 3 can also recognize the position of the object from the point cloud data. The object recognition device 3 outputs the object recognition result to the device 4. An example of the configuration of the object recognition device 3 will be described in detail later.

[0014] The device 4 performs various controls based on the object recognition result output from the object recognition device 3.

[0015] An example of the configuration of the object recognition device 3 according to the first embodiment will be described. FIG. 2 is a diagram illustrating an example of the configuration of the object recognition device 3 according to the first embodiment. The object recognition device 3 includes a data acquisition unit 31, a data generation unit 32, an object recognition unit 33, and a recognition result output unit . The data generation unit 32 includes a three-dimensional voxel grid data generation unit 32a, a recognition target region detection unit 32b, a pseudo three-dimensional data generation unit 32c, and a development drawing generation unit 32d.

[0016] The data acquisition unit 31 acquires the sensing data output from the three-dimensional sensor 2, in other words, point cloud data. The data acquisition unit 31 outputs the acquired point cloud data to the data generation unit 32.

[0017] The data generation unit 32 generates object recognition data from the point cloud data output from the data acquisition unit 31. In the first embodiment, the object recognition data includes pseudo three-dimensional data and development view data. Details of the pseudo three-dimensional data and development view data will be described later. The data generation unit 32 outputs the generated object recognition data to the object recognition unit 33.

[0018] The data generating unit 32 will now be described in detail. The three-dimensional voxel grid data generation unit 32a of the data generation unit 32 generates three-dimensional voxel grid data based on the point cloud data acquired by the data acquisition unit 31. Specifically, the three-dimensional voxel grid data is data represented by a three-dimensional model composed of a plurality of grids of a predetermined size W×H×Z.

[0019] FIG. 3 is a diagram schematically showing an example of the three-dimensional voxel grid data generated by the three-dimensional voxel grid data generating unit 32a in the first embodiment. 3, the three-dimensional voxel grid data is represented by a three-dimensional model divided into multiple grids (with a predetermined size of W×H×Z). For example, the three-dimensional voxel grid data is represented by a three-dimensional model in which the field of view of the three-dimensional sensor 2 is virtually divided into multiple grids. In FIG. 3, as an example, the three-dimensional voxel grid data is assumed to be composed of 15×15×8 grids. In addition, in Figure 3, the 3D model shows how objects existing within the target region appear within a 15 x 15 x 8 grid. Grids where objects exist are shown in gray. That is, the gray grids are grids for which distances were obtained by the 3D sensor 2. Grids for which distances were not obtained by the 3D sensor 2, i.e., the white grids in the example of Figure 3, are also called blank grids. In the 3D voxel grid data, gray grids contain values, but blank grids do not contain values.

[0020] The three-dimensional voxel grid data generating unit 32a outputs the generated three-dimensional voxel grid data to the recognition target region detecting unit 32b.

[0021] Based on the three-dimensional voxel grid data output from the three-dimensional voxel grid data generation unit 32a, the recognition target region detection unit 32b detects, as a recognition target region, a region in which an object exists among a region consisting of a plurality of grids represented by the three-dimensional voxel grid data. Note that the recognition target region detection unit 32b detects a recognition target region for each object. The recognition target area detection unit 32b may detect the recognition target area using a known technique. For example, the recognition target area detection unit 32b detects the outline of the grid area containing values ​​from the differences between grids, more specifically, the differences in values ​​contained in the grids, and extracts the grid surrounded by the detected outline. The recognition target area detection unit 32b then detects the circumscribed cubic rectangle of the outline of the extracted grid area as the recognition target area. Alternatively, for example, the recognition target area detection unit 32b may detect a cubic rectangle obtained by adding a margin to the circumscribed cubic rectangle of the extracted grid outline as the recognition target area. The recognition-target-area detecting unit 32b outputs data indicating the detected recognition-target area (hereinafter referred to as "recognition-target-area data") to the pseudo three-dimensional data generating unit 32c and the development drawing generating unit 32d.

[0022] The pseudo 3D data generation unit 32c generates pseudo 3D data that can be subjected to 2D convolution based on the recognition target area data output from the recognition target area detection unit 32b. More specifically, the pseudo 3D data generation unit 32c acquires the recognition target area data output from the recognition target area detection unit 32b, slices the recognition target area in three directions (x direction, y direction, and z direction), and regards a specific axis as a channel to generate data to which 2D convolution can be applied. In the first embodiment, the data to which 2D convolution can be applied that is generated from the recognition target area by the pseudo 3D data generation unit 32c is referred to as "slice data." Slice data is one channel, in other words, one piece of 2D data. One channel corresponds to a specific position on a specific axis (x axis, y axis, or z axis) in the recognition target area. In other words, one channel (one piece of 2D data) represents a cross section on a specific axis at that position in the recognition target area, which is the original 3D data. The specific position indicates the location of the cross section within the recognition target area, which is the original 3D data. For example, if the specific axis is the x-axis, the features such as the shape of the object at a specific position on the x-axis are expressed as a slice on the yz plane. The pseudo 3D data generator 32c generates multiple channels (multiple 2D data) at multiple positions on the specific axis within the recognition target area as multiple slice data. A 2D convolutional neural network can capture multiple cross-sectional data at different axial positions by using multiple slice data, and can capture 3D data information by processing these multiple slice data simultaneously. The pseudo three-dimensional data generating unit 32c generates a plurality of slice data for each of the three axes (x-axis, y-axis, and z-axis).

[0023] The pseudo three-dimensional data generator 32c then generates pseudo three-dimensional data by combining the plurality of slice data for each of the three axes. The pseudo three-dimensional data generator 32c may generate the pseudo three-dimensional data by applying a known combining method such as Contact, Mul, or Add. Like slice data, pseudo three-dimensional data obtained by combining multiple slice data is also data to which two-dimensional convolution can be applied.

[0024] FIG. 4 is a diagram illustrating an example of a process in which pseudo three-dimensional data generating unit 32c generates pseudo three-dimensional data from recognition target area data in the first embodiment. In Fig. 4, the three-dimensional model indicated by 401 represents the recognition target area. In Fig. 4, 402a, 402b, and 402c represent multiple slice data. In Fig. 4, 403 represents pseudo three-dimensional data. Note that in Fig. 4, the grid in the height direction of the recognition target area is indicated by "H", the grid in the width direction by "W", and the grid in the depth direction by "D". Also, in Fig. 4, the grid in the axis regarded as a channel in the slice data is indicated by "C". For convenience, in the recognition target area, grids where objects exist and blank grids are both shown as white grids.

[0025] First, the pseudo three-dimensional data generating unit 32c regards the cross section on the yz plane obtained by slicing the recognition target area (see 401 in FIG. 4) detected by the recognition target area detecting unit 32b at a specific position in the x-axis direction as one channel, and generates a plurality of slice data (hereinafter referred to as "x-axis slice data") expressed as two-dimensional data (see 402a in FIG. 4). Each image represents slice data at a specific position on the x-axis. Furthermore, the pseudo three-dimensional data generating unit 32c regards the cross section on the xz plane obtained by slicing the recognition target area (see 401 in FIG. 4) detected by the recognition target area detecting unit 32b at a specific position in the y-axis direction as one channel, and generates slice data (hereinafter referred to as "y-axis slice data") expressed as a two-dimensional image (see 402b in FIG. 4). Each image represents slice data at a specific position on the y-axis. Furthermore, the pseudo three-dimensional data generating unit 32c regards the cross section on the xy plane obtained by slicing the recognition target area (see 401 in FIG. 4) detected by the recognition target area detecting unit 32b at a specific position in the z-axis direction as one channel, and generates slice data (hereinafter referred to as "z-axis slice data") expressed as a two-dimensional image (see 402c in FIG. 4). Each image represents slice data at a specific position on the z-axis. Then, the pseudo three-dimensional data generating unit 32c generates pseudo three-dimensional data by combining the generated multiple x-axis slice data, multiple y-axis slice data, and multiple z-axis slice data (see 403 in FIG. 4). The pseudo three-dimensional data generating unit 32c may generate the x-axis slice data, the y-axis slice data, and the z-axis slice data in any order. 4, the pseudo three-dimensional data generator 32c generates 13 pieces of x-axis slice data, 13 pieces of y-axis slice data, and 5 pieces of z-axis slice data, but this is just an example. The pseudo three-dimensional data generator 32c can generate any number of slice data in the x-axis direction, the y-axis direction, and the z-axis direction.

[0026] The pseudo three-dimensional data generating unit 32c outputs the generated pseudo three-dimensional data to the object recognizing unit 33.

[0027] The development generation unit 32d generates, as a development image, data including a development indicating the reflectance of the surface of a grid area in which an object exists, based on the recognition target area data output from the recognition target area detection unit 32b. More specifically, the development generation unit 32d acquires the recognition target area data output from the recognition target area detection unit 32b, and views the recognition target area from a specific direction, and generates, as a development image, two-dimensional data including a development obtained by developing the grids on the surface of the grid area indicative of an object included in the recognition target area. The point cloud data includes intensity data, and in the 3D voxel grid data, each grid is assigned intensity data. In the development view, each square is assigned the intensity data, in other words, data indicating reflectance.

[0028] FIG. 5 is a diagram for explaining an example of a development image generated by the development generating unit 32d based on the recognition target area data in the first embodiment. In Fig. 5, the three-dimensional model 501a indicates the recognition target area. Note that in Fig. 5, the recognition target area detection unit 32b detects, as the recognition target area, a cubic rectangle that is a cubic rectangle with a margin added to the circumscribed cubic rectangle of the outline of the grid extracted as the area where the object exists. In FIG. 5, 502a shows a development image including a development (shown as 502b in FIG. 5) in which only the surface data of an area consisting of a grid (shown as 501b in FIG. 5) indicating an object included in the recognition target area shown in 501a is developed when the recognition target area is viewed from the direction indicated by the arrow. In Figure 5, the development image is an image in which data corresponding to the surface of the blank grid around the development is provided, but this is merely an example. The development image may be an image that includes at least the development. The part corresponding to the surface of the blank grid around the development, represented by the white grid in Figure 5, may be clearly a grid with no value, and a uniform value such as "0" or "-1" may be assigned to this part.

[0029] The development drawing generation unit 32d outputs the generated development drawing image to the object recognition unit 33.

[0030] The intensity data included in the sensing data acquired from the three-dimensional sensor 2 represents reflectance. The intensity data varies depending on the material of the object's surface, and contains many of the object's characteristics. By having the development image generation unit 32d generate a development image, the object recognition device 3 can perform object recognition by referring to the intensity data of the entire surface of the object, rather than just from one direction; in other words, by taking the object's characteristics into greater consideration. In the object recognition device 3, object recognition is performed by the object recognition unit 33.

[0031] The object recognition unit 33 recognizes an object by inputting the object recognition data generated by the data generation unit 32 into a two-dimensional convolutional neural network to obtain data related to the object. Specifically, the object recognition unit 33 recognizes an object by inputting the pseudo three-dimensional data generated by the pseudo three-dimensional data generation unit 32c and the development image generated by the development generation unit 32d into two-dimensional convolutional neural networks to obtain data related to the object. More specifically, the object recognition unit 33 inputs the pseudo 3D data and the development image to a 2D convolutional neural network. When the pseudo 3D data and the development image are input, the 2D convolutional neural network outputs data related to the object. Here, the data related to the object output by the 2D convolutional neural network is data in which the type of object is associated with the probability that the object is considered to be that type. Based on the data related to the object, the object recognition unit 33 determines, for example, the type with the highest association probability as the type of object. In this way, the object recognition unit 33 recognizes the object.

[0032] An example of an object recognition method performed by the object recognition unit 33 will now be described in detail. FIG. 6 is a diagram showing an example of the object recognition process performed by the object recognition unit 33 using a two-dimensional convolutional neural network in the first embodiment. First, the object recognition unit 33 inputs the pseudo three-dimensional data output from the pseudo three-dimensional data generation unit 32c into a two-dimensional convolutional neural network, and obtains a feature amount (hereinafter referred to as a "first feature amount") output from the two-dimensional convolutional neural network (see 601 in FIG. 6). The object recognition unit 33 also inputs the development image output from the development generation unit 32d into the two-dimensional convolutional neural network, and obtains a feature amount (hereinafter referred to as a "second feature amount") output from the two-dimensional convolutional neural network (see 602 in FIG. 6). The order in which the object recognition unit 33 obtains the first feature amount and the second feature amount does not matter. Next, the object recognition unit 33 combines the obtained first feature amount and second feature amount. Then, the object recognition unit 33 inputs the combined feature amount (hereinafter referred to as "combined feature amount") back into the two-dimensional convolutional neural network to obtain data related to the object. Then, based on the data related to the object, the object recognition unit 33 determines, for example, the type that is most likely to be associated with the data related to the object as the type of the object (see 603 in FIG. 6).

[0033] The two-dimensional convolutional neural network has been trained in advance using training data with correct answer labels. The trained two-dimensional convolutional neural network is stored in a location that can be referenced by the object recognition unit 33. The learning data is an existing data set that has been generated in advance by an administrator or the like.

[0034] The object recognition unit 33 generates an object recognition result indicating the type of object. The object recognition result may include data indicating the position of the object. The position of the object can be determined from point cloud data. The object recognition unit 33 may acquire the point cloud data from the data acquisition unit 31 via the data generation unit 32. The object recognition unit 33 outputs the object recognition result to the recognition result output unit .

[0035] The recognition result output unit 34 outputs the object recognition result output from the object recognition unit 33 to the device 4.

[0036] The operation of the object recognition device 3 according to the first embodiment will be described. FIG. 7 is a flowchart for explaining the operation of the object recognition device 3 according to the first embodiment. For example, when the object recognition device 3 is powered on, the object recognition device 3 starts the operation shown in the flowchart of FIG. 7, and repeats the operation shown in the flowchart of FIG. 7 until the power is turned off.

[0037] The data acquisition unit 31 acquires the sensing data output from the three-dimensional sensor 2, in other words, point cloud data (step ST1). The data acquisition unit 31 outputs the acquired point cloud data to the data generation unit 32.

[0038] The three-dimensional voxel grid data generating unit 32a generates three-dimensional voxel grid data based on the point cloud data acquired by the data acquiring unit 31 in step ST1 (step ST2). The three-dimensional voxel grid data generating unit 32a outputs the generated three-dimensional voxel grid data to the recognition target region detecting unit 32b.

[0039] Based on the three-dimensional voxel grid data output from the three-dimensional voxel grid data generation unit 32a in step ST2, the recognition target area detection unit 32b detects an area where an object exists as a recognition target area from among the areas consisting of multiple grids represented by the three-dimensional voxel grid data (step ST3). The recognition target area detection unit 32b outputs the recognition target area data to the pseudo three-dimensional data generation unit 32c and the development drawing generation unit 32d.

[0040] The pseudo three-dimensional data generating unit 32c generates pseudo three-dimensional data that can be two-dimensionally convoluted based on the recognition target region data output from the recognition target region detecting unit 32b in step ST3 (step ST4a). The pseudo three-dimensional data generating unit 32c outputs the generated pseudo three-dimensional data to the object recognizing unit 33.

[0041] The development drawing generating unit 32d generates a development drawing image based on the recognition target area data output from the recognition target area detecting unit 32b in step ST3 (step ST4b). The development drawing generation unit 32d outputs the generated development drawing image to the object recognition unit 33.

[0042] Based on the pseudo 3D data output from the pseudo 3D data generation unit 32c in step ST4a and the development image output from the development image generation unit 32d in step ST4b, the object recognition unit 33 inputs the pseudo 3D data and the development image into a 2D convolutional neural network, respectively, to obtain data related to the object and thereby recognize the object (step ST5). The object recognition unit 33 outputs the object recognition result to the recognition result output unit .

[0043] The recognition result output unit 34 outputs the object recognition result output from the object recognition unit 33 in step ST5 to the device 4 (step ST6).

[0044] 7, the processing of step ST4a and the processing of step ST4b are performed in parallel, but this is merely an example. For example, the processing may be performed in the order of step ST3, step ST4a, step ST4b, and step ST5, or in the order of step ST3, step ST4b, step ST4a, and step ST5.

[0045] In this way, the object recognition device 3 generates three-dimensional voxel grid data based on the point cloud data acquired from the three-dimensional sensor 2, and detects a recognition target area based on the generated three-dimensional voxel grid data. The object recognition device 3 generates pseudo three-dimensional data and a development image based on the recognition target area data, and inputs the generated pseudo three-dimensional data and development image into a two-dimensional convolutional neural network to obtain data related to the object, thereby recognizing the object. The object recognition device 3 generates three-dimensional voxel grid data to which two-dimensional convolution can be applied and performs object recognition using a two-dimensional convolutional neural network, thereby reducing the amount of calculation and performing object recognition faster than object recognition using a three-dimensional convolutional neural network, and enabling more accurate object recognition than conventional techniques that recognize objects by performing image processing using a convolutional neural network based on multiple two-dimensional information. Furthermore, by generating a development image and performing object recognition using a two-dimensional convolutional neural network, the object recognition device 3 can perform object recognition by referring to intensity data on the entire surface of the object, rather than just from one direction, in other words, by taking the object's features into greater consideration. As a result, the object recognition device 3 can further improve the object recognition accuracy.

[0046] In the above-described first embodiment, the object recognition device 3 is provided with the development drawing generation unit 32d, but this is merely an example, and the object recognition device 3 does not necessarily have to include the development drawing generation unit 32d. When the object recognition device 3 is configured without the development view generating unit 32d, the process of step ST4b in the operation of the object recognition device 3 shown in the flowchart of FIG. 7 can be omitted.

[0047] Furthermore, in the above-described first embodiment, the object recognition device 3 acquires point cloud data generated by the three-dimensional sensor 2 from the three-dimensional sensor 2, but this is merely an example. The source of point cloud data is not limited to the three-dimensional sensor 2, and the object recognition device 3 may, for example, acquire point cloud data that has been generated in advance and stored in a storage device (not shown) that the object recognition device 3 can refer to from the storage device, or the object recognition device 3 may be connected to a simulator (not shown), and the object recognition device 3 may acquire point cloud data from the simulator that has been generated by the simulator running a simulation.

[0048] In the first embodiment, the object recognition device 3 is mounted on, for example, an autonomous driving vehicle, but this is merely an example. The object recognition device 3 may be mounted on, for example, a server, or some of the data acquisition unit 31, the 3D voxel grid data generation unit 32a, the recognition target area detection unit 32b, the pseudo 3D data generation unit 32c, the development view generation unit 32d, the object recognition unit 33, and the recognition result output unit 34 may be provided in the in-vehicle device, and the rest may be provided in the server.

[0049] In the first embodiment, the function of the recognition result output unit 34 may be included in the object recognition unit 33. In this case, the object recognition device 3 may be configured without the recognition result output unit 34.

[0050] In the first embodiment, for example, the object recognition device 3 may store the object recognition results, and the object recognition system 1 may be configured by the three-dimensional sensor 2 and the object recognition device 3.

[0051] 8A and 8B are diagrams illustrating an example of a hardware configuration of the object recognition device 3 according to the first embodiment. In the first embodiment, the functions of the data acquisition unit 31, the 3D voxel grid data generation unit 32a, the recognition target area detection unit 32b, the pseudo 3D data generation unit 32c, the development drawing generation unit 32d, the object recognition unit 33, and the recognition result output unit 34 are realized by the processing circuit 101. That is, the object recognition device 3 includes the processing circuit 101 for generating pseudo 3D data to which 2D convolution can be applied based on sensor data, in other words, point cloud data, and for controlling object recognition using the pseudo 3D data and a 2D convolutional neural network. The processing circuit 101 may be dedicated hardware as shown in FIG. 8A, or may be a processor 104 that executes a program stored in memory as shown in FIG. 8B.

[0052] When the processing circuitry 101 is dedicated hardware, the processing circuitry 101 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0053] When the processing circuit is a processor 104, the functions of the data acquisition unit 31, the 3D voxel grid data generation unit 32a, the recognition target area detection unit 32b, the pseudo 3D data generation unit 32c, the development view generation unit 32d, the object recognition unit 33, and the recognition result output unit 34 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 105. The processor 104 reads and executes the program stored in the memory 105 to execute the functions of the data acquisition unit 31, the 3D voxel grid data generation unit 32a, the recognition target area detection unit 32b, the pseudo 3D data generation unit 32c, the development view generation unit 32d, the object recognition unit 33, and the recognition result output unit 34. That is, the object recognition device 3 includes a memory 105 for storing a program that, when executed by the processor 104, results in the execution of steps ST1 to ST6 of FIG. 7 described above. It can also be said that the program stored in memory 105 causes the computer to execute the processing procedures or methods of data acquisition unit 31, 3D voxel grid data generation unit 32a, recognition target area detection unit 32b, pseudo 3D data generation unit 32c, development view generation unit 32d, object recognition unit 33, and recognition result output unit 34. Here, memory 105 corresponds to, for example, non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc), etc.

[0054] It is also possible to realize some of the functions of data acquisition unit 31, 3D voxel grid data generation unit 32a, recognition target area detection unit 32b, pseudo 3D data generation unit 32c, development view generation unit 32d, object recognition unit 33, and recognition result output unit 34 with dedicated hardware and some with software or firmware. For example, the functions of data acquisition unit 31 and recognition result output unit 34 can be realized by processing circuit 101 as dedicated hardware, and the functions of 3D voxel grid data generation unit 32a, recognition target area detection unit 32b, pseudo 3D data generation unit 32c, development view generation unit 32d, and object recognition unit 33 can be realized by processor 104 reading and executing programs stored in memory 105. The object recognition device 3 also includes an input interface device 102 and an output interface device 103 that perform wired or wireless communication with devices such as the three-dimensional sensor 2 or the equipment 4.

[0055] As described above, according to the first embodiment, the object recognition device 3 is configured to include a data acquisition unit 31 that acquires point cloud data (sensor data), a 3D voxel grid data generation unit 32a that generates 3D voxel grid data based on the point cloud data acquired by the data acquisition unit 31, a recognition target region detection unit 32b that detects, as a recognition target region, a region consisting of a plurality of grids represented by the 3D voxel grid data, in which an object exists, based on the 3D voxel grid data generated by the 3D voxel grid data generation unit 32a, a pseudo 3D data generation unit 32c that generates pseudo 3D data that can be 2D convolved based on the data of the recognition target region detected by the recognition target region detection unit 32b, and an object recognition unit 33 that recognizes the object by inputting the pseudo 3D data generated by the pseudo 3D data generation unit 32c into a 2D convolutional neural network to obtain data related to the object. Therefore, the object recognition device 3 can reduce the amount of calculations and perform object recognition faster than object recognition using a 3D convolutional neural network that uses 3D information as input, and can perform object recognition with higher accuracy than conventional technology that recognizes objects by performing image processing using a convolutional neural network based on multiple pieces of 2D information.

[0056] In addition to the above configuration, the object recognition device 3 also includes a development image generator 32d that generates a development image including a development showing the surface reflectance of a grid-like area where an object exists, based on the data of the recognition target area detected by the recognition target area detector 32b. The object recognition unit 33 can recognize an object by inputting the pseudo 3D data generated by the pseudo 3D data generator 32c and the development image generated by the development image generator 32d into a 2D convolutional neural network to obtain data related to the object. This allows the object recognition device 3 to recognize objects by referring to intensity data of the entire surface of the object, rather than just from one direction. In other words, it can perform object recognition that takes into consideration the characteristics of the object. As a result, the object recognition device 3 can further improve the object recognition accuracy.

[0057] Any of the components of the embodiments may be modified or omitted. [Industrial Applicability]

[0058] The object recognition device disclosed herein can reduce the amount of calculations and perform object recognition faster than object recognition using a 3D convolutional neural network, and can perform object recognition with higher accuracy than technology that recognizes objects by performing image processing using a convolutional neural network based on multiple pieces of 2D information. [Explanation of symbols]

[0059] 1 Object recognition system, 2 3D sensor, 3 Object recognition device, 31 Data acquisition unit, 32 Data generation unit, 32a 3D voxel grid data generation unit, 32b Recognition target area detection unit, 32c Pseudo 3D data generation unit, 32d Development drawing generation unit, 33 Object recognition unit, 34 Recognition result output unit, 4 Equipment, 101 Processing circuit, 102 Input interface device, 103 Output interface device, 104 Processor, 105 Memory.

Claims

1. A data acquisition unit that acquires point cloud data, A 3D voxel grid data generation unit generates 3D voxel grid data based on the point cloud data acquired by the data acquisition unit, A recognition target area detection unit detects, based on the three-dimensional voxel grid data generated by the three-dimensional voxel grid data, the region in which an object exists among the regions consisting of multiple grids represented by the three-dimensional voxel grid data as a recognition target area, A pseudo-3D data generation unit generates pseudo-3D data that can be 2D convolved based on the data of the recognition target region detected by the recognition target region detection unit, The object recognition unit recognizes the object by inputting the pseudo-3D data generated by the pseudo-3D data generation unit into a 2D convolutional neural network to obtain data about the object. An object recognition device equipped with the following.

2. The system includes a development diagram generation unit that generates a development diagram image, which includes a development diagram showing the reflectance of the surface of the grid region where the object is located, based on the data of the recognition target region detected by the recognition target region detection unit. The object recognition unit recognizes the object by inputting the pseudo-3D data generated by the pseudo-3D data generation unit and the unfolded image generated by the unfolded image generation unit into the 2D convolutional neural network to obtain data about the object. The object recognition device according to claim 1, characterized in that it is a feature of the present invention.

3. The pseudo-3D data generation unit generates the pseudo-3D data by slicing the data of the recognition target region in three directions and treating a specific axis as a channel, and then synthesizing multiple slice data that represent cross-sections within the recognition target region that show the characteristics of the object at a specific position on the axis. The object recognition device according to claim 1 or 2, characterized in that it is a feature of the present invention.

4. The unfolded diagram generation unit generates two-dimensional data as the unfolded diagram image, which includes an unfolded diagram of the grid on the surface of the region consisting of the grid where the object in the recognition target region is located, viewed from a certain direction. The object recognition device according to claim 2.

5. The data acquisition unit performs the step of acquiring point cloud data, The 3D voxel grid data generation unit generates 3D voxel grid data based on the point cloud data acquired by the data acquisition unit, The recognition target area detection unit detects, based on the three-dimensional voxel grid data generated by the three-dimensional voxel grid data generation unit, the area in which an object exists among the areas consisting of multiple grids represented by the three-dimensional voxel grid data, as the recognition target area. The pseudo-3D data generation unit generates pseudo-3D data that can be 2D convolved based on the data of the recognition target region detected by the recognition target region detection unit, The object recognition unit recognizes the object by inputting the pseudo-3D data generated by the pseudo-3D data generation unit into a 2D convolutional neural network to obtain data about the object. An object recognition method equipped with the following.

6. The object recognition device according to claim 1, claim 2, or claim 4, The 3D sensor that generates the aforementioned point cloud data An object recognition system equipped with [the following].

7. The aforementioned three-dimensional sensor is a LiDAR or millimeter-wave sensor. The object recognition system according to claim 6, characterized in that it is a feature of the present invention.

Images