Object recognition device, object recognition method, and object recognition system

Abstract

This invention is provided with: a data acquisition unit (31) for acquiring point cloud data; a three-dimensional voxel grid data generation unit (32a) for generating three-dimensional voxel grid data on the basis of the point cloud data acquired by the data acquisition unit (31); a recognition target region detection unit (32b) for detecting, as a recognition target region, a region in which an object is present and which is from among regions formed of a plurality of grids represented by the three-dimensional voxel grid data, such detection performed on the basis of the three-dimensional voxel grid data generated by the three-dimensional voxel grid data generation unit (32a); a pseudo three-dimensional data generation unit (32c) for generating pseudo three-dimensional data on the basis of data of the recognition target region detected by the recognition target region detection unit (32b); and an object recognition unit (33) for recognizing the object by inputting the pseudo three-dimensional data generated by the pseudo three-dimensional data generation unit (32c) to a two-dimensional convolutional neural network and obtaining data pertaining to the object.

Description

Object recognition device, object recognition method, and object recognition system

[0001] The present disclosure relates to an object recognition device, an object recognition method, and an object recognition system that use a three-dimensional sensor.

[0002] Object recognition technologies using three-dimensional sensors are known. For example, one method for object recognition using a three-dimensional sensor involves mapping three-dimensional point cloud data obtained from the three-dimensional sensor onto a three-dimensional space of a predetermined size to generate three-dimensional information, and then inputting the three-dimensional information into a three-dimensional convolutional neural network to recognize an object. However, object recognition using a three-dimensional convolutional neural network that inputs three-dimensional information requires a significant amount of computation. Therefore, for example, Patent Document 1 discloses a technology for recognizing an object by generating, based on three-dimensional information including the three-dimensional position of at least a portion of the object's outline, multiple pieces of two-dimensional information representing two-dimensional drawings of a solid object represented by the three-dimensional positions viewed from multiple directions, and then performing image processing using a convolutional neural network based on the multiple pieces of two-dimensional information.

[0003] International Publication No. 2020 / 008684

[0004] In the conventional technology disclosed in Patent Document 1, even though the amount of calculation can be reduced compared to object recognition using a three-dimensional convolutional neural network that uses three-dimensional information as input, there is a problem in that, since two-dimensional drawings cannot acquire information about space other than the surface, in other words, information about space in the depth direction, the object recognition accuracy may be lower than that of object recognition using the above three-dimensional convolutional neural network.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an object recognition device that reduces the amount of calculation compared to object recognition using a three-dimensional convolutional neural network that uses three-dimensional information as input, and that can perform object recognition with higher accuracy compared to conventional object recognition techniques such as those disclosed in Patent Document 1.

[0006] The object recognition device according to the present disclosure includes a data acquisition unit that acquires point cloud data, a three-dimensional voxel grid data generation unit that generates three-dimensional voxel grid data based on the point cloud data acquired by the data acquisition unit, a recognition target region detection unit that detects, as a recognition target region, a region consisting of a plurality of grids represented by the three-dimensional voxel grid data, in which an object exists, based on the three-dimensional voxel grid data generated by the three-dimensional voxel grid data generation unit, a pseudo three-dimensional data generation unit that generates pseudo three-dimensional data that can be two-dimensionally convolved, based on the data of the recognition target region detected by the recognition target region detection unit, and an object recognition unit that recognizes the object by inputting the pseudo three-dimensional data generated by the pseudo three-dimensional data generation unit into a two-dimensional convolutional neural network to obtain data related to the object.

[0007] According to the present disclosure, the object recognition device reduces the amount of calculation compared to object recognition using a three-dimensional convolutional neural network that uses three-dimensional information as input, and can perform object recognition with higher accuracy compared to conventional object recognition techniques such as those disclosed in Patent Document 1.

[0008] 8A and 8B are diagrams illustrating an example of the configuration of an object recognition system according to embodiment 1. FIG. 8B is a diagram illustrating an example of the configuration of an object recognition device according to embodiment 1. FIG. 8A is a diagram illustrating an example of three-dimensional voxel grid data generated by a three-dimensional voxel grid data generation unit according to embodiment 1. FIG. 8B is a diagram illustrating an example of a process in which a pseudo three-dimensional data generation unit generates pseudo three-dimensional data from recognition target area data according to embodiment 1. FIG. 8B is a diagram illustrating an example of a development image generated by a development generation unit based on recognition target area data according to embodiment 1. FIG. 8A is a diagram illustrating an example of object recognition processing performed by an object recognition unit using a two-dimensional convolutional neural network according to embodiment 1. FIG. 8B is a flowchart illustrating the operation of an object recognition device according to embodiment 1. FIG. 8A and FIG. 8B are diagrams illustrating an example of the hardware configuration of an object recognition device according to embodiment 1.

[0009] In the present disclosure, an object recognition device acquires data (hereinafter referred to as "sensing data") obtained by sensing an area targeted by the three-dimensional sensor (hereinafter referred to as a "target area") from the three-dimensional sensor, and recognizes objects present in the target area based on the acquired sensing data. In the present disclosure, the three-dimensional sensor is, for example, a Light Detection and Ranging (LiDAR) or a millimeter-wave sensor. The object recognition device outputs the object recognition results (hereinafter referred to as "object recognition results") to various devices. For example, the target area is the area ahead of a traveling autonomous vehicle, and the device is an autonomous driving control device that controls the autonomous driving of the autonomous vehicle. In this case, the object recognition device recognizes objects present in the area ahead of the autonomous vehicle based on the sensing data acquired from the three-dimensional sensor, and outputs the object recognition results to the autonomous driving control device. The autonomous driving control device performs autonomous driving control of the autonomous vehicle, such as avoiding objects present ahead, based on the object recognition results. This is merely an example, and the object recognition device can be applied to any situation where an object is recognized based on sensing data acquired from a three-dimensional sensor. Furthermore, the object recognition results obtained by the object recognition device can be used in an appropriate manner in various devices.

[0010] 1 is a diagram showing an example of the configuration of an object recognition system 1 according to embodiment 1. An object recognition device 3 is connected to a three-dimensional sensor 2 and a device 4 via a network, and the object recognition device 3, the three-dimensional sensor 2, and the device 4 constitute the object recognition system 1. For example, if the target area is an area ahead of an autonomously driven vehicle (not shown) and the device 4 is an autonomous driving control device, the object recognition device 3, the three-dimensional sensor 2, and the device 4 are mounted on, for example, the autonomously driven vehicle.

[0011] The three-dimensional sensor 2 senses a target area and outputs point cloud data within the target area. More specifically, the three-dimensional sensor 2 acquires reflected light or waves that are generated when light or radio waves are emitted toward the target area and reflected by objects within the target area. As a result, the three-dimensional sensor 2 outputs point cloud data. Specifically, the three-dimensional sensor 2 measures the distance to an object, the object's position, the object's shape, or the object's direction based on the time difference between when the reflected light or reflected waves hit an object within the target area and bounce back. The three-dimensional sensor 2 generates sensing data based on the measured distance, etc. The sensing data is point cloud data. Each point in the point cloud data includes distance data represented by three-dimensional coordinates (x, y, z) and intensity data that represents the reflectivity of the object's surface.

[0012] While the method by which the three-dimensional sensor 2 generates point cloud data is known, an example of the method by which the three-dimensional sensor 2 generates point cloud data will be described here. Here, as an example, the three-dimensional sensor 2 will be described as a LiDAR. The three-dimensional sensor 2 sequentially emits light in the x and y directions of a target area. For example, the light is emitted to six points in the x direction and two points in the y direction of the target area, for a total of 12 points. Note that, as viewed from the three-dimensional sensor 2, the x direction is the horizontal direction relative to the target area, and the y direction is the upward direction relative to the target area. The light sequentially emitted from the three-dimensional sensor 2 in the x and y directions of the target area is irradiated and reflected by an object within the target area to form reflected light. In the above example, the three-dimensional sensor 2 can be said to have a resolution of six pixels in the x direction and two pixels in the y direction, for a total of 12 pixels. The three-dimensional sensor 2 receives the reflected light and calculates the distance to the object based on the time difference between light emission and light reception, thereby obtaining distance data. In the above example, the three-dimensional sensor 2 individually calculates the distance for each of the irradiated positions distributed across the target area (six points in the x direction and two points in the y direction), a total of 12 points, and uses this distance data as distance data. Furthermore, the three-dimensional sensor 2 calculates the reflectance at each point on the reflective target based on the ratio between the amount of light emitted and the amount of light received for each of the irradiated positions, and uses this as intensity data. The distance data and intensity data calculated by the three-dimensional sensor 2 are referred to as point cloud data. The point cloud data is output to the object recognition device 3. The point cloud data is output to the object recognition device 3 on a frame-by-frame basis. The three-dimensional sensor 2 outputs point cloud data obtained by sensing the entire target area once, i.e., in the above example, the point cloud data obtained by sensing six points in the x direction and two points in the y direction, a total of 12 points, to the object recognition device 3 as point cloud data for one frame, or in other words, sensing data. The three-dimensional sensor 2 always distributes light in the horizontal and vertical directions at the same angle. In this way, the three-dimensional sensor 2 obtains the following three-dimensional information of x, y, and z: The three-dimensional sensor 2 also obtains intensity data (i) at each point.x; horizontal coordinate y; vertical coordinate z; distance data i; intensity data The horizontal coordinate x is a value indicating the horizontal position of each pixel, and the vertical coordinate y is a value indicating the vertical position of each pixel. The distance data z is depth information in the z-axis direction obtained at the position of each pixel. With regard to the distance data z, distance data z is measured one by one for each of the multiple pixels identified by all combinations of the horizontal coordinate x and the vertical coordinate y. Therefore, with regard to the multiple combinations of the horizontal coordinate x, the vertical coordinate y, and the distance data z in the three-dimensional information, each combination indicates a specific spatial position.

[0013] The object recognition device 3 recognizes objects by processing using a two-dimensional convolutional neural network based on sensing data acquired from the three-dimensional sensor 2, in other words, point cloud data. More specifically, the object recognition device 3 inputs data generated based on the point cloud data (hereinafter referred to as "object recognition data") into the two-dimensional convolutional neural network and recognizes objects by obtaining data related to the object output from the two-dimensional convolutional neural network. The object recognition device 3 recognizes the type of object by processing using the two-dimensional convolutional neural network. The object recognition device 3 can also recognize the position of the object from the point cloud data. The object recognition device 3 outputs the object recognition result to the device 4. A detailed configuration example of the object recognition device 3 will be described later.

[0014] The device 4 performs various controls based on the object recognition result output from the object recognition device 3.

[0015] An example configuration of the object recognition device 3 according to embodiment 1 will be described. Fig. 2 is a diagram showing an example configuration of the object recognition device 3 according to embodiment 1. The object recognition device 3 includes a data acquisition unit 31, a data generation unit 32, an object recognition unit 33, and a recognition result output unit 34. The data generation unit 32 includes a 3D voxel grid data generation unit 32a, a recognition target region detection unit 32b, a pseudo 3D data generation unit 32c, and a development drawing generation unit 32d.

[0016] The data acquisition unit 31 acquires sensing data, in other words, point cloud data, output from the three-dimensional sensor 2. The data acquisition unit 31 outputs the acquired point cloud data to the data generation unit 32.

[0017] The data generation unit 32 generates object recognition data from the point cloud data output from the data acquisition unit 31. In the first embodiment, the object recognition data includes pseudo three-dimensional data and development view data. Details of the pseudo three-dimensional data and development view data will be described later. The data generation unit 32 outputs the generated object recognition data to the object recognition unit 33.

[0018] The data generation unit 32 will now be described in detail. A three-dimensional voxel grid data generation unit 32a of the data generation unit 32 generates three-dimensional voxel grid data based on the point cloud data acquired by the data acquisition unit 31. Specifically, the three-dimensional voxel grid data is data represented by a three-dimensional model composed of a plurality of grids of a predetermined size W × H × Z.

[0019] FIG. 3 is a diagram schematically illustrating an example of 3D voxel grid data generated by the 3D voxel grid data generator 32a in the first embodiment. As shown in FIG. 3 , the 3D voxel grid data is represented by a 3D model divided into multiple grids (each having a predetermined size of W×H×Z). For example, the 3D voxel grid data is represented by a 3D model in which the field of view of the 3D sensor 2 is virtually divided into multiple grids. In FIG. 3 , as an example, the 3D voxel grid data is assumed to be composed of 15×15×8 grids. Also, in FIG. 3 , the 3D model shows how objects present in the target area appear within the 15×15×8 grids. Grids in which objects exist are shown in gray. That is, the gray grids are grids for which distances were obtained by the 3D sensor 2. Grids for which distances were not obtained by the 3D sensor 2, i.e., the white grids in the example of FIG. 3 , are also referred to as blank grids. In the 3D voxel grid data, the grey grids have values, while the blank grids have no values.

[0020] The three-dimensional voxel grid data generating unit 32a outputs the generated three-dimensional voxel grid data to the recognition target region detecting unit 32b.

[0021] Based on the 3D voxel grid data output from the 3D voxel grid data generator 32a, the recognition target area detection unit 32b detects, as the recognition target area, an area where an object exists among the areas consisting of multiple grids represented by the 3D voxel grid data. The recognition target area detection unit 32b detects the recognition target area for each object. The recognition target area detection unit 32b may detect the recognition target area using a known technique. For example, the recognition target area detection unit 32b detects the outline of the grid area containing values ​​based on the difference between the grids, more specifically, the difference between the values ​​contained in the grids, and extracts the grid surrounded by the detected outline. The recognition target area detection unit 32b then detects the circumscribed cubic rectangle of the outline of the extracted grid area as the recognition target area. Alternatively, for example, the recognition target area detection unit 32b may detect, as the recognition target area, a cubic rectangle obtained by adding a margin to the circumscribed cubic rectangle of the outline of the extracted grid. The recognition target area detection unit 32b outputs data indicating the detected recognition target area (hereinafter referred to as "recognition target area data") to the pseudo three-dimensional data generation unit 32c and the development drawing generation unit 32d.

[0022] The pseudo 3D data generator 32c generates pseudo 3D data that can be subjected to 2D convolution based on the recognition target area data output from the recognition target area detector 32b. More specifically, the pseudo 3D data generator 32c acquires the recognition target area data output from the recognition target area detector 32b, slices the recognition target area in three directions (x-direction, y-direction, and z-direction), and regards a specific axis as a channel to generate data to which 2D convolution can be applied. In the first embodiment, the data to which 2D convolution can be applied, generated from the recognition target area by the pseudo 3D data generator 32c, is referred to as "slice data." Slice data is one channel, in other words, one piece of 2D data. One channel corresponds to a specific position on a specific axis (x-axis, y-axis, or z-axis) within the recognition target area. In other words, one channel (one piece of 2D data) represents a cross section along a specific axis at that position within the recognition target area, which is the original 3D data. The specific position indicates the location of the cross section within the recognition target area, which is the original 3D data. For example, if a specific axis is the x-axis, the features such as the shape of the object at a specific position on the x-axis are expressed as a slice on the y-z plane. The pseudo 3D data generator 32c generates multiple slice data from multiple channels (multiple 2D data) at multiple positions on the specific axis within the recognition target area. By using multiple slice data, the 2D convolutional neural network can capture multiple cross-sectional data at positions on different axes. The 2D convolutional neural network can capture information from the 3D data by simultaneously processing these multiple slice data. The pseudo 3D data generator 32c generates multiple slice data for each of the three axes (x-axis, y-axis, and z-axis).

[0023] The pseudo three-dimensional data generator 32c then generates pseudo three-dimensional data by combining multiple slice data for each of the three axes. The pseudo three-dimensional data generator 32c may generate pseudo three-dimensional data by applying a known combining method such as Contact, Mul, or Add. Like slice data, the pseudo three-dimensional data obtained by combining multiple slice data is also data to which two-dimensional convolution can be applied.

[0024] FIG. 4 is a diagram illustrating an example of a process in which the pseudo 3D data generator 32c generates pseudo 3D data from the recognition target area data in the first embodiment. In FIG. 4, the three-dimensional model indicated by 401 represents the recognition target area. In FIG. 4, 402a, 402b, and 402c represent multiple slice data. In FIG. 4, 403 represents pseudo 3D data. Note that in FIG. 4, grids in the height direction of the recognition target area are indicated by "H," grids in the width direction are indicated by "W," and grids in the depth direction are indicated by "D." Also, in FIG. 4, grids on axes considered to be channels in the slice data are indicated by "C." Note that, for convenience, grids in which objects exist and blank grids in the recognition target area are both illustrated as white grids.

[0025] First, the pseudo 3D data generation unit 32c regards a cross section on the y-z plane obtained by slicing the recognition target area (see 401 in FIG. 4 ) detected by the recognition target area detection unit 32b at a specific position in the x-axis direction as one channel, and generates a plurality of slice data (hereinafter referred to as "x-axis slice data") expressed as two-dimensional data (see 402a in FIG. 4 ). Each image represents the slice data at a specific position on the x-axis. Furthermore, the pseudo 3D data generation unit 32c regards a cross section on the x-z plane obtained by slicing the recognition target area (see 401 in FIG. 4 ) detected by the recognition target area detection unit 32b at a specific position in the y-axis direction as one channel, and generates slice data (hereinafter referred to as "y-axis slice data") expressed as two-dimensional images (see 402b in FIG. 4 ). Each image represents the slice data at a specific position on the y-axis. Furthermore, the pseudo 3D data generation unit 32c regards the cross section on the x-y plane obtained by slicing the recognition target region detected by the recognition target region detection unit 32b (see 401 in FIG. 4) at a specific position in the z-axis direction as one channel, and generates slice data (hereinafter referred to as "z-axis slice data") expressed as a two-dimensional image (see 402c in FIG. 4). Each image represents slice data at a specific position on the z-axis. The pseudo 3D data generation unit 32c then generates pseudo 3D data by combining the generated multiple x-axis slice data, multiple y-axis slice data, and multiple z-axis slice data (see 403 in FIG. 4). Note that the order in which the pseudo 3D data generation unit 32c generates the x-axis slice data, y-axis slice data, and z-axis slice data is not important. 4, the pseudo 3D data generator 32c generates 13 pieces of x-axis slice data, 13 pieces of y-axis slice data, and 5 pieces of z-axis slice data, but this is merely an example. The pseudo 3D data generator 32c can generate any number of slice data in the x-axis direction, the y-axis direction, and the z-axis direction.

[0026] The pseudo three-dimensional data generating unit 32 c outputs the generated pseudo three-dimensional data to the object recognizing unit 33 .

[0027] Based on the recognition target area data output from the recognition target area detection unit 32b, the development generation unit 32d generates, as a development image, data including a development showing the reflectance of the surface of a grid area in which an object exists. More specifically, the development generation unit 32d acquires the recognition target area data output from the recognition target area detection unit 32b, views the recognition target area from a specific direction, and generates, as a development image, two-dimensional data including a development obtained by developing the grids on the surface of the grid area in which the object is included in the recognition target area. Note that the point cloud data includes intensity data, and in the three-dimensional voxel grid data, intensity data is assigned to each grid. In the development, the intensity data, in other words, data indicating reflectance, is assigned to each square.

[0028] FIG. 5 illustrates an example of a development image generated by the development generation unit 32d based on the recognition target area data in the first embodiment. In FIG. 5, the three-dimensional model 501a indicates the recognition target area. In FIG. 5, the recognition target area detection unit 32b detects a cubic rectangle, which is a cubic rectangle with a margin around the circumscribed cubic rectangle of the outline of the grid extracted as the area where the object exists, as the recognition target area. In FIG. 5, 502a indicates a development image including a development (denoted by 502b in FIG. 5) in which only the surface data of the area consisting of the grid (denoted by 501b in FIG. 5) representing the object included in the recognition target area is developed when the recognition target area shown in 501a is viewed from the direction indicated by the arrow. In FIG. 5, the development image is an image in which data corresponding to the surface of the blank grid is provided around the development, but this is merely an example. It is sufficient for the development image to include at least a development. The part corresponding to the surface of the blank grid around the unfolded drawing, represented by the white grid in Figure 5, only needs to be clearly marked as a grid with no values, and this part is assigned a uniform value, such as "0" or "-1".

[0029] The development generating unit 32 d outputs the generated development image to the object recognizing unit 33 .

[0030] The intensity data included in the sensing data acquired from the three-dimensional sensor 2 represents reflectance. The intensity data varies depending on the material of the object's surface, and contains many of the object's characteristics. By having the development image generation unit 32d generate a development image, the object recognition device 3 can perform object recognition by referring to the intensity data of the entire surface of the object, rather than just from one direction; in other words, by taking the object's characteristics into greater consideration. Note that in the object recognition device 3, object recognition is performed by the object recognition unit 33.

[0031] The object recognition unit 33 recognizes an object by inputting the object recognition data generated by the data generation unit 32 into a two-dimensional convolutional neural network to obtain data related to the object. Specifically, the object recognition unit 33 recognizes an object by inputting the pseudo three-dimensional data generated by the pseudo three-dimensional data generation unit 32c and the developed view image generated by the developed view generation unit 32d into a two-dimensional convolutional neural network to obtain data related to the object. More specifically, the object recognition unit 33 inputs the pseudo three-dimensional data and the developed view image into a two-dimensional convolutional neural network. When the pseudo three-dimensional data and the developed view image are input, the two-dimensional convolutional neural network outputs data related to the object. Here, the object data output by the two-dimensional convolutional neural network is data in which the type of object is associated with the probability that the object is considered to be that type. Based on the object data, the object recognition unit 33 determines, for example, the type with the highest association probability as the object type. In this way, the object recognition unit 33 recognizes the object.

[0032] An example of an object recognition method by the object recognition unit 33 will be described in detail. FIG. 6 is a diagram showing an example of object recognition processing performed by the object recognition unit 33 using a two-dimensional convolutional neural network in the first embodiment. First, the object recognition unit 33 inputs the pseudo three-dimensional data output from the pseudo three-dimensional data generation unit 32c into the two-dimensional convolutional neural network and obtains a feature (hereinafter referred to as a "first feature") output from the two-dimensional convolutional neural network (see 601 in FIG. 6). The object recognition unit 33 also inputs the development image output from the development generation unit 32d into the two-dimensional convolutional neural network and obtains a feature (hereinafter referred to as a "second feature") output from the two-dimensional convolutional neural network (see 602 in FIG. 6). The order in which the object recognition unit 33 obtains the first feature and the second feature does not matter. Next, the object recognition unit 33 combines the obtained first feature and second feature. The object recognition unit 33 then inputs the synthesized feature (hereinafter referred to as the "synthesized feature") back into the two-dimensional convolutional neural network to obtain data related to the object. Based on the data related to the object, the object recognition unit 33 then determines, for example, the type that is most likely to be associated with the data related to the object as the type of the object (see 603 in FIG. 6).

[0033] The two-dimensional convolutional neural network has been trained in advance using training data with correct answer labels. The trained two-dimensional convolutional neural network is stored in a location that can be referenced by the object recognition unit 33. The training data is an existing data set that has been generated in advance by an administrator or the like.

[0034] The object recognition unit 33 generates an object recognition result indicating the type of object. The object recognition result may include data indicating the position of the object. The position of the object can be determined from point cloud data. The object recognition unit 33 may acquire the point cloud data from the data acquisition unit 31 via the data generation unit 32. The object recognition unit 33 outputs the object recognition result to the recognition result output unit 34.

[0035] The recognition result output unit 34 outputs the object recognition result output from the object recognition unit 33 to the device 4 .

[0036] The operation of the object recognition device 3 according to embodiment 1 will now be described. Fig. 7 is a flowchart for explaining the operation of the object recognition device 3 according to embodiment 1. For example, when the object recognition device 3 is powered on, the object recognition device 3 starts the operation shown in the flowchart of Fig. 7 and repeats the operation shown in the flowchart of Fig. 7 until the power is turned off.

[0037] The data acquisition unit 31 acquires sensing data, in other words, point cloud data, output from the three-dimensional sensor 2 (step ST1). The data acquisition unit 31 outputs the acquired point cloud data to the data generation unit 32.

[0038] The three-dimensional voxel grid data generator 32a generates three-dimensional voxel grid data (step ST2) based on the point cloud data acquired by the data acquirer 31 in step ST1. The three-dimensional voxel grid data generator 32a outputs the generated three-dimensional voxel grid data to the recognition target region detector 32b.

[0039] Based on the three-dimensional voxel grid data output from the three-dimensional voxel grid data generator 32a in step ST2, the recognition-target-region detector 32b detects, as a recognition-target region, a region where an object exists among a region consisting of a plurality of grids represented by the three-dimensional voxel grid data (step ST3). The recognition-target-region detector 32b outputs the recognition-target-region data to the pseudo three-dimensional data generator 32c and the development-drawing generator 32d.

[0040] The pseudo three-dimensional data generating unit 32c generates pseudo three-dimensional data that can be two-dimensionally convoluted based on the recognition target region data output from the recognition target region detecting unit 32b in step ST3 (step ST4a). The pseudo three-dimensional data generating unit 32c outputs the generated pseudo three-dimensional data to the object recognizing unit 33.

[0041] The development generating unit 32d generates a development image based on the recognition target area data output from the recognition target area detecting unit 32b in step ST3 (step ST4b). The development generating unit 32d outputs the generated development image to the object recognizing unit 33.

[0042] Based on the pseudo 3D data output from the pseudo 3D data generating unit 32c in step ST4a and the development image output from the development generating unit 32d in step ST4b, the object recognizing unit 33 inputs the pseudo 3D data and the development image into a 2D convolutional neural network, respectively, to obtain data related to the object, thereby recognizing the object (step ST5). The object recognizing unit 33 outputs the object recognition result to the recognition result output unit 34.

[0043] The recognition result output unit 34 outputs the object recognition result output from the object recognition unit 33 in step ST5 to the device 4 (step ST6).

[0044] 7, the processing of step ST4a and the processing of step ST4b are performed in parallel, but this is merely an example. For example, the processing may be performed in the order of step ST3, step ST4a, step ST4b, and step ST5, or in the order of step ST3, step ST4b, step ST4a, and step ST5.

[0045] In this way, the object recognition device 3 generates three-dimensional voxel grid data based on the point cloud data acquired from the three-dimensional sensor 2, and detects a recognition target area based on the generated three-dimensional voxel grid data. The object recognition device 3 generates pseudo three-dimensional data and a development image based on the recognition target area data, and inputs the generated pseudo three-dimensional data and the development image into a two-dimensional convolutional neural network to obtain data related to the object, thereby recognizing the object. By generating three-dimensional voxel grid data to which two-dimensional convolution can be applied and performing object recognition using a two-dimensional convolutional neural network, the object recognition device 3 can reduce the amount of calculation and perform object recognition faster than object recognition using a three-dimensional convolutional neural network, and can perform object recognition with higher accuracy than conventional techniques that recognize objects by performing image processing using a convolutional neural network based on multiple pieces of two-dimensional information. Furthermore, by generating a development image and performing object recognition using a two-dimensional convolutional neural network, the object recognition device 3 can perform object recognition by referring to intensity data on the entire surface of the object, rather than just from one direction, in other words, by taking the object's features into greater consideration. As a result, the object recognition device 3 can further improve the object recognition accuracy.

[0046] In the above-described first embodiment, the object recognition device 3 is provided with the development drawing generation unit 32d, but this is merely an example, and the object recognition device 3 is not required to include the development drawing generation unit 32d. When the object recognition device 3 is configured not to include the development drawing generation unit 32d, the processing of step ST4b can be omitted from the operation of the object recognition device 3 shown in the flowchart of FIG.

[0047] Furthermore, in the above-described first embodiment, the object recognition device 3 acquires point cloud data generated by the three-dimensional sensor 2 from the three-dimensional sensor 2, but this is merely an example. The source of point cloud data is not limited to the three-dimensional sensor 2, and the object recognition device 3 may, for example, acquire point cloud data that has been generated in advance and stored in a storage device (not shown) that the object recognition device 3 can refer to from the storage device, or the object recognition device 3 may be connected to a simulator (not shown), and the object recognition device 3 may acquire point cloud data from the simulator that has been generated by the simulator running a simulation.

[0048] In the first embodiment, the object recognition device 3 is mounted on, for example, an autonomous driving vehicle, but this is merely an example. The object recognition device 3 may be mounted on, for example, a server, or some of the data acquisition unit 31, the 3D voxel grid data generation unit 32a, the recognition target area detection unit 32b, the pseudo 3D data generation unit 32c, the development view generation unit 32d, the object recognition unit 33, and the recognition result output unit 34 may be provided in the in-vehicle device, and the rest may be provided in the server.

[0049] In the first embodiment, the function of the recognition result output unit 34 may be included in the object recognition unit 33. In this case, the object recognition device 3 may be configured without the recognition result output unit 34.

[0050] In the first embodiment, for example, the object recognition device 3 may store the object recognition results, and the three-dimensional sensor 2 and the object recognition device 3 may constitute the object recognition system 1 .

[0051] 8A and 8B are diagrams illustrating an example of a hardware configuration of an object recognition device 3 according to the first embodiment. In the first embodiment, the functions of the data acquisition unit 31, the 3D voxel grid data generation unit 32a, the recognition target area detection unit 32b, the pseudo 3D data generation unit 32c, the development view generation unit 32d, the object recognition unit 33, and the recognition result output unit 34 are realized by a processing circuit 101. That is, the object recognition device 3 includes the processing circuit 101 for generating pseudo 3D data to which 2D convolution can be applied based on sensor data, in other words, point cloud data, and for controlling object recognition using the pseudo 3D data and a 2D convolutional neural network. The processing circuit 101 may be dedicated hardware as shown in FIG. 8A, or a processor 104 that executes a program stored in a memory as shown in FIG. 8B.

[0052] When processing circuitry 101 is dedicated hardware, processing circuitry 101 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0053] When the processing circuit is a processor 104, the functions of the data acquisition unit 31, the 3D voxel grid data generation unit 32a, the recognition target area detection unit 32b, the pseudo 3D data generation unit 32c, the development view generation unit 32d, the object recognition unit 33, and the recognition result output unit 34 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 105. The processor 104 reads and executes the program stored in the memory 105 to perform the functions of the data acquisition unit 31, the 3D voxel grid data generation unit 32a, the recognition target area detection unit 32b, the pseudo 3D data generation unit 32c, the development view generation unit 32d, the object recognition unit 33, and the recognition result output unit 34. In other words, the object recognition device 3 includes a memory 105 for storing a program that, when executed by the processor 104, results in the execution of steps ST1 to ST6 of FIG. 7 described above. In addition, it can also be said that the program stored in memory 105 causes the computer to execute the processing procedures or methods of the data acquisition unit 31, the three-dimensional voxel grid data generation unit 32a, the recognition target area detection unit 32b, the pseudo three-dimensional data generation unit 32c, the development drawing generation unit 32d, the object recognition unit 33, and the recognition result output unit 34. Here, the memory 105 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

[0054] Note that the functions of the data acquisition unit 31, the 3D voxel grid data generation unit 32a, the recognition target area detection unit 32b, the pseudo 3D data generation unit 32c, the development view generation unit 32d, the object recognition unit 33, and the recognition result output unit 34 may be partially implemented by dedicated hardware and partially implemented by software or firmware. For example, the functions of the data acquisition unit 31 and the recognition result output unit 34 may be implemented by a processing circuit 101 as dedicated hardware, while the functions of the 3D voxel grid data generation unit 32a, the recognition target area detection unit 32b, the pseudo 3D data generation unit 32c, the development view generation unit 32d, and the object recognition unit 33 may be implemented by a processor 104 reading and executing programs stored in a memory 105. The object recognition device 3 also includes an input interface device 102 and an output interface device 103 that communicate with devices such as the 3D sensor 2 or the device 4 via wired or wireless communication.

[0055] As described above, according to the first embodiment, the object recognition device 3 is configured to include: a data acquisition unit 31 that acquires point cloud data (sensor data); a three-dimensional voxel grid data generation unit 32 a that generates three-dimensional voxel grid data based on the point cloud data acquired by the data acquisition unit 31; a recognition target region detection unit 32 b that detects, as a recognition target region, a region consisting of a plurality of grids represented by the three-dimensional voxel grid data, in which an object exists, based on the three-dimensional voxel grid data generated by the three-dimensional voxel grid data generation unit 32 a; a pseudo three-dimensional data generation unit 32 c that generates pseudo three-dimensional data that can be two-dimensionally convoluted, based on the data of the recognition target region detected by the recognition target region detection unit 32 b; and an object recognition unit 33 that inputs the pseudo three-dimensional data generated by the pseudo three-dimensional data generation unit 32 c into a two-dimensional convolutional neural network to obtain data related to the object, thereby recognizing the object. Therefore, the object recognition device 3 can reduce the amount of calculations and perform object recognition faster than object recognition using a three-dimensional convolutional neural network that uses three-dimensional information as input, and can perform object recognition with higher accuracy than conventional technology that recognizes objects by performing image processing using a convolutional neural network based on multiple pieces of two-dimensional information.

[0056] In addition to the above configuration, the object recognition device 3 also includes a development image generator 32d that generates a development image including a development showing the surface reflectance of a grid-based area in which an object exists, based on the data of the recognition target area detected by the recognition target area detector 32b. The object recognition unit 33 can recognize an object by inputting the pseudo 3D data generated by the pseudo 3D data generator 32c and the development image generated by the development image generator 32d into a 2D convolutional neural network to obtain data related to the object. This allows the object recognition device 3 to perform object recognition by referring to intensity data for the entire surface of the object, rather than just from one direction. In other words, it can perform object recognition that takes the object's characteristics into greater consideration. As a result, the object recognition device 3 can further improve the object recognition accuracy.

[0057] Any of the components of the embodiments may be modified or omitted.

[0058] The object recognition device disclosed herein can reduce the amount of calculations and perform object recognition faster than object recognition using a three-dimensional convolutional neural network, and can perform object recognition with higher accuracy than technology that recognizes objects by performing image processing using a convolutional neural network based on multiple two-dimensional information.

[0059] 1 Object recognition system, 2 Three-dimensional sensor, 3 Object recognition device, 31 Data acquisition unit, 32 Data generation unit, 32a Three-dimensional voxel grid data generation unit, 32b Recognition target area detection unit, 32c Pseudo three-dimensional data generation unit, 32d Development drawing generation unit, 33 Object recognition unit, 34 Recognition result output unit, 4 Equipment, 101 Processing circuit, 102 Input interface device, 103 Output interface device, 104 Processor, 105 Memory.

Claims

1. An object recognition device comprising: a data acquisition unit that acquires point cloud data; a three-dimensional voxel grid data generation unit that generates three-dimensional voxel grid data based on the point cloud data acquired by the data acquisition unit; a recognition target region detection unit that detects, as a recognition target region, a region consisting of multiple grids represented by the three-dimensional voxel grid data, where an object exists, based on the three-dimensional voxel grid data generated by the three-dimensional voxel grid data generation unit; a pseudo three-dimensional data generation unit that generates pseudo three-dimensional data that can be two-dimensionally convolved based on the data of the recognition target region detected by the recognition target region detection unit; and an object recognition unit that inputs the pseudo three-dimensional data generated by the pseudo three-dimensional data generation unit into a two-dimensional convolutional neural network to obtain data related to the object, thereby recognizing the object.

2. The object recognition device according to claim 1, further comprising a development generation unit that generates a development image including a development showing the reflectance of the surface of the area consisting of the grid in which the object exists, based on the data of the recognition target area detected by the recognition target area detection unit, and wherein the object recognition unit recognizes the object by inputting the pseudo three-dimensional data generated by the pseudo three-dimensional data generation unit and the development image generated by the development generation unit into the two-dimensional convolutional neural network to obtain data related to the object.

3. An object recognition device as described in claim 1 or claim 2, characterized in that the pseudo three-dimensional data generation unit generates the pseudo three-dimensional data by slicing the data of the recognition target area in three directions and treating a specific axis as a channel, and synthesizing multiple slice data that show cross sections within the recognition target area that represent the characteristics of the object at a specific position on the axis.

4. The object recognition device according to claim 2, characterized in that the unfolded view generation unit generates as the unfolded view image two-dimensional data including an unfolded view of the grid on the surface of the area consisting of the grid in which the object in the recognition target area exists, viewed from a certain direction.

5. An object recognition method comprising: a data acquisition unit acquiring point cloud data; a three-dimensional voxel grid data generation unit generating three-dimensional voxel grid data based on the point cloud data acquired by the data acquisition unit; a recognition target area detection unit detecting, based on the three-dimensional voxel grid data generated by the three-dimensional voxel grid data generation unit, an area where an object exists from an area consisting of multiple grids represented by the three-dimensional voxel grid data as a recognition target area; a pseudo three-dimensional data generation unit generating pseudo three-dimensional data that can be two-dimensionally convolved based on the data of the recognition target area detected by the recognition target area detection unit; and an object recognition unit inputting the pseudo three-dimensional data generated by the pseudo three-dimensional data generation unit into a two-dimensional convolutional neural network to obtain data related to the object, thereby recognizing the object.

6. An object recognition system comprising: an object recognition device according to any one of claims 1 to 4; and a three-dimensional sensor that generates the point cloud data.

7. The object recognition system according to claim 6, wherein the three-dimensional sensor is a LiDAR or millimeter wave sensor.

Images