DATA PROCESSING APPARATUS, DATA PROCESSING METHOD, DATA PROCESSING PROGRAM, AND LEARNING MODEL GENERATION METHOD

Abstract

To provide a data processing apparatus, a data processing method, a data processing program, and a learning mode creation method that can perform data processing on object data representing an object in consideration of a reference direction of the object.SOLUTION: An image processing apparatus 150 as a data processing apparatus includes: a receiving unit 60 that receives object data representing an object and a direction corresponding to a reference direction of the object on the object data; and a data processing unit 70 that, based on the direction corresponding to the reference direction of the object, performs data processing by performing convolution processing on the object data by using a filter after change obtained by changing the coefficient of the filter.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a data processing device, a data processing method, a data processing program, and a learning model generating method, and in particular to a data processing device, a data processing method, a data processing program, and a learning model generating method for performing data processing on target data representing an object. [Background technology]

[0002] For the purpose of automatically recognizing a scene captured in an image, technologies have been researched and developed to divide an image into regions for each of multiple objects or regions for each of multiple parts captured in the image, and to recognize the objects or parts captured in each region. Hereinafter, the captured objects or parts are called subjects. Region division accompanied by subject recognition is called semantic segmentation.

[0003] In particular, in recent years, techniques for performing the above-mentioned segmentation and recognition based on learning have been actively researched. For example, Non-Patent Document 1 describes a method of preparing a large number of learning images in which a class representing the subject is assigned to each pixel of an area previously segmented for each subject, and having a computer perform machine learning on these learning images. The information assigned in advance is called an annotation or the like. If an arbitrary image is input to a trained model generated by this learning, a class for each pixel of the input image is output. In other words, the input image is segmented into areas (label areas) labeled with a class for each subject. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] “Fully Convolutional Networks for Semantic Segmentation”, Jonathan Long, Evan Shelhamer, and Trevor Darrell (Proceedings of the IEEE conference on computer vision and pattern recognition, 2015) Summary of the Invention [Problem to be solved by the invention]

[0005] Traditionally, these machine learning methods often use a dataset consisting of learning images made of perspective projection images with little lens distortion and their annotations (see Fig. 11). On the other hand, perspective projection images show objects in a significantly different way than fisheye images taken by fisheye cameras used for surveillance (see Fig. 2). Therefore, when a fisheye image is input to a trained model created with a dataset of perspective projection images, the accuracy may be significantly reduced, especially when there is a misalignment between the top and bottom of the image and the top and bottom of the real world, people cannot be detected correctly on the left and right or at the bottom of the fisheye image, or the floor may be mistaken for a wall (see Fig. 12). Fig. 12 shows an example of segmentation into incorrectly labeled areas as shown in Fig. 12(B) as a result of segmentation processing on the fisheye image shown in Fig. 12(A).

[0006] One method to prevent this kind of decrease in accuracy is to convert the fisheye image into a perspective projection image before processing it. In this case, multiple perspective projection images are obtained from one fisheye image, so the processing time increases because all of the results must be processed and integrated, and integration is difficult when inconsistencies occur in the results.

[0007] Another approach is to rotate the dataset 360 degrees in every direction during training, so that the trained model can handle objects in every orientation and process fisheye images. However, this approach tries to find objects in every orientation across the entire fisheye image, which can lead to false positives, such as people being upside down.

[0008] In addition, when using a convolutional neural network to estimate segmentation, it may be possible to estimate weights with high accuracy for fisheye images by rotating the filter used for convolution according to the inclination of the object in the image. However, the filters used in the convolution operation generally have small widths and heights, and it is difficult to define continuous rotations for this weight array.

[0009] Therefore, an object of the present invention is to provide a data processing device, a data processing method, a data processing program, and a learning model generation method that can perform data processing on target data representing an object while taking into account the reference direction of the object. [Means for solving the problem]

[0010] In order to achieve the above-mentioned object, the data processing device of the present invention is a data processing device that processes data using a filter using coefficients that have been pre-learned using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object, and is configured to include a receiving unit that receives object data representing an object and a direction on the object data that corresponds to the reference direction of the object, and a data processing unit that performs a convolution process on the object data and processes data using a modified filter in which the coefficients of the filter have been modified based on the direction that corresponds to the reference direction of the object.

[0011] In the data processing device according to the present invention, the receiving unit receives object data representing an object and a direction on the object data corresponding to a reference direction of the object, and the data processing unit performs data processing by performing a convolution process on the object data using a modified filter in which the coefficients of the filter are modified based on the direction corresponding to the reference direction of the object.

[0012] In this way, by performing a convolution process on the target data using a modified filter whose coefficients have been changed based on the direction on the target data that corresponds to the reference direction of the object, data processing can be performed on the target data representing the object, taking into account the reference direction of the object.

[0013] In addition, the target data is data in which the reference direction of the object differs for each element, and the receiving unit receives an angle map that stores a direction corresponding to the reference direction of the object for each element, and the data processing unit can perform a convolution process on the target data using a modified filter in which the coefficient of the filter has been changed based on the angle map, to perform data processing.

[0014] In addition, the data processing unit can change the coefficient of the filter for each element based on the direction, and perform convolution processing on the target data using the changed filter for each element to process the data.

[0015] The data processing unit changes the coefficients of the filters generated using the filters using the pre-learned coefficients, corresponding to each of a plurality of specific directions, by weighting the filters according to the relative angle between the direction and the specific direction, and can perform a convolution process on the target data using the changed filters to process the data.

[0016] The data processing unit performs data processing by performing a convolution process on the target data using a filter corresponding to each of a plurality of specific directions, which is generated using a filter using the pre-learned coefficients, and can integrate the results of the data processing using the filter corresponding to each of the plurality of specific directions by weighting them according to the relative angle between the direction corresponding to the reference direction of the target object and the specific direction.

[0017] The data processing unit can perform data processing by performing convolution processing on the target data using the changed filter as a filter for at least a portion of a layer of a convolutional neural network.

[0018] The target data may be a fisheye image captured by a fisheye camera.

[0019] The data processing method of the present invention is a data processing method that performs data processing using a filter using coefficients that have been pre-learned using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object, in which a receiving unit receives object data representing an object and a direction on the object data that corresponds to the reference direction of the object, and a data processing unit performs data processing by performing a convolution process on the object data using a modified filter in which the coefficients of the filter have been modified based on the direction that corresponds to the reference direction of the object.

[0020] The data processing program of the present invention is a data processing program for performing data processing using a filter using coefficients that have been pre-learned using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object, and is a program for causing a computer to function as a reception unit that receives object data representing an object and a direction on the object data that corresponds to the reference direction of the object, and a data processing unit that performs convolution processing on the object data using a modified filter in which the coefficients of the filter have been modified based on the direction that corresponds to the reference direction of the object.

[0021] In the learning model generation method of the present invention, a learning data generation unit generates a training data set including learning data representing an object, a correct data processing result, and a direction on the learning data that corresponds to a reference direction of the object, and a learning unit learns by performing a convolution process on the learning data using a modified filter in which the coefficients of the filter are changed based on the direction, so that the result of the data processing matches the correct data processing result. Effect of the Invention

[0022] The data processing device, data processing method, data processing program, and learning model generation method of the present invention have the advantage that data processing can be performed on target data representing an object while taking into account the reference direction of the object. [Brief description of the drawings]

[0023] [Figure 1] 1 is a schematic diagram showing a configuration of an image processing system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a diagram illustrating an example of a fisheye image. [Diagram 3] 1 is a schematic diagram showing a configuration of a learning device according to an embodiment of the present invention; [Figure 4] 2 is a schematic diagram showing a configuration of a learning unit of the learning device according to the embodiment of the present invention; FIG. [Diagram 5] 1 is a schematic diagram showing a configuration of a data processing unit of a learning device and an image processing device according to an embodiment of the present invention. [Figure 6] 1 is a schematic diagram illustrating a configuration of an image processing device according to an embodiment of the present invention. [Figure 7] 5 is a flowchart showing the operation of a learning process by the learning device according to the embodiment of the present invention. [Figure 8] 4 is a flowchart showing an image processing operation by the image processing device according to the embodiment of the present invention. [Figure 9]FIG. 2A is a diagram showing an example of a fisheye image, and FIG. 2B is a diagram showing an example of a segmentation processing result. [Figure 10] FIG. 13 is a schematic diagram showing the configuration of a data processing unit of a learning device and an image processing device according to a modified example. [Figure 11] FIG. 2 is a diagram illustrating an example of a perspective projection image. [Figure 12] FIG. 1A is a diagram showing an example of a fisheye image, and FIG. 1B is a diagram showing an example of a segmentation processing result in the prior art. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In this embodiment, the present invention will be described as an example in which the present invention is applied to an image processing system that classifies fisheye images taken by a fisheye photography device according to subject classes and divides them into label regions. A fisheye image is an example of target data, and a subject is an example of an object.

[0025] <System configuration> Hereinafter, the configuration of an embodiment of the present invention will be described with reference to FIG. 1 which shows a schematic configuration of an image processing system 1000 to which the present invention is applied.

[0026] (Image processing system 1000) The image processing system 1000 includes a fisheye photography device 1100, a network 1200, a learning device 100, and an image processing device 150. The image processing device 150 is an example of a data processing device.

[0027] (Fisheye photography device 1100)

[0028] The fisheye photographing device 1100 captures a fisheye image as shown in Fig. 2. Specifically, the fisheye photographing device 1100 is a surveillance camera installed for the purpose of monitoring a specified surveillance space, and is installed, for example, on a ceiling, with its optical center facing approximately vertically downward to capture images downward. In a fisheye image, the reference direction of the subject (e.g., the upward direction) differs for each pixel, and the reference direction of the subject for each pixel is determined in advance. The fisheye image captured by the fisheye photographing device 1100 is transmitted to the image processing device 150. Note that a pixel is an example of an element.

[0029] (Network 1200) Network 1200 is a line used for transmitting and receiving data between fisheye photography device 1100, image processing device 150, and learning device 100. A LAN (Local Area Network) or a public line such as the Internet can be used as network 1200 of the present invention. It is desirable to take security measures for messages on network 1200, such as encrypting the messages using known VPN technology or the like.

[0030] (Learning device 100) The parameters of the model used in the image processing device 150 are learned by the learning device 100.

[0031] The learning device 100 is composed of a CPU, a GPU, an MPU, peripheral circuits, terminals, various memories, etc., and learns model parameters for receiving a learning dataset and performing segmentation processing to classify the objects of an image by class and divide them into label regions. The segmentation processing is an example of data processing.

[0032] As shown in FIG. 3, the learning device 100 functionally includes a receiving unit 10 and a calculation unit 20.

[0033] The receiving unit 10 receives a plurality of learning data sets. For example, the learning data set includes a set of learning images of perspective projection images and correct labels for segmentation processing. Here, in the perspective projection images, the reference direction of the subject is the upward direction. The correct labels for segmentation processing are an array having the same width and height as the learning images, and each element indicates a segmentation label of a position of the corresponding image.

[0034] As shown in FIG. 3, the calculation unit 20 includes a learning image generation unit 22, a learning unit 24, and a model storage unit .

[0035] Hereinafter, each of the learning image generating unit 22, the learning unit 24, and the model storage unit 26 that configure the calculation unit 20 will be described in detail.

[0036] The learning image generating unit 22 compares the learning image and the correct label at an angle θ aug At this time, rotate it by θ aug 0°≦θ aug <360°. The learning image generating unit 22 also creates an angle map, which is an array with the same width and height as the learning image, and sets the values ​​of all elements to θ aug The output is as follows:

[0037] As shown in FIG. 4, the learning section 24 includes a data processing section 30, an error calculation section 40, and an update section 50.

[0038] The data processing unit 30, the error calculation unit 40, and the update unit 50 that constitute the learning unit 24 will be described in detail below.

[0039] The data processing unit 30 performs a segmentation process on the learning image using the learning image and the angle map. Here, the data processing unit 30 is assumed to be configured with a convolutional neural network, and inputs the learning image to a convolutional neural network using a filter whose coefficients have been changed based on the angle map, performs a convolution process on the learning image to perform a segmentation process, and outputs a segmentation result in which the learning image is classified by subject class and divided into label regions.

[0040] Hereinafter, each part of the filter synthesis unit 32 and the convolution calculation unit 34 constituting the data processing unit 30 shown in FIG. 5 will be described in detail.

[0041] In the data processing unit 30, a filter w is applied to the area where the upper direction of the object is the upper direction of the image. ↑ and a filter w for the area where the bottom direction of the subject is the top direction of the image. ↓ and a filter w for the area where the right direction of the subject is the top direction of the image. → and a filter w that targets the area where the left direction of the subject is the top direction of the image. ← The base filters are stored as N base filters. These base filters are synthesized in the filter synthesis unit 32 based on the angle map. The upward direction, downward direction, rightward direction, and leftward direction are examples of the specific direction. FIG. 5 shows an example in which N base filters are synthesized based on the angle map.

[0042] The filter synthesis unit 32 changes the filter coefficients for each pixel based on the values ​​of the angle map.

[0043] Specifically, the value of the angle map at pixel i in the learning image is expressed as θ i Then, the synthesis filter w i is calculated as follows:

[0044]

number

[0045] As shown in the above equation (1), the data processing unit 30 obtains a composite filter by weighting the basis filters corresponding to each of the multiple specific directions according to the relative angle between the direction indicated by the angle map value and the specific direction, and by changing the filter coefficients, it is possible to create a filter that changes within the image according to the angle map value.

[0046] In addition, the rotation of the filter used for convolution can be defined by transposing the array and inverting it up, down, left, and right if the rotation is 90 degrees, 180 degrees, or 270 degrees. ↑ Only the basis filter is w ← , w ↓ , w → Regarding w ↑ In this case, it is possible to prevent an increase in the learning parameters and to impose natural constraints on the relationship between the basis filters.

[0047] In each layer, the convolution calculation unit 34 performs convolution processing on the learning image for each pixel based on the synthesis filter obtained for that pixel by the filter synthesis unit 32, and outputs the result to the next layer.

[0048] The convolution calculation unit 34 finally outputs the probability of each class at each pixel as the segmentation processing result.

[0049] The error calculation unit 40 calculates the cross-entropy error at each pixel based on the accuracy of each class at each pixel and the rotated correct label.

[0050] The update unit 50 updates parameters including the filter coefficients of the convolutional neural network based on the calculated error.

[0051] The learning unit 24 repeats the processes of the data processing unit 30, the error calculation unit 40, and the update unit 50 to minimize the calculated error and learn the parameters of the convolutional neural network that constitutes the data processing unit 30.

[0052] In this embodiment, the error is calculated for a plurality of training data sets, and the average of the errors is used to determine the amount of parameter update. The angle by which the training images are rotated in the training image generating unit 22 may be different for each training data set.

[0053] The model storage unit 26 stores parameters of the model learned by the learning unit 24 .

[0054] (Image processing device 150) The image processing device 150 is composed of a CPU, GPU, MPU, peripheral circuits, terminals, various memories, etc., and receives fisheye images sent by the fisheye photography device 1100, and performs segmentation processing to classify the fisheye images by subject class and divide them into label areas.

[0055] As shown in FIG. 6, the image processing device 150 functionally comprises a receiving unit 60, a data processing unit 70, and an output unit 80.

[0056] The receiving unit 60 receives the received fisheye image. In addition, the reference direction (upward) of the subject in the fisheye image is assumed to be known for each pixel, and the receiving unit 60 receives an angle map that stores the reference direction of the subject in the fisheye image for each pixel.

[0057] To accept a fisheye image such as that shown in FIG. 2 above, the value of pixel i in the angle map is set according to the following equation, where c is the image center:

[0058]

number

[0059]

number

[0060] The data processing unit 70 is equipped with a convolutional neural network trained by the learning device 100. A fisheye image is input to the convolutional neural network using a modified filter whose filter coefficients have been changed based on the angle map, and a convolution process is performed on the fisheye image to perform segmentation processing. The segmentation results in which the fisheye image is classified by subject class and divided into label regions are output by the output unit 80.

[0061] Specifically, the data processing unit 70 includes a filter synthesis unit 72 and a convolution calculation unit 74, as shown in FIG.

[0062] Similar to the filter synthesis unit 32, the filter synthesis unit 72 changes the filter coefficient for each pixel of the fisheye image based on the value of the angle map.

[0063] Specifically, the angle map value at pixel i in the fisheye image is set to θi, and the synthesis filter wi is calculated according to the above equation (1).

[0064] Similar to the convolution calculation unit 34, the convolution calculation unit 74 performs convolution processing on the fisheye image for each pixel in each layer based on the synthesis filter determined by the filter synthesis unit 72, and outputs the result to the next layer.

[0065] The convolution calculation unit 74 ultimately determines for each pixel the class with the highest probability as the class of that pixel, and outputs the segmentation processing result from the output unit 80.

[0066] <Image processing system operation> Hereinafter, the operation of the image processing system 1000 to which the present invention is applied will be described with reference to the flowcharts shown in Figures 7 and 8. Note that the description will be given taking as an example a case in which a plurality of learning data sets prepared in advance are input to the learning device 100 by the receiving unit 10.

[0067] 7 is executed in advance. In the learning process, the learning image generating unit 22 generates a learning image and a correct label for each learning data set at a randomly selected angle θ aug and rotate each pixel by an angle θ aug Then, an angle map is generated (step S100).

[0068] The data processing unit 30 performs segmentation processing on the learning images for each learning data set using the learning images and the angle map. Here, the data processing unit 30 inputs the learning images to a convolution neural network using a filter whose coefficients have been changed based on the angle map, performs convolution processing on the learning images to perform segmentation processing, and outputs the segmentation results (step S102).

[0069] Then, for each learning data set, the error calculation unit 40 calculates the cross-entropy error at each pixel based on the accuracy of each class at each pixel and the rotated correct label (step S104).

[0070] The update unit 50 updates various parameters including the filter coefficients of the convolutional neural network based on the error calculated for each learning data set (step S106).

[0071] Then, it is determined whether or not a predetermined iteration end condition is satisfied (step S108). If the predetermined iteration end condition is not satisfied, the process returns to step S102, whereas if the predetermined iteration end condition is satisfied, the process proceeds to step S110.

[0072] The iteration end condition may be that the calculated error value has converged, that the number of iterations has reached an upper limit, or the like.

[0073] Then, the parameters of the finally trained convolutional neural network are stored in the model storage unit 26 (step S110), and the training process ends.

[0074] Then, the parameters of the convolutional neural network trained by the training device 100 are set in the data processing unit 70 of the image processing device 150.

[0075] The image processing device 150 that receives the fisheye images from the fisheye photography device 1100 performs the image processing shown in Fig. 8. Note that the operation of the image processing device 150 in Fig. 8, which will be described below, is executed each time a fisheye image captured by the fisheye photography device 1100 is received.

[0076] First, the receiving unit 60 receives a received fisheye image. The receiving unit 60 also receives an angle map that stores, for each pixel, a reference direction of a subject captured in the fisheye image (step S110).

[0077] The filter synthesis unit 72 changes the filter coefficient for each pixel of the fisheye image based on the value of the angle map (step S112).

[0078] In each layer, the convolution calculation unit 74 performs convolution processing on the fisheye image for each pixel based on the filter calculated by the filter synthesis unit 72, and outputs the result to the next layer. Finally, the convolution calculation unit 74 determines the class with the highest probability for each pixel as the class of the pixel, and outputs the segmentation processing result by the output unit 80 (step S114).

[0079] As described above, in the image processing device 150 according to the embodiment of the present invention, a convolution process is performed on the fisheye image using a modified filter whose coefficient is changed based on the direction corresponding to the reference direction of the subject on the fisheye image, thereby making it possible to perform segmentation processing on the fisheye image while taking into account the reference direction of the subject. For example, as shown in FIG. 9, it is possible to perform correct segmentation processing on the fisheye image. This makes it unnecessary to perform processing to expand the fisheye image into multiple images. FIG. 9 shows an example in which the fisheye image shown in FIG. 9(A) is correctly divided into labeled regions as a result of the segmentation processing on the fisheye image shown in FIG. 9(B).

[0080] In addition, a machine learning model can be used that changes the processing depending on the orientation of the subject in the image by changing the coefficients of the filter used in the convolution process based on the direction corresponding to the reference direction of the subject in the fisheye image.

[0081] <Modification> Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments. For example, in the present embodiment, the data processing units 30 and 70 are configured as shown in FIG. 5, but the present invention is not limited to this. For example, instead of the data processing units 30 and 70, data processing units 230 and 270 may be configured as shown in FIG. 10. FIG. 10 shows an example in which the data processing unit 230 of the learning device 100 includes N convolution calculation units 234 and an output integration unit 236 in each layer of the neural network, and the data processing unit 270 of the image processing device 150 includes N convolution calculation units 274 and an output integration unit 276 in each layer of the neural network.

[0082] The N convolution calculation units 234 in each layer of the data processing unit 230 of the learning device 100 perform convolution processing on an image using basis filters corresponding to each of a plurality of specific directions. For example, the first convolution calculation unit 234 performs convolution processing on the learning image based on the first basis filter for each pixel, and outputs the result to the corresponding output integration unit 236. Similarly, the Nth convolution calculation unit 234 performs convolution processing on the learning image based on the Nth basis filter for each pixel, and outputs the result to the corresponding output integration unit 236. Furthermore, the N convolution calculation units 274 in each layer of the image processing device 150 perform convolution processing on a fisheye image, similar to the convolution calculation unit 234 of the learning device 100.

[0083] The output integration unit 236 of the learning device 100 integrates the results of the convolution processing obtained from the corresponding N convolution calculation units 234 for each pixel by weighting them according to the relative angle between the reference direction of the subject of the pixel and the specific direction. For example, the output integration unit 236 integrates the results of the convolution processing as in the above formula (1) and outputs the integrated result to the next layer. Finally, based on the integration result in the last layer, the accuracy of each class in each pixel is output as the segmentation processing result. Also, the output integration unit 276 of the image processing device 150 integrates the results of the convolution processing performed by the N convolution calculation units 274 as in the output integration unit 236 of the learning device 100. By such processing, it is possible to output the same result as the configuration of FIG. 5 above. In this configuration, since the filter does not change depending on the position when performing the convolution operation, it is possible to perform calculations using a GPU or the like at high speed.

[0084] The number of basis filters used in the data processing units 30 and 70 may be three or less, or may be five or more. In this case, for example, the basis filters are arranged so that their orientations are equally spaced, and at any angle, a basis filter can be defined as the weighted sum of the two basis filters closest to that angle. By setting the number of basis filters to a multiple of four, a set of basis filters spaced at 90-degree intervals can be created, and the rotation relationship of the basis filters can be defined within the set by transposing or inverting the basis filters.

[0085] In addition, the convolution operation using the angle map may be used in all layers of the convolutional neural network, or may be used only in a part of it, such as only the first layer, only the first half, or only the second half.

[0086] In addition, the techniques described in the above embodiments relate to convolution operations and can be used for data processing other than segmentation processing, such as posture estimation and detection.

[0087] In addition, some or all of the images used for learning may be fisheye images. In this case, the angle map used in the data processing for the fisheye images in the image processing device 150 is input.

[0088] The techniques described in the above embodiments can also be applied to three-dimensional convolution. In this case, for example, six basis filters are taken for the positive and negative directions of the x, y, and z axes, and the angle map can be given as the coordinates of points on a unit sphere.

[0089] Three-dimensional convolution is applied to the analysis of time-series images such as videos. Consider a space-time in which images are arranged in the x and y directions, and the time axis is set in the z direction, and the time-series images are arranged three-dimensionally. In this case, when an object moves, the object will exist in various orientations and distortions in the space-time. The orientation of the object in this space-time can be found by calculating the shooting interval of the time-series images and the movement distance between two consecutive images. By inputting this orientation as an angle map and performing 3D convolution, it becomes possible to analyze time-series images with higher accuracy. In addition, during learning, a single still image is repeatedly input and learned, and by changing the angle map during data processing, it becomes possible to respond to time-series images.

[0090] In the present embodiment, the image is used as the target data for data processing, but the present invention is not limited thereto. For example, point cloud data measured using LiDAR (Light Detection and Ranging) may be used as the target data for data processing. For example, as in the above embodiment, by using an angle map that stores the reference direction of the subject, it is possible to identify the subject from the point cloud data from a LiDAR equipped with a fisheye lens.

[0091] Furthermore, the learning unit 24 may be provided in the image processing device 150 instead of in a learning device 100 separate from the image processing device 150 .

[0092] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]

[0093] 10, 60 Reception 20 Arithmetic section 22 Learning image generation unit 24 Learning Department 26 Model Memory Section 30, 70, 230, 270 Data processing unit 32, 72 Filter synthesis section 34, 74, 234, 274 Convolution calculation section 80 Output section 100 Learning Device 150 Image Processing Device 236, 276 Output integration section 1000 Image Processing System 1100 Fisheye camera 1200 Network

Claims

1. 1. A data processing device that processes data using a filter using a coefficient that is learned in advance using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object, a receiving unit that receives object data representing an object and a direction on the object data that corresponds to a reference direction of the object; a data processing unit that performs data processing by performing a convolution process on the target data using a modified filter obtained by modifying a coefficient of the filter based on a direction corresponding to a reference direction of the target object; Including, The data processing unit changes the coefficients of the filters corresponding to each of a plurality of specific directions, which are generated using a filter using the pre-learned coefficients, by weighting the filters according to the relative angle between the specific directions and a direction corresponding to a reference direction of the object, and performs a convolution process on the target data using the changed filters to process data.

2. 1. A data processing device that processes data using a filter using a coefficient that is learned in advance using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object, a receiving unit that receives object data representing an object and a direction on the object data that corresponds to a reference direction of the object; a data processing unit that performs data processing by performing a convolution process on the target data using a modified filter obtained by modifying a coefficient of the filter based on a direction corresponding to a reference direction of the target object; Including, the data processing unit performs a convolution process on the target data using a filter corresponding to each of a plurality of specific directions, the filter being generated using the filter using the pre-learned coefficient, to perform data processing, a data processing device that weights and integrates the results of the data processing using the filters corresponding to each of the plurality of specific directions in accordance with a relative angle between the direction corresponding to a reference direction of the object and the specific directions.

3. The target data is data in which a reference direction of an object differs for each element, The reception unit receives an angle map that stores a direction corresponding to a reference direction of the object for each of the elements; 3 . The data processing device according to claim 1 , wherein the data processing unit performs data processing by performing a convolution process on the target data using a filter in which a coefficient of the filter is changed based on the angle map.

4. The data processing unit changes a coefficient of the filter for each of the elements based on the direction; 4. The data processing apparatus according to claim 3, further comprising: a data processing unit for performing a convolution process on the target data for each of the elements using the changed filter.

5. 5. The data processing device according to claim 1, wherein the data processing unit performs a convolution process on the target data using the changed filter as a filter for at least a portion of a layer of a convolutional neural network to perform data processing.

6. 6. The data processing device according to claim 1, wherein the target data is a fisheye image taken by a fisheye camera.

7. 1. A data processing method for processing data using a filter using a coefficient that is learned in advance using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object, comprising: a receiving unit receiving object data representing an object and a direction on the object data corresponding to a reference direction of the object; A data processing unit performs data processing by performing a convolution process on the target data using a modified filter in which the coefficients of the filter are modified based on a direction corresponding to a reference direction of the target object. Including, The data processing method includes changing the coefficients of the filters corresponding to each of a plurality of specific directions, the filters being generated using the filters that use the pre-learned coefficients, by weighting the filters according to the relative angle between the specific directions and a direction corresponding to a reference direction of the object, and performing a convolution process on the target data using the changed filters to process data.

8. 1. A data processing method for processing data using a filter using a coefficient that is learned in advance using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object, comprising: a receiving unit receiving object data representing an object and a direction on the object data corresponding to a reference direction of the object; A data processing unit performs data processing by performing a convolution process on the target data using a modified filter in which the coefficients of the filter are modified based on a direction corresponding to a reference direction of the target object. Including, the data processing unit performs a convolution process on the target data using a filter corresponding to each of a plurality of specific directions, the filter being generated using the filter using the pre-learned coefficient, to perform data processing, A data processing method in which the results of the data processing using filters corresponding to each of the plurality of specific directions are weighted and integrated according to a relative angle between the direction corresponding to a reference direction of the object and the specific directions.

9. 1. A data processing program for performing data processing using a filter using coefficients that have been learned in advance using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object, comprising: Computer, a receiving unit that receives object data representing an object and a direction on the object data that corresponds to a reference direction of the object; and a data processing unit that performs data processing by performing convolution processing on the target data using a modified filter in which the coefficients of the filter are modified based on a direction corresponding to a reference direction of the target object. A data processing program for causing the device to function as a The data processing unit changes the coefficients of the filters corresponding to each of a plurality of specific directions, which are generated using a filter using the pre-learned coefficients, by weighting the filters according to the relative angle between the specific directions and a direction corresponding to a reference direction of the object, and uses the changed filters to perform a convolution process on the target data and process the data.

10. 1. A data processing program for performing data processing using a filter using coefficients that have been learned in advance using learning data representing an object and a direction on the learning data that corresponds to a reference direction of the object, comprising: Computer, a receiving unit that receives object data representing an object and a direction on the object data that corresponds to a reference direction of the object; and a data processing unit that performs data processing by performing convolution processing on the target data using a modified filter in which the coefficients of the filter are modified based on a direction corresponding to a reference direction of the target object. A data processing program for causing the device to function as a the data processing unit performs a convolution process on the target data using a filter corresponding to each of a plurality of specific directions, the filter being generated using the filter using the pre-learned coefficient, to perform data processing, A data processing program that integrates the results of the data processing using the filters corresponding to each of the plurality of specific directions by weighting them according to a relative angle between the specific direction and a direction corresponding to a reference direction of the object.

11. A learning data generation unit generates a training data set including learning data representing an object, a correct data processing result, and a direction on the learning data corresponding to a reference direction of the object; A learning unit performs convolution processing on the learning data using a filter whose coefficients are changed based on a direction corresponding to a reference direction of the object, and learns the result of the data processing so that the result of the data processing coincides with the correct data processing result. Including, When the learning unit performs the data processing, the learning model generation method changes the coefficients of the filters generated using the filters that use the coefficients, by weighting the filters corresponding to each of a plurality of specific directions according to the relative angle between the direction corresponding to the reference direction of the object and the specific directions, and performs a convolution process on the learning data using the changed filters to process the data.

12. A learning data generation unit generates a training data set including learning data representing an object, a correct data processing result, and a direction on the learning data corresponding to a reference direction of the object; A learning unit performs convolution processing on the learning data using a filter whose coefficients are changed based on a direction corresponding to a reference direction of the object, and learns the result of the data processing so that the result of the data processing coincides with the correct data processing result. Including, When the learning unit performs the data processing, the learning unit performs a convolution process on the learning data using a filter corresponding to each of a plurality of specific directions, the filter being generated using a filter using the coefficient, and performs data processing on each of the learning data; A learning model generation method in which the results of the data processing using filters corresponding to each of the multiple specific directions are weighted and integrated according to the relative angle between the direction corresponding to the reference direction of the object and the specific directions.

Images