Information processing method and information processing device

Abstract

To provide an image processing method capable of improving accuracy in converting an image from a shooting viewpoint to another viewpoint, and an image processing device.SOLUTION: An image processing method includes: acquiring image data of a first image captured from a first viewpoint output from an in-vehicle camera; inputting the image data of the first image into a trained second CNN (convolutional neural network); applying a convolution operation to the image data of the first image to identify a plurality of road markings included in the first image and generating image data in which a label for each type of the plurality of road markings is set in each pixel of the first image; inputting the image data of the first image in which a label for each type of the plurality of road markings is set in each pixel into a first CNN in which a conversion model has been constructed and applying a convolution operation to the image data of the first image to generate a second image; inputting the results of accumulating a weight map to feature quantities for each label of road markings output from an encoder layer into a decoder layer; and outputting the image data of the second image externally from the decoder layer.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing method and an image processing device. [Background technology]

[0002] Patent Document 1 discloses an object identification device that outputs a converted image obtained by converting image data into a viewpoint different from the viewpoint from which the image was taken by passing the image data through a convolutional neural network. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-21900 A Summary of the Invention [Problem to be solved by the invention]

[0004] The object identification device of Patent Document 1 converts an image from the viewpoint of a camera mounted on a vehicle (hereinafter referred to as a driving image) into a projectively transformed overhead image using a convolutional neural network and outputs the image. Even though the object identification device of Patent Document 1 uses a convolutional neural network, the output overhead image is a projectively transformed image, so the accuracy of the overhead image decreases as the distance from the vehicle increases.

[0005] The present invention has been made in view of the above problems, and its object is to provide an image processing method and an image processing device that can improve the accuracy when converting an image from a shooting viewpoint to another viewpoint. [Means for solving the problem]

[0006] An image processing method according to one aspect of the present invention applies a weight map, which indicates weights of feature quantities for each type of a plurality of road markings included in a first image captured from a first viewpoint, to image data of the first image, assigning each pixel in the first image to each pixel in a second image, and generating and outputting a second image converted from the first image to a second viewpoint different from the first viewpoint. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to improve the accuracy when converting an image from a shooting viewpoint to another viewpoint. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an image processing system including an image processing apparatus according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing the configuration of a first convolutional neural network in the image processing device according to the first embodiment. [Figure 3] FIG. 3 is a diagram showing a method for calculating a weight map in the image processing device according to the first embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of processing by the image processing device according to the first embodiment. [Figure 5] FIG. 5 is a diagram showing an example of a second image generated by applying a weight map to image data of a first image in the image processing device according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing an example of a second image generated without applying a weight map to image data of a first image in the image processing device according to the first embodiment. [Figure 7] FIG. 7 is a flowchart showing an example of processing by the image processing device according to the second embodiment. [Figure 8] FIG. 8 is a diagram for explaining a method for calculating a weight map in the image processing device according to the second and third embodiments. [Figure 9]FIG. 9 is a flowchart showing an example of the operation of the image processing device according to the third embodiment. [Figure 10] FIG. 10 is a diagram for explaining a trapezoidal convolution filter when identifying road markings in an image processing device according to an embodiment. [Figure 11] FIG. 11 is a diagram illustrating a trapezoidal convolution filter used when identifying road markings in an image processing device according to an embodiment. [Figure 12] FIG. 12 is a diagram for explaining a method for calculating a trapezoidal convolution filter. [Figure 13] FIG. 13 is a diagram for explaining a method for calculating a trapezoidal convolution filter for each region of the first image. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are designated by the same reference numerals and the description thereof will be omitted.

[0010] (First embodiment) First, a configuration example of an image processing device according to a first embodiment of the present invention will be described. Fig. 1 is a block diagram showing a configuration example of an image processing system including an image processing device 100 according to the first embodiment. As shown in Fig. 1, the image processing system includes an in-vehicle camera 1 and the image processing device 100.

[0011] The vehicle-mounted camera 1 is mounted on a vehicle and sequentially captures the external environment of the vehicle from a first capturing viewpoint (hereinafter referred to as the first viewpoint) based on its placement, and generates image data of a first image. For example, the vehicle-mounted camera 1 is placed at the front of the vehicle and captures the external environment ahead of the vehicle.

[0012] The vehicle-mounted camera 1 has an imaging element, a lens, and a circuit unit serving as a control unit. The imaging element is an element that converts light into an electrical signal through photoelectric conversion, and may be, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging element forms a rectangular area in which pixels are arranged two-dimensionally. The lens is disposed between the imaging object and the imaging element to form an image of the object on the imaging element.

[0013] The circuit unit is electrically connected to the imaging element, thereby controlling the imaging element, generating the first image as image data consisting of the detection results of each pixel, and outputting the image data of the first image to the image processing device 100 as an electrical signal.

[0014] The image processing device 100 includes an image acquisition unit 2, a training teacher image storage unit 3, an output unit 7, and a controller 8. The controller 8 is connected to the image acquisition unit 2, the training teacher image storage unit 3, and the output unit 7 via a wired or wireless communication path.

[0015] The image acquisition unit 2 acquires image data of a first image captured from a first viewpoint by the vehicle-mounted camera 1, which is output from the vehicle-mounted camera 1. The image acquisition unit 2 stores the acquired image data of the first image in the training teacher image storage unit 3. The image acquisition unit 2 also outputs the acquired image data of the first image to the controller 8.

[0016] The training teacher image storage unit 3 stores image data of the first image acquired by the image acquisition unit 2. The training teacher image storage unit 3 also stores training teacher images used for learning by the first convolutional neural network in the controller 8. The training teacher images stored in the training teacher image storage unit 3 include, for example, image data of a plurality of captured images captured from a first viewpoint and including a plurality of road markings, and image data of a plurality of converted images obtained by converting the plurality of captured images to a second viewpoint, which are ground truth data corresponding to each of the image data of the plurality of captured images. Note that each pixel constituting the plurality of captured images prepared as training teacher images is assigned a label that identifies the plurality of road markings included in each of the plurality of captured images.

[0017] The output unit 7 outputs various information generated by the controller 8. For example, the output unit 7 outputs to the outside image data of a second image generated by the controller 8 and obtained by converting the first image into a second viewpoint different from the first viewpoint.

[0018] The controller 8 is a general-purpose computer equipped with a CPU (Central Processing Unit), a memory, and an input / output unit. A computer program for causing the controller 8 to function as part of the image processing device 100 is installed in the controller 8. By executing the computer program, the controller 8 functions as multiple information processing circuits equipped in the image processing device 100.

[0019] Here, an example is shown in which the multiple information processing circuits provided in the image processing device 100 are realized by software. However, it is also possible to configure the information processing circuits by providing dedicated hardware for executing each of the information processes described below. Also, the multiple information processing circuits may be configured by individual hardware.

[0020] The controller 8 includes a model learning unit 4, a road marking recognition unit 5, and a converted image generation unit 6 as a plurality of information processing circuits.

[0021] The model learning unit 4 includes a first convolutional neural network. The model learning unit 4 uses the training image stored in the training image storage unit 3 to adjust the parameters of the first convolutional neural network and construct a transformation model for generating a second image converted from a first image captured from a first viewpoint to a second viewpoint.

[0022] The road marking recognition unit 5 identifies multiple road markings included in the first image acquired by the image acquisition unit 2 and generates image data in which labels for each type of road marking are assigned to each pixel in the first image. In this embodiment, the road marking recognition unit 5 includes a pre-trained second convolutional neural network different from the first convolutional neural network. The unit inputs image data of the first image to the second convolutional neural network, applies a convolution operation to the image data of the first image to identify multiple road markings included in the first image, and assigns labels for each type of road marking to each pixel in the first image. Types of road markings include road dividing lines, arrows indicating specific meanings, symbols, letters, and numbers. Types of road markings also include regulatory markings and directional markings.

[0023] The converted image generation unit 6 acquires image data of a first image, in which a label for each of a plurality of road marking types is set for each pixel, from the road marking recognition unit 5. The converted image generation unit 6 also acquires a first convolutional neural network in which a conversion model has been constructed, from the model learning unit 4. The converted image generation unit 6 inputs the image data of the first image, in which a label for each of a plurality of road marking types is set for each pixel, to the first convolutional neural network in which the conversion model has been constructed, and applies a convolution operation to the image data of the first image to generate a second image from the first image.

[0024] The training teacher image storage unit 3 and the model training unit 4 may be included in an external device separate from the image processing device 100. The converted image generation unit 6 may acquire a first convolutional neural network in which a conversion model trained using an external device is constructed, or may acquire a conversion model trained using an external device and apply it to a first convolutional neural network that has been stored in advance.

[0025] The configuration of the first convolutional neural network will be described with reference to Fig. 3. As shown in Fig. 3, the first convolutional neural network 20 has an encoder layer and a decoder layer.

[0026] The encoder layer has multiple (e.g., five) feature extraction units connected in series from the input side to the output side. Each feature extraction unit has multiple (e.g., two) convolution layers and a pooling layer arranged on the output side of the convolution layers, connected in series with each other. The width and height (i.e., the size of the image data matrix) of each layer belonging to the same feature extraction unit are substantially the same. On the other hand, the size becomes smaller as the feature extraction unit is located closer to the output side.

[0027] Each convolutional layer performs a convolution operation on the input data by convolving a local filter (called a kernel or feature extractor) having a predetermined size (e.g., 3x3, 5x5) determined for each convolutional layer. For example, an input value input to a position corresponding to each element of the kernel is multiplied by a weighting coefficient preset in the kernel for each element, and a linear sum of these multiplication values ​​for each element is calculated. The convolution operation is performed by adding a bias preset in the kernel to this linear sum.

[0028] The pooling layer individually corresponds to each layer and calculates local statistics of the feature map output from the upper layer, the convolutional layer connected to the input side. Specifically, a window of a predetermined size (e.g., 2x2 or 3x3) corresponding to the position of the upper layer is set, and local statistics are calculated from the input values ​​within the window. For example, the maximum value can be used as the local statistics. In other words, by using the maximum value of the input values ​​within the window as the pooling result, the input values ​​within the window can be integrated. Unlike the kernels set in the convolutional operation, the windows set in this pooling operation often do not overlap with each other (although they may overlap partially). The size of the feature map output from the pooling layer is reduced according to the size of the window. This is called downsampling.

[0029] The structure of the encoder layer and the processing performed by the encoder layer are the same as those of an encoder layer in a general convolutional neural network, and therefore a detailed description thereof will be omitted. In this embodiment, the encoder layer receives image data of a first image 40, in which a label for each of a plurality of road marking types is set for each pixel, and outputs a feature map representing the feature amounts for each of the plurality of road marking types (labels). The converted image generation unit 6 multiplies the feature map representing the feature amounts for each of the plurality of road marking types (labels), output from the encoder layer, by the weight map 23, and inputs the result to the decoder layer. A method for calculating the weight map 23 in this embodiment will be described with reference to FIG. 3.

[0030] In this embodiment, the converted image generation unit 6 calculates a weight map 23 that represents the weights of the feature quantities for each of the multiple road marking types included in the first image 40, based on the feature map 21 output from the encoder layer. As shown in FIG. 3 , the feature map 21 output from the final layer of the encoder layer has maps (matrices) that represent the feature quantities for each of the multiple road marking types included in the first image 40, arranged in rows equal to the number i of road marking types. Note that the multiple maps that represent the feature quantities for each of the multiple road marking types all have the same size. In the following description, the direction D1 in which the maps that represent the feature quantities for each of the multiple road marking types included in the first image 40 are arranged is referred to as the channel direction D1 of the feature map 21.

[0031] The converted image generation unit 6 calculates the weight map 23, for example, by using at least the maximum, average, and minimum values ​​in the channel direction D1 of the feature map 21. Specifically, the converted image generation unit 6 performs a maximum value pooling operation on the feature map 21 to extract the maximum value in the channel direction D1 (maximum value feature), an average value pooling operation on the feature map 21 to extract the average value in the channel direction D1 (average value feature), and a minimum value pooling operation on the feature map 21 to extract the minimum value in the channel direction D1 (minimum value feature). As a result, the number of maps in the feature map 21 that represent the feature amounts for each type of multiple road markings included in the first image 40 is reduced from i to 3.

[0032] The converted image generating unit 6 may calculate the weight map 23 using at least the upper limit, average, and lower limit of the 3σ variance in the channel direction D1 of the feature map 21.

[0033] The converted image generation unit 6 overlays three maps representing the maximum value feature, average value feature, and minimum value feature extracted by each pooling calculation to generate a feature map 22 representing the feature amounts of multiple road markings included in the first image 40.

[0034] Furthermore, the transformed image generation unit 6 calculates a weight map 23 by performing a convolution operation on the feature map 22 using an odd-numbered size convolution filter. The transformed image generation unit 6 calculates the weight map 23 by, for example, performing an operation of convolving a 7×7×3 median filter on the feature map 22. Thereafter, the transformed image generation unit 6 multiplies the calculated weight map 23 by the feature map 21 output from the encoder layer, and inputs the result to the decoder layer of the first convolutional neural network 20.

[0035] The decoder layer receives the result of multiplying the weight map 23 by the feature map 21, which is output from the encoder layer and indicates the feature amounts for each type (label) of a plurality of road markings, and outputs image data of the second image.

[0036] The decoder layer has multiple classification units (e.g., five, the same number as the feature extraction units) connected in series from the input side to the output side, and a softmax layer at the output side. Each classification unit has a depooling layer and a deconvolution layer arranged on the output side of the depooling layer, connected in series with each other.

[0037] Each identification unit is configured to form a pair with a corresponding feature extraction unit. The nth (n is a natural number) feature extraction unit counting from the feature extraction unit closest to the input side to the output side and the nth identification unit counting from the identification unit closest to the output side to the input side are paired and individually associated one-to-one.

[0038] In this way, each inverse pooling layer is individually associated with each pooling layer of the feature extraction unit in a one-to-one correspondence, and the associated pairs are substantially the same size. The inverse pooling layer re-enlarges the size of the feature map downsampled in the encoder layer. That is, in this embodiment, the inverse pooling layer re-enlarges the size of the feature map obtained by multiplying the feature map 21 output from the encoder layer by the calculated weight map 23. This is called upsampling.

[0039] The deconvolution layer performs deconvolution (also called transposed convolution) on the input feature map. Deconvolution is a process of restoring the feature map to its pre-convolution state, assuming that the input feature map is the result of a convolution using a specific kernel. This process can generate an output feature map by multiplying the input feature map by the transpose of a matrix generated from a specific kernel.

[0040] The structure of the decoder layer and the processing by the decoder layer are the same as those of a decoder layer in a general convolutional neural network, so detailed description will be omitted. In this embodiment, the feature map output from the decoder layer is a viewpoint conversion map in which road markings are identified in a second image obtained by converting a first image captured from a first viewpoint to a second viewpoint. In other words, the viewpoint conversion map output via the second convolutional neural network is image data of the second image obtained by converting the first image to the second viewpoint.

[0041] With the above configuration, the converted image generation unit 6 applies a weight map representing the weights of the feature amounts for each type of multiple road markings included in the first image to the image data of the first image, and assigns each pixel in the first image to each pixel of the second image to generate the second image.

[0042] The converted image generating unit 6 may store the generated image data of the second image as correct answer data for the image data of the first image in the training teacher image storage unit 3. The model training unit 4 may retrain the conversion model in the first convolutional neural network using the image data of the first image and the image data of the second image.

[0043] Next, an example of processing by the image processing device 100 will be described with reference to Fig. 4. In step S21, the image acquisition unit 2 acquires image data of a first image captured by the vehicle-mounted camera 1 from a first viewpoint, which is output from the vehicle-mounted camera 1.

[0044] The process proceeds to step S22, where the road marking recognition unit 5 inputs the image data of the first image to the trained second convolutional neural network (second CNN). The process proceeds to step S23, where the road marking recognition unit 5 applies a convolution operation to the image data of the first image to identify the multiple road markings included in the first image, and generates image data in which labels for each type of the multiple road markings are set for each pixel in the first image.

[0045] The process proceeds to step S24, where the converted image generation unit 6 inputs the image data of the first image, in which labels for each of the multiple road marking types are set for each pixel, to the first convolutional neural network (first CNN) in which the conversion model is constructed. In this embodiment, the image data of the first image, in which labels for each of the multiple road marking types are set for each pixel, is input to the encoder layer of the first convolutional neural network in which the conversion model is constructed.

[0046] The process proceeds to step S25, where the transformed image generation unit 6 applies a convolution operation to the image data of the first image in the first convolutional neural network to generate a second image from the first image. That is, in the first convolutional neural network, the encoder layer outputs feature quantities for each label of a plurality of road markings. The transformed image generation unit 6 inputs the result of multiplying the feature quantities for each label of the road markings output from the encoder layer by a weight map to the decoder layer. The decoder layer then outputs the image data of the second image.

[0047] The process proceeds to step S26, the output unit 7 outputs the image data of the second image to the outside, and the image processing device 100 ends the process of FIG.

[0048] Here, in this embodiment, the effect of applying a weight map to image data of a first image will be described with reference to Figures 5 and 6. In Figures 5 and 6, it is assumed that image data of a first image, which is captured from a first viewpoint by an on-board camera 1 disposed at the front of a vehicle and captures an image of the area ahead of the vehicle, and which includes an intersection ahead, is input to a first convolutional neural network. It is also assumed that a transformation model is constructed in the first convolutional neural network so as to generate a second image by transforming the first image from the first viewpoint to a second viewpoint, which is a bird's-eye viewpoint.

[0049] FIG. 5 shows an example of a second image generated by applying a weight map to image data of a first image in the image processing device 100. That is, in FIG. 5, the transformed image generation unit 6 multiplies the weight map by the feature map output from the encoder layer in the first convolutional neural network, and inputs the result to the decoder layer. On the other hand, FIG. 6 shows an example of a second image generated by the image processing device 100 without applying a weight map to the image data of the first image. That is, in FIG. 6, the transformed image generation unit 6 does not multiply the weight map by the feature map output from the encoder layer in the first convolutional neural network, but inputs the feature map output from the encoder layer directly to the decoder layer. In this case, FIG. 5, in which a weight map is applied to the feature map, can generate a second image that more accurately estimates information about the road markings at the intersection than FIG. 6, in which a weight map is not applied.

[0050] (Operation and effect of the first embodiment) As described above, according to the first embodiment, the following advantageous effects can be obtained.

[0051] The image processing method according to the first embodiment generates and outputs a second image by converting a first image captured from a first viewpoint into a second viewpoint different from the first viewpoint. In the image processing method according to the first embodiment, a weight map representing weights of feature amounts for each type of multiple road markings included in the first image 40 is applied to the image data of the first image, and each pixel in the first image is assigned to each pixel of the second image to generate the second image.

[0052] According to the image processing method of the first embodiment, by performing viewpoint conversion by applying weights of feature amounts for each label (type) of road markings included in the first image, the viewpoint of the image can be converted without losing information about the road markings in the first image, thereby improving the accuracy when converting an image from one shooting viewpoint to another.

[0053] Furthermore, the image processing method according to the first embodiment identifies a plurality of road markings included in a first image and assigns a label for each type of the plurality of road markings to each pixel in the first image. Then, image data of the first image is input to a first convolutional neural network, and a convolution operation is applied to the image data of the first image to generate and output a second image from the first image. The first convolutional neural network has an encoder layer that receives the image data of the first image, in which a label is assigned to each pixel, and outputs features for each label, and a decoder layer that receives the result of multiplying the features for each label output from the encoder layer by a weight map, and outputs image data of the second image.

[0054] By multiplying the output of the final layer of the encoder layer, which contains features for each label of multiple road markings, by a weight map that represents the weights of the feature amounts for each label of multiple road markings, the first convolutional neural network can determine the positions that should be noted in the first image and can primarily learn important information in the positions that should be noted. This makes it possible to create a convolutional neural network that is appropriate for converting the first image from the first viewpoint to the second viewpoint.

[0055] In the image processing method according to the first embodiment, the weight map is calculated based on the output of the encoder layer. By calculating the weight map based on the output of the final layer of the encoder layer, it is possible to calculate a weight map that represents information correlated with features of the first image using a smaller amount of information than the original first image.

[0056] Furthermore, in the image processing method according to the first embodiment, the weight map is calculated using at least the maximum, average, and minimum values ​​in the channel direction of the output of the encoder layer, which makes it possible to easily calculate the weight map representing the feature amount for each label of the road markings included in the first image.

[0057] In the image processing method according to the first embodiment, the weight map may be calculated using at least the upper limit, average, and lower limit of the 3σ variance in the channel direction of the output of the encoder layer. In this case, the feature amount for each label of the road markings included in the first image 40 can be calculated more accurately.

[0058] Furthermore, in the image processing method according to the first embodiment, a weight map is calculated by performing a convolution operation on the output of the encoder layer using a convolution filter of odd size. By performing the convolution operation, a weight map with higher accuracy can be calculated.

[0059] (Second embodiment) A second embodiment to which the present invention is applied will be described below with reference to the drawings. In the drawings, the same parts are given the same reference numerals and detailed description will be omitted. The image processing device according to the second embodiment differs from the image processing device 100 according to the first embodiment in the method of calculating the weight map and the method of applying the weight map to image data of the first image in the converted image generation unit 6. The other configurations are the same as those of the image processing device 100 shown in FIG. 1. Therefore, the different parts will be mainly described, and a repeated description of the same parts will be omitted.

[0060] A method for calculating a weight map and a method for applying a weight map according to the second embodiment will be described with reference to Fig. 7 and Fig. 8. In step S31 of Fig. 7, the converted image generation unit 6 acquires, from the training teacher image storage unit 3, image data of a plurality of captured images 30 including a plurality of road markings stored in the training teacher image storage unit 3.

[0061] 8, the converted image generation unit 6 identifies a plurality of road markings included in each of the plurality of captured images 30 from the plurality of captured images 30, and generates a plurality of road marking images 41-4i for each of the plurality of road marking types. For example, the converted image generation unit 6 inputs each of the image data of the plurality of captured images 30 to a pre-trained second convolutional neural network, and applies a convolution operation to each of the image data of the plurality of captured images 30 to identify the plurality of road markings included in each of the plurality of captured images 30, and generates a plurality of road marking images 41-4i for each of the plurality of road marking types. Here, i corresponds to the number of the plurality of road marking types.

[0062] The process proceeds to step S32, where the converted image generation unit 6 converts the image data of the multiple road marking images 41-4i into first matrices C1-Ci. The process proceeds to step S33, where the converted image generation unit 6 performs principal component analysis on the converted first matrices C1-Ci for each of the multiple road marking types to calculate first principal components A1-Ai.

[0063] The process proceeds to step S34, where the converted image generation unit 6 calculates a weight map based on the first principal components A1 to Ai. The converted image generation unit 6 uses, for example, a matrix in which the first principal components A1 to Ai are arranged as the weight map. Thereafter, the image processing device according to this embodiment ends the process of FIG. 7.

[0064] In this embodiment, the converted image generation unit 6 multiplies the image data of the first image by a weight map before inputting the image data of the first image into the first convolutional neural network, and inputs the result of multiplying the image data of the first image by the weight map into the first convolutional neural network to generate a second image from the first image.

[0065] (Operation and effect of the second embodiment) As described above, according to the second embodiment, the following advantageous effects can be obtained.

[0066] The image processing method according to the second embodiment prepares image data of a plurality of captured images 30 that are captured from a first viewpoint and include a plurality of road markings, identifies the plurality of road markings included in each of the plurality of captured images 30, and generates a plurality of road marking images 41-4i for each of the plurality of road marking types. The image data of the plurality of road marking images 41-4i is converted into first matrices C1-Ci, and a weight map 23 is calculated based on first principal components A1-Ai obtained by performing principal component analysis on the converted first matrices C1-Ci for each of the plurality of road marking types.

[0067] According to the image processing method of the second embodiment, a weight map can be calculated by simple calculation from image data of a plurality of captured images 30 including a plurality of road markings that have been prepared in advance, thereby reducing the processing load on the image processing device.

[0068] (Third embodiment) A third embodiment to which the present invention is applied will be described below with reference to the drawings. In the drawings, the same parts are given the same reference numerals and detailed description will be omitted. The image processing device according to the third embodiment differs from the image processing device 100 according to the first embodiment in the method of calculating the weight map in the converted image generation unit 6 and the method of applying the weight map to the image data of the first image. The other configurations are the same as those of the image processing device 100 shown in FIG. 1. Therefore, the different parts will be mainly described, and a repeated description of the same parts will be omitted.

[0069] A method for calculating a weight map and a method for applying a weight map according to the third embodiment will be described with reference to Fig. 8 and Fig. 9. The processing of steps S41 to S43 in Fig. 9 is the same as the processing of steps S31 to S33 in Fig. 7, and therefore a description thereof will be omitted here.

[0070] 9, the converted image generation unit 6 calculates a second matrix in which the first principal components A1 to Ai are arranged. The process proceeds to step S45, where the converted image generation unit 6 acquires, from the training teacher image storage unit 3, image data of a plurality of converted images obtained by converting the plurality of captured images 30 to a second viewpoint, which are correct data corresponding to each of the image data of the plurality of captured images 30 stored in the training teacher image storage unit 3.

[0071] The converted image generation unit 6 identifies the road markings included in each of the converted images and generates road marking converted images 51-5i for each type of road marking. Although not shown, the road marking converted images 51-5i are images obtained by converting the road marking images 41-4i from a second viewpoint.

[0072] 8 inputs each of the image data of the plurality of converted images into a pre-trained second convolutional neural network, applies a convolution operation to each of the image data of the plurality of converted images, identifies the plurality of road markings included in each of the plurality of converted images, and generates a plurality of road marking converted images 51 to 5i for each of the plurality of road marking types, where i corresponds to the number of the plurality of road marking types.

[0073] The process proceeds to step S46, where the converted image generation unit 6 converts the image data of the plurality of road marking converted images 51-5i into third matrices c1-ci. The process proceeds to step S47, where the converted image generation unit 6 performs principal component analysis on the converted third matrices c1-ci for each of the plurality of road marking types to calculate first principal components a1-ai.

[0074] The process proceeds to step S48, where the converted image generation unit 6 calculates a fourth matrix in which the first principal components a1 to ai are arranged. The process proceeds to step S49, where the converted image generation unit 6 calculates a weight map based on the second matrix and the fourth matrix. The converted image generation unit 6 calculates a transformation matrix Mi using, for example, the following equation (1), and uses the calculated transformation matrix Mi as the weight map. Thereafter, the image processing device according to this embodiment ends the process of FIG. 7. MiAi=ai (1)

[0075] In this embodiment, the converted image generation unit 6 inputs image data of the first image into a pre-trained second convolutional neural network and applies a convolution operation to each of the image data of the multiple converted images to identify the multiple road markings included in the first image and generate first road marking images P1-Pi for each type of road marking.The converted image generation unit 6 then multiplies the image data of the first road marking images P1-Pi by the corresponding transformation matrices M1-Mi to generate first road marking converted images p1-pi by transforming the first road marking images P1-Pi to a second viewpoint.The converted image generation unit 6 then generates a second image by overlaying the image data of the first road marking converted images p1-pi.That is, in the third embodiment, the second image is generated from the first image without using a first convolutional neural network.

[0076] (Operation and effect of the third embodiment) As described above, according to the third embodiment, the following advantageous effects can be obtained.

[0077] An image processing method according to the third embodiment prepares image data of a plurality of captured images 30 captured from a first viewpoint and including a plurality of road markings, identifies the plurality of road markings included in each of the plurality of captured images 30, and generates a plurality of road marking images 41-4i for each of the plurality of road marking types. Then, the image data of the plurality of road marking images 41-4i is converted into first matrices C1-Ci, and the converted first matrices C1-Ci are subjected to principal component analysis for each of the plurality of road marking types to calculate a second matrix in which first principal components A1-Ai are arranged. The image processing method according to the third embodiment also prepares image data of a plurality of converted images obtained by converting the plurality of captured images to a second viewpoint, which are ground truth data corresponding to each of the image data of the plurality of captured images 30, and identifies the plurality of road markings included in each of the converted images to generate a plurality of road marking converted images 51-5i for each of the plurality of road marking types. Then, the image data of the plurality of road surface marking converted images 51-5i are converted into third matrices c1-ci, and the converted third matrices c1-ci are subjected to principal component analysis for each of the plurality of road surface marking types to calculate a fourth matrix in which first principal components a1-ai are arranged. After that, a weight map is calculated based on the second matrix and the fourth matrix.

[0078] According to the image processing method of the third embodiment, a weight map can be calculated by simple calculation from a plurality of captured images 30 captured from a first viewpoint and a plurality of converted images corresponding to each of the captured images 30, each of which is converted to a second viewpoint.

[0079] Although the embodiments of the present invention have been described above, the descriptions and drawings that form part of this disclosure should not be understood to limit the present invention. Various alternative embodiments, examples, and operating techniques will become apparent to those skilled in the art from this disclosure.

[0080] For example, in the image processing method according to the first or second embodiment, the image data of the first image may be converted into image data of the second viewpoint by projective transformation before inputting the image data of the first image into the first convolutional neural network. By converting the first image into an image of the second viewpoint by projective transformation before inputting the first image into the first convolutional neural network, the inference time by the first convolutional neural network when converting the first image from the first viewpoint to the second viewpoint can be shortened and the inference accuracy can be improved.

[0081] Furthermore, in the first to third embodiments, for example, as shown in FIG. 10 , if the first viewpoint from which first image 71 was captured is a viewpoint having a line of sight direction inclined with respect to a vertically downward vector (hereinafter referred to as a projective viewpoint), when image data of first image 71 from the first viewpoint is input to a second convolutional neural network and a rectangular convolution filter 72 is used for the convolution operation when identifying road markings included in first image 71, an area 74 unnecessary for recognizing road markings may be included in convolution filter 72, as shown in FIG. 11 .

[0082] Therefore, in the image processing methods according to the first to third embodiments, when the first viewpoint is a viewpoint having a line of sight direction inclined with respect to a vertically downward vector, a trapezoidal convolution filter may be calculated based on the imaging direction when the first image is captured, and the trapezoidal convolution filter may be used in the convolution calculation in the second convolutional neural network to identify a plurality of road markings included in the first image.

[0083] 12, for example, a viewpoint transformation matrix for the entire image may be calculated based on a projection image Q captured in the same imaging direction as when a first image 71 was captured in advance, and a transformed image P obtained by transforming the projection image Q from a second viewpoint (a bird's-eye viewpoint in FIG. 12), which is ground truth data corresponding to the projection image Q. Then, based on the calculated viewpoint transformation matrix, a trapezoidal convolution filter 73 suitable for identifying road markings may be calculated by inputting image data of the first image 71 from the first viewpoint into a second convolutional neural network. In FIG. 12, the X' direction in the transformed image P and the X direction in the projection image Q correspond to the width direction of the road marking, and the Y' direction in the transformed image P and the Y direction in the projection image Q correspond to the length direction of the road marking. In FIG. 12, the origin of the transformed image P is O', and the origin of the projection image Q is O.

[0084] In this case, as shown in FIG. 12, four feature points A', B', C', and D' in the converted image P and four feature points A, B, C, and D in the projected image Q corresponding to the feature points A', B', C', and D' are set, and the viewpoint transformation matrix H0 for the entire image can be calculated using the following equations (2) to (4). P=(A',B',C',D') t ···(2) Q=(A,B,C,D) t ···(3) Q=H0P (4)

[0085] 10 is then transformed by the viewpoint transformation matrix H0 of the entire image calculated using equations (2) to (4), thereby calculating a trapezoidal convolution filter 73. As a result, when the trapezoidal convolution filter 73 is used in the second neural network, more unnecessary information can be omitted than when the rectangular convolution filter 72 is used, and the accuracy of identifying road markings can be improved.

[0086] Furthermore, in the image processing methods according to the first to third embodiments, the area of ​​the first image may be divided, a different trapezoidal convolution filter may be calculated for each divided area, and in the convolution calculation in the second convolutional neural network, the plurality of road markings included in the first image may be identified using the trapezoidal convolution filter for each divided area.

[0087] For example, as shown in Fig. 13, converted image P is divided into four regions, and four frames P1-P4, each with the same character size as a default road marking (width 0.8 m, length 2.4 m), are placed within each region. Then, frames P1-P4 are multiplied by the viewpoint transformation matrix H0 for the entire image calculated based on equations (2)-(4) to calculate the shapes of frames Q1-Q4 in projected image Q corresponding to each of frames P1-P4. Note that in Fig. 13, the X' direction in converted image P and the X direction in projected image Q are the width direction of the road marking, and the Y' direction in converted image P and the Y direction in projected image Q are the length direction of the road marking. Also, in Fig. 13, the origin in converted image P is O', and the origin in projected image Q is O.

[0088] Based on the relationship between the frames P1 to P4 and the frames Q1 to Q4 corresponding to the frames P1 to P4, the following formulas (5) to (7) are used to calculate the frame P n Frame Q n A viewpoint transformation matrix H that transforms n Here, n is a natural number, and in the example of FIG. P n =(A n ',B n ',C n ',D n ') t ···(5) Q n =(A n ,B n ,C n ,D n ) t ···(6) Q n =H n P n ···(7)

[0089] Then, the first image 71 in FIG. 10 is divided into four regions in the same manner as in FIG. 13, and the convolution filter for each region is calculated using the viewpoint transformation matrix H n Four trapezoidal convolution filters are calculated using the following formula:

[0090] In the second neural network, a trapezoidal convolution filter suitable for each region obtained by dividing the first image can be calculated, thereby further improving the accuracy of identifying road markings. do. [Explanation of symbols]

[0091] 100 Image processing device

Claims

1. 1. An image processing method for an image processing device that generates and outputs a second image by converting a first image captured from a first viewpoint into a second viewpoint different from the first viewpoint, the method comprising: A weight map representing weights of feature amounts for each type of a plurality of road markings included in the first image is applied to the image data of the first image, and each pixel in the first image is assigned to each pixel in the second image to generate the second image. Image processing methods.

2. inputting image data of the first image into a first convolutional neural network, and applying a convolution operation to the image data of the first image to generate and output the second image from the first image; The image processing method according to claim 1 .

3. Identifying the plurality of road markings included in the first image and setting labels for each type of the plurality of road markings to each pixel in the first image. The image processing method according to claim 2 .

4. The first convolutional neural network an encoder layer that receives image data of the first image in which the label is set for each pixel and outputs a feature amount for each label; a decoder layer that receives as input a result of multiplying the feature amount for each label output from the encoder layer by the weight map and outputs image data of the second image; The image processing method according to claim 3 .

5. Calculating the weight map based on the output of the encoder layer The image processing method according to claim 4.

6. The weight map is calculated using at least the maximum, average, and minimum values ​​in the channel direction of the output of the encoder layer. The image processing method according to claim 5 .

7. The weight map is calculated using at least the upper limit, the average value, and the lower limit value of the 3σ variance in the channel direction of the output of the encoder layer. The image processing method according to claim 5 .

8. To calculate the weight map, a convolution operation is performed on the output of the encoder layer using an odd-size convolution filter.

8. The image processing method according to claim 6 or 7.

9. preparing image data of a plurality of captured images captured from the first viewpoint and including a plurality of road markings; identifying the plurality of road markings included in each of the plurality of captured images and generating a plurality of road marking images for each type of the plurality of road markings; converting image data of the plurality of road marking images into a first matrix; The weight map is calculated based on first principal components obtained by performing principal component analysis on the transformed first matrix for each of the plurality of road marking types. The image processing method according to claim 2 .

10. Before inputting the image data of the first image into the first convolutional neural network, the image data of the first image is transformed into image data of the second viewpoint by projective transformation. The image processing method according to any one of claims 2 to 7.

11. preparing image data of a plurality of captured images captured from the first viewpoint and including a plurality of road markings; identifying the plurality of road markings included in each of the plurality of captured images and generating a plurality of road marking images for each type of the plurality of road markings; converting image data of the plurality of road marking images into a first matrix; calculating a second matrix in which first principal components obtained by performing principal component analysis on the transformed first matrix for each of the plurality of road marking types are arranged; further preparing image data of a plurality of converted images obtained by converting the plurality of captured images to a second viewpoint, the converted image data being correct data corresponding to each of the image data of the plurality of captured images; identifying the plurality of road markings included in each of the plurality of converted images and generating a plurality of road marking converted images for each type of the plurality of road markings; converting the image data of the plurality of road marking converted images into a third matrix; calculating a fourth matrix in which first principal components obtained by performing principal component analysis on the transformed third matrix for each of the plurality of road marking types are arranged; Calculating the weight map based on the second matrix and the fourth matrix The image processing method according to claim 1 .

12. inputting image data of the first image into a second convolutional neural network, applying a convolution operation to the image data of the first image to identify the plurality of road markings included in the first image, and setting the labels for each type of the plurality of road markings to each pixel in the first image; The image processing method according to claim 3 .

13. When the first viewpoint is a viewpoint having a line of sight direction inclined with respect to a vertically downward vector, a trapezoidal convolution filter is calculated based on an imaging direction when the first image is captured; In a convolution operation in the second convolutional neural network, the trapezoidal convolution filter is used to identify the plurality of road markings included in the first image. The image processing method according to claim 12.

14. Dividing an area of ​​the first image and calculating a different trapezoidal convolution filter for each divided area; In the convolution operation in the second convolutional neural network, the plurality of road markings included in the first image are identified using the trapezoidal convolution filter for each of the divided regions. The image processing method according to claim 13.

15. An image processing device including an acquisition unit and a controller, the acquisition unit acquires image data of a first image captured from a first viewpoint; The controller a weight map representing weights of feature amounts for each type of road marking included in the first image is applied to image data of the first image, and each pixel in the first image is assigned to each pixel in a second image, thereby generating the second image obtained by converting the first image to a second viewpoint different from the first viewpoint; Image processing device.

Images