Image processing device, recording medium, and image processing method

Abstract

The present disclosure provides an image processing device, a recording medium, and an image processing method that can improve the inference accuracy without preparing a large amount of training data. The image processing device has a processor with hardware, and the processor performs the following processing, namely: using a pre-learned recognizer to infer a semantic label for each pixel of an input image, thereby generating a semantic label image; inferring the original image based on the semantic label image, thereby generating a restored image; calculating a first difference between the input image and the restored image; based on the first difference, updating the inference parameter when inferring the semantic label or the inference parameter when inferring the original image.

Description

Technical Field

[0001] The present disclosure relates to an image processing device, an image processing program, and an image processing method. Background Art

[0002] Patent document 1 discloses a technology that improves the inference accuracy of semantic labels by inferring semantic labels based on input images, creating training data (correctly labeled images) based on the difficulty of inferring semantic labels, and learning the training data.

[0003] Prior Art Literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-194912 Summary of the invention

[0006] Problems to be solved by the invention

[0007] In the technology of Patent Document 1, in order to ensure accuracy in a wide range of scenes, it is necessary to produce training data for a large number of images. Usually, the production of training data requires a high cost. Therefore, a technology is sought that can improve the inference accuracy without preparing a large amount of training data.

[0008] The present disclosure is an invention made in view of the above circumstances, and an object of the present invention is to provide an image processing device, an image processing program, and an image processing method that can improve the estimation accuracy without preparing a large amount of training data.

[0009] Means for solving problems

[0010] The image processing device involved in the present disclosure has a processor with hardware, wherein the processor performs the following processing, namely: generating a semantic label image by inferring a semantic label for each pixel of an input image by using a pre-learned identifier; generating a restored image by inferring the original image based on the semantic label image; calculating a first difference between the input image and the restored image; and updating the inference parameters when inferring the semantic label or the inference parameters when inferring the original image based on the first difference.

[0011] The image processing program involved in the present disclosure enables a processor with hardware to perform the following processing, namely: generating a semantically labeled image by inferring a semantic label for each pixel of an input image using a pre-learned identifier; generating a restored image by inferring the original image based on the semantically labeled image; calculating a first difference between the input image and the restored image; and updating the inference parameters when inferring the semantic label or the inference parameters when inferring the original image based on the first difference.

[0012] In the image processing method involved in the present disclosure, a processor with hardware performs the following processing, namely: generating a semantic label image by inferring a semantic label for each pixel of an input image by using a pre-learned identifier; generating a restored image by inferring the original image based on the semantic label image; calculating a first difference between the input image and the restored image; and updating the inference parameters when inferring the semantic label or the inference parameters when inferring the original image based on the first difference.

[0013] Effects of the Invention

[0014] According to the present disclosure, it is possible to improve inference accuracy without creating a large amount of training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a block diagram showing the structure of the image processing device according to the first embodiment.

[0016] Figure 2 It is a block diagram showing the structure of an image processing device according to a second embodiment.

[0017] Figure 3 It is a block diagram showing the structure of an image processing device according to a third embodiment.

[0018] Figure 4 It is a block diagram showing the structure of an image processing device according to a fourth embodiment.

[0019] Figure 5 It is a block diagram showing the structure of an image processing device according to a fifth embodiment.

[0020] Figure 6 It is a block diagram showing the structure of an image processing device according to a sixth embodiment.

[0021] Figure 7 It is a block diagram showing the structure of an image processing device according to the seventh embodiment.

[0022] Figure 8It is a block diagram showing the structure of an image processing device involved in the eighth embodiment.

[0023] Fig. 9 It is a block diagram showing the structure of an image processing device involved in the ninth embodiment. DETAILED DESCRIPTION

[0024] The image processing device, image processing program, and image processing method according to the embodiments of the present disclosure are described with reference to the accompanying drawings. In addition, the structural elements in the following embodiments include structural elements that can be easily replaced by those skilled in the art, or substantially the same structural elements.

[0025] The image processing device involved in the present disclosure is a device for performing semantic segmentation processing (Semantic Segmentation) on an input image (hereinafter referred to as "input image"). Each embodiment of the image processing device described below is realized by, for example, a processor composed of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field-Programmable Gate Array), a memory (main storage device, auxiliary storage device) composed of a RAM (Random Access Memory), a ROM (Read Only Memory), etc., and a communication unit (communication interface) and other general-purpose computers such as workstations or personal computers.

[0026] In addition, each part of the image processing device can be implemented by the function of a single computer, or can be implemented by the functions of multiple computers with different functions. In addition, although the following example of applying the image processing device to the vehicle field is described, as long as the image processing device is in a field that requires semantic region segmentation processing, it can also be widely used in fields other than vehicles.

[0027] (First Embodiment)

[0028] Reference Figure 1 Next, an image processing device 1 according to the first embodiment will be described. The image processing device 1 includes a semantic label estimation unit 11 , an original image estimation unit 12 , a difference calculation unit 13 , and a parameter update unit 14 .

[0029] The semantic label inference unit 11 generates a semantic label image by inferring a semantic label for each pixel of the input image using a pre-learned recognizer and learned parameters. Specifically, the semantic label inference unit 11 infers a semantic label for each pixel of the input image using a recognizer and learned parameters, and assigns the semantic label. Thus, the semantic label inference unit 11 converts the input image into a semantic label image, and outputs the semantic label image to the original image inference unit 12. In addition, the input image input to the semantic label inference unit 11 can be, for example, an image captured by a vehicle-mounted camera mounted on a vehicle, or can be an image that is actually captured.

[0030] The semantic label inference unit 11 is constructed, for example, as a network in which elements such as convolution layers, activation layers (ReLU Layer, Softmax Layer, etc.), pooling layers, and upsampling layers are stacked into a multi-layered network using a method based on deep learning (especially CNN (Convolutional Neural Network)). In addition, as a learning method for the identifier and learning completion parameters used by the semantic label inference unit 11, for example, a method based on CRF (Conditional random field), a method combining deep learning and CRF (Conditional random field), a method of performing inference in real time using multiple resolution images, etc. can be listed.

[0031] The original image inference unit 12 generates a restored image by inferring the original image using the previously learned recognizer and the learned parameters based on the semantic label image generated by the semantic label inference unit 11. Specifically, the original image inference unit 12 restores the original image based on the semantic label image using the recognizer and the learned parameters. Thus, the original image inference unit 12 converts the semantic label image into a restored image and outputs the restored image to the difference calculation unit 13.

[0032] The original image inference unit 12 is constructed, for example, as a network that stacks elements such as convolution layers, activation layers (ReLU Layer, Softmax Layer, etc.), pooling layers, and upsampling layers into a multi-layered structure using a method based on deep learning (especially CNN (Convolutional Neural Network)). In addition, as a learning method for the recognizer and learning completion parameters used by the original image inference unit 12, for example, a method based on CRN (Cascaded Refinement Network), a method based on Pix2PixHD, etc. can be cited.

[0033] The difference calculation unit 13 calculates the difference (first difference) between the input image and the restored image generated by the original image estimation unit 12, and outputs the calculation result to the parameter updating unit 14. For example, the difference calculation unit 13 may calculate a simple difference (I(x, y)-P(x, y)) for each pixel with respect to the image information I(x, y) of the input image and the image information P(x, y) of the restored image. In addition, the difference calculation unit 13 may calculate the correlation of each pixel with respect to the image information I(x, y) of the input image and the image information P(x, y) of the restored image by the following formula (1).

[0034] Mathematical formula 1

[0035] ||I(x, y)-P(x, y)||n(n=1 or 2) ...(1)

[0036] In addition, the difference calculation unit 13 may also perform a difference comparison after implementing a predetermined image transformation F(·) for the image information I(x, y) of the input image and the image information P(x, y) of the restored image. That is, the difference calculation unit 13 may also calculate "F(I(x, y))-F(P(x, y))". In addition, as the image transformation F(·), for example, the "perceptual loss" output by the hidden layer of a deep learner (such as vgg16, vgg19, etc.) can be listed. In addition, when using any of the above methods, the difference calculated by the difference calculation unit 13 is also output as an image. Moreover, in the present disclosure, the image representing the difference calculated by the difference calculation unit 13 is defined as a "reconstructed error image".

[0037] The parameter updating unit 14 updates the estimation parameters used by the semantic label estimation unit 11 when estimating the semantic label from the input image, based on the difference (reconstructed error image) calculated by the difference calculating unit 13 .

[0038] Here, in Figure 1 In the figure, an example of an input image is shown at the upper left, an example of a semantically labeled image is shown at the upper right, an example of a restored image is shown at the lower left, and an example of a reconstructed error image is shown at the lower right. As shown in part A of the input image, for example, a warning sign is projected at the lower right of the input image. In this case, in the semantic label inference unit 11, when learning of an image (correctly labeled image) containing the warning sign is not performed, label inference loss may occur for the portion of the warning sign (refer to Figure 1 Moreover, when such label inference loss occurs, restoration loss also occurs in the restored image generated by the original image inference unit 12 (refer to the lower right part of the restored image in the figure), and as a result, the reconstruction error of the reconstructed error image becomes larger (refer to the lower right part of the reconstructed error image in the figure).

[0039] Therefore, in the image processing device 1, the inference parameters of the semantic label inference unit 11 are updated in the parameter updating unit 14 so that the reconstruction error of the reconstructed error image becomes smaller. For example, in deep learning, the inference parameters are updated by an error back propagation method or the like. Thus, even when an input image for which no training data (correct label image) is used, the inference accuracy of the semantic label can be improved.

[0040] That is, in the image processing device 1, a limited amount of training data (correctly labeled images) is first used to simply perform initial learning, and then the inference parameters of the semantic label inference unit 11 are updated based on the difference between the input image and the restored image. Therefore, in the image processing device 1, the inference accuracy of the semantic label can be improved without using a large amount of training data. In addition, in the image processing device 1, since it is not necessary to prepare a large amount of training data (for example, to assign correct labels to input images by manual operation), the production cost of the training data can be reduced.

[0041] (Second Embodiment)

[0042] Reference Figure 2 The image processing device 1A involved in the second embodiment is described. In addition, in this figure, the same symbols are marked for the same structures as those in the above-mentioned embodiment, and the description is omitted. In addition, in this figure, the structure different from the first embodiment is shown in a manner surrounded by dotted lines. The image processing device 1A includes a semantic label inference unit 11, an original image inference unit 12, a difference calculation unit 13, a parameter updating unit 14, a difference calculation unit 15 and a parameter updating unit 16.

[0043] The difference calculation unit 15 calculates a difference (second difference) between a correct label image prepared in advance and the semantic label image estimated by the semantic label estimation unit 11 , and outputs the calculation result to the parameter updating unit 16 .

[0044] Here, the "correct label image" refers to a semantic label image corresponding to the input image, and represents a semantic label image in which the inference accuracy of each semantic label is 100%. Normally, the semantic label image generated by the semantic label inference unit 11 is set with the inference accuracy of each semantic label for each pixel, such as "the accuracy of weather is 80%, the accuracy of road is 20%..." On the other hand, in the correct label image, the inference accuracy of each semantic label is set to 100%, such as "the accuracy of weather is 100%". The correct label image can be produced manually, or it can be automatically produced by an advanced learner.

[0045] Similar to the differential calculation unit 13, the differential calculation unit 15 can calculate a simple difference for each pixel between the image information of the input image and the image information of the correct label image, calculate the correlation between the two pixels using the above formula (1), or perform a differential comparison between the two after implementing a predetermined image transformation F(·).

[0046] The parameter updating unit 16 updates the inference parameters used by the semantic label inference unit 11 when inferring the semantic label from the input image, based on the difference calculated by the difference calculating unit 15. For example, in deep learning, the inference parameters are updated by error back propagation method or the like.

[0047] In the image processing device 1A, when a correct label image with respect to the input image can be obtained, in addition to the parameter update by the reconstruction error in the parameter updating unit 14, the parameter updating unit 16 updates the inference parameters of the semantic label inference unit 11 so that the label data (correct label data) included in the correct label image is consistent with the semantic label inferred by the semantic label inference unit 11. At this time, the parameter updating unit 14 and the parameter updating unit 16 can be operated separately, or the weighted sum of the update amounts of the two can be taken to update simultaneously.

[0048] According to the image processing device 1A, by performing parameter updates by the correct label image in addition to parameter updates by the reconstruction error, the inference accuracy of the semantic label can be further improved. In addition, according to the image processing device 1A, by performing learning by the reconstruction error, the inference accuracy of the semantic label can be improved compared with the case where learning is performed using only the input image and the correct label image.

[0049] (Third Embodiment)

[0050] Reference Figure 3 The image processing device 1B involved in the third embodiment is described. In addition, in this figure, the same symbols are marked for the same structures as those in the above-mentioned embodiment, and the description is omitted. In addition, in this figure, the structure different from the first embodiment is shown in a manner surrounded by dotted lines. The image processing device 1B includes a semantic label inference unit 11, an original image inference unit 12, a difference calculation unit 13, a parameter update unit 14 and a parameter update unit 17.

[0051] The parameter updating unit 17 updates the estimation parameters used by the original image estimation unit 12 when estimating the original image from the semantically labeled image, based on the difference (first difference) calculated by the difference calculating unit 13 .

[0052] In the image processing device 1B, in addition to updating the inference parameters of the semantic label inference unit 11 in the parameter updating unit 14 so as to reduce the reconstruction error of the reconstructed error image, the inference parameters of the original image inference unit 12 are also updated in the parameter updating unit 17 so as to reduce the reconstruction error of the reconstructed error image. For example, in deep learning, the updating of the inference parameters is implemented by an error back propagation method or the like. Thus, even when an input image without a correct label image is used, the inference accuracy of the original image can be improved.

[0053] In addition, the image processing device 1B can also be implemented in combination with the image processing device 1A. In this case, the update of the inference parameters of the semantic label realized by the reconstruction error, the update of the inference parameters of the semantic label realized by the correct label image, and the update of the inference parameters of the original image realized by the reconstruction error are respectively implemented. By implementing the image processing device 1B in combination with the image processing device 1A, the inference accuracy of the original image can be further improved.

[0054] (Fourth Embodiment)

[0055] Reference Figure 4The image processing device 1C involved in the fourth embodiment is described. In addition, in this figure, the same symbols are marked for the same structures as those in the above-mentioned embodiment, and the description is omitted. In addition, in this figure, the structure different from the first embodiment is shown in a manner surrounded by dotted lines. The image processing device 1C includes a semantic label inference unit 11, a label synthesis unit 18, an original image inference unit 12, a difference calculation unit 13, a parameter update unit 14 and a parameter update unit 17.

[0056] The label synthesis unit 18 synthesizes the correct label of the correct label image and the semantic label of the semantic label image generated by the semantic label inference unit 11, and outputs the image containing the synthesized label to the original image inference unit 12. As the synthesis method in the label synthesis unit 18, for example, the weighted sum of the correct label image and the semantic label image, random selection of images (random selection of the correct label image or the semantic label image), local synthesis (balanced or random selection of a part of the image), etc. can be listed. In addition, the original image inference unit 12 generates a restored image by inferring the original image based on the image synthesized by the label synthesis unit 18.

[0057] In the image processing device 1C, when a correct label image corresponding to the input image can be obtained, the correct label image and the semantic label image generated by the semantic label inference unit 11 are synthesized, and a restored image is generated in the original image inference unit 12 based on the synthesized image. In this way, by implementing the parameter update of the original image inference unit 12 realized by the correct label image, the inference accuracy of the original image can be further improved.

[0058] (Fifth Embodiment)

[0059] Reference Figure 5 The image processing device 1D involved in the fifth embodiment is described. In addition, in this figure, the same symbols are marked for the same structures as those in the above-mentioned embodiment, and the description is omitted. In addition, in this figure, the structure different from the first embodiment is shown in a manner surrounded by dotted lines. The image processing device 1D includes a semantic label inference unit 11, an original image inference unit 12, a difference calculation unit 13, a region synthesis unit 20, a parameter update unit 14, and an update region calculation unit 19.

[0060] The update region calculation unit 19 calculates a specific region in the input image as the update region. The update region calculation unit 19 masks, for example, a region that does not require learning (e.g., the upper half, the lower half, etc.) or a region that has low brightness and thus requires time to learn in the input image, and outputs information other than the masked region to the region synthesis unit 20 as the update region.

[0061] The region synthesis unit 20 synthesizes the reconstructed error image calculated by the difference calculation unit 13 and the update region calculated by the update region calculation unit 19, and outputs them to the parameter update unit 14. In the region synthesis unit 20, for example, the reconstructed error image and the update region are synthesized by performing multiplication, addition, logical AND, or logical OR operations. Furthermore, the parameter update unit 14 updates the inference parameters when inferring the semantic label for the update region of the synthesized image.

[0062] In the image processing device 1D, when the inference parameter in the semantic tag inference unit 11 is updated, the region to update the inference parameter is not limited, and unnecessary learning is omitted. This can improve the inference accuracy of the part that needs to be learned and increase the learning speed.

[0063] (Sixth Embodiment)

[0064] Reference Figure 6 The image processing device 1E involved in the sixth embodiment is described. In addition, in this figure, the same symbols are marked for the same structures as those in the above-mentioned embodiment, and the description is omitted. In addition, in this figure, the structure different from the first embodiment is shown in a manner surrounded by dotted lines. The image processing device 1E includes a semantic label inference unit 11, an original image inference unit 12, a difference calculation unit 13, a region synthesis unit 22, a parameter updating unit 14, and a semantic label inference difficult region calculation unit 21.

[0065] The semantic label inference difficult region calculation unit 21 calculates the inference difficult region in the input image where the inference of the semantic label is difficult. Specifically, the semantic label inference difficult region calculation unit 21 uses the information of the semantic label inferred by the semantic label inference unit 11 to calculate the region having the value of updating the inference parameter, and outputs the information of the region as the inference difficult region to the region synthesis unit 22.

[0066] For example, when the inference accuracy of each semantic tag is set to "p i In the case of ", the index of the difficult area to infer can be obtained by, for example, the entropy of the inference accuracy of each semantic tag "Σ i p i logp i "、Standard deviation of the inference accuracy of each semantic tag STD (p i ), the maximum difference of the inference accuracy of each semantic tag "max i , j(p i -p j )" etc.

[0067] The region synthesis unit 22 synthesizes the reconstructed error image calculated by the difference calculation unit 13 and the inference difficulty region calculated by the semantic label inference difficulty region calculation unit 21, and outputs them to the parameter update unit 14. In the semantic label inference difficulty region calculation unit 21, for example, the reconstructed error image and the inference difficulty region are synthesized by performing multiplication, addition, logical AND, or logical OR operations. Furthermore, the parameter update unit 14 updates the inference parameters when the semantic label inference unit 11 infers the semantic label based on the input image, with respect to the inference difficulty region of the synthesized image.

[0068] In the image processing device 1E, when the inference parameter in the semantic tag inference unit 11 is updated, the area for updating the inference parameter is limited to the area where the semantic tag inference is difficult, and unnecessary learning is omitted. Thus, the inference accuracy of the part that needs to be learned can be improved, and the learning speed can be increased.

[0069] (Seventh Embodiment)

[0070] Reference Figure 7 The image processing device 1F involved in the seventh embodiment is described. In addition, in this figure, the same symbols are marked for the same structures as those in the above-mentioned embodiment, and the description is omitted. In addition, in this figure, the structure different from the first embodiment is shown in a manner surrounded by dotted lines. The image processing device 1F includes a semantic label inference unit 11, an original image inference unit 12, a difference calculation unit 13, and a parameter update unit 14.

[0071] In the semantic label inference unit 11, a method based on deep learning is used as a learning method for the identifier and the learning completion parameter. In addition to the semantic label image generated in the final layer of deep learning (i.e., the inference result of the semantic label inferred in the final layer), the semantic label inference unit 11 also outputs the semantic label image generated in the intermediate layer (hidden layer) of deep learning (i.e., the inference result of the semantic label inferred in the intermediate layer) to the original image inference unit 12. In addition, the original image inference unit 12 infers the original image by using either or both of the semantic label image generated in the intermediate layer and the semantic label image generated in the final layer, thereby generating a restored image.

[0072] In the image processing device 1F, in addition to the completely abstracted semantic label image generated in the final layer of deep learning, the original image is also inferred based on the semantic label image that is not completely abstracted generated in the intermediate layer of deep learning. As a result, since the restoration degree of the semantic label image in the intermediate layer is improved, the quality of the restored image is improved for the part where the semantic label is correctly inferred, and the detection accuracy (S / N) of the part where the semantic label is incorrectly inferred is improved.

[0073] (Eighth Embodiment)

[0074] Reference Figure 8 The image processing device 1G involved in the eighth embodiment is described. In addition, in this figure, the same symbols are marked for the same structures as those in the above-mentioned embodiment, and the description is omitted. In addition, in this figure, the structure different from the first embodiment is shown in a manner surrounded by dotted lines. The image processing device 1G includes a semantic label inference unit 11, an original image inference unit 12, a plurality of difference calculation units 13 and a parameter updating unit 14.

[0075] In the image processing device 1G, a plurality (N) of original image inference units 12 and difference calculation units 13 are provided. The plurality of original image inference units 12 may be composed of networks of different structures, or the identifiers and learning completion parameters may be learned by different learning methods (CRN, Pix2PixHD, other deep learning algorithms, etc.).

[0076] The multiple original image inference units 12 generate multiple restored images by inferring the original image using, for example, multiple different restoration methods and based on the semantically labeled images. In addition, the semantically labeled images input to the multiple original image inference units 12 may be different, for example, the i-th semantically labeled image (e.g., only the car label) may be input only to the i-th original image inference unit 12.

[0077] In the image processing device 1G, the reconstruction error can be accurately inferred by integrating the inference results of the original images in the multiple original image inference units 12. In addition, when specific semantic tags are separately input to the original image inference unit 12, the image category that each original image inference unit 12 should be responsible for is limited, thereby improving the restoration ability of the original image.

[0078] (Ninth Embodiment)

[0079] Reference Fig. 9The image processing device 1H involved in the ninth embodiment is described. In addition, in this figure, the same symbols are marked for the same structures as those in the above-mentioned embodiment, and the description is omitted. In addition, in this figure, the structure different from the first embodiment is shown in a manner surrounded by dotted lines. The image processing device 1H includes a semantic label inference unit 11, an original image inference unit 12, a plurality of difference calculation units 13, a parameter updating unit 14, and a semantic label area summary information generation unit 23.

[0080] The semantic label region summary information generating unit 23 generates region summary information of the semantic label based on the input image and the semantic label image generated by the semantic label inference unit 11, and outputs it to the original image inference unit 12. Examples of the region summary information include the average value, maximum value, minimum value, standard deviation, region area, spatial frequency, edge image (e.g., an algorithm for approximately extracting edge images from images, i.e., Canny algorithm, etc.), and local mask image of the colors of each semantic label.

[0081] Furthermore, when restoring the original image from the semantic label image, the original image estimation unit 12 generates a restored image by using the region summary information generated by the semantic label region summary information generation unit 23 to estimate the original image from the semantic label image.

[0082] In the image processing device 1H, since the original image is inferred by using the region summary information, the quality of the restored image is improved for the part where the semantic label is correctly inferred, and thus the detection accuracy (S / N) of the part where the semantic label is incorrectly inferred can be improved.

[0083] The image processing devices 1 to 1H described so far are specifically used as "learning devices for semantic label inference units" for learning the semantic label inference unit 11 in a low-cost and simple manner. That is, the image processing devices 1 to 1H are not vehicle-mounted devices, and in the development environment of the center, the semantic label inference unit 11 learned by the image processing devices 1 to 1H is introduced into an obstacle recognition device configured in a vehicle or a center (for example, initially installed, or updated in OTA (Over The Air)). Furthermore, by inputting images from, for example, a vehicle-mounted camera into the semantic label inference unit 11 (which can be either vehicle-mounted or at the center), obstacles on the road are recognized.

[0084] Further effects or variations can be easily derived by those skilled in the art. Therefore, the broader aspects of the present invention are not limited to the specific details and representative embodiments expressed and recorded above. Therefore, various changes can be made under the conditions of the spirit or scope of the concept of the comprehensive invention defined by the attached technical solutions and their equivalents.

[0085] Explanation of symbols

[0086] 1. 1A, 1B, 1C, 1D, 1E, 1F, 1G, 1H image processing device;

[0087] 11. Semantic tag inference unit;

[0088] 12. Original image inference unit;

[0089] 13, 15 differential calculation unit;

[0090] 14, 16, 17 parameter updating unit;

[0091] 18. Label synthesis department;

[0092] 19 Updated the regional computing department;

[0093] 20, 22 Regional Synthesis Department;

[0094] 21 Semantic label inference difficult area calculation department;

[0095] 23. Semantic label area summary information generation unit.

Claims

1. An image processing device comprising a processor, wherein the processor has hardware, wherein: The processor performs the following processing, namely: Generate a semantically labeled image by inferring a semantic label for each pixel of an input image using a pre-learned recognizer; Synthesizing the correct label image and the semantic label image; Generate a restored image by inferring the original image based on the synthesized image; Calculating a first difference between the input image and the restored image; Based on the first difference, an inference parameter when inferring the semantic tag or an inference parameter when inferring the original image is updated, The correct label image refers to a semantic label image corresponding to the input image, and indicates a semantic label image in which the inference accuracy of each semantic label is 100%.

2. The image processing device according to claim 1, wherein: The processor performs the following processing, namely: Calculating a second difference between a pre-prepared correct label image and the semantic label image; An inference parameter for inferring the semantic tag is updated based on the first difference and the second difference.

3. The image processing apparatus according to claim 1, wherein: The processor performs the following processing, namely: The specific area in the input image is used as the update area for calculation, Regarding the update area, the inference parameters when inferring the semantic tag are updated.

4. The image processing apparatus according to claim 1, wherein: The processor performs the following processing, namely: In the input image, calculating the inference difficulty area, the index of the inference difficulty area can be shown by the entropy of the inference accuracy of each semantic tag, the standard deviation of the inference accuracy of each semantic tag, or the difference between the maximum values ​​of the inference accuracy of each semantic tag; synthesizing the inference difficult region and a reconstructed error image representing the first difference; The inference parameters used in inferring the semantic tags are updated based on the synthesized image.

5. The image processing apparatus according to claim 1, wherein: The processor performs the following processing, namely: By using a plurality of different restoration methods and inferring the original image according to the semantically labeled image, a plurality of restored images are generated; Calculating first differences between the input image and the plurality of restored images respectively; Based on the multiple first differences, inference parameters when inferring the semantic tag are updated.

6. The image processing apparatus according to claim 1, wherein: The processor performs the following processing, namely: Generate regional summary information of the semantic label, The restored image is generated by using the region summary information and inferring the original image according to the semantic label image.

7. A recording medium having an image processing program recorded thereon, wherein: The image processing program enables a processor having hardware to perform the following processing, namely: Generate a semantically labeled image by inferring a semantic label for each pixel of an input image using a pre-learned recognizer; Synthesizing the correct label image and the semantic label image; Generate a restored image by inferring the original image based on the synthesized image; Calculating a first difference between the input image and the restored image; Based on the first difference, an inference parameter when inferring the semantic tag or an inference parameter when inferring the original image is updated, The correct label image refers to a semantic label image corresponding to the input image, and indicates a semantic label image in which the inference accuracy of each semantic label is 100%.

8. The recording medium according to claim 7, wherein The image processing program enables the processor to perform the following processing, namely: Calculating a second difference between a pre-prepared correct label image and the semantic label image; An inference parameter for inferring the semantic tag is updated based on the first difference and the second difference.

9. The recording medium according to claim 7, wherein The image processing program enables the processor to perform the following processing, namely: The specific area in the input image is used as the update area for calculation, Regarding the update area, the inference parameters when inferring the semantic tag are updated.

10. The recording medium according to claim 7, wherein The image processing program enables the processor to perform the following processing, namely: In the input image, calculating the inference difficulty area, the index of the inference difficulty area can be shown by the entropy of the inference accuracy of each semantic tag, the standard deviation of the inference accuracy of each semantic tag, or the difference between the maximum values ​​of the inference accuracy of each semantic tag; synthesizing the inference difficult region and a reconstructed error image representing the first difference; The inference parameters used in inferring the semantic tags are updated based on the synthesized image.

11. The recording medium according to claim 7, wherein The image processing program enables the processor to perform the following processing, namely: By using a plurality of different restoration methods and inferring the original image according to the semantically labeled image, a plurality of restored images are generated; Calculating first differences between the input image and the plurality of restored images respectively; Based on the multiple first differences, inference parameters when inferring the semantic tag are updated.

12. The recording medium according to claim 7, wherein The image processing program enables the processor to perform the following processing, namely: Generate regional summary information of the semantic label, The restored image is generated by using the region summary information and inferring the original image according to the semantic label image.

13. An image processing method, wherein: The processor with hardware performs the following processing, namely: Generate a semantically labeled image by inferring a semantic label for each pixel of an input image using a pre-learned recognizer; Synthesizing the correct label image and the semantic label image; Generate a restored image by inferring the original image based on the synthesized image; Calculating a first difference between the input image and the restored image; Based on the first difference, an inference parameter when inferring the semantic tag or an inference parameter when inferring the original image is updated, The correct label image refers to a semantic label image corresponding to the input image, and indicates a semantic label image in which the inference accuracy of each semantic label is 100%.

Images