Image processing method, device and system, storage medium and learning model manufacturing method

Abstract

The invention discloses an image processing method, device and system, a storage medium and a learning model manufacturing method. An image processing method includes obtaining input data including an input image and a noise map representing an amount of noise in the input image based on an optical black region corresponding to the input image, and inputting the input data into a neural network to perform a task of recognition or regression.

Description

[0001] This application is a divisional application of the Chinese invention patent application with application number 202010141298.3, application date March 4, 2020, and titled “Image processing method, device, system, storage medium and learning model manufacturing method”. Technical Field

[0002] The present invention relates to an image processing method that can improve the robustness against noise in a neural network. Background Art

[0003] Japanese Patent Publication No. (“JP”) 2016-110232 discloses a method for determining the position of a recognition target in an image with high accuracy using a neural network.

[0004] However, when the image has a low S / N ratio, the method disclosed in JP2016-110232 reduces the determination accuracy. Since the noise in the image depends on the ISO sensitivity during imaging and the performance of the image sensor, images with noise of various intensities may be input to the neural network. Since the object feature quantity cannot be exclusively extracted due to the influence of noise, the low S / N ratio will reduce the recognition accuracy.

[0005] There are two conceivable methods to improve the robustness of a neural network against noise. The first method is to include images with various noise amounts in the learning data for the neural network. This method can improve the robustness against noise, but may reduce the maximum accuracy value. The second method is to learn the neural network for each different noise amount. This method requires storing the learning results for each noise amount and may increase the amount of data. Summary of the invention

[0006] The present invention provides image processing methods and the like, each of which can maintain high accuracy and a small amount of data and improve robustness against noise in a neural network.

[0007] An image processing method according to one aspect of the present invention comprises: obtaining input data, the input data comprising an input image and a noise map representing the amount of noise in the input image based on an optical black area corresponding to the input image, and inputting the input data into a neural network to perform a recognition or regression task. An image processing device corresponding to the image processing method and a non-transitory computer-readable storage medium storing a program for causing a computer to execute the above-mentioned image processing method also constitute another aspect of the present invention.

[0008] An image processing system according to another aspect of the present invention includes a first device and a second device configured to communicate with the first device. The first device includes a transmitter configured to send a request for causing the second device to process a captured image. The second device includes: a receiver configured to receive the request sent by the transmitter; an acquirer configured to acquire input data, the input data including a captured image and a noise map representing the amount of noise in the captured image based on an optical black area corresponding to the captured image; a processor configured to input the input data into a neural network to perform a recognition or regression task; and a transmitter configured to send a result of the task.

[0009] According to another aspect of the present invention, an image processing method or a method for making a learned model includes obtaining ground truth data and input data, the input data including a training image and a noise map representing the amount of noise in the training image based on an optical black area corresponding to the training image, and learning a neural network for performing a recognition or regression task using the input data and the ground truth data. A non-transitory computer-readable storage medium storing a program for causing a computer to execute the above-mentioned image processing method and an image processing device also constitute another aspect of the present invention.

[0010] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 The configuration of the neural network according to the first embodiment is shown.

[0012] Figure 2 is a block diagram of an image processing system according to the first embodiment.

[0013] Figure 3 is an external diagram of the image processing system according to the first embodiment.

[0014] Figure 4 is a flowchart of weight learning according to the first embodiment.

[0015] Figure 5A and Figure 5B Exemplary training images and exemplary ground-truth class maps according to the first embodiment are shown.

[0016] Figure 6 is a flow chart of a manner of generating an estimated class map according to a first embodiment.

[0017] Figure 7 is a block diagram of an image processing system according to a second embodiment.

[0018] Figure 8is an external diagram of the image processing system according to the second embodiment.

[0019] Fig. 9 is a flowchart of weight learning according to the second embodiment.

[0020] Fig.10 A manner of generating an input image and a noise representation image according to the second embodiment is shown.

[0021] Fig.11 A configuration of a neural network according to the second embodiment is shown.

[0022] Fig.12 is a flowchart of a manner of generating an estimated image according to the second embodiment.

[0023] Fig.13 is a block diagram of an image processing system according to a third embodiment.

[0024] Fig.14 is a flowchart of a manner of generating an output image according to the third embodiment. DETAILED DESCRIPTION

[0025] Now, with reference to the accompanying drawings, a detailed description of embodiments according to the present invention will be given. Corresponding elements in the various drawings will be denoted by the same reference numerals, and their repeated description will be omitted.

[0026] Before describing the embodiment in detail, the main points of the present invention will be described. The present invention improves the robustness against noise in the task of recognition or regression using a neural network. Now, where "x" (a d-dimensional vector, where d is a natural number) is input data to be input to the neural network, recognition is the task of finding a category "y" corresponding to the vector x. For example, there is a task of classifying objects in an image into people, dogs, cars, etc., and a task of recognizing facial expressions such as smiling faces or crying faces from facial images. Category y is generally a discrete variable and can be a vector when generating a segmentation map, etc. On the other hand, regression is the task of finding a continuous variable y corresponding to the vector x. For example, there is a task of estimating a noise-free image based on an image including noise, and a task of estimating a high-resolution image before downsampling based on a downsampled image.

[0027] In the following embodiments, an input image and a noise representative image (noise map or two-dimensional noise distribution information) corresponding to the input image are used for input data in order to improve robustness against noise. The noise representative image is an image representing the amount of noise in the input image. The noise representative image is an image based on optical black corresponding to the input image, or an image based on the noise characteristics of an image sensor that captured the input image. Since the neural network can specify the amount of noise in the input image even if learning is performed by mixing a plurality of noise amounts, robustness against noise can be improved while maintaining accuracy by inputting both the noise representative image and the input image.

[0028] In the following description, the step of learning the weights of the neural network will be referred to as a learning phase, and the identification or regression step using the learned weights will be referred to as an estimation phase. First embodiment

[0029] A description will now be given of an image processing system according to a first embodiment of the present invention. In the first embodiment, a neural network performs a recognition task of detecting a human region (human segmentation) in an image. However, the present invention is not limited to this embodiment and is similarly applicable to other recognition or regression tasks.

[0030] Figure 2 is a block diagram of an image processing system 100 according to this embodiment. Figure 3 1 is an external diagram of the image processing system 100. The image processing system 100 includes a learning device 101, an imaging device (image pickup device) 102, and a network 103. The learning device 101 includes a memory 111, an acquirer (acquisition unit) 112, a detector (detection unit) 113, and an updater (update unit) 114, and the learning device 101 learns the weight of a neural network that detects a human region. The details of this learning will be described later. The weight information learned by the learning device 101 is stored in the memory 111. The imaging device 102 acquires a captured image and detects a human region using a neural network.

[0031] The imaging device 102 includes an optical system 121 and an image sensor 122. The optical system 121 collects light entering the imaging device 102 from the object space. The image sensor 122 receives an optical image (object image) formed via the optical system 121, and acquires (photoelectrically converts) a captured image. The image sensor 122 includes, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0032] The image processor 123 includes an acquirer 123a and a detector 123b, and detects a human region by setting at least a portion of a captured image as an input image. At this time, information about weights stored in the memory 124 is used. The weight information is read out from the learning device 101 via the wired or wireless network 103 in advance and stored in the memory 124. The stored weight information may be a numerical value of the weight itself or in a coded form. Details of the detection process for the human region will be described later. The image processor 123 performs different processing for the human region and other regions to generate an output image. For example, by blurring the region outside the human region, a portrait image with blur is generated. The output image is stored in the recording medium 125. Alternatively, the captured image may be stored in the recording medium 125 as it is, and then the image processor 123 may read the captured image from the recording medium 125 and detect the human region. According to the user's instruction, the output image stored in the recording medium 125 is displayed on the display unit 126. A series of operations are controlled by the system controller 127.

[0033] Reference now Figure 4 , a description will be given of weight learning (production of a learned model) performed by the learning device 101 according to this embodiment. Figure 4 It is a flowchart of weight learning. Figure 4 Each step in is mainly performed by the acquirer 112, the detector 113 or the updater 114 in the learning device 101.

[0034] Initially, in step S101 , the acquirer 112 acquires one or more pairs of training images and true class maps (also called true segmentation maps or true data) and noise representation images. The training images are input images in the learning phase of the neural network.

[0035] Figure 5A Examples of training images and ground-truth class maps are shown. Figure 5A An example of a training image is shown, and Figure 5B The corresponding ground-truth class map is shown. Figure 5B The white color in represents the category representing the human area, and the gray color represents the category representing other areas. In this embodiment, the noise representative image is an image having a signal value set to the standard deviation of the noise present in the training image. However, the present invention is not limited to the standard deviation, and for example, the variance may be used as long as the (degree of discreteness) discreteness represents the noise amount. Alternatively, the value of ISO sensitivity may be used instead of discreteness.

[0036] The learning phase uses a plurality of training images with various imaging scenes and noise amounts so as to stably detect a human region even in an unknown imaging scene or in an image with a noise amount in the estimation phase. The first method is to prepare a plurality of training images with different noise amounts by simulating a method of adding noise to images of various imaging scenes. This method obtains a standard deviation of the noise set to be added, and a noise representation image having a standard deviation value for each element (pixel). Since the shot noise component in the noise depends on the amount of light, the standard deviation of the noise changes depending on the magnitude of the signal value (amount of light). If the noise characteristics of the image sensor 122 are known, a noise representation image with a standard deviation depending on the amount of light can be obtained in both the learning phase and the estimation phase. More specifically, the signal value of each pixel in the input image or the training image is converted into the amount of light (if necessary, then inverse gamma correction is applied), and the standard deviation of the noise based on the amount of light converted for each pixel in the noise representation image is stored. Since the input image and the training image already have noise, the value slightly deviates from the standard deviation calculated based on the real signal. If the noise characteristics of the image sensor 122 are unknown, the estimation phase calculates the standard deviation of the noise based on the optical black area (black level area). Therefore, the standard deviation of the noise at zero light amount is used to obtain the noise representation image in the learning stage.

[0037] The second method is a method of capturing various imaging scenes by changing the ISO sensitivity or using an image sensor with different sensitivities. In this case, an image sensor with known noise characteristics is used for imaging, and the standard deviation of the noise of the training image is obtained from the known noise characteristics. Alternatively, when the training image is captured, the standard deviation of the noise at zero light is obtained from the signal value of the optical black area in the image sensor. As with the first method, the noise representation image is an image in which the signal value of each pixel has a standard deviation. In the estimation stage, the training image has the same format as the input image. If the input image in the estimation stage is an undeveloped original image, then the training image is also an undeveloped original image. If the input image in the estimation stage is a developed image, then the training image is also a developed image. The number of pixels in the training image in the estimation stage and the number of pixels in the input image do not have to be equal to each other.

[0038] Next, in Figure 4 In step S102, the detector (learner or learning unit) 113 inputs the training image and the noise representation image into the neural network and generates an estimated class map. In this embodiment, the neural network is Figure 1The U-Net shown in (for details, see O. Ronneberger, P. Fischer and T. Brox, "U-net: Convolutional networks for biomedical image segmentation", arXiv: 1505.04597), but the present invention is not limited to this embodiment.

[0039] Input data 201 is data obtained by concatenating a training image and a noise representation image in a channel direction. The training image may have multiple channels of RGB (red, green, blue). The noise representation image may have only one channel or the same number of channels as that of the training image. The number of pixels (number of elements) of each channel of the training image and the noise representation image is the same. Although training images with various noise amounts are input to the neural network, the neural network can specify the noise amount in the training image by including the noise representation image in the input data. Therefore, robustness against noise can be ensured while suppressing a decrease in estimation accuracy.

[0040] The input data can be normalized as needed. When the input image is a RAW image, the black level may be different depending on the image sensor and ISO sensitivity. Therefore, after subtracting the black level from the signal value of the input image, the signal is input to the neural network. Normalization can also be performed after subtracting the black level. Figure 1 In , "convolution" represents one or more convolution layers, "max pooling" represents max pooling, "up-convolution" represents one or more convolution layers including upsampling, and "cascade" represents connection in the channel direction. In the initial learning, the weights of the filters used for each convolution layer are randomly determined. An estimated class map 202 is calculated as the output of the U-Net corresponding to the training image.

[0041] Next, in step S103, the updater (learner or learning unit) 114 updates the weight of the neural network according to the estimated class map and the true class map. Although this embodiment uses the cross entropy between the estimated class map and the true class map for the loss function, the present invention is not limited to this embodiment. The weight is updated according to the calculated value of the loss function by the error back propagation method (back propagation) or the like.

[0042] Next, in step S104, the updating unit 114 determines whether the weight learning has been completed. The completion of the learning can be determined based on whether the number of iterations of the learning (weight update) has reached a specified value or whether the weight change during the update is less than a specified value. If it is determined that the learning has not been completed, the process returns to step S101, and one or more new training images, noise representation images, and true category images are obtained. On the other hand, when it is determined that the learning has been completed, the learning is terminated, and the weight information is stored in the memory 111.

[0043] Reference now Figure 6 , a description will be given of a manner (a manner of generating an estimated class map or an estimation stage) of a person region in an input image by the image processor 123 according to this embodiment. Figure 6 is a flow chart of the way in which the estimated category map is generated. Figure 6 Each step in is mainly performed by the acquirer 123a or the detector 123b in the image processor 123.

[0044] First, in step S201, the acquirer 123a acquires an input image and a standard deviation of noise corresponding to the input image. The input image is at least a portion of an image captured by the image sensor 122. When the noise characteristics of the image sensor 122 are known, the standard deviation of the noise is obtained based on imaging conditions (such as ISO sensitivity) during imaging. Alternatively, the standard deviation of the noise can be calculated based on the signal value of an optical black area under the same imaging conditions as those of the input image. For example, when the captured image is an undeveloped RAW image, an area in the captured image where information about the object space is obtained is set as the input image, and an area where light is blocked is set as an optical black area corresponding to the input image. The standard deviation of the noise at zero light intensity can be obtained from the change in the signal value in the optical black area.

[0045] Next, in step S202, the acquirer 123a generates a noise representation image based on the standard deviation. The noise representation image is an image in which the signal value of each pixel has a standard deviation. Next, in step S203, the detector 123b inputs the input image and the noise representation image into the neural network and generates an estimated class map. At this time, the noise representation image is generated by the neural network in the learning phase. Figure 1 The weights obtained by the neural network in .

[0046] This embodiment can realize an image processing system that can generate a segmentation map that is robust against noise while maintaining high accuracy and a small data volume. Second embodiment

[0047] A description will be given of an image processing system according to a second embodiment of the present invention. In this embodiment, a neural network performs a regression task (deblurring) for correcting aberration and blurring due to diffraction of a captured image. However, the present invention is not limited to this embodiment and may be applied to other recognition or regression tasks.

[0048] Figure 7 is a block diagram of an image processing system 300 according to this embodiment. Figure 8 300 is an external diagram of the image processing system 300. The image processing system 300 includes a learning device 301, an imaging device 302, an image estimation device 303, a display device 304, a recording medium 305, an output device 306, and a network 307.

[0049] The learning device 301 includes a memory 301a, an acquirer 301b, a generator 301c, and an updater 301d. The imaging device 302 includes an optical system 302a and an image sensor 302b. The image captured by the image sensor 302b is blurred due to aberration and diffraction of the optical system 302a and noise caused by the image sensor 302b. The image estimation device 303 includes a memory 303a, an acquirer 303b, and a generator 303c, corrects blur in an input image as at least a part of the captured image, and generates an estimated image. A neural network is used to correct blur, and information about weights is read out from the memory 303a. The weights are learned by the learning device 301, and the image estimation device 303 reads the weight information from the memory 301a via the network 307 in advance and stores the information in the memory 303a. The details of the blur correction process and weight learning using the weights will be described later. The image estimation device 303 performs a development process on the estimated image to generate an output image. The output image is output to at least one of a display device 304, a recording medium 305, and an output device 306. The display device 304 is, for example, a liquid crystal display or a projector. The user can perform editing operations and the like via the display device 304 while checking the processed image. The recording medium 305 includes, for example, a semiconductor memory, a hard disk drive, a server on a network, and the like. The output device 306 is a printer and the like.

[0050] Reference now Fig. 9 , a description will be given of weight learning performed by the learning device 301. Fig. 9 It is a flowchart of weight learning. Fig. 9 Each step in is mainly performed by the acquirer 301b, the generator 301c or the updater 301d in the learning device 301.

[0051] Initially, in step S301, the acquirer 301b acquires one or more sets of (one or more) source images and (one or more) imaging conditions. Learning blur correction by aberration and diffraction requires a pair of images: a blurred image (first training image) and an image without blur (real image). The second embodiment generates the pair of images from the source image by imaging simulation. However, the present invention is not limited to this embodiment, and the pair of images can be prepared by capturing the same object using a lens that corrects blur caused by aberration and diffraction and a lens of higher performance.

[0052] This embodiment performs learning and blur correction for a RAW image, but the present invention is not limited to this embodiment, and a developed image may be used. The source image is a RAW image, and the imaging conditions are parameters for performing imaging simulation using the source image as an object. The parameters include an optical system for imaging, a state of the optical system (such as zoom, F number, focusing distance), image height, the presence and type of an optical low-pass filter, noise characteristics of the image sensor, pixel pitch, ISO sensitivity, color filter array, dynamic range, black level, etc. The second embodiment performs learning for each optical system. Multiple combinations of states, image height, pixel pitch, ISO sensitivity, etc. are set for a specific optical system, and a pair of images are generated under different imaging conditions: a first training image and a real image (real data).

[0053] Next, in step S302, the generator 301c generates a first training image, a second training image, and a real image from the source image based on the imaging conditions. The first training image and the real image are an image obtained by adding blur caused by aberration and diffraction generated in the optical system to the source image and an image obtained by not adding blur to the source image, respectively. Noise is added to the first training image. Noise is added to the real image as needed. If noise is not added to the real image, or if noise unrelated to the noise of the first training image is added, the neural network learns blur correction and denoising. When noise related to the noise of the first training image is added to the real image, blur correction that suppresses noise changes is learned.

[0054] This embodiment uses a Wiener filter for a first training image to generate a second training image (an intermediate blur-corrected image in the learning phase) in which blur is corrected to a certain extent. The Wiener filter is a filter calculated based on the blur applied to the first training image. However, the correction method is not limited to the Wiener filter, and other methods such as an inverse filter-based method and a Richardson-Lucy method may be used. The use of a second training image can improve the robustness of blur correction against blur changes in a neural network. The source image is reduced as needed during the imaging simulation. When the source image is prepared not by CG (computer graphics) but by actual imaging, the source image is an image captured by a certain optical system. Therefore, blur already exists due to aberrations and diffraction. However, reduction can reduce the influence of blur and generate a realistic image that exists even up to high frequencies.

[0055] Next, in step S303, the generator 301c generates a noise representative image based on the imaging conditions. In this embodiment, the noise representative image is generated by simulating the signal value of the optical black area (black level area) for reproduction. The optical black signal value is generated by adding the noise corresponding to the ISO sensitivity to the black level set under the imaging conditions.

[0056] Fig.10 The generation of the input image and the noise representation image is shown. In the estimation stage, as Fig.10 As shown in , a noise representative image 405 is generated by extracting and arranging one or more partial optical black images 404 from an optical black region 403 in a captured image 401. Therefore, a noise representative image can be generated in a similar manner in the learning phase. Assume that K (K is a natural number) is the number of partial optical black images to be extracted, and m k ×n k (k=1, ..., K) is the number of pixels in the partially optically black image. At this time, even this step is performed by generating pixels each having m k ×n k K optical blacks are arranged to generate a noise representation image. The number of pixels of the noise representation image is the same as the number of pixels of each of the first training image and the second training image. Fig. 9 The order of steps S302 and S303 in .

[0057] Next, in step S304, the generator 301c inputs the first training image, the second training image, and the noise representation image into the neural network to generate an estimated image (blur-corrected image). Fig.11 The configuration of the neural network according to this embodiment is shown. This embodiment uses Fig.11The neural network shown in , but the present invention is not limited to this embodiment. For example, a GAN (Generative Adversarial Network) can be used. The input data 511 is data obtained by connecting the first training image 501, the second training image, and the noise representation image in the channel direction. There is no restriction on the order of connection in the channel direction. "Convolution" means one or more convolution layers, and "deconvolution" means one or more deconvolution layers. The second skip connection, the third skip connection, and the fourth skip connection 522, 523, and 524 take the sum of two feature maps for each element. Alternatively, they can be connected to each other in the channel direction. The first skip connection 521 obtains an estimated image 512 by taking the sum of the first training image 501 (or the second training image) and the residual component output from the last layer. However, the number of skip connections is not limited to Fig.11 When making the resolution and contrast of an image high such as blur correction, not only the object but also the noise is emphasized. However, by inputting a noise representation image, a neural network can easily separate the object from the noise and achieve a high resolution or high contrast that is robust against noise.

[0058] Next, in step S305, the updater 301d updates the weights of the neural network based on the estimated image and the real image. This embodiment sets the Euclidean norm of the difference between the signal values ​​of the estimated image and the real image as the loss function. However, the loss function is not limited to this embodiment. Next, in step S306, the updater 301d determines whether the learning is completed. If the learning is not completed, the process returns to step S301 to obtain a new set of one or more source images and imaging conditions. On the other hand, when the learning is completed, the weight information is stored in the memory 301a.

[0059] Reference now Fig.12 , a description will be given of a manner of correcting blur caused by aberration and diffraction in an input image (a manner of generating an estimated image) performed by the image estimation device 303. Fig.12 is a flowchart of the way in which the estimated image is generated. Fig.12 Each step in is mainly performed by the acquirer 303b and the generator 303c in the image estimation device 303.

[0060] Initially, in step S401, the acquirer 303b acquires an input image and an optical black area corresponding to the input image from a captured image. Fig.10As shown in FIG. 4 , the second embodiment takes an area where information about the object space exists from a captured image 401 as a RAW image as an input image 402 and takes an area where no light enters as an optical black area 403. However, the optical black area can be obtained from another image under the same imaging conditions as those of the input image.

[0061] Next, in step S402, the generator 303c generates an intermediate blur-corrected image from the input image. Information of the Wiener filter that corrects blur caused by aberration and diffraction of the optical system 302a is called from the memory 303a and applied to the input image to generate the intermediate blur-corrected image. Since the input image has different blur at each image height, shift variant correction is performed.

[0062] Next, in step S403, the acquirer 303b extracts a portion of the optical black image from the optical black area. Fig.10 As shown in FIG. 4 , one or more partial optical black images 404 are extracted from the optical black region 403. K is the number of partial optical black images to be extracted, and m is k ×n k is the number of pixels in each partially optically black image.

[0063] Next, in step S404, the generator 303c generates a noise representative image based on the partial optical black image and the number of pixels in the input image. Fig.10 As shown in FIG. 4 , partial optical black images 404 are arranged to generate a noise representative image 405. The partial optical black images 404 are arranged so that the number of pixels is equal to or greater than the number of pixels in the input image 402, and the noise representative image 405 is generated by extracting the number of pixels of the input image 402. When the partial optical black images 404 are arranged, a single optical black image 404 may be inverted or rotated before being arranged.

[0064] Next, in step S405, the generator 303c inputs the input image, the intermediate blur-corrected image, and the noise-representing image into a neural network to generate an estimated image. Fig.11 The configuration shown in and input data formed by connecting an input image (corresponding to the first training image), an intermediate blur-corrected image (corresponding to the second training image), and a noise representation image in the channel direction in the same order as in learning is input. Information on the weight corresponding to the optical system 302a is read out from the memory 303a, and an estimated image is generated. If normalization or black level subtraction is performed during input to the neural network, a process of returning the ratio of the signal value to the estimated image and performing black level addition is performed.

[0065] This embodiment can realize an image processing system that can suppress blurring caused by aberration and diffraction and is robust against noise while maintaining high accuracy and a small data amount.

[0066] Therefore, the image processing method according to each embodiment has a first step and a second step. The first step acquires input data including an input image and a noise map representing the amount of noise in the input image (S201, S202; S401 to S404). The second step is a step of inputting the input data to a neural network and performing a recognition or regression task (S203; S405). Third embodiment

[0067] A description will now be given of an image processing system according to a third embodiment of the present invention. The image processing system according to this embodiment differs from the image processing system in each of the first and second embodiments in having a processing device (computer) that sends a captured image as an image processing target to an image estimation device and receives a processed output image from the image estimation device.

[0068] Fig.13 6 is a block diagram of an image processing system 600 according to the embodiment. The image processing system 600 includes a learning device 601, an imaging device 602, an image estimation device 603, and a processing device (computer) 604. The learning device 601 and the image estimation device 603 are, for example, servers. The computer 604 is, for example, a user terminal such as a personal computer and a smart phone. The computer 604 is connected to the image estimation device 603 via a network 605. The image estimation device 603 is connected to the learning device 601 via a network 606. That is, the computer 604 and the image estimation device 603 can communicate with each other, and the image estimation device 603 and the learning device 601 can communicate with each other. The computer 604 corresponds to the first device, and the image estimation device 603 corresponds to the second device. The learning device 601 is configured similarly to the learning device 301 of the second embodiment, and its description will be omitted. The imaging device 602 is configured similarly to the image sensor 302 of the second embodiment, and its description will be omitted.

[0069] The image estimation device 603 includes a memory 603a, an acquirer 603b, a generator 603c, and a communicator (receiver, transmitter) 603d. Each of the memory 603a, the acquirer 603b, and the generator 603c is the same as the memory 103a, the acquirer 103b, and the generator 103c in the image estimation device 303 of the second embodiment. The communicator 603d is used to receive a request sent from the computer 604, and send the output image generated by the image estimation device 603 to the computer 604.

[0070] The computer 604 includes a communicator (transmitter) 604a, a display unit 604b, an image processor 604c, and a recorder 604d. The communicator 604a is used to send a request for the image estimation device 603 to process a captured image to the image estimation device 603, and receive an output image processed by the image estimation device 603. The display unit 604b is used to display various information. The information displayed by the display unit 604b includes, for example, a captured image to be sent to the image estimation device 603 and an output image received from the image estimation device 603. The image processor 604c is also used to perform image processing on the output image received from the image estimation device 603. The recorder 604d records the captured image obtained from the imaging device 602, the output image received from the image estimation device 603, and the like.

[0071] Reference now Fig.14 , a description will be given of the image processing according to this embodiment. The image processing according to this embodiment is equivalent to the blur correction (deblurring) processing ( Fig.12 ). Fig.14 is a flow chart of the method of generating an output image. When the user issues an instruction to start image processing via computer 604, the process starts. Fig.14 A description will now be given of the operation of the computer 604.

[0072] In step S701, the computer 604 sends a request to process a captured image to the image estimation device 603. The method of sending the captured image to be processed to the image estimation device 603 is not important. For example, the captured image may be uploaded from the computer 604 to the image estimation device 603 at the same time as step S701, or may be uploaded to the image estimation device 603 before step S701. Instead of the image recorded in the computer 604, the captured image may be an image stored on a server different from the image estimation device 603. In step S701, the computer 604 may send ID information for authenticating a user, etc. together with the request to process the captured image. In step S702, the computer 604 receives the output image generated in the image estimation device 603. As in the second embodiment, the output image is an estimated image obtained by performing blur correction on the captured image.

[0073] Next, the operation of the image estimation device 603 is described. In step S801, the image estimation device 603 receives a request for processing a captured image sent from the computer 604. The image estimation device 603 determines that processing (blur correction processing) for the captured image is instructed, and performs the processing after step S802. Steps S802 to S806 are the same as steps S401 to S405 in the second embodiment. In step S807, the image estimation device 603 sends the estimated image as a result of the regression task as an output image to the computer 604.

[0074] This embodiment performs the blur correction processing of the second embodiment, but is similarly applicable to the detection of the human region of the first embodiment ( Figure 6 ). This embodiment describes that the image estimation device 603 performs all the processing corresponding to S401 to S405 of the second embodiment, but the present invention is not limited to this embodiment. For example, one or more of S401 to S404 of the second embodiment (steps S802 to S806 in this embodiment) may be performed in the computer 604, and the result may be sent from the computer 604 to the image estimation device 603.

[0075] As described above, as in this embodiment, the image estimation device 603 may be configured to be controlled using the computer 604 communicably connected to the image estimation device 603 .

[0076] The noise map is based on an optical black area (black level area) corresponding to an input image. The image processing method may include obtaining an optical black area corresponding to the input image. The image processing method may include extracting one or more partial optical black images as at least a portion of the optical black area. The image processing method may include generating the noise map based on the partial optical black images and the number of pixels in the input image. The noise map may be generated by arranging the partial optical black images.

[0077] The image processing method may include obtaining an optical black area corresponding to an input image, calculating a discreteness (variance, standard deviation) of a signal value in the optical black area, and calculating a noise map based on the discreteness. The input image and the optical black area may be extracted from the same captured image. The number of pixels in one channel (each of the R, G, B channels) of the input image and the noise map may be equal to each other. The input data may include data obtained by connecting the input image and the noise map in a channel direction. The image processing method may input the input data into a neural network after subtracting a black level from a signal value of the input data. The task may be to improve the resolution or contrast of the image.

[0078] In each embodiment, the image processing device (image processor 123, image estimation device 303) includes an acquirer (acquirers 123a and 303b and generator 303c) and a processor (detector 123b and generator 303c). The acquirer acquires input data including an input image and a noise map representing the amount of noise in the input image. The processor inputs the input data into a neural network to perform a recognition or regression task. The image processing device has a memory (memory 124 and 303a) for storing information about weights used for the neural network.

[0079] In each embodiment, the image processing system includes a first device and a second device configured to communicate with the first device. The first device has a transmitter that sends a request for causing the second device to process a captured image. The second device has a receiver, an acquirer, a processor, and a transmitter. The receiver receives the request sent by the transmitter. The acquirer acquires input data including a captured image and a noise map representing the amount of noise in the captured image. The processor inputs the input data into a neural network to perform a recognition or regression task. The transmitter sends the result of the task.

[0080] In each embodiment, the method of making a learned model includes an acquisition step and a learning step. The acquisition step acquires input data including a training image and a noise map representing the amount of noise in the training image, and real data. The learning step learns a neural network for performing a recognition or regression task using the input data and the real data. Other embodiments

[0081] The embodiment(s) of the present invention may also be implemented by a computer of a system or device that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more completely referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more of the above-mentioned embodiment(s) and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more of the above-mentioned embodiment(s), and by a method executed by a computer of a system or device, for example, by reading out and executing computer executable instructions from a storage medium to perform the functions of one or more of the above-mentioned embodiment(s) and / or controlling one or more circuits to perform the functions of one or more of the above-mentioned embodiment(s). The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read out and execute computer executable instructions. Computer executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage device of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD)) TM ), one or more of flash memory devices, memory cards, etc. Other embodiments

[0082] The embodiments of the present invention may also be implemented by providing software (program) for executing the functions of the above-mentioned embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0083] The present invention can provide image processing methods and the like, each of which can maintain high accuracy and a small amount of data and improve robustness against noise in a neural network.

[0084] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An image processing method, characterized in that include: obtaining input data including (i) an input image and (ii) a noise map representing an amount of noise in the input image based on an optically black region corresponding to the input image, and Input data is fed into a neural network to perform the task of recognition or regression.

2. The image processing method according to claim 1, further comprising: Obtaining an optical black area corresponding to an input image; extracting one or more partial optically black images that are at least a portion of the optically black region; as well as A noise map is generated based on the partial optical black image and the number of pixels of the input image.

3. The image processing method according to claim 2, characterized in that: A noise map is generated by arranging parts of the optical black image.

4. The image processing method according to claim 1, further comprising: Obtaining an optical black area corresponding to an input image; Calculate the dispersion of signal values ​​in the optical black region; as well as Generates a noise plot based on the dispersion.

5. The image processing method according to claim 2, characterized in that: The input image and the optical black area are extracted from the same captured image.

6. The image processing method according to claim 1, characterized in that: The input image and the noise map have the same number of pixels in a single channel.

7. The image processing method according to claim 1, characterized in that: The input data includes data obtained by concatenating the input image and the noise map in the channel direction. 8 . The image processing method according to claim 1 , comprising subtracting a black level from a signal value of the input data before inputting the input data.

9. The image processing method according to claim 1, characterized in that: The task is to make the image have a higher resolution or contrast.

10. An image processing device, characterized in that include: An acquirer configured to acquire input data, the input data comprising (i) an input image and (ii) a noise map representing an amount of noise in the input image based on an optical black area corresponding to the input image; as well as A processor is configured to input input data into a neural network to perform a recognition or regression task.

11. An image processing system, comprising: The image processing device according to claim 10 and the control device capable of communicating with the image processing device, wherein the control device comprises a transmitter configured to transmit a request for causing the image processing device to process the captured image, and wherein the image processing device comprises a receiver configured to receive the request transmitted by the transmitter.

12. A non-transitory computer-readable storage medium storing a program for causing a computer to execute an image processing method, It is characterized in that The image processing method comprises: Obtaining input data, the input data comprising (i) an input image and (ii) a noise map representing an amount of noise in the input image based on an optically black region corresponding to the input image; and Input data is fed into a neural network to perform the task of recognition or regression.

13. An image processing method, characterized in that include: Acquire real data and input data, wherein the input data includes a training image and a noise map representing the amount of noise in the training image based on an optical black area corresponding to the training image; as well as Use input data and real data to learn a neural network for performing recognition or regression tasks.

14. A non-transitory computer-readable storage medium storing a program for causing a computer to execute an image processing method, It is characterized in that The image processing method comprises: Acquire real data and input data, the input data including a training image and a noise map representing the amount of noise in the training image based on an optical black area corresponding to the training image; and Use input data and real data to learn a neural network for performing recognition or regression tasks.

15. A method for generating a learned model, characterized in that include: Acquire real data and input data, wherein the input data includes a training image and a noise map representing the amount of noise in the training image based on an optical black area corresponding to the training image; as well as Use input data and real data to learn a neural network for performing recognition or regression tasks.

16. An image processing device, characterized in that include: an acquirer configured to acquire real data and input data, the input data including a training image and a noise map representing a noise amount in the training image based on an optical black area corresponding to the training image; as well as A learner is configured to use the input data and the real data to learn a neural network for performing the task of recognition or regression.

Images