An image processing apparatus calculates estimated pixel values of respective pixels of a provisional high-resolution image by interpolation on basis of pixel values in a reference frame, segmentalizes interior of the reference frame into an edge region, a texture region or a flat region and other on basis of pixel values of respective pixels, sets the respective pixels in the reference frame as target pixels one by one in sequence, calculates corresponding positions on the provisional high-resolution image, of the respective target pixels in fractional pixel accuracy on basis of information on the segmented regions including the target values, modifies the estimated pixel values so that difference from the pixel values of the target pixels to provisionally estimated pixel values obtained from the estimated pixel values of the provisional high-resolution image for pixels around the corresponding positions of the target pixels, becomes smaller, and obtains modified pixel values.