Image processing device, image processing method, and imaging device

Abstract

To provide an image processing device capable of suppressing at least a conventional technology problem occurring when generating an image where a partial region of the original image has been upscaled using a method for upscaling an image per block.SOLUTION: An image processing device performs upscaling processing per block to synthesize an upscaling image per block into one upscaling image. When it generates an upscaling image of a partial region of the original image, the image processing device divides the partial region into a plurality of blocks in a predetermined position without reference to the location of the partial region in the original image.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and an imaging device, and in particular to a technique for enlarging an image. [Background technology]

[0002] The technology of enlarging an image (increasing the number of pixels) is also called upscaling and is widely used. In recent years, upscaling using a trained machine learning model (hereinafter referred to as machine learning (ML)-based upscaling) has been attracting attention (Patent Document 1).

[0003] The larger the image size to be upscaled (the more pixels there are), the greater the amount of memory and calculation required for upscaling. In particular, with ML-based upscaling, the amount of memory and calculation required increases exponentially with increasing image size. In contrast, by upscaling images in smaller, specific-sized blocks, it is possible to upscale images of various sizes with the amount of memory required for upscaling a single block. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2022-505275 Summary of the Invention [Problem to be solved by the invention]

[0005] If you want to upscale only a partial area (desired part) of the original image, (1) After upscaling the entire original image, cut out the area corresponding to the desired part. (2) Cut out the desired part from the original image and upscale the cut-out part In these processing methods, when block-based upscaling as described in Patent Document 1 is applied, the following problems arise.

[0006] In method (1), upscaling of blocks that do not contain the desired part is wasted. In method (2), the extracted desired part is divided into blocks, so the positional relationship between the original image and each block changes depending on the position of the desired part. As a result, the upscaling result of the same area included in the desired part changes depending on the extraction position of the desired part. These problems are not limited to ML-based upscaling, but also occur in upscaling based on pixel value interpolation, such as bicubic interpolation.

[0007] In one aspect, the present invention provides an image processing device that can at least alleviate the problems of the prior art that arise when generating an image by upscaling a partial region of an original image using a method of upscaling an image on a block-by-block basis. [Means for solving the problem]

[0008] In one aspect, the present invention comprises: Target The image processing apparatus includes a dividing means for dividing an image into a plurality of blocks, a processing means for performing an upscaling process for each block, and a combining means for combining the upscaled images for each block into one upscaled image. An image processing device , With image processing equipment When generating an upscaled image of a partial region of the original image, the dividing means divides the partial region into As the target image , regardless of the location of the subregion in the original image, For the original image An image processing device is provided that divides an image into a plurality of blocks at predetermined positions. [Effects of the Invention]

[0009] According to the present invention, an image processing device can be provided that can at least alleviate the problems of the prior art that arise when generating an image by upscaling a partial region of an original image using a method of upscaling an image on a block-by-block basis. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an imaging device as an example of an image processing device according to an embodiment; [Figure 2] Flowchart for upscaling processing in the first embodiment [Figure 3] Flowchart of upscaling processing in the second embodiment [Figure 4] FIG. 10 is a diagram illustrating block division positions in upscaling processing; [Figure 5] FIG. 10 is a schematic diagram illustrating an upscaling process according to the second embodiment; [Figure 6] Flowchart of upscaling processing in the third embodiment [Figure 7] FIG. 10 is a diagram for explaining block division in the third embodiment. [Figure 8] Flowchart for block size determination processing in the fourth embodiment [Figure 9] FIG. 13 is a diagram for explaining a calculation process of the number of surplus pixels in the fourth embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0011] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] In the following embodiments, the present invention will be described with reference to an imaging device (digital camera). However, imaging functionality is not essential to the present invention. The present invention can be implemented in any electronic device capable of handling image data. In addition to imaging devices, such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0013] ●(First embodiment) FIG. 1 is a block diagram showing an example of the basic functional configuration of an imaging device 100 as an example of an image processing device according to a first embodiment of the present invention. Each functional block of the imaging device 100 can be implemented by software or a combination of software and hardware, except for parts that can clearly be realized only by hardware (e.g., lenses included in the optical system 101, pixels of the image sensor 102, etc.). For example, a functional block may be realized by dedicated hardware such as an ASIC. Alternatively, a functional block may be realized by a processor such as a CPU executing a program stored in memory. Note that multiple functional blocks may be realized by a common configuration (e.g., a single ASIC). Also, hardware that realizes part of the functions of one functional block may be included in hardware that realizes another functional block.

[0014] The optical system 101 has a plurality of lenses including a movable lens, a shutter, an aperture, and motors and actuators for driving the movable members. The movable lenses include a focus lens for adjusting the focal length of the optical system, and a zoom lens for adjusting the focal length (angle of view) of the optical system 101. The operation of the optical system 101 is controlled by a control circuit of the imaging device 100 or a CPU 103 serving as a controller.

[0015] The optical system 101 forms an optical image of a subject on the imaging surface of the image sensor 102. The image sensor 102 may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The image sensor 102 has a pixel array in which multiple pixels are arranged two-dimensionally, and peripheral circuits for reading out signals from each pixel. Each pixel accumulates charge according to the amount of incident light through photoelectric conversion. A signal having a voltage according to the amount of charge accumulated during the exposure period is read out from each pixel, thereby obtaining a group of pixel signals (analog image signals) representing the subject image formed on the imaging surface. The analog image signals are A / D converted by the CPU 103 and stored in the RAM 104. Note that the image sensor 102 may also have an A / D conversion function.

[0016] The CPU 103 loads a program stored in the ROM 107 into the RAM 104 and executes it to control the operation of each functional block of the imaging device 100 and realize the functions of the imaging device 100. The CPU 103 also performs autofocus detection (AF) and autoexposure control (AE), and controls the operation of the optical system 101 and the imaging element 102 in accordance with the results of AF and AE.

[0017] The ROM 107 is a rewritable nonvolatile memory that stores programs executed by the CPU 103, various setting values ​​of the imaging device 100, image data (GUI data) for menu screens and on-screen displays (OSDs), and the like.

[0018] The RAM 104 is used as a main memory for the CPU 103, a buffer for temporarily storing captured image data, a work memory for the image processing device 105, etc. Furthermore, a part of the RAM 104 may be used as a video memory for storing image data for display.

[0019] The image processing device 105 applies predetermined image processing to image data stored in the RAM 104, generates signals and image data according to the application, and acquires and / or generates various types of information. The image processing device 105 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing device 105 may be configured such that a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) executes software to realize a specific function. The image processing device 105 outputs the acquired or generated information and data to the CPU 103 or the RAM 104 according to the application.

[0020] The image processing applied by the image processing device 105 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation, also known as demosaicing, is performed when the image sensor is equipped with a color filter, and is a process of interpolating the values ​​of color components that are not included in the individual pixel data that make up the image data. Correction processing includes white balance adjustment, tone correction, light This may include processing such as correction of image degradation caused by optical aberrations in the optical system 101 (image restoration), correction of the effects of vignetting in the optical system 101, and color correction. The detection process may include detection of characteristic regions (for example, face regions or human body regions) and their movements, person recognition processing, and the like. Data processing can include processes such as area extraction (trimming), compositing, scaling, encoding and decoding, header information generation (data file generation), etc. Data processing also includes the generation of image data for display or image data for recording. The evaluation value calculation process can include processes such as generating signals and evaluation values ​​used in autofocus (AF) detection, and generating evaluation values ​​used in automatic exposure control (AE). Special effect processing can include adding a blur effect, changing color tones, relighting, and the like. Note that these are examples of processes that can be applied by the image processing device 105, and do not limit the processes that can be applied by the image processing device 105. Some of the processes listed here may be performed by the CPU 103 executing a program.

[0021] In this embodiment, upscaling is performed by the image processing device 105. Upscaling can be performed using any known method. When performing ML-based upscaling, the image processing device 105 is provided with a trained machine learning model configured to upscale an input image of a specific size at a predetermined magnification and output the upscaled image. The machine learning model can be implemented using, for example, a two-dimensional convolutional neural network (CNN). Note that the machine learning model for performing ML-based upscaling is known, as described in, for example, Patent Document 1, and therefore a detailed description thereof will be omitted. When performing non-ML-based upscaling, the image processing device 105 performs upscaling using a known method that does not use a machine learning model, such as bicubic interpolation.

[0022] Regardless of the method used for upscaling, in this embodiment, upscaling is performed in units of blocks of a predetermined size. A block is a rectangular image area, and is sufficiently small compared to the size of an image (e.g., a captured image) handled by the imaging device 100. Here, as an example, a block is assumed to be a square area with 64 or 128 pixels on a side, which is sufficiently small compared to the resolution of commonly used images (several million pixels or more).

[0023] The recording medium 106 is, for example, a semiconductor memory card that is detachable from the imaging device 100. The CPU 103 records image data files for recording that are stored in the RAM 104 onto the recording medium 106. The CPU 103 also reads out image data files that are recorded on the recording medium 106 and stores them in the RAM 104. If the imaging device 100 has a communication interface, the CPU 103 may record and read out image data files to and from an external storage device that can be accessed via the communication interface.

[0024] Display unit 108 is provided, for example, on the surface of the housing of imaging device 100, and displays captured or played back images, information about imaging device 100, a graphical user interface, etc. By capturing moving images and displaying the captured moving images in parallel and continuously, display unit 108 can function as an electronic viewfinder (EVF). The operation of display unit 108 functioning as an EVF is called live view display, and the moving image displayed in live view is called a live view image.

[0025] The operation unit 109 is a collective term for input devices (such as buttons, switches, and dials) provided for the user to input various instructions to the imaging device 100. The input devices constituting the operation unit 109 are named according to the assigned functions. For example, the operation unit 109 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, and a confirmation key. The release switch is a switch for recording still images, and the CPU 103 recognizes a half-pressed state of the release switch as an instruction to prepare for shooting and a full-pressed state as an instruction to start shooting. The CPU 103 also recognizes a press of the video recording switch in shooting standby mode as an instruction to start video recording, and a press of the video recording switch during video recording as an instruction to stop recording. Note that the functions assigned to the same input device may be variable. The input device may also be software buttons or keys using a touch display. The operation unit 109 may also include an input device compatible with non-contact input methods such as voice input and eye-gaze input.

[0026] The imaging device 100 has a function of generating a separate image by upscaling a part (partial region) of an image (hereinafter referred to as a partial upscaling function). have The partial upscaling function can be used in a variety of applications, such as to realize an electronic zoom function or as one of the image editing functions.

[0027] The electronic zoom function realizes a zoom magnification exceeding the maximum zoom magnification of the optical system 101 by using a partial upscaling function. The size of the partial area to be upscaled becomes smaller as the zoom magnification increases. When the electronic zoom function is enabled and the zoom magnification specified via the operation unit 109 exceeds the maximum zoom magnification of the optical system 101, the CPU 103 determines a partial area to be cropped from the captured image and notifies the image processing device 105. The image processing device 105 generates an upscaled image of the partial area notified by the CPU 103. The image processing device 105 also generates image data for display or image data for recording based on the upscaled image.

[0028] The image editing function may be, for example, 6 When image data recorded in the image file is displayed on the display unit 108, the CPU 103 provides the user with functions to change the color tone and brightness of the image, and to apply rotation, cropping, etc. When cropping is applied, for example, the CPU 103 notifies the image processing device 105 of the position and size of the cropped partial area. The image processing device 105 generates an upscaled image of the partial area notified by the CPU 103. The image processing device 105 generates image data for display or image data for recording based on the upscaled image.

[0029] Note that these are only a few examples of uses of the partial upscaling function and do not limit the uses of the partial upscaling function. The partial upscaling function executed by the image processing device 105 will be described in detail below.

[0030] Before describing the details of the partial upscaling function performed by the image processing device 105, problems that may arise in the prior art will be described.

[0031] As described above, when generating an image by upscaling a partial region of an image, (1) After upscaling the entire input image, cut out the area corresponding to the original partial area. (2) Extract a subregion from the input image and upscale the extracted region. There are two possible ways to deal with this.

[0032] Figures 4(a) and 4(b) show the difference in block division positions when upscaling is performed on a block-by-block basis and an upscaling image of the same partial region is generated using methods (1) and (2).

[0033] In the case of method (1), the entire original image is divided into blocks, so the division position of the original image does not change regardless of the position of the partial region. On the other hand, in the case of method (2), the partial region is divided into blocks, so the division position of the original image changes depending on the position of the partial region. In other words, in method (1), the image included in each block does not change depending on the position of the partial region, but in method (2), the image included in each block changes depending on the position of the partial region.

[0034] The result of upscaling depends on the original image to which the upscaling is applied. Therefore, when using method (2), different positions of subregions will result in different upscaling results for the same region in the original image.

[0035] Method (1) does not have the problem of method (2). On the other hand, method (1) has a problem that does not occur in method (2), namely, unnecessary upscaling processing is performed on blocks that do not contain partial regions.

[0036] In this embodiment, these problems are taken into consideration, and it is possible to obtain an upscaling result that is independent of the position of a partial region while avoiding unnecessary upscaling processes. Fig. 4(c) shows a schematic diagram of the upscaling process in this embodiment. In this embodiment, block division is performed in the same way as in method (1), but upscaling processes are not performed on blocks that do not include partial regions that ultimately require an upscaling image.

[0037] 2 is a flowchart of the upscaling process in this embodiment. The upscaling process is executed by the image processing device 105 under the control of the CPU 103, for example.

[0038] Here, it is assumed that data for one frame of the original image is stored in RAM 104, and information identifying the partial area (hereinafter also referred to as the cut-out area) of the original image from which the upscaled image is to be generated is notified to image processing device 105 by CPU 103.

[0039] The original image may be, for example, a frame image of a moving image or a still image of one frame. 6 The partial area may be an image already recorded in the memory. The partial area may be specified by the user or determined by the CPU 103. The partial area is a rectangular area, and information identifying the partial area may be, for example, the image coordinates of two diagonal vertices. Note that the shape of the partial area and the information identifying the partial area are merely examples.

[0040] In S201, the image processing device 105 (division means) divides the entire original image into blocks of a predetermined size. As described above, the block size can be determined according to the available capacity of the RAM 104 and the input image size that the machine learning model used for ML-based upscaling can handle (for example, equal to the maximum size of the input image). The block size may be notified by the CPU 103 or may be set in advance.

[0041] Since the blocks are rectangular, the image processing device 105 divides the original image horizontally and vertically. The image processing device 105 stores image data for each block in the RAM 104 in association with information identifying the block (e.g., a raster order number, block coordinates (horizontal number, vertical number), etc.).

[0042] For example, the raster order number is given to the block in the upper left corner as number 1, and numbers are assigned horizontally from the second block onwards until the rightmost block is reached, and numbers are assigned horizontally from the leftmost block in the next row of blocks. Also, the block coordinates can be set as (horizontal number, vertical number), with the block coordinates of the leftmost block being (1,1).

[0043] In S202, the image processing device 105 targets the nth block (n is an integer equal to or greater than 1) in accordance with a predetermined processing order as the upscaling processing target. At this point, the upscaling processing is not executed on the nth block.

[0044] In S203, the image processing device 105 determines whether the block to be processed includes a cutout area, and if it is determined that the block includes a cutout area, executes S204, and if it is not determined that the block includes a cutout area, executes S205. The image processing device 105 can determine whether the block to be processed includes a cutout area by using the position information of the cutout area, the block size, and the number or position information of the block to be processed.

[0045] In S204, the image processing device 105 (processing means) performs upscaling processing on the block to be processed, and stores the processing result in the RAM 104. Thereafter, the image processing device 105 executes S205.

[0046] In S205, the image processing device 105 increments the block number of the processing target stored in, for example, the RAM 104 by 1. After that, the image processing device 105 executes S206.

[0047] In S206, the image processing device 105 determines whether or not upscaling processing has been performed on all blocks. If it is determined that upscaling processing has been performed on all blocks, the image processing device 105 executes S207, and if not, executes S202.

[0048] In S207, the image processing device 105 (combining means) composites (stitches) the upscaled images for each block stored in the RAM 104 according to the block number. This results in an upscaled image of the portion of the original image excluding blocks that do not include the cropped area. The image processing device 105 then crops out the area of ​​the obtained upscaled image that corresponds to the cropped area in the original image, and uses this as the final upscaled image.

[0049] Note that the synthesis process can be omitted by storing the upscaling results for each block in the RAM 104 so that, for example, the pixel arrangement is maintained. The image processing device 105 stores the finally obtained upscaling image data of a partial region (cut-out region) of the original image in the RAM 104. Thereafter, the image processing device 105 generates at least one of image data for display and image data for recording based on the upscaling image data.

[0050] According to this embodiment, when generating an upscaled image of a partial region of an original image using block-based upscaling, the entire original image is divided into blocks, and then upscaling is performed only on the block containing the partial region. This avoids the problems that arise when dividing a partial region into blocks, and also makes it possible to avoid unnecessary upscaling.

[0051] ●(Second embodiment) Next, a second embodiment of the present invention will be described. This embodiment differs from the first embodiment in the upscaling process executed by the image processing device 105. Therefore, the following description will focus on the upscaling process in this embodiment.

[0052] In this embodiment, a partial region for generating an upscaled image is extracted from the original image, and upscaling processing is performed in units of blocks. (2) However, the method of dividing the blocks is different.

[0053] Fig. 5(a) shows the correspondence between the cut-out partial region and the blocks in this embodiment when generating an upscaled image for the same partial region as in Fig. 4. In this embodiment, the cut-out partial region is divided into blocks at the same positions as when the entire image is divided into blocks.

[0054] This allows us to reduce the number of divisions due to the difference in the division position depending on the position of the partial region. (2) This can avoid the above problem. On the other hand, by dividing the block at a position independent of the position of the partial region, the outermost blocks may not be filled with pixels (may have areas where no pixels exist). For blocks where the ratio of non-existent pixels (or the ratio of areas where no pixels exist) to the block size (total number of pixels) is small (below a predetermined threshold), upscaling processing may be performed as is.

[0055] On the other hand, for blocks where the ratio of missing pixels relative to the block size (total number of pixels) is small (below a predetermined threshold), the missing pixels are filled in before upscaling. Any method can be used to fill in the pixels, such as filling in achromatic pixels. Figure 5(b) shows an example of filling in missing pixels by moving the position of the outermost block to a position included in the partial region.

[0056] In the example of Figure 5(b), the position of a block having an area without pixels is moved in a direction that reduces the area without pixels until the block is filled with pixels. This can also be said to move the position of the block within a partial area. For example, a block with an area without pixels at the top is moved downward, and a block with an area without pixels at the bottom is moved upward. Furthermore, a block with an area without pixels at the left is moved right, and a block with an area without pixels at the right is moved left. Blocks with areas without pixels in multiple directions, such as the blocks at the four corners, are moved diagonally. For example, the block in the upper left corner is moved downward and right because there are areas without pixels at the top and left (or it is moved downward until the area at the top is filled with pixels, and then moved right until the area at the left is filled with pixels).

[0057] The left side of Figure 5(b) shows the direction of movement of the outermost block, and the right side shows the position of the block after movement. (2) However, the upscaling result of the moved block uses only the part where the partial region existed at the position before the movement, so the effect on the entire upscaling image is small.

[0058] 5(b) shows the case where all the outermost blocks are moved, but it is also possible to move only the blocks where the proportion of non-existent pixels exceeds a threshold. If the entire periphery of the partial region coincides with the block division position, there is no block with an area where pixels do not exist, so it goes without saying that pixel filling is not necessary.

[0059] 3 is a flowchart relating to the upscaling process in this embodiment. The upscaling process is assumed to be executed by the image processing device 105 under the control of, for example, the CPU 103. In FIG. 3, steps that execute the same processes as in the first embodiment are given the same reference numerals as in FIG. 2, and descriptions thereof will be omitted. In this embodiment as well, it is assumed that the CPU 103 has notified the image processing device 105 of information that specifies a partial region for generating an upscaled image.

[0060] In S301, the image processing device 105 (cutting means) cuts out a partial region from the original image, and stores data of the cut out partial region in the RAM 104.

[0061] In S302, the image processing device 105 divides the cut-out partial region into blocks. Here, the image processing device 105 divides the partial region into blocks at the same positions as when the entire original image is divided into blocks. Specifically, when the partial region is expressed in the coordinate system of the original image, the image processing device 105 divides the partial region at positions that correspond to multiples of the horizontal size of the block in the horizontal direction and at positions that correspond to multiples of the vertical size of the block in the vertical direction. The image processing device 105 treats the outermost blocks as normal-sized blocks even if at least one of the horizontal and vertical directions is smaller than the block size.

[0062] In S303, the image processing device 105 determines whether the block to be processed requires pixel filling. For example, if the ratio of areas where no pixels exist to the entire block to be processed exceeds a threshold, the image processing device 105 can determine that the block to be processed requires pixel filling. If it is determined that the block to be processed requires pixel filling, the image processing device 105 executes S304, and if not, executes S204.

[0063] In S304, the image processing device 105 performs pixel filling on the block to be processed. As described above, pixel filling can be performed in various ways. DeviceThe image processing device 105 stores information specifying an area where pixels have been filled or an area where pixels have not been filled (an area where pixels originally existed) in association with the block. For example, the image processing device 105 can record position information of two diagonal vertices of a rectangular area where pixels existed in association with the block. When a block is moved for pixel filling, the image processing device 105 also stores the horizontal and vertical movement amounts in association with the block.

[0064] If it is determined in S206 that upscaling processing has been performed on all blocks, the image processing device 105 executes S308; if not, it executes S202.

[0065] In S308, the image processing device 105 synthesizes (stitches) the upscaled images for each block stored in the RAM 104 according to the block numbers, thereby obtaining an upscaled image of the portion of the original image excluding blocks that do not include the clipped region.

[0066] For a block on which pixel filling has been performed, the image processing device 105 uses only the image area of ​​the upscaled image of the block that corresponds to the portion that was included in the block before the pixel filling. If the block has been moved, the image processing device 105 uses only the image area of ​​the upscaled image of the block that corresponds to the portion that was included in the block at its position before the movement.

[0067] In this embodiment, even when a partial region for generating an upscaled image is extracted and then upscaling processing is performed for each block, the same effect as in the first embodiment can be obtained for blocks filled with pixels. Furthermore, since there are very few blocks, if any, that include regions without pixels, the impact on the upscaled image is sufficiently small.

[0068] ●(Third embodiment) Next, a third embodiment of the present invention will be described. This embodiment can be selectively performed together with the first or second embodiment. Since this embodiment can be performed by the image capture device 100, the following description will be given assuming that it is performed by the image capture device 100.

[0069] In the first and second embodiments, the original image is divided at fixed positions to reduce the effect of the position of the clipped area on the upscaling result. However, since upscaling processing must be performed on blocks that contain even a small amount of the clipped area, more efficient upscaling processing may be performed when, for example, processing in a short time is required.

[0070] This embodiment is an upscaling process that can be performed instead of the upscaling processes according to the first and second embodiments. The upscaling process according to this embodiment can be performed, for example, when a user explicitly selects from a menu screen to perform the upscaling process according to this embodiment, when a predetermined shooting mode is set, etc. The upscaling process according to this embodiment can be performed not only when the settings of the imaging device 100 satisfy predetermined conditions, but also when the CPU 103 determines that a dynamic factor, such as the processing load of the image processing device 105, satisfies predetermined conditions.

[0071] 6 is a flowchart of the upscaling process according to this embodiment. The upscaling process is executed by the image processing device 105 under the control of the CPU 103, for example.

[0072] In S601, the image processing device 105 acquires information for identifying a cutout region of the original image. In this embodiment, the cutout region may be, for example, specified by the user or determined by the CPU 103. The partial region is a rectangular region, and the information for identifying the partial region may be, for example, the image coordinates of two diagonal vertices.

[0073] In S602, the image processing device 105 identifies a cutout region from the information acquired in S601 and divides the cutout region into multiple blocks. As described above, the image processing device 105 of this embodiment can divide the cutout region into multiple blocks at a position corresponding to the position of the cutout region. FIG. 7 is a schematic diagram showing an example of a cutout region and block division. Reference numeral 501 denotes the original image, and 502 denotes the cutout region. The blocks are squares with A pixels on a side, and each block is indicated by a dotted line. Here, the cutout region is divided into multiple blocks so that adjacent blocks have an overlapping region (number of pixels da) of about several pixels. However, the overlapping region da is not necessarily required. Reference numeral 503 denotes an area used for upscaling the cutout region.

[0074] Here, an image coordinate system is used in which the upper left vertex of the original image 501 is set as the origin (0,0), the x-axis extends horizontally to the right from the origin, and the y-axis extends vertically downward from the origin. Also, the image coordinates of the upper left vertex of the cutout area are (x0,y0), and the coordinates of the lower right vertex are (x1,y1).

[0075] The area 503 used in the upscaling process is determined so as to include the cutout area 502. Therefore, if the size of one side of a block is A and the number of pixels in the overlapping area with an adjacent block is da, the number of blocks in the horizontal direction Nx and the number of blocks in the vertical direction Ny are respectively given by Nx>(x1-x0-2da) / (A-da) (1) Ny>(y1-y0-2da) / (A-da) (2) If there is no overlapping area between blocks, da = 0 in equations (1) and (2).

[0076] In S603, the image processing device 105 performs upscaling processing for each block divided in S602, and stores the result in, for example, the RAM 104. The upscaling processing may or may not be ML-based upscaling processing.

[0077] In S604, the image processing device 105 determines whether or not the upscaling process has been performed on all blocks. If it is determined that the upscaling process has been performed on all blocks, the image processing device 105 executes S605, and if not, it repeatedly executes S603.

[0078] In S605, the image processing device 105 composites (stitches) the upscaled images for each block stored in the RAM 104 according to the block positions. At this time, if there is an overlapping area between the blocks, the overlapping area is deleted before composite. Furthermore, if at least one of the horizontal size (x1-x0) and vertical size (y1-y0) of the cutout area is not a multiple of A, the composite image includes an area outside the cutout area. In this case, the image processing device 105 removes the area outside the cutout area (cuts out the portion corresponding to the cutout area from the composite image) to obtain the final upscaled image.

[0079] In the upscaling process according to this embodiment, the division position of the blocks can change depending on the cropped region, and therefore the effect achieved by the upscaling process according to the first or second embodiment cannot be obtained. However, since the upscaling process according to this embodiment is easier to process than the upscaling process according to the first or second embodiment, by making it possible to selectively execute the upscaling process according to this embodiment, flexible upscaling can be performed according to the user's intentions, the state of the imaging device 100, and the like.

[0080] ●(Fourth embodiment) Next, the present invention 4 An embodiment will be described. This embodiment relates to a method for determining a block size in block-based upscaling processing in the first to third embodiments. Since this embodiment can be implemented by the imaging device 100, the following description will be given assuming that it is implemented by the imaging device 100.

[0081] As explained above, performing upscaling on unnecessary regions wastes time and power required for the upscaling process. Therefore, it is important to use an appropriate block size. This section explains how to determine the block size when first and second machine learning models with different input image sizes are prepared in advance to perform ML-based upscaling.

[0082] 8 is a flowchart showing the block size determination process according to this embodiment. This determination process can be executed by the image processing device 105 at the beginning of S204 or S603 (before the upscaling process starts).

[0083] In S801, the image processing device 105 divides the original image (when performed in S204) or the cutout region (when performed in S603) into a first block size and a second block size. The first block size is the input image size of the first machine learning model, and the second block size is the input image size of the second machine learning model.

[0084] In S802, the image processing device 105 calculates the number of pixels (number of excess pixels) outside the clipping area where upscaling processing is performed for the case where the image is divided into blocks of the first size and the case where the image is divided into blocks of the second size. Then, the image processing device 105 compares the calculated numbers of excess pixels.

[0085] 9 is a diagram related to the calculation process of the number of excess pixels in S802. Here, the case where the same block division method as in the third embodiment is executed is shown. 701 indicates the original image, 702 indicates the cropped area, 703 indicates the area used for upscaling processing when divided into the first block size, and 704 indicates the area used for upscaling processing when divided into the second block size. The image coordinate system and the coordinates of the cropped area are the same as in the third embodiment.

[0086] The number of pixels M in the cutout area is calculated using the following formula (3). M=(x1-x0)*(y1-y0) (3)

[0087] Furthermore, the number of pixels M1 in the area 703 used for upscaling processing when divided by the first block size is calculated using the number of blocks in the horizontal direction Nx, the number of blocks in the vertical direction Ny, the block size A, and the number of overlapping pixels da with adjacent blocks, using the following equation (4): M1 = [Nx1 * A - (Nx1-1)*da] * [Ny1 * A - (Ny1-1)*da] (4) The number of extra pixels is then given by M1 - M.

[0088] The area used for upscaling processing when divided by the second block size area The number of pixels M2 of 704 is calculated by the following equation (5) using the number of blocks in the horizontal direction Nx2, the number of blocks in the vertical direction Ny2, the block size B, and the number of overlapping pixels db with adjacent blocks. M2 = [Nx2 * B - (Nx2-1)*db] * [Ny2 * B - (Ny2-1)*db] (5) The number of extra pixels is then given by M2 - M.

[0089] In S803, the image processing device 105 determines whether the number of excess pixels when dividing by the first block size is smaller than the number of excess pixels when dividing by the second block size. If it is determined that the number of excess pixels when dividing by the first block size is smaller than the number of excess pixels when dividing by the second block size, the image processing device 105 executes S804, and if not, executes S805.

[0090] In S804, the image processing device 105 determines to perform upscaling processing on each of the blocks divided by the first block size using the first machine learning model.

[0091] In S805, the image processing device 105 determines to perform upscaling processing on each block divided by the second block size using the second machine learning model.

[0092] According to this embodiment, for example, when multiple machine learning models with different input image sizes are prepared for upscaling processing, it is possible to use a more appropriate block size, thereby realizing efficient upscaling processing. Although the above description has been given of a case where there are two types of block sizes, the present invention is equally applicable to a case where there are three or more types of block sizes.

[0093] As with the first and second embodiments, the upscaling processing according to the third and fourth embodiments can be used to realize cropping or electronic zooming functions, or can be provided as one of the image editing functions. Cropping or electronic zooming may have a predetermined number of pixels after cropping. In this case, since the extra pixels described in the fourth embodiment are known in advance, it is also possible to switch the learning model referenced in upscaling depending on the number of cropped pixels. Furthermore, although the embodiments using an imaging device have been described, the processing can also be performed on electronic devices without imaging functions, such as information processing devices.

[0094] (Other embodiments) In the above embodiment, the partial regions are divided at the same positions as when the entire original image is divided. (2) The problem of (1) can be solved by dividing the partial regions at fixed positions in the coordinate system of the original image, without necessarily dividing the regions at the same positions as when the entire original image is divided.

[0095] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0096] The disclosure of the present embodiment includes the following image processing device, image processing method, imaging device, and program. (Item 1) a dividing means for dividing an image into a plurality of blocks; processing means for performing an upscaling process for each of the blocks; a synthesis means for synthesizing the upscaled images for each block into one upscaled image, When generating an upscaled image of a partial region of the original image, the dividing means divides the partial region into a plurality of blocks at predetermined positions regardless of the position of the partial region in the original image; 1. An image processing device comprising: (Item 2) When generating an upscaled image of the partial region, the dividing means divides the entire original image into a plurality of blocks at the predetermined positions; the processing means does not perform the upscaling process on any block among the plurality of blocks that does not include the partial region. 2. The image processing device according to item 1, (Item 3) 3. The image processing device according to item 2, further comprising: a cutting means for cutting out a portion corresponding to the partial region from the single upscaled image synthesized by the synthesis means. (Item 4) When generating an upscaled image of the partial region, the dividing means divides the partial region cut out from the original image into a plurality of blocks at the predetermined positions; 2. The image processing device according to item 1, (Item 5) 5. The image processing device according to item 4, wherein the processing means performs the upscaling process after filling in the missing pixels for blocks among the plurality of blocks that include areas where no pixels exist. (Item 6) 6. The image processing device according to item 5, wherein the processing means compensates for the absent pixels by moving the position of a block including an area where the pixels do not exist into the partial area. (Item 7) 7. The image processing device according to item 5 or 6, characterized in that for an upscaled image of a block in which pixels have been filled, the synthesis means uses only an area corresponding to a portion that was included in the block before the pixels were filled. (Item 8) 8. The image processing device according to any one of items 5 to 7, wherein the processing means does not fill in the missing pixels for blocks in which the proportion of the area in which the pixels do not exist is equal to or less than a threshold value among the plurality of blocks. (Item 9) 9. The image processing device according to any one of items 1 to 8, wherein the predetermined position is based on a multiple of the size of the block. (Item 10) the processing means performs the upscaling process using a trained machine learning model that uses the block as an input image; The size of the block is determined according to the size of the input image that the trained machine learning model can handle. 10. The image processing device according to any one of items 1 to 9, characterized in that: (Item 11) 11. The image processing device according to any one of items 1 to 10, wherein the dividing means is capable of selectively dividing the partial region into a plurality of blocks at positions according to the positions of the partial region. (Item 12) Item 12. The image processing device according to item 11, wherein the dividing means divides the partial region into a plurality of blocks at a position corresponding to the position of the partial region when a setting or a dynamic factor of the image processing device satisfies a predetermined condition. (Item 13) 13. The image processing device according to any one of items 1 to 12, wherein the dividing means selects a size of the plurality of blocks from a plurality of sizes. (Item 14) Item 14. The image processing device according to item 13, wherein the plurality of sizes are input image sizes of a plurality of machine learning models that perform the upscaling process. (Item 15) 15. The image processing device according to item 13 or 14, characterized in that the dividing means selects the first block size when, when dividing into a plurality of blocks of a first block size, the number of pixels that are not in the partial region contained in a block that includes the partial region among the plurality of blocks is smaller than when dividing into a plurality of blocks of a second block size. (Item 16) An imaging element; 16. The image processing device according to any one of items 1 to 15, wherein an image captured by the imaging element is used as an original image; An imaging device comprising: (Item 17) 17. The imaging device according to item 16, wherein the upscaling process is used to realize an electronic zoom function. (Item 18) An image processing method executed by an image processing device, Dividing an image into a plurality of blocks; performing an upscaling process for each of the blocks; combining the block-by-block upscaled images into one upscaled image; When generating an upscaled image of a partial region of the original image, The dividing step divides the partial region into a plurality of blocks at predetermined positions regardless of the position of the partial region in the original image. An image processing method comprising: (Item 19) 16. A program for causing a computer to function as each of the means possessed by the image processing device according to any one of items 1 to 15.

[0097] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0098] 100...imaging device, 101...optical system, 102...imaging element, 103...CPU, 104...RAM, 105...image processing device, 106...recording medium, 107...ROM, 108...display unit, 109...operation unit

Claims

1. A dividing means for dividing a target image into a plurality of blocks; processing means for performing an upscaling process for each of the blocks; a synthesis means for synthesizing the upscaled images for each block into one upscaled image, When the image processing device generates an upscaled image of a partial region of an original image, the dividing means divides the partial region into a plurality of blocks at predetermined positions in the original image, regardless of the position of the partial region in the original image, as the target image; 1. An image processing device comprising:

2. When the image processing device generates an upscaled image of the partial region, the dividing means divides the entire original image into a plurality of blocks at the predetermined positions as the target image; the processing means does not perform the upscaling process on any block among the plurality of blocks that does not include the partial region.

2. The image processing device according to claim 1, wherein:

3. 3. The image processing apparatus according to claim 2, further comprising: cutting means for cutting out a portion corresponding to the partial region from the single upscaled image synthesized by the synthesis means.

4. When the image processing device generates an upscaled image of the partial region, the dividing means divides the partial region cut out from the original image into a plurality of blocks at the predetermined position as the target image; 2. The image processing device according to claim 1, wherein:

5. 5. The image processing device according to claim 4, wherein the processing means performs the upscaling process after filling in the missing pixels for a block that includes an area where no pixels exist among the plurality of blocks.

6. 6. The image processing apparatus according to claim 5, wherein said processing means compensates for the non-existent pixels by moving a position of a block including the region where the pixel does not exist into the partial region.

7. 6. The image processing device according to claim 5, wherein the synthesizing means uses, for the upscaled image of a block in which pixels have been filled, only an area corresponding to a portion that was included in the block before the pixels were filled.

8. 6. The image processing device according to claim 5, wherein the processing means does not perform filling of the absent pixels in a block where the ratio of the area where the pixels do not exist is equal to or less than a threshold value among the plurality of blocks.

9. 2. The image processing apparatus according to claim 1, wherein the dividing means is capable of dividing the partial area into a plurality of blocks at positions corresponding to positions of the partial area in the original image as the target image.

10. The image processing device according to claim 9, characterized in that, when the settings of the image processing device satisfy predetermined conditions, the division means divides the partial area into a plurality of blocks at a position corresponding to the position of the partial area in the original image as the target image.

11. The image processing device described in claim 1, characterized in that when the target image is divided into a plurality of blocks of a first block size, if the number of pixels that are not in the partial area contained in a block containing the partial area among the plurality of blocks is smaller than when the target image is divided into a plurality of blocks of a second block size, the division means sets the block size for dividing the target image to the first block size.

12. The image processing device according to claim 11 , wherein the first block size and the second block size are input image sizes of a plurality of machine learning models that perform the upscaling process.

13. An imaging element; an image processing device according to claim 1 , wherein the original image is an image captured by the imaging element; An imaging device comprising:

14. 14. The imaging device according to claim 13, wherein the upscaling process is used to realize an electronic zoom function.

15. An image processing method executed by an image processing device, Dividing a target image into a plurality of blocks; performing an upscaling process for each of the blocks; combining the block-by-block upscaled images into one upscaled image; When generating an upscaled image of a partial region of the original image, The dividing step divides the partial region into a plurality of blocks at predetermined positions in the original image, regardless of the position of the partial region in the original image, using the partial region as the target image. An image processing method comprising:

16. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 12.

Images