Image processing device, image processing method, and imaging device

Abstract

To provide an image processing device capable of suppressing at least a conventional technology problem occurring when generating an image where a partial region of the original image has been upscaled using a method for upscaling an image per block.SOLUTION: An image processing device performs upscaling processing per block to synthesize an upscaling image per block into one upscaling image. When it generates an upscaling image of a partial region of the original image, the image processing device divides the partial region into a plurality of blocks in a predetermined position without reference to the location of the partial region in the original image.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and an imaging device, and in particular to a technique for enlarging an image. [Background technology]

[0002] The technology of enlarging an image (increasing the number of pixels) is also called upscaling and is widely used. In recent years, upscaling using a trained machine learning model (hereinafter, also referred to as machine learning (ML)-based upscaling) has been attracting attention (Patent Document 1).

[0003] The larger the image size to be upscaled (the more pixels there are), the greater the amount of memory and calculation required for upscaling. In particular, with ML-based upscaling, the amount of memory and calculation required increases exponentially with the increase in image size. In response to this, by upscaling the image in units of smaller blocks of a specific size, it is possible to upscale images of various sizes with the amount of memory required for upscaling processing of one block. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2022-505275 Summary of the Invention [Problem to be solved by the invention]

[0005] If you want to upscale only a partial area (desired part) of the original image, (1) After upscaling the entire original image, cut out the area corresponding to the desired part. (2) Cut out the desired part from the original image and upscale the cut-out part In these processing methods, when block-based upscaling as described in Patent Document 1 is applied, the following problems arise.

[0006] In method (1), upscaling processing for blocks that do not contain the desired part is wasted. In method (2), the extracted desired part is divided into blocks, so the positional relationship between the original image and each block changes depending on the position of the desired part. Therefore, the upscaling result of the same area included in the desired part changes depending on the extraction position of the desired part. These problems occur not only in ML-based upscaling, but also in upscaling based on pixel value interpolation, such as bicubic interpolation.

[0007] In one aspect, the present invention provides an image processing device that can at least alleviate the problems of the prior art that arise when generating an image by upscaling a partial region of an original image using a method of upscaling an image on a block-by-block basis. [Means for solving the problem]

[0008] In one aspect, the present invention provides an image processing device comprising a division means for dividing an image into a plurality of blocks, a processing means for performing an upscaling process for each block, and a synthesis means for synthesizing the upscaling images for each block into a single upscaling image, wherein when generating an upscaling image of a partial region of an original image, the division means divides the partial region into a plurality of blocks at a predetermined position regardless of the position of the partial region in the original image. Effect of the Invention

[0009] According to the present invention, it is possible to provide an image processing device that can at least alleviate the problems of the conventional technology that arise when generating an image by upscaling a partial region of an original image using a method of upscaling an image on a block-by-block basis. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an imaging apparatus as an example of an image processing apparatus according to an embodiment; [Diagram 2] Flowchart regarding upscaling processing in the first embodiment [Diagram 3] Flowchart regarding upscaling processing in the second embodiment [Figure 4] FIG. 13 is a diagram for explaining block division positions in upscaling processing; [Diagram 5] FIG. 11 is a schematic diagram for explaining an upscaling process in the second embodiment; [Figure 6] Flowchart regarding upscaling processing in the third embodiment [Figure 7] FIG. 13 is a diagram for explaining block division in the third embodiment; [Figure 8] Flowchart of block size determination processing in the fourth embodiment [Figure 9] FIG. 13 is a diagram for explaining a calculation process of the number of surplus pixels in the fourth embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0012] In the following embodiment, the present invention will be described with respect to a case where the present invention is implemented in an imaging device (digital camera). However, imaging functionality is not essential to the present invention. The present invention can be implemented in any electronic device capable of handling image data. In addition to imaging devices, such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0013] ●(First embodiment) FIG. 1 is a block diagram showing an example of a basic functional configuration of an imaging device 100 as an example of an image processing device according to a first embodiment of the present invention. Each functional block of the imaging device 100 can be implemented by software or a combination of software and hardware, except for parts that can obviously be realized only by hardware (for example, a lens of the optical system 101, pixels of the image sensor 102, etc.). For example, the functional block may be realized by dedicated hardware such as an ASIC. Also, the functional block may be realized by a processor such as a CPU executing a program stored in a memory. Note that a plurality of functional blocks may be realized by a common configuration (for example, one ASIC). Also, hardware that realizes a part of the function of a certain functional block may be included in hardware that realizes another functional block.

[0014] The optical system 101 has a plurality of lenses including a movable lens, a shutter, an aperture, and a motor and an actuator for driving the movable members. The movable lenses include a focus lens for adjusting the focal length of the optical system, and a zoom lens for adjusting the focal length (angle of view) of the optical system 101. The operation of the optical system 101 is controlled by a control circuit of the imaging device 100 or a CPU 103 serving as a controller.

[0015] The optical system 101 forms an optical image of a subject on the imaging surface of the imaging element 102. The imaging element 102 may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The imaging element 102 has a pixel array in which a plurality of pixels are arranged two-dimensionally, and a peripheral circuit for reading out a signal from each pixel. Each pixel accumulates a charge according to the amount of incident light by photoelectric conversion. A pixel signal group (analog image signal) representing the subject image formed on the imaging surface is obtained by reading out from each pixel a signal having a voltage according to the amount of charge accumulated during the exposure period. The analog image signal is A / D converted by the CPU 103 and stored in the RAM 104. Note that the imaging element 102 may have an A / D conversion function.

[0016] The CPU 103 loads a program stored in the ROM 107 into the RAM 104 and executes it to control the operation of each functional block of the imaging device 100, thereby realizing the functions of the imaging device 100. The CPU 103 also executes autofocus detection (AF) and autoexposure control (AE), as well as control of the operation of the optical system 101 and the imaging element 102 according to the results of AF and AE.

[0017] The ROM 107 is a rewritable non-volatile memory, and stores programs executed by the CPU 103, various setting values ​​of the imaging device 100, image data (GUI data) for a menu screen or an on-screen display (OSD), and the like.

[0018] The RAM 104 is used as a main memory for the CPU 103, a buffer for temporarily storing captured image data, a work memory for the image processing device 105, etc. Furthermore, a part of the RAM 104 may be used as a video memory for storing image data for display.

[0019] The image processing device 105 applies predetermined image processing to image data stored in the RAM 104, generates signals and image data according to the application, and acquires and / or generates various information. The image processing device 105 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing device 105 may be configured such that a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) executes software to realize a specific function. The image processing device 105 outputs the acquired or generated information and data to the CPU 103 or the RAM 104 according to the application.

[0020] The image processing applied by the image processing device 105 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when an image sensor is provided with a color filter, and is a process for interpolating values ​​of color components that are not included in the individual pixel data that constitutes the image data. Color interpolation processing is also called demosaic processing. The correction processing can include white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the imaging optical system 101 (image restoration), correction of the effects of peripheral light falloff of the optical system 101, color correction, and the like. The detection process may include detection of feature regions (for example, face regions or human body regions) and their movements, person recognition processing, and the like. Data processing may include processes such as area extraction (trimming), synthesis, scaling, encoding and decoding, header information generation (data file generation), etc. Data processing also includes the generation of image data for display or image data for recording. The evaluation value calculation process can include processes such as generation of a signal and evaluation value used in automatic focus detection (AF) and generation of an evaluation value used in automatic exposure control (AE). The special effects processing may include adding a blur effect, changing color tones, relighting, and the like. Note that these are examples of processes that can be applied by the image processing device 105, and do not limit the processes that can be applied by the image processing device 105. Some of the processes listed here may be performed by the CPU 103 executing a program.

[0021] In this embodiment, the image processing device 105 performs upscaling. The upscaling can be performed using any known method. When performing ML-based upscaling, the image processing device 105 is provided with a trained machine learning model configured to upscale an input image of a specific size at a predetermined magnification and output the upscaling. The machine learning model can be implemented using, for example, a two-dimensional convolutional neural network (CNN). Note that the machine learning model that performs ML-based upscaling is known, as described in, for example, Patent Document 1, and therefore a detailed description thereof will be omitted. When performing non-ML-based upscaling, the image processing device 105 performs upscaling using a known method that does not use a machine learning model, such as bicubic interpolation.

[0022] Regardless of the method used for upscaling, in this embodiment, upscaling is performed in units of blocks of a predetermined size. A block is a rectangular image area, and has a size that is sufficiently small compared to the size of an image (e.g., a captured image) handled by the imaging device 100. Here, as an example, the block is a square area with 64 or 128 pixels on a side. This is sufficiently small compared to the resolution of images that are generally used (several million pixels or more).

[0023] The recording medium 106 is, for example, a semiconductor memory card that is detachable from the imaging device 100. The CPU 103 records image data files for recording that are stored in the RAM 104 onto the recording medium 106. The CPU 103 also reads out image data files recorded onto the recording medium 106 and stores them into the RAM 104. If the imaging device 100 has a communication interface, the CPU 103 may record and read out image data files onto an external storage device accessible via the communication interface.

[0024] Display unit 108 is provided, for example, on the surface of the housing of imaging device 100, and displays captured or played images, information about imaging device 100, a graphical user interface, etc. By capturing moving images and displaying the captured moving images in parallel and continuously, display unit 108 can function as an electronic viewfinder (EVF). The operation of causing display unit 108 to function as an EVF is called live view display, and a moving image displayed in live view is called a live view image.

[0025] The operation unit 109 is a general term for input devices (such as buttons, switches, and dials) that are provided for a user to input various instructions to the imaging device 100. The input devices constituting the operation unit 109 have names according to the functions assigned to them. For example, the operation unit 109 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, direction keys, and a decision key. The release switch is a switch for recording still images, and the CPU 103 recognizes the half-pressed state of the release switch as a shooting preparation instruction and the full-pressed state as a shooting start instruction. In addition, the CPU 103 recognizes the video recording switch pressed in a shooting standby state as a video recording start instruction, and recognizes the video recording switch pressed during video recording as a recording stop instruction. Note that the functions assigned to the same input device may be variable. In addition, the input device may be a software button or key using a touch display. In addition, the operation unit 109 may include an input device compatible with a non-contact input method such as voice input or gaze input.

[0026] The imaging device 100 has a function to generate a different image by upscaling a part (partial region) of an image (hereinafter, referred to as a partial upscaling function). The partial upscaling function can be used for various purposes. For example, it can be used to realize an electronic zoom function or provided as one of the image editing functions.

[0027] The electronic zoom function realizes a zoom magnification exceeding the maximum zoom magnification of the optical system 101 by using a partial upscaling function. The size of the partial area to be upscaled becomes smaller as the zoom magnification increases. When the electronic zoom function is enabled and the zoom magnification specified through the operation unit 109 exceeds the maximum zoom magnification of the optical system 101, the CPU 103 determines a partial area to be cut out from the captured image and notifies the image processing device 105. The image processing device 105 generates an upscaling image of the partial area notified by the CPU 103. The image processing device 105 also generates display image data and recording image data based on the upscaling image.

[0028] The image editing function provides the user with the ability to change the color tone and brightness of an image, or to rotate or crop the image, for example, when image data recorded on the recording medium 109 is displayed on the display unit 108. When cropping is applied, for example, the CPU 103 notifies the image processing device 105 of the position and size of the cropped partial area. The image processing device 105 generates an upscaled image of the partial area notified by the CPU 103. The image processing device 105 generates display image data or recording image data based on the upscaled image.

[0029] Note that these are only a few examples of uses of the partial upscaling function and are not intended to limit the uses of the partial upscaling function. Details of the partial upscaling function executed by the image processing device 105 will be described below.

[0030] Before describing the details of the partial upscaling function performed by the image processing device 105, a problem that may arise in the prior art will be described.

[0031] As described above, when generating an image by upscaling a partial region of an image, (1) After upscaling the entire input image, cut out the area corresponding to the original partial area. (2) Extract a subregion from the input image and upscale the extracted region. There are two possible processing methods:

[0032] Figures 4(a) and 4(b) show the difference in block division positions when upscaling is performed on a block-by-block basis to generate upscaling images of the same partial region using method (1) and method (2).

[0033] In the case of method (1), the entire original image is divided into blocks, so the division positions of the original image do not change regardless of the position of the partial region. On the other hand, in the case of method (2), the partial regions are divided into blocks, so the division positions of the original image change depending on the position of the partial region. In other words, in method (1), the image included in each block does not change depending on the position of the partial region, but in method (2), the image included in each block changes depending on the position of the partial region.

[0034] The result of upscaling depends on the original image to which the upscaling is applied, so when using method (2), different positions of a subregion will result in different upscaling results for the same region in the original image.

[0035] Method (1) does not have the problem of method (2), but it does have a problem that does not occur in method (2), that is, upscaling processing is performed unnecessarily on blocks that do not include partial regions.

[0036] In this embodiment, in consideration of these problems, it is possible to obtain an upscaling result that is not dependent on the position of a partial region while avoiding unnecessary upscaling processes. Fig. 4(c) shows a schematic diagram of the upscaling process in this embodiment. In this embodiment, the division of blocks is performed in the same manner as in method (1), but the upscaling process is not performed on blocks that do not include a partial region that ultimately requires an upscaling image.

[0037] 2 is a flowchart of the upscaling process in this embodiment. The upscaling process is executed by the image processing device 105 under the control of the CPU 103, for example.

[0038] Here, it is assumed that data for one frame of an original image is stored in RAM 104, and information identifying a partial area (hereinafter also referred to as a cut-out area) of the original image from which an upscaling image is to be generated is notified to image processing device 105 by CPU 103.

[0039] The original image may be, for example, a frame image of a moving image or a still image of one frame. Also, it may be an image before recording or an image already recorded on the recording medium 109. Also, the partial region may be one specified by the user or one determined by the CPU 103. The partial region is a rectangular region, and information identifying the partial region may be, for example, the image coordinates of two diagonal vertices. Note that the shape of the partial region and the information identifying the partial region are merely examples.

[0040] In S201, the image processing device 105 (division means) divides the entire original image into blocks of a predetermined size. As described above, the block size can be determined according to the available capacity of the RAM 104 and the input image size that the machine learning model used for ML-based upscaling can handle (for example, equal to the maximum size of the input image). The block size may be notified by the CPU 103 or may be set in advance.

[0041] Since the blocks are rectangular, the image processing device 105 divides the original image in the horizontal and vertical directions. The image processing device 105 stores image data for each block in the RAM 104 in association with information that identifies the block (e.g., a raster order number, block coordinates (horizontal number, vertical number), etc.).

[0042] For example, the numbering in raster order is such that the block in the upper left corner is numbered 1, and numbers are assigned horizontally from the second block onwards up to the rightmost block, and then numbers are assigned horizontally starting from the leftmost block in the next row of blocks. Also, the block coordinates can be (horizontal number, vertical number) with the block in the left corner being (1,1).

[0043] In S202, the image processing device 105 targets the nth block (n is an integer equal to or greater than 1) in accordance with a predetermined processing order as the upscaling process target. At this point, the upscaling process is not executed on the nth block.

[0044] In S203, the image processing device 105 determines whether the block to be processed includes a cut-out area, and if it is determined that the block includes the cut-out area, executes S204, and if not, executes S205. The image processing device 105 can determine whether the block to be processed includes a cut-out area by using the position information of the cut-out area, the block size, and the number or position information of the block to be processed.

[0045] In S204, the image processing device 105 (processing means) executes upscaling processing on the block to be processed, and stores the processing result in the RAM 104. After that, the image processing device 105 executes S205.

[0046] In S205, the image processing device 105 increments the block number of the processing target stored in, for example, the RAM 104 by 1. After that, the image processing device 105 executes S206.

[0047] In S206, the image processing device 105 determines whether or not the upscaling process has been performed on all blocks. If it is determined that the upscaling process has been performed on all blocks, the image processing device 105 executes S207, and if not, executes S202.

[0048] In S207, the image processing device 105 (compositing means) composites (stitches) the upscaled images for each block stored in the RAM 104 according to the block numbers. This results in an upscaled image of the original image excluding blocks that do not include the cropped area. Then, the image processing device 105 crops out an area corresponding to the cropped area in the original image from the obtained upscaled image, and creates a final upscaled image.

[0049] Note that the synthesis process can be omitted by storing the upscaling results for each block in the RAM 104 so that, for example, the pixel arrangement is maintained. The image processing device 105 stores the finally obtained upscaling image data of a partial region (cut-out region) of the original image in the RAM 104. After that, the image processing device 105 generates at least one of image data for display and image data for recording based on the data of the upscaling image.

[0050] According to this embodiment, when generating an upscaled image of a partial region of an original image using block-based upscaling, the entire original image is divided into blocks, and then upscaling is performed only on the block that contains the partial region. This avoids the problems that arise when dividing a partial region into blocks, and also makes it possible to avoid unnecessary upscaling.

[0051] ●(Second embodiment) Next, a second embodiment of the present invention will be described. This embodiment differs from the first embodiment in the upscaling process executed by the image processing device 105. Therefore, the following description will focus on the upscaling process in this embodiment.

[0052] This embodiment is similar to method 2 in that a partial region for generating an upscaled image is extracted from the original image and upscaling processing is performed in block units, but the method of dividing the blocks is different.

[0053] Fig. 5(a) shows the correspondence between the cut-out partial region and the blocks in this embodiment when generating an upscaled image for the same partial region as in Fig. 4. In this embodiment, the cut-out partial region is divided into blocks at the same positions as when the entire image is divided into blocks.

[0054] This makes it possible to avoid the problems of method 2, which are caused by the division position differing depending on the position of the partial region. On the other hand, by dividing the block at a position independent of the position of the partial region, the outermost block may not be filled with pixels (may have an area where no pixels exist). For blocks where the ratio of non-existent pixels (or the ratio of areas where no pixels exist) relative to the block size (total number of pixels) is small (equal to or less than a predetermined threshold), upscaling processing may be performed as is.

[0055] On the other hand, for blocks in which the ratio of non-existent pixels to the block size (total number of pixels) is small (below a predetermined threshold), the non-existent pixels are filled before the upscaling process is performed. Any method can be used to fill in the pixels, for example, filling in achromatic pixels. Figure 5(b) shows an example in which the non-existent pixels are filled in by moving the position of the outermost block to a position included in the partial region.

[0056] In the example of FIG. 5(b), the position of a block having an area where no pixels exist is moved in a direction that reduces the area where no pixels exist until the block is filled with pixels. This can also be said to move the position of the block into a partial area. For example, a block with an area where no pixels exist on the upper side is moved downward, and a block with an area where no pixels exist on the lower side is moved upward. Also, a block with an area where no pixels exist on the left side is moved right, and a block with an area where no pixels exist on the right side is moved left. Blocks where areas where no pixels exist in multiple directions, such as blocks at the four corners, are moved diagonally. For example, the block at the upper left corner is moved in the lower right direction because there are areas where no pixels exist on the upper and left sides (or moved downward until the upper area is filled with pixels, and then moved right until the left area is filled with pixels).

[0057] The left side of Fig. 5(b) shows the direction of movement of the outermost block, and the right side shows the position of the block after movement. The same problem occurs for the moved block as in method 2, but the upscaling result of the moved block uses only the part where the partial region existed at the position before the movement, so the impact on the entire upscaling image is small.

[0058] In addition, although Fig. 5(b) shows a case where all the outermost blocks are moved, it is also possible to move only the blocks where the ratio of non-existent pixels exceeds a threshold. If the entire outer periphery of the partial region coincides with the dividing position of the blocks, there is no block having an area where pixels do not exist, so it goes without saying that pixel filling is not necessary.

[0059] Fig. 3 is a flowchart relating to the upscaling process in this embodiment. The upscaling process is executed by the image processing device 105 under the control of the CPU 103, for example. In Fig. 3, the steps in which the same processes as those in the first embodiment are executed are given the same reference numerals as those in Fig. 2, and the explanation thereof will be omitted. In this embodiment, too, it is assumed that the CPU 103 has notified the image processing device 105 of information specifying a partial region in which an upscaling image is to be generated.

[0060] In S301, the image processing device 105 (cutting means) cuts out a partial region from an original image. The image processing device 105 stores data of the cut out partial region in the RAM 104.

[0061] In S302, the image processing device 105 divides the cut-out partial region into blocks. Here, the image processing device 105 divides the partial region into blocks at the same positions as when the entire original image is divided into blocks. Specifically, when the partial region is expressed in the coordinate system of the original image, the image processing device 105 divides the partial region at positions corresponding to multiples of the horizontal size of the block in the horizontal direction and at positions corresponding to multiples of the vertical size of the block in the vertical direction. The image processing device 105 treats the outermost blocks as normal-sized blocks even if at least one of the horizontal and vertical directions is smaller than the block size.

[0062] In S303, the image processing device 105 determines whether the block to be processed requires pixel filling. For example, the image processing device 105 can determine that the block to be processed requires pixel filling when the ratio of an area where no pixels exist to the entire block to be processed exceeds a threshold. If it is determined that the block to be processed requires pixel filling, the image processing device 105 executes S304, and if not, executes S204.

[0063] In S304, the image processing device 105 performs pixel filling on the block to be processed. As described above, pixel filling can be performed in various ways. The image processing unit 105 stores information specifying an area where pixels have been filled or an area where pixels have not been filled (an area where pixels originally existed) in association with the block. The image processing device 105 can, for example, record position information of two diagonal vertices of a rectangular area where pixels existed in association with the block. When a block is moved for pixel filling, the image processing device 105 also stores the horizontal and vertical movement amounts in association with the block.

[0064] If it is determined in S206 that the upscaling process has been performed on all the blocks, the image processing device 105 executes S308, and if not, executes S202.

[0065] In S308, the image processing device 105 synthesizes (stitches) the upscaled images for each block stored in the RAM 104 according to the block numbers, thereby obtaining an upscaled image of the portion of the original image excluding blocks that do not include the clipped region.

[0066] For a block on which pixel filling has been performed, the image processing device 105 uses only an image area of ​​the upscaled image of the block that corresponds to a portion that was included in the block before the pixel filling for synthesis. If the block has been moved, the image processing device 105 uses only an image area of ​​the upscaled image of the block that corresponds to a portion that was included in the block at the position before the movement for synthesis.

[0067] In this embodiment, even if a partial region for generating an upscaled image is extracted and then upscaling processing is performed for each block, the same effect as in the first embodiment can be obtained for blocks filled with pixels. Moreover, since there are very few blocks including regions without pixels, if any, their influence on the upscaled image is sufficiently small.

[0068] ●(Third embodiment) Next, a third embodiment of the present invention will be described. This embodiment can be selectively performed with the first or second embodiment. Since this embodiment can be performed by the imaging device 100, the following description will be given assuming that the embodiment is performed by the imaging device 100.

[0069] In the first and second embodiments, the original image is divided at a fixed position to suppress the influence of the position of the cutout region on the upscaling result. However, since upscaling processing is required for blocks that include even a small amount of the cutout region, a more efficient upscaling processing may be performed, for example, when processing is required in a short time.

[0070] This embodiment is an upscaling process that can be performed instead of the upscaling process according to the first or second embodiment. The upscaling process according to this embodiment can be performed, for example, when a user explicitly selects to perform the upscaling process according to this embodiment from a menu screen, when a predetermined shooting mode is set, etc. The upscaling process according to this embodiment can be performed not only when the settings of the imaging device 100 satisfy predetermined conditions, but also when the CPU 103 determines that a dynamic factor such as the processing load of the image processing device 105 satisfies predetermined conditions.

[0071] 6 is a flowchart of the upscaling process according to this embodiment. The upscaling process is executed by the image processing device 105 under the control of the CPU 103, for example.

[0072] In S601, the image processing device 105 acquires information for identifying a cut-out region of an original image. In this embodiment, the cut-out region may be, for example, a region designated by a user or a region determined by the CPU 103. The partial region is a rectangular region, and the information for identifying the partial region may be, for example, image coordinates of two diagonal vertices.

[0073] In S602, the image processing device 105 identifies the cut-out region from the information acquired in S601, and divides the cut-out region into a plurality of blocks. As described above, the image processing device 105 of this embodiment can divide the cut-out region into a plurality of blocks at a position corresponding to the position of the cut-out region. FIG. 7 is a schematic diagram showing an example of the cut-out region and block division. 501 is the original image, and 502 is the cut-out region. Moreover, the blocks are squares with A pixels on each side, and each block is indicated by a dotted line. Here, a case is shown in which the cut-out region is divided into a plurality of blocks so that adjacent blocks have an overlapping region (number of pixels da) of about several pixels. However, the overlapping region da is not necessarily required. 503 indicates an area used for upscaling processing of the cut-out region.

[0074] In this example, the upper left vertex of the original image 501 is set as the origin (0,0), and an image coordinate system is used that has an x-axis extending horizontally to the right from the origin and a y-axis extending vertically downward from the origin. Also, the image coordinates of the upper left vertex of the cutout area are (x0,y0), and the coordinates of the lower right vertex are (x1,y1).

[0075] The area 503 used for the upscaling process is determined so as to include the cut-out area 502. If the size of one side of the block is A and the number of pixels in the overlapping area with an adjacent block is da, then the number of blocks in the horizontal direction Nx and the number of blocks in the vertical direction Ny are respectively given by: Nx>(x1-x0-2da) / (A-da) (1) Ny>(y1-y0-2da) / (A-da) (2) If no overlapping area is provided between blocks, then da = 0 in equations (1) and (2).

[0076] In S603, the image processing device 105 performs upscaling processing for each block divided in S602, and stores the result in, for example, the RAM 104. The upscaling processing may or may not be ML-based upscaling processing.

[0077] In S604, the image processing device 105 determines whether or not the upscaling process has been performed on all blocks. If it is determined that the upscaling process has been performed on all blocks, the image processing device 105 executes S605, and if not, it repeatedly executes S603.

[0078] In S605, the image processing device 105 synthesizes (stitches) the upscaled images for each block stored in the RAM 104 according to the block positions. At this time, if there is an overlapping area between the blocks, the overlapping area is deleted before synthesizing. Also, if at least one of the horizontal size (x1-x0) and vertical size (y1-y0) of the cut-out area is not a multiple of A, the synthesized image includes an area outside the cut-out area. In this case, the image processing device 105 removes the area outside the cut-out area (cuts out the part corresponding to the cut-out area from the synthesized image) to obtain the final upscaled image.

[0079] In the upscaling process according to this embodiment, the division position of the blocks may change depending on the cropped region, and therefore the effect achieved by the upscaling process according to the first or second embodiment cannot be obtained. However, since the upscaling process according to this embodiment is easier to process than the upscaling process according to the first or second embodiment, by making it possible to selectively execute the upscaling process according to this embodiment, flexible upscaling can be performed according to the user's intention, the state of the imaging device 100, and the like.

[0080] ●(Fourth embodiment) Next, a third embodiment of the present invention will be described. This embodiment relates to a method for determining a block size in the block-unit upscaling process in the first to third embodiments. Since this embodiment can be implemented by the imaging device 100, the following description will be given assuming that it is implemented by the imaging device 100.

[0081] As described above, performing an upscaling process on an unnecessary region leads to waste of time and power required for the upscaling process. Therefore, it is important to use an appropriate block size. Here, a method for determining a block size in a case where a first and a second machine learning model with different input image sizes are prepared in advance to perform an ML-based upscaling process will be described.

[0082] 8 is a flowchart of a block size determination process according to the present embodiment. This determination process can be executed by the image processing device 105 at the beginning of S204 or S603 (before the upscaling process starts).

[0083] In S801, the image processing device 105 divides an original image (when performed in S204) or a cutout region (when performed in S603) into a first block size and a second block size. The first block size is the input image size of a first machine learning model, and the second block size is the input image size of a second machine learning model.

[0084] In S802, the image processing device 105 calculates the number of pixels (the number of extra pixels) outside the cutout area where the upscaling process is performed for the case where the image is divided into the first block size and the case where the image is divided into the second block size. Then, the image processing device 105 compares the calculated number of extra pixels.

[0085] 9 is a diagram related to the calculation process of the number of surplus pixels in S802. Here, a case is shown in which the same block division method as in the third embodiment is executed. 701 indicates the original image, 702 indicates the cut-out region, 703 indicates the region used for the upscaling process when divided into the first block size, and 704 indicates the region used for the upscaling process when divided into the second block size. The image coordinate system and the coordinates of the cut-out region are the same as in the third embodiment.

[0086] The number of pixels M in the cut-out area is calculated using the following formula (3). M = (x1-x0) * (y1-y0) (3)

[0087] In addition, the number of pixels M1 of the area 703 used for upscaling processing when divided by the first block size is calculated using the number of blocks in the horizontal direction Nx, the number of blocks in the vertical direction Ny, the block size A, and the number of overlapping pixels da with adjacent blocks, according to the following equation (4). M1 = [Nx1 * A - (Nx1-1)*da] * [Ny1 * A - (Ny1-1)*da] (4) And the number of extra pixels is given by M1 - M.

[0088] The number of pixels M2 of the area 704 used for upscaling processing when divided by the second block size is calculated using the number of blocks in the horizontal direction Nx2, the number of blocks in the vertical direction Ny2, the block size B, and the number of overlapping pixels with adjacent blocks db, as shown in the following equation (5). M2 = [Nx2 * B - (Nx2-1)*db] * [Ny2 * B - (Ny2-1)*db] (5) And the number of extra pixels is given by M2 - M.

[0089] In S803, the image processing device 105 judges whether the number of surplus pixels when dividing by the first block size is smaller than the number of surplus pixels when dividing by the second block size. If it is judged that the number of surplus pixels when dividing by the first block size is smaller than the number of surplus pixels when dividing by the second block size, the image processing device 105 executes S804, and if not, executes S805.

[0090] In S804, the image processing device 105 determines to perform upscaling processing on each block divided by the first block size using a first machine learning model.

[0091] In S805, the image processing device 105 determines to perform upscaling processing on each block divided by the second block size using a second machine learning model.

[0092] According to this embodiment, for example, when multiple machine learning models with different input image sizes are prepared for upscaling processing, it is possible to use a more appropriate block size, and efficient upscaling processing can be realized. Although the above description has been given of a case where there are two types of block sizes, the present invention is similarly applicable to a case where there are three or more types of block sizes.

[0093] The upscaling process according to the third and fourth embodiments can be used to realize a crop or electronic zoom function, or can be provided as one of the image editing functions, similar to the first and second embodiments. In the case of cropping or electronic zoom, the number of pixels after cropping may be fixed. In this case, since the extra pixels described in the fourth embodiment are known in advance, it is also possible to switch the learning model to be referred to in upscaling according to the number of cropped pixels. In addition, although the embodiment using an imaging device has been described, it is also possible to execute the process in an electronic device that does not have an imaging function, such as an information processing device.

[0094] (Other embodiments) In the above embodiment, the partial regions are divided at the same positions as when the entire original image is divided. However, the problem of Method 2 can be solved by dividing the partial regions at fixed positions in the coordinate system of the original image, without necessarily dividing the partial regions at the same positions as when the entire original image is divided.

[0095] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0096] The disclosure of the present embodiment includes the following image processing device, image processing method, imaging device, and program. (Item 1) A dividing means for dividing an image into a plurality of blocks; a processing means for performing an upscaling process for each of the blocks; a synthesis means for synthesizing the upscaled images for each block into one upscaled image, When generating an upscaled image of a partial region of the original image, the dividing means divides the partial region into a plurality of blocks at a predetermined position regardless of a position of the partial region in the original image; 13. An image processing device comprising: (Item 2) When generating an upscaled image of the partial region, The dividing means divides the entire original image into a plurality of blocks at the predetermined positions, the processing means does not perform the upscaling process on any block among the plurality of blocks that does not include the partial region. 2. The image processing device according to item 1, (Item 3) 3. The image processing device according to item 2, further comprising a cutout means for cutting out a portion corresponding to the partial region from the one upscaled image combined by the combination means. (Item 4) When generating an upscaled image of the partial region, the dividing means divides the partial region cut out from the original image into a plurality of blocks at the predetermined position; 2. The image processing device according to item 1, (Item 5) 5. The image processing device according to item 4, wherein the processing means performs the upscaling process after filling in the non-existent pixels for blocks among the plurality of blocks that include areas where no pixels exist. (Item 6) 6. The image processing device according to item 5, wherein the processing means compensates for the non-existent pixels by moving a position of a block including an area where the pixel does not exist into the partial area. (Item 7) 7. The image processing device according to item 5 or 6, characterized in that for an upscaling image of a block in which pixels have been filled, the synthesis means uses only an area corresponding to a portion that was included in the block before the pixels were filled. (Item 8) The image processing device according to any one of items 5 to 7, characterized in that the processing means does not fill in the non-existent pixels for blocks among the plurality of blocks in which the proportion of the area in which the pixels do not exist is equal to or less than a threshold value. (Item 9) 9. The image processing device according to any one of items 1 to 8, wherein the predetermined position is based on a multiple of a size of the block. (Item 10) The processing means performs the upscaling process using a trained machine learning model that uses the block as an input image; The size of the block is determined according to the size of an input image that the trained machine learning model can handle. 10. The image processing device according to any one of items 1 to 9, (Item 11) 11. The image processing device according to any one of items 1 to 10, wherein the dividing means is capable of selectively dividing the partial region into a plurality of blocks at a position according to a position of the partial region. (Item 12) The image processing device according to item 11, characterized in that the dividing means divides the partial region into a plurality of blocks at a position according to the position of the partial region when a setting or a dynamic factor of the image processing device satisfies a predetermined condition. (Item 13) 13. The image processing device according to any one of items 1 to 12, wherein the division means selects a size of the plurality of blocks from a plurality of sizes. (Item 14) Item 14. The image processing device according to item 13, wherein the multiple sizes are input image sizes of multiple machine learning models that perform the upscaling process. (Item 15) 15. The image processing device according to item 13 or 14, characterized in that the division means selects the first block size when, when dividing into a plurality of blocks of a first block size, a block including the partial region among the plurality of blocks contains a smaller number of pixels that are not in the partial region than when dividing into a plurality of blocks of a second block size. (Item 16) An imaging element; 16. The image processing device according to any one of items 1 to 15, wherein an image captured by the imaging element is used as an original image; An imaging device comprising: (Item 17) 17. The imaging device according to item 16, wherein the upscaling process is used to realize an electronic zoom function. (Item 18) An image processing method executed by an image processing device, comprising: Dividing an image into a number of blocks; performing an upscaling process for each of the blocks; combining the block-wise upscaled images into one upscaled image; When generating an upscaled image of a partial region of the original image, The dividing step divides the partial region into a plurality of blocks at a predetermined position, regardless of a position of the partial region in the original image. 13. An image processing method comprising: (Item 19) A program for causing a computer to function as each of the means possessed by the image processing device according to any one of items 1 to 15.

[0097] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0098] 100: imaging device, 101: optical system, 102: imaging element, 103: CPU, 104: RAM, 105: image processing device, 106: recording medium, 107: ROM, 108: display unit, 109: operation unit

Claims

1. A dividing means for dividing an image into a plurality of blocks; a processing means for performing an upscaling process for each of the blocks; a synthesis means for synthesizing the upscaled images for each block into one upscaled image, When generating an upscaled image of a partial region of the original image, the dividing means divides the partial region into a plurality of blocks at a predetermined position regardless of a position of the partial region in the original image; 13. An image processing device comprising:

2. When generating an upscaled image of the partial region, The dividing means divides the entire original image into a plurality of blocks at the predetermined positions, the processing means does not perform the upscaling process on any block among the plurality of blocks that does not include the partial region.

2. The image processing device according to claim 1,

3. 3. The image processing apparatus according to claim 2, further comprising: a cutout means for cutting out a portion corresponding to the partial region from the single up-scaled image combined by the combining means.

4. When generating an upscaled image of the partial region, the dividing means divides the partial region cut out from the original image into a plurality of blocks at the predetermined position; 2. The image processing device according to claim 1,

5. 5. The image processing apparatus according to claim 4, wherein the processing means performs the upscaling process after filling in, for a block among the plurality of blocks, an area in which no pixels exist, the non-existent pixels.

6. 6. The image processing apparatus according to claim 5, wherein said processing means compensates for the non-existent pixels by moving a position of a block including the region where no pixels exist into the partial region.

7. 6. The image processing device according to claim 5, wherein the synthesizing means uses, for the upscaling image of a block in which pixels have been filled, only an area corresponding to a portion that was included in the block before the pixels were filled.

8. 6. The image processing apparatus according to claim 5, wherein said processing means does not perform compensation for non-existent pixels for blocks in which the ratio of the area in which the pixels do not exist is equal to or less than a threshold value among the plurality of blocks.

9. 2. The image processing apparatus of claim 1, wherein the predetermined position is based on a multiple of a size of the block.

10. The processing means performs the upscaling process using a trained machine learning model that uses the block as an input image; The size of the block is determined according to the size of an input image that the trained machine learning model can handle.

2. The image processing device according to claim 1,

11. 2. The image processing apparatus according to claim 1, wherein the dividing means is capable of selectively dividing the partial area into a plurality of blocks at a position corresponding to the position of the partial area.

12. The image processing device according to claim 11, characterized in that the dividing means divides the partial region into a plurality of blocks at a position corresponding to a position of the partial region when a setting or a dynamic factor of the image processing device satisfies a predetermined condition.

13. 2. The image processing apparatus according to claim 1, wherein the dividing means selects a size of the plurality of blocks from a plurality of sizes.

14. The image processing device according to claim 13 , wherein the multiple sizes are input image sizes of multiple machine learning models that execute the upscaling process.

15. The image processing device according to claim 13, characterized in that the division means selects the first block size when, when the image is divided into a plurality of blocks of a first block size, a block including the partial region among the plurality of blocks contains a smaller number of pixels that are not the partial region than when the image is divided into a plurality of blocks of a second block size.

16. An imaging element; The image processing device according to claim 1 , wherein an image captured by the imaging element is used as an original image; An imaging device comprising:

17. 17. The image pickup apparatus according to claim 16, wherein the upscaling process is used to realize an electronic zoom function.

18. An image processing method executed by an image processing device, comprising: Dividing an image into a number of blocks; performing an upscaling process for each of the blocks; combining the block-wise upscaled images into one upscaled image; When generating an upscaled image of a partial region of the original image, The dividing step divides the partial region into a plurality of blocks at a predetermined position, regardless of a position of the partial region in the original image.

13. An image processing method comprising:

19. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 15.

Images