Image processing apparatus, image processing method, imaging apparatus, storage medium, and computer program product

Abstract

The invention discloses an image processing apparatus, an image processing method, an image pickup apparatus, a storage medium, and a computer program product. The image processing apparatus enlarges a partial region of an image using an enlargement method in which enlargement is performed in units of blocks. The apparatus divides an image into a plurality of blocks, performs an enlargement process on each block, and then combines the enlarged images of the plurality of blocks into one enlarged image. In a case where an enlarged image of a partial region of an original image is to be generated, the apparatus divides the partial region into a plurality of blocks at predetermined positions irrespective of a position of the partial region in the original image.

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method and an image pickup device, and particularly to a technique for enlarging an image. Background Art

[0002] The technique of enlarging an image (increasing the number of pixels), also known as "upscaling", is widely used. In recent years, upscaling using a trained machine learning model (hereinafter also referred to as "upscaling based on machine learning (ML)") has attracted attention (Japanese Patent Application No. 2022-505275).

[0003] As the size (number of pixels) of the image to be enlarged increases, the enlargement requires a larger amount of memory and computation. In particular, in ML-based enlargement, the required amount of memory and computation increases exponentially with the image size. In response to this, the image is enlarged in smaller units of a specific size (i.e., blocks), so that images of various sizes can be enlarged using only the amount of memory required to enlarge a single block.

[0004] However, differences in processing accuracy, processing time, and the like of the enlargement process occur depending on a method for setting blocks.

[0005] These problems are not limited to ML-based upscaling, and also occur in upscaling based on pixel value interpolation such as bicubic interpolation. Summary of the invention

[0006] In one aspect, the present invention provides an image processing device that is capable of at least suppressing the problems described in the prior art that occur when generating an image in which a partial area of ​​an original image is enlarged using a method for enlarging an image in units of blocks.

[0007] According to one aspect of the present invention, there is provided an image processing device, comprising: a segmentation component for segmenting an image into a plurality of blocks; a processing component for enlarging each of the plurality of blocks; and a combining component for combining the enlarged images of the plurality of blocks into one enlarged image, wherein, in a case where an enlarged image of a partial area of ​​an original image is to be generated, the segmentation component segments the partial area into a plurality of blocks at a predetermined position, regardless of the position of the partial area in the original image.

[0008] According to another aspect of the present invention, there is provided an image processing device, comprising: a segmentation component for segmenting an image into a plurality of blocks; a processing component for performing an enlargement process on each of the plurality of blocks; and a combining component for combining the enlarged images of the plurality of blocks into one enlarged image, wherein, in the case of generating an enlarged image of a partial area of ​​an original image, the processing component sets a processing area that includes the partial area and is smaller than the original image and larger than the partial area, and performs the enlargement process on each of the plurality of blocks obtained by segmenting the processing area according to the position of the partial area, and crops the partial area of ​​the one enlarged image obtained by the combining component.

[0009] According to one aspect of the present invention, there is provided an imaging device, comprising: an image sensor; and an image processing device according to the present invention, the image processing device using an image captured by the image sensor as an original image.

[0010] According to another aspect of the present invention, there is provided an image processing method, which is performed by an image processing device, and the image processing method comprises: dividing an image into a plurality of blocks; performing enlargement processing on each of the plurality of blocks; and combining the enlarged images of the plurality of blocks into one enlarged image, wherein, in the case of generating an enlarged image of a partial area of ​​an original image, the dividing divides the partial area into a plurality of blocks at a predetermined position regardless of the position of the partial area in the original image.

[0011] According to another aspect of the present invention, there is provided an image processing method, which is performed by an image processing device, and the image processing method includes: dividing an image into a plurality of blocks; performing an enlargement process on each of the plurality of blocks; and combining the enlarged images of the plurality of blocks into one enlarged image, wherein, in the case of generating an enlarged image of a partial area of ​​an original image, the enlargement process is performed by setting a processing area that includes the partial area and is smaller than the original image and larger than the partial area, and the enlargement process is performed on each of the plurality of blocks obtained by dividing the processing area according to the position of the partial area, and the partial area of ​​the one enlarged image obtained from the combination is cropped.

[0012] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram showing an example of a functional configuration of an image pickup apparatus serving as an example of an image processing apparatus according to the embodiment.

[0014] Figure 2is a flowchart showing the expansion process according to the first embodiment.

[0015] Figure 3 is a flowchart showing the expansion process according to the second embodiment.

[0016] FIG. 4A to FIG. 4C It is a diagram showing block division positions in the expansion process.

[0017] Figure 5A and Figure 5B is a diagram illustrating an expansion process according to the second embodiment.

[0018] Figure 6 is a flowchart showing the expansion process according to the third embodiment.

[0019] Figure 7 is a diagram showing block division in the third embodiment.

[0020] Figure 8 is a flowchart showing a block size determination process according to the fourth embodiment.

[0021] Fig. 9 is a diagram showing a process for calculating the number of excess pixels according to the fourth embodiment. DETAILED DESCRIPTION

[0022] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but the invention does not require the limitation of all these features, and a plurality of such features may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are assigned to the same or similar structures, and redundant descriptions thereof are omitted.

[0023] The following embodiments will describe the case where the present invention is applied to a camera device (digital camera). However, the present invention does not require a camera function. The present invention can be performed in any electronic device that can process image data. Examples of such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), smart phones, game consoles, robots, drones, and dashboard cameras in addition to camera devices. However, these are only examples, and the present invention can also be applied to other electronic devices.

[0024] First embodiment

[0025] Figure 11 is a block diagram showing an example of a basic functional configuration of an image pickup device 100 used as an example of an image processing device according to the first embodiment of the present invention. Except for components that can obviously be realized only by hardware (e.g., a lens provided in the optical system 101, and pixels in the image sensor 102, etc.), the functional blocks in the image pickup device 100 can be realized by software or by a combination of software and hardware. For example, the functional blocks can be realized by dedicated hardware such as an ASIC. Alternatively, the functional blocks can be realized by a processor such as a CPU executing a program stored in a memory. It is also noted that multiple functional blocks can be realized by a common configuration (e.g., a single ASIC). In addition, hardware for realizing some functions of a given functional block may be included in hardware for realizing other functional blocks.

[0026] The optical system 101 includes a plurality of lenses including a movable lens, a shutter, an aperture, a motor, an actuator, and the like, and the motor drives the movable member. The movable lens includes a focus lens for adjusting the focusing distance of the optical system and a zoom lens for adjusting the focal length (angle of view) of the optical system 101. The CPU 103 serving as a control circuit or controller of the imaging apparatus 100 controls the operation of the optical system 101.

[0027] The optical system 101 forms an optical image of a subject on an imaging surface of an image sensor 102. The image sensor 102 may be a well-known CCD or CMOS color image sensor having, for example, a primary color Bayer array color filter. The image sensor 102 includes a pixel array having a plurality of pixels arranged two-dimensionally and a peripheral circuit for reading out signals from the pixels. Each pixel accumulates a charge corresponding to the amount of incident light by photoelectric conversion. A group of pixel signals (analog image signals) representing an image of a subject formed on the imaging surface is obtained by reading out a signal having a voltage corresponding to the amount of charge accumulated during an exposure period from each pixel. The analog image signal is A / D converted by the CPU 103 and stored in the RAM 104. The image sensor 102 may have an A / D conversion function.

[0028] The CPU 103 controls the operation of each functional block of the imaging apparatus 100, for example, by loading a program stored in the ROM 107 into the RAM 104 and executing the program, so as to realize the functions of the imaging apparatus 100. In addition, the CPU 103 performs automatic focus detection (AF), automatic exposure control (AE), and control of the operations of the optical system 101 and the image sensor 102 according to the results of AF and AE.

[0029] The ROM 107 is a rewritable nonvolatile memory, and stores programs executed by the CPU 103, various types of setting values ​​of the imaging apparatus 100, and image data (GUI data) for menu screens and on-screen display (OSD), and the like.

[0030] The RAM 104 is used as a main memory of the CPU 103, a buffer for temporarily storing captured image data, and a work memory of the image processing device 105. A part of the RAM 104 can also be used as a video memory for storing image data for display.

[0031] The image processing device 105 generates signals and image data for different purposes, obtains and / or generates various types of information, etc. by applying predetermined image processing to the image data stored in the RAM 104. The image processing device 105 may be, for example, a dedicated hardware circuit such as an application-specific integrated circuit (ASIC) designed to implement a specific function. Alternatively, the image processing device 105 may be configured by executing software to implement a specific function through a processor such as a digital signal processor (DSP) or a graphics processing unit (GPU). The image processing device 105 outputs the obtained or generated information and data, etc. to the CPU 103 or the RAM 104, etc. according to the purpose of use.

[0032] The image processing applied by the image processing device 105 may include, for example, preprocessing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like.

[0033] Preprocessing includes signal amplification, baseline level adjustment and defective pixel correction.

[0034] The color interpolation process is performed when the image sensor is provided with a color filter, and the values ​​of the color components not included in the individual pixel data constituting the image data are interpolated. The color interpolation process is also called "demosaicing".

[0035] The correction processing may include white balance adjustment, color tone adjustment, correction of image degradation caused by optical aberration of the optical system 101 (image restoration), correction of the influence of vignetting in the optical system 101, color correction, and the like.

[0036] The detection process includes detecting a feature area (eg, a face area or a human body area) or motion in such an area, or a process for recognizing a person, and the like.

[0037] Data processing may include cropping (cropping), combining, scaling, encoding and decoding, and header information generation (data file generation). Data processing also includes the generation of display image data and recording image data.

[0038] The evaluation value calculation process may include processes such as generating a signal and an evaluation value used in automatic focus detection (AF) and generating an evaluation value used in automatic exposure control (AE).

[0039] Special effects processing includes adding bokeh effects, changing color tones and relighting.

[0040] Note that these are merely examples of processing that can be applied by the image processing apparatus 105, and are not intended to limit the processing applied by the image processing apparatus 105. Some of the processing described here can be realized by the CPU 103 executing a program.

[0041] In the present embodiment, the image processing device 105 performs enlargement. Any well-known method can be used for the enlargement. In the case of ML-based enlargement, a trained machine learning model configured to enlarge an input image of a specific size at a predetermined magnification and output the enlarged image is provided to the image processing device 105. For example, the machine learning model can be implemented by using a two-dimensional convolutional neural network (CNN). Note that the machine learning model for performing ML-based enlargement is well-known, such as the machine learning model described in Japanese Patent Application No. 2022-505275, etc., and therefore will not be described in detail here. In the case of non-ML-based enlargement, the image processing device 105 performs enlargement using, for example, a well-known method that does not use a machine learning model, such as bicubic interpolation.

[0042] Regardless of the method used for enlargement, the present embodiment assumes that the enlargement is performed in units of blocks of a predetermined size. A "block" is a rectangular image area and has a size that is sufficiently small relative to the size of the image (e.g., captured image) being processed by the camera device 100. Here, as an example, it is assumed that a block is a square area with 64 pixels or 128 pixels on a side. This is sufficiently small for the resolution of images commonly used (millions of pixels or more).

[0043] The recording medium 106 is, for example, a semiconductor memory card that can be attached to and detached from the imaging apparatus 100. The CPU 103 records the image data files for recording stored in the RAM 104 in the recording medium 106. The CPU 103 also reads out the image data files recorded in the recording medium 106 and stores them in the RAM 104. Note that if the imaging apparatus 100 has a communication interface, the CPU 103 can record image data files to and read out image data files from an external storage device that can be accessed through the communication interface.

[0044] The display unit 108 is provided on the surface of the housing of the imaging apparatus 100, for example, and displays a captured or reproduced image, information of the imaging apparatus 100, and a graphical user interface, etc. By simultaneously and continuously capturing a moving image and displaying the moving image, the display unit 108 can be used as an electronic viewfinder (EVF). The operation of causing the display unit 108 to function as an EVF is referred to as "live view display", and the moving image displayed in the live view display is referred to as a "live view image".

[0045] The "operation unit 109" is a general term for input devices (buttons, switches, dials, etc.) provided to the user to input various types of instructions to the camera apparatus 100. The input devices constituting the operation unit 109 are named according to the functions assigned thereto. For example, the operation unit 109 includes a release switch, a moving image recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, a direction key, an OK key, and the like. The release switch is a switch for recording a still image, and the CPU 103 recognizes a half-pressed state of the release switch as a shooting preparation instruction, and recognizes a fully pressed state of the release switch as a shooting start instruction. In addition, the CPU 103 recognizes pressing the moving image recording switch in a shooting standby state as a moving image recording start instruction, and recognizes pressing the moving image recording switch during recording a moving image as a recording stop instruction. Note that the function assigned to the same input device may be variable. In addition, the input device may include a software button or key using a touch screen. In addition, the operation unit 109 may include an input device corresponding to a non-contact input method such as voice input or gaze input.

[0046] The imaging device 100 has a function for generating another image by enlarging a part (partial area) of one image (hereinafter referred to as a "partial enlargement function"). The partial enlargement function can be used for various purposes. For example, the partial enlargement function can be used to implement an electronic zoom function, or can be provided as an image editing function.

[0047] The electronic zoom function uses a partial enlargement function to achieve a zoom ratio exceeding the maximum zoom ratio of the optical system 101. The size of the partial area to be enlarged decreases as the zoom ratio increases. In the case where the electronic zoom function is set to be effective and the zoom ratio specified by the operation unit 109 exceeds the maximum zoom ratio of the optical system 101, the CPU 103 determines a partial area to be cut out from the captured image and communicates the partial area to the image processing device 105. The image processing device 105 generates an enlarged image of the partial area communicated from the CPU 103. The image processing device 105 also generates image data for display and image data for recording based on the enlarged image.

[0048] For example, in the case where image data that has been recorded in the recording medium 106 is displayed in the display unit 108, the image editing function provides the user with functions for changing the color and brightness of the image, applying rotation and cropping, etc. For example, in the case where cropping is applied, the CPU 103 communicates the position and size of the cropped partial area to the image processing device 105. The image processing device 105 generates an enlarged image of the partial area communicated from the CPU 103. The image processing device 105 generates display image data and recording image data based on the enlarged image.

[0049] Note that these are only a few examples of the use of the partial enlargement function, and are not intended to limit the use of the partial enlargement function. The partial enlargement function performed by the image processing apparatus 105 will be described in detail below.

[0050] Before describing in detail a portion of the enlargement function performed by the image processing device 105, problems that may occur using the prior art will be described.

[0051] As mentioned above, when generating an image by enlarging a portion of an image, two processing methods can be considered:

[0052] (1) enlarge the entire input image and then crop out the region corresponding to the original partial region; or

[0053] (2) Cropping out a portion of the input image and then enlarging the cropped portion.

[0054] Figure 4A and Figure 4B The following shows the difference in the block division positions when enlargement is performed in units of blocks when an enlarged image of the same partial area is generated by method (1) and method (2).

[0055] With method (1), the entire original image is divided into blocks, so the division position of the original image does not change regardless of the position of the partial area. On the other hand, with method (2), the partial area is divided into blocks, so the division position of the original image changes according to the position of the partial area. In other words, with method (1), the image in each block does not change according to the position of the partial area, but with method (2), the image in each block changes according to the position of the partial area.

[0056] The result of the dilation depends on the original image to which the dilation is applied. Therefore, in the case of using method (2), if the position of the partial area is different, the result of the dilation for the same area in the original image will also be different.

[0057] The problem of method (2) does not occur in method (1). However, method (1) has a problem that does not occur in method (2), that is, the block that does not include the partial area is unnecessarily expanded.

[0058] In view of these problems, the present embodiment can obtain an expansion result that does not depend on the position of the partial region while avoiding unnecessary expansion processing. Figure 4C The enlargement process according to this embodiment is schematically shown. In this embodiment, block division is performed in the same manner as method (1), but the enlargement process is not performed on blocks that ultimately do not include a partial area that requires an enlarged image.

[0059] Figure 2 1 is a flowchart showing the enlargement process according to the present embodiment. For example, it is assumed that the enlargement process is performed by the image processing device 105 under the control of the CPU 103.

[0060] Here, it is assumed that data of an original image of one frame is stored in RAM 104 , and information for specifying a partial region of the original image where an expanded image is to be generated (hereinafter also referred to as “target region”) is communicated to image processing device 105 via CPU 103 .

[0061] For example, the original image may be a frame image from a moving image, or may be a still image from a single frame. Alternatively, the original image may be an image before recording, or may be an image that has been recorded in the recording medium 106. The partial area may be an area specified by the user, or may be an area determined by the CPU 103. The partial area is a rectangular area, and the information for specifying the partial area may be, for example, the image coordinates of the vertices at the two opposite corners of the rectangle. Note that the shape of the partial area and the information for specifying the partial area are only examples.

[0062] In step S201, the image processing device 105 (segmentation means) segments the entire original image into blocks of a predetermined size. Note that, as described above, the block size can be determined based on the available capacity of the RAM 104 and the size of the input image that the machine learning model used for ML-based enlargement can handle (e.g., equal to the maximum size of the input image), etc. The block size can be communicated through the CPU 103, or can be set in advance.

[0063] The blocks are rectangular, so the original image is divided in the horizontal and vertical directions by the image processing device 105. The image processing device 105 associates the image data of each block with information for specifying the block (for example, a raster order number or block coordinates (a number in the horizontal direction and a number in the vertical direction), etc.), and stores the image data in the RAM 104.

[0064] Raster sequential numbering, for example, takes the block at the upper left corner as the first block, and assigns numbers to the second and subsequent blocks in the horizontal direction. Once the rightmost block in the row has been assigned a number, numbers are assigned again in order in the horizontal direction starting from the leftmost block in the next row of blocks. At the same time, the block coordinates may be set so that the block coordinates of the block at the upper left corner are (1,1) (number in the horizontal direction, number in the vertical direction).

[0065] In step S202, the image processing apparatus 105 takes the nth block (where n is an integer of 1 or greater) in a predetermined processing order as a block to be subjected to expansion processing. However, expansion processing is not performed for the nth block at this time.

[0066] In step S203, the image processing device 105 determines whether the block to be processed includes the object area. If yes, proceed to step S204, otherwise proceed to step S205. The image processing device 105 can use the location information of the object area, the block size, and the number or location information of the block to be processed to determine whether the block to be processed includes the object area.

[0067] In step S204, the image processing apparatus 105 (processing means) performs enlargement processing on the block to be processed, and stores the processing result in the RAM 104. Then, the image processing apparatus 105 performs step S205.

[0068] In step S205 , the image processing apparatus 105 increases the block number to be processed stored in the RAM 104 by, for example, 1. Then, the image processing apparatus 105 proceeds to step S206 .

[0069] In step S206, the image processing device 105 determines whether all blocks have been expanded. If the image processing device 105 determines that all blocks have been expanded, it proceeds to step S207, otherwise it proceeds to step S202.

[0070] In step S207, the image processing device 105 (combining means) combines (joins) the enlarged images of the blocks stored in the RAM 104 according to the block numbers. As a result, an enlarged image of a part of the original image is obtained, thereby excluding the block that does not include the object area. Then, the image processing device 105 cuts out the area corresponding to the object area of ​​the original image in the obtained enlarged image, and takes the cut-out area as the final enlarged image.

[0071] Note that the combination process can be skipped by storing the expansion results for each block in the RAM 104 so that, for example, the order of pixels is maintained. The image processing device 105 stores the data of the expanded image of the partial area (object area) of the original image finally obtained in the RAM 104. The image processing device 105 then generates at least one of the image data for display and the image data for recording based on the data of the expanded image.

[0072] According to the present embodiment, in the case of generating an enlarged image of a partial area of ​​an original image by using an enlargement process performed in units of blocks, the entire original image is divided into blocks, and thereafter only the blocks including the partial area are enlarged. As a result, the problem that occurs in the case of dividing the partial area into blocks does not occur, and unnecessary enlargement processing can also be avoided.

[0073] Second embodiment

[0074] Next, a second embodiment of the present invention will be described. This embodiment is different from the first embodiment in the expansion process performed by the image processing device 105. Therefore, the following description will focus on the expansion process according to this embodiment.

[0075] This embodiment is similar to method (2) in that a partial area of ​​the original image to be used to generate the enlarged image is cropped and enlarged in units of blocks, but the block segmentation method is different.

[0076] Figure 5A Shown in the context of FIG. 4A to FIG. 4C Correspondence between the cropped partial region and the blocks in this embodiment when the enlarged image is generated by the same partial region In this embodiment, the cropped partial region is divided into blocks at the same position used when dividing the entire image into blocks.

[0077] This can avoid the problem of method (2) caused by the fact that the segmentation position is different depending on the position of the partial area. On the other hand, segmenting the block at a position irrelevant to the position of the partial area may result in the outermost block not being filled with pixels (i.e., having an area without pixels). For a block where the percentage of non-existent pixels relative to the block size (total number of pixels) (or the percentage of an area without pixels) is small (not greater than a predetermined threshold), the enlargement process can be performed as it is.

[0078] On the other hand, for a block where the percentage of non-existent pixels relative to the block size (total number of pixels) is small (not greater than a predetermined threshold), the expansion process is performed with the non-existent pixels supplemented. For example, any desired method such as supplementing non-color pixels may be used to supplement the pixels. Figure 5B An example is shown in which a non-existent pixel is supplemented by moving the position of the outermost block to a position within the partial area.

[0079] exist Figure 5B In the example of , the position of the block with an area without pixels is shifted in a direction that reduces the area without pixels until the block is filled with pixels. This can also be said to be shifting the position of the block into a partial area. For example, the block with an area without pixels at the top of the block is shifted downward, and the block with an area without pixels at the bottom of the block is shifted upward. Similarly, the block with an area without pixels on the left side of the block is shifted to the right, and the block with an area without pixels on the right side of the block is shifted to the left. Blocks with areas without pixels in multiple directions, such as the blocks at the four corners, are shifted diagonally. For example, for the block in the upper left corner, there are areas without pixels at the top and left, so the block is shifted in the lower right direction (or, downward until the area at the top is filled with pixels, and then right until the area on the left is filled with pixels).

[0080] Figure 5B The left side shows the shift direction of the outermost block, and the right side shows the position of the block after the shift. Although the same problem as method (2) occurs for the shifted block, the enlargement result for the shifted block is used only in the part where there is a partial area at the position before the shift, so there is almost no effect on the overall enlarged image.

[0081] Although Figure 5B The case where all outermost blocks are shifted is shown, but a configuration may be used in which only blocks whose percentage of non-existent pixels exceeds a threshold are shifted. It goes without saying that if the entire periphery of the partial area coincides with the division position of the block, there will be no block with an area without pixels, so supplementary pixels will not be necessary.

[0082] Figure 31 is a flowchart showing the enlargement processing according to the present embodiment. For example, it is assumed that the enlargement processing is performed by the image processing device 105 under the control of the CPU 103. Figure 3 In the embodiment, the steps of performing the same processing as that of the first embodiment have been given Figure 2 The present embodiment also assumes that information for specifying a partial region for which an expanded image is to be generated is communicated to the image processing device 105 by the CPU 103 .

[0083] In step S301 , the image processing device 105 (cropping means) cuts out a partial area from the original image and stores the data of the cut-out partial area in the RAM 104 .

[0084] In step S302, the image processing device 105 divides the cropped partial area into blocks. Here, the image processing device 105 divides the partial area into blocks at the same position used when dividing the entire original image into blocks. Specifically, when the partial area is represented by the coordinate system of the original image, the image processing device 105 divides the partial area in the horizontal direction at a position corresponding to a multiple of the horizontal size of the block, and divides the partial area in the vertical direction at a position corresponding to a multiple of the vertical size of the block. Even if the outermost block is smaller than the block size in at least one of the horizontal direction and the vertical direction, the image processing device 105 regards the outermost block as a block of normal size.

[0085] In step S303, the image processing device 105 determines whether the block to be processed needs to be supplemented with pixels. For example, the image processing device 105 may determine that the block to be processed needs to be supplemented with pixels when the percentage of the area without pixels relative to the entire block to be processed exceeds a threshold. If the image processing device 105 determines that the block to be processed needs to be supplemented with pixels, step S304 is performed, otherwise step S204 is performed.

[0086] In step S304, the image processing device 105 supplements pixels for the block to be processed. As described above, pixel supplementation can be performed by various methods. The image processing device 105 stores information for specifying an area to which pixels have been supplemented or an area to which pixels have not been supplemented (an area where pixels originally existed) in association with the block. For example, the image processing device 105 can record position information of vertices at two diagonal corners of a rectangular area where pixels exist in association with the block. In the case where the block is shifted for pixel supplementation, the image processing device 105 also stores the shift amounts in the horizontal and vertical directions in association with the block.

[0087] If the image processing device 105 determines in step S206 that the expansion process has been performed on all blocks, the process proceeds to step S308 ; otherwise, the process proceeds to step S202 .

[0088] In step S308, the image processing device 105 combines (joins) the expanded images of the blocks stored in the RAM 104 according to the block numbers. As a result, an expanded image of a part of the original image is obtained, excluding the blocks not including the object area.

[0089] Note that, for a block to which pixels have been supplemented, the image processing device 105 uses only the image area corresponding to the portion included in the block before the pixels are supplemented in the enlarged image of the block when combining. If the block has been shifted, the image processing device 105 uses only the image area corresponding to the portion included in the block at the position before the shift in the enlarged image of the block when combining.

[0090] According to the present embodiment, even in the case where each block is enlarged after first cutting out a partial area to generate an enlarged image, the same effect as that of the first embodiment can be achieved for blocks filled with pixels. In addition, there are very few blocks (if any) including areas without pixels, so these blocks will have almost no effect on the enlarged image.

[0091] Third embodiment

[0092] Next, a third embodiment of the present invention will be described. This embodiment is an embodiment that can be selectively performed with the first embodiment or the second embodiment. This embodiment can be performed by the imaging apparatus 100, so it will be described below as being performed by the imaging apparatus 100.

[0093] In the first and second embodiments, the influence of the position of the object area on the enlargement result is suppressed by dividing the original image at a fixed position. However, it is necessary to perform the enlargement processing on a block including even a small part of the object area, so that, for example, in the case where processing is required in a short period of time, more efficient enlargement processing can be performed.

[0094] The present embodiment is an enlargement process that can be performed instead of the enlargement process according to the first embodiment and the second embodiment. For example, when the user explicitly selects the enlargement process according to the present embodiment from the menu screen, or when a predetermined shooting mode is set, etc., the enlargement process according to the present embodiment can be performed. Not only when the setting of the camera 100 satisfies the predetermined condition, but also when, for example, the CPU 103 determines that a dynamic factor such as the processing load of the image processing device 105 satisfies the predetermined condition, the enlargement process according to the present embodiment can be performed.

[0095] Figure 61 is a flowchart showing the enlargement process according to the present embodiment. For example, it is assumed that the enlargement process is performed by the image processing device 105 under the control of the CPU 103.

[0096] In step S601, the image processing device 105 obtains information for specifying the object area of ​​the original image. In the present embodiment as well, for example, the object area may be an area specified by the user, or may be an area determined by the CPU 103. In addition, the partial area is a rectangular area, and the information for specifying the partial area may be, for example, the image coordinates of the vertices at the two diagonal corners of the rectangle.

[0097] In step S602, the image processing device 105 specifies the object region based on the information obtained in step S601 and divides the object region into a plurality of blocks. As described above, the image processing device 105 according to this embodiment can divide the object region into a plurality of blocks at positions based on the position of the object region. Figure 7 501 indicates an original image, and 502 indicates an object region. Each block is a square having A pixels on one side, and each block is indicated by a dotted line. Figure 7 The case where the object area is divided into a plurality of blocks so that adjacent blocks have an overlapping area (pixel number da) overlapping by a number of pixels is shown. However, the overlapping area da is not essential. 503 indicates an area (processing area) used for the expansion process of the object area.

[0098] Here, the following image coordinate system is assumed, in which the upper left vertex of the original image 501 is the origin (0,0), the x-axis extends horizontally to the right from the origin, and the y-axis extends vertically downward from the origin. In addition, (x0, y0) represents the image coordinates at the upper left vertex of the object area, and (x1, y1) represents the coordinates at the lower right vertex.

[0099] The area 503 used for the expansion processing is judged to contain the object area 502. Therefore, assuming that the size of each side of the block is A and the number of pixels in the overlapping area between adjacent blocks is da, the number of blocks Nx in the horizontal direction and the number of blocks Ny in the vertical direction are the minimum integers that satisfy the following corresponding formulas.

[0100] Nx > (x1 - x0 - 2da) / (A- da) (1)

[0101] Ny > (y1 - y0 - 2da) / (A-da) (2)

[0102] Note that if no overlapping area is provided between blocks, da=0 in equations (1) and (2).

[0103] In step S603, the image processing apparatus 105 performs expansion processing on each block obtained from the division performed in step S602, and stores the result in, for example, the RAM 104. The expansion processing may or may not be ML-based expansion processing.

[0104] In step S604, the image processing device 105 determines whether the expansion process has been performed on all blocks. If the image processing device 105 determines that the expansion process has been performed on all blocks, step S605 is performed, otherwise step S603 is repeated.

[0105] In step S605, the image processing device 105 combines (joins) the enlarged images of the blocks stored in the RAM 104 according to the block positions. If there are overlapping areas between the blocks at this time, the overlapping areas are deleted before combining. If at least one of the horizontal size (X1-X0) and the vertical size (Y1-Y0) of the object area is not a multiple of A, the combined image will include an area outside the object area. In this case, the image processing device 105 removes the area outside the object area (crops out the portion corresponding to the object area from the combined image), and takes the result as the final enlarged image.

[0106] With the expansion processing according to the present embodiment, the block division position differs depending on the object area, and thus the effect provided by the expansion processing according to the first embodiment and the second embodiment is not achieved. However, the expansion processing according to the present embodiment is easier than the expansion processing according to the first embodiment or the second embodiment, and thus can be selectively performed, which enables flexible expansion processing according to the user's intention and the state of the imaging apparatus 100, etc.

[0107] Fourth embodiment

[0108] Next, a fourth embodiment of the present invention will be described. This embodiment relates to a method for determining a block size in the enlargement process performed in units of blocks in the first to third embodiments. This embodiment can be performed by the imaging device 100, so the following will be described as being performed by the imaging device 100.

[0109] As mentioned above, enlarging an unnecessary area will cause a waste of time and power required for the enlargement process. Therefore, it is important to use an appropriate block size. Here, a method for determining a block size in the case of preparing a first machine learning model and a second machine learning model with different input image sizes in advance for ML-based enlargement processing will be described.

[0110] Figure 82 is a flowchart showing the block size determination processing according to the present embodiment. The image processing apparatus 105 may perform the determination processing at the start of step S204 or step S603 (before starting the expansion processing).

[0111] In step S801, the image processing device 105 divides the original image (if performed in step S204) or the object area (if performed in step S603) into blocks of a first block size and blocks of a second block size, respectively. The first block size is the input image size for the first machine learning model, and the second block size is the input image size for the second machine learning model.

[0112] In step S802, the image processing device 105 calculates the number of pixels outside the target area for expansion processing (excess pixel number) for the case of segmentation with the first block size and for the case of segmentation with the second block size. Then, the image processing device 105 compares the calculated excess pixel numbers.

[0113] Fig. 9 3 is a diagram showing the processing performed in step S802 for calculating the number of excess pixels. Fig. 9 701 indicates an original image, 702 indicates an object region, 703 indicates an area used for enlargement processing in the case of segmentation with a first block size, and 704 indicates an area used for enlargement processing in the case of segmentation with a second block size. The image coordinate system and the coordinates of the object region are the same as those of the third embodiment.

[0114] The number of pixels M in the target area is obtained by the following equation (3).

[0115] M=(x1-x0)*(y1-y0) (3)

[0116] The number of pixels M1 of the area 703 used for the expansion process when segmented with the first block size is obtained by the following equation (4) using the number of blocks Nx1 in the horizontal direction, the number of blocks Ny1 in the vertical direction, the block size A, and the number of overlapping pixels da between adjacent blocks.

[0117] M1=[Nx1*A-(Nx1-1)*da]*[Ny1*A-(Ny1-1)*da] (4)

[0118] The excess pixel number is obtained as M1-M.

[0119] Using the number of blocks Nx2 in the horizontal direction, the number of blocks Ny2 in the vertical direction, the block size B and the number of overlapping pixels db between adjacent blocks, the number of pixels M2 of the area 704 used for the expansion processing when divided with the second block size is obtained by the following formula (5).

[0120] M2=[Nx2*B-(Nx2-1)*db]*[Ny2*B-(Ny2-1)*db] (5)

[0121] The number of excess pixels is obtained as M2-M.

[0122] In step S803, the image processing device 105 determines whether the number of excess pixels in the case of segmentation with the first block size is smaller than the number of excess pixels in the case of segmentation with the second block size. If the image processing device 105 determines that the number of excess pixels in the case of segmentation with the first block size is smaller than the number of excess pixels in the case of segmentation with the second block size, step S804 is performed, otherwise step S805 is performed.

[0123] In step S804 , the image processing apparatus 105 determines that each block obtained by segmentation at the first block size is to be enlarged using the first machine learning model.

[0124] In step S805 , the image processing apparatus 105 determines that each block obtained by segmentation at the second block size is to be enlarged using the second machine learning model.

[0125] According to the present embodiment, for example, in the case where a plurality of machine learning models having different input image sizes are prepared for the enlargement process, a more appropriate block size can be used, which enables more efficient enlargement processing. Although the case where there are two types of block sizes has been described above, the technology can be applied in the same manner even in the case where there are three or more block sizes.

[0126] Note that, in the same manner as the first and second embodiments, the enlargement processing according to the third and fourth embodiments can be used to implement a cropping or electronic zoom function, or be provided as an image editing function, etc. There is a case where the number of pixels after cropping is fixed for cropping and electronic zoom, etc. In this case, the excess pixels described in the fourth embodiment are known in advance, which makes it possible to switch the learning model referenced by the enlargement according to the number of pixels after cropping. Although the embodiments in the camera apparatus have been described, the technology can also be applied to electronic devices that do not have a camera function, such as information processing equipment.

[0127] Other embodiments

[0128] The above embodiment describes the case where the partial region is segmented at the same position used when segmenting the entire original image. However, if the partial region is segmented at a fixed position in the coordinate system of the original image, even if it is not necessary to segment the partial region at the same position used when segmenting the entire original image, the problem of method (2) can still be solved.

[0129] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0130] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An image processing device, comprising: A segmentation component, used for segmenting the image into multiple blocks; A processing component, used for performing an enlargement process on each of the plurality of blocks; as well as a combining component, used for combining the enlarged images of the plurality of blocks into one enlarged image, Here, when an enlarged image of a partial region of an original image is to be generated, the segmentation means segments the partial region into a plurality of blocks at predetermined positions regardless of the position of the partial region in the original image.

2. The image processing device according to claim 1, in, In the case where an enlarged image of the partial area is to be generated, The segmentation component segments the entire original image into a plurality of blocks at the predetermined positions, and The processing means does not perform the expansion process on a block among the plurality of blocks that does not include the partial area.

3. The image processing device according to claim 2, further comprising: A cropping component is used to crop a portion corresponding to the partial area from the one enlarged image obtained by the combining component.

4. The image processing device according to claim 1, in, In a case where an enlarged image of a partial region of an original image is to be generated, the segmentation means segments the partial region cut out from the original image into a plurality of blocks at the predetermined position.

5. The image processing device according to claim 4, in, For a block including an area where pixels do not exist among the plurality of blocks, the processing section performs the expansion process after supplementing the non-existent pixels.

6. The image processing device according to claim 5, in, The processing section complements the non-existent pixel by shifting the position of a block including an area where the pixel does not exist into the partial area.

7. The image processing device according to claim 5, in, For the expanded image of the block including the supplemented pixels, the combining section uses only the area corresponding to the portion included in the block before the supplementation of the pixels when combining.

8. The image processing device according to claim 5, in, For a block among the plurality of blocks in which the percentage of the area where pixels do not exist is not greater than a threshold value, the processing unit does not supplement the non-existent pixels.

9. The image processing device according to any one of claims 1 to 8, in, The predetermined position is based on a multiple of the size of the block.

10. The image processing apparatus according to any one of claims 1 to 8, wherein: The processing means performs the dilation process using a trained machine learning model in which the input image is one of the plurality of blocks, and The segmentation component segments the image into the plurality of blocks so that the size of each of the plurality of blocks is determined according to the size of the input image that the trained machine learning model can handle.

11. The image processing device according to any one of claims 1 to 8, in, The dividing unit can selectively divide the partial region into a plurality of blocks at a position based on the position of the partial region.

12. The image processing device according to claim 11, in, In a case where a setting or a dynamic factor of the image processing device satisfies a predetermined condition, the segmentation means segments the partial region into a plurality of blocks at positions based on the positions of the partial region.

13. The image processing device according to any one of claims 1 to 8, in, The dividing means divides the image into a plurality of blocks so that the size of each of the plurality of blocks is selected from a plurality of sizes.

14. The image processing device according to claim 13, in, The multiple sizes are sizes of input images of multiple machine learning models used to perform the expansion process.

15. The image processing device according to claim 13, in, When the number of pixels that are not in the partial area in the blocks including the partial area among the multiple blocks is smaller when the image is divided into multiple blocks of a first block size than when the image is divided into multiple blocks of a second block size, the dividing component selects the first block size.

16. An image processing device, comprising: A segmentation component, used for segmenting the image into multiple blocks; A processing component, used for performing an enlargement process on each of the plurality of blocks; as well as a combining component, used for combining the enlarged images of the plurality of blocks into one enlarged image, In the case where an enlarged image of a partial area of ​​an original image is to be generated, The processing section sets a processing area that includes the partial area and is smaller than the original image and larger than the partial area, and performs the expansion processing on each of a plurality of blocks obtained by dividing the processing area according to the position of the partial area, and The partial area of ​​the one enlarged image obtained by the combining component is cropped.

17. A camera device, comprising: Image sensor; as well as The image processing device according to any one of claims 1 to 16, The image processing device will use the image captured by the image sensor as an original image.

18. The imaging device according to claim 17, in, The enlargement process is used to realize an electronic zoom function.

19. An image processing method, which is performed by an image processing device, the image processing method comprising: Split the image into multiple blocks; performing an enlargement process on each of the plurality of blocks; as well as combining the enlarged images of the plurality of blocks into one enlarged image, Here, when an enlarged image of a partial region of an original image is to be generated, the segmentation segments the partial region into a plurality of blocks at predetermined positions regardless of the position of the partial region in the original image.

20. An image processing method, which is performed by an image processing device, the image processing method comprising: Split the image into multiple blocks; performing an enlargement process on each of the plurality of blocks; as well as combining the enlarged images of the plurality of blocks into one enlarged image, In the case where an enlarged image of a partial area of ​​an original image is to be generated, The enlargement process is performed by setting a processing area that includes the partial area and is smaller than the original image and larger than the partial area, and performing the enlargement process on each of a plurality of blocks obtained by dividing the processing area according to the position of the partial area, and The partial area of ​​the one enlarged image obtained from the combination is cropped.

21. A computer-readable storage medium storing a program for causing a computer to execute the image processing method according to claim 19 or 20.

22. A computer program product comprising a program for causing a computer to execute the image processing method according to claim 19 or 20.

Images