Image processing system, image processing apparatus, image processing method, and image processing program

Abstract

To reduce a difference in image characteristics between a plurality of divided images whose image values are respectively changed.SOLUTION: Dividing an image to generate a plurality of divided images; changing a pixel value of each of the plurality of divided images; and reducing the image; The image processing apparatus includes a reduced-image generating unit that generates a reduced image of the image, a second changing unit that changes pixel values of the reduced image, a corresponding-region specifying unit that specifies regions respectively corresponding to the plurality of divided images changed by the first changing unit in the reduced image changed by the second changing unit, a correcting unit that corrects the pixel values of the divided images changed by the first changing unit on the basis of pixel values of pixels included in the specified regions, and a combining unit that combines the plurality of divided images corrected by the correcting unit.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an image processing system, an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] There is known a technique for changing the pixel values ​​of a low-brightness image to a high-brightness image using artificial intelligence (AI), etc. This technique can change the pixel values ​​of an image captured in a dark environment to a high-brightness image, thereby improving visibility. Summary of the Invention [Problem to be solved by the invention]

[0003] On the other hand, with the above technology, for large images, after dividing the image into resizable sizes, the pixel values ​​of each divided image are individually changed to a brighter image using generation AI, etc. (In other words, the pixel values ​​are individually optimized for each divided image.) For this reason, when the divided images are combined after the changes, differences in image characteristics may occur between the divided images.

[0004] The present disclosure aims to reduce differences in image characteristics among a plurality of segmented images in which image values ​​have been changed. [Means for solving the problem]

[0005] According to one aspect, an image processing system includes: a divided image generation unit that divides an image and generates a plurality of divided images; a first change unit that changes pixel values ​​of each of the plurality of divided images; a reduced image generating unit that reduces the image and generates a reduced image of the image; a second change unit that changes pixel values ​​of the reduced image; a corresponding area specifying unit that specifies areas in the reduced image that have been changed by the second changing unit that correspond to each of the plurality of divided images that have been changed by the first changing unit; a correction unit that corrects pixel values ​​of the divided image changed by the first change unit based on pixel values ​​of pixels included in the specified region; and a combining unit that combines the plurality of divided images corrected by the correcting unit. [Effects of the Invention]

[0006] According to the present disclosure, it is possible to reduce differences in image characteristics between a plurality of divided images whose image values ​​have been changed. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of an entire image generation system. [Figure 2] FIG. 1 illustrates an example of a hardware configuration of an image processing system. [Figure 3] FIG. 2 is a diagram illustrating an example of a functional configuration of an image processing unit. [Figure 4] 10A and 10B are diagrams illustrating a specific example of processing by a divided image generating unit. [Figure 5] 10A and 10B are diagrams illustrating a specific example of processing by a first changing unit. [Figure 6] 10A and 10B are diagrams illustrating a specific example of processing by a reduced image generating unit. [Figure 7] 10A and 10B are diagrams illustrating a specific example of processing by a second change unit. [Figure 8] 10 is a first diagram showing a specific example of processing by a corresponding area specifying unit and a correction unit. FIG. [Figure 9] FIG. 10 is a second diagram showing a specific example of processing by the corresponding area specifying unit and the correction unit. [Figure 10] FIG. 10 is a third diagram showing a specific example of processing by the corresponding area specifying unit and the correction unit. [Figure 11] FIG. 4 is a fourth diagram showing a specific example of processing by the corresponding area specifying unit and the correcting unit. [Figure 12]1 is a flowchart showing the flow of image processing by the image processing system. [Figure 13] 10 is a flowchart showing the flow of correction processing by the image processing system. [Figure 14] FIG. 10 is a diagram showing an example of an image after image processing performed by the image processing system. [Figure 15] FIG. 10 is a diagram showing an example of the luminance of pixels included in an image after image processing performed by the image processing system. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0009] [First embodiment] <System configuration of image generation system> First, a description will be given of the overall system configuration of an image generation system including an image processing system according to Embodiment 1. Fig. 1 is a diagram showing an example of the overall system configuration of the image generation system.

[0010] The image generation system 100 is a system that generates an image by capturing an image using an imaging device, and performs various processes on the generated image to adjust image characteristics, thereby generating an image with high visibility. Note that image characteristics refer to characteristics that affect the visibility of the image, such as brightness and color difference. In this embodiment, color difference refers to the color components (Cb value, Cr value) when the pixel values ​​(R value, G value, B value) of pixels in an RGB format image are converted into a brightness signal (Y value) and color difference signals (Cb value, Cr value).

[0011] 1, the image generation system 100 includes an imaging device 110, an information terminal 120, and an image processing system 140. In the image generation system 100, the information terminal 120 and the image processing system 140 are communicably connected via a communication network 130.

[0012] The imaging device 110 generates an image by capturing an image and transmits the generated image to the information terminal 120. The type of imaging device 110 is arbitrary, and may be, for example, a spherical camera. The representation format of the image generated by the imaging device 110 is arbitrary, and may be, for example, RGB format or YCbCr format. The compression format of the image generated by the imaging device 110 is arbitrary, and may be, for example, JPEG format.

[0013] The information terminal 120 transmits the image transmitted from the imaging device 110 to the image processing system 140 via the communication network 130, and instructs the image processing system 140 to perform various processes for adjusting the image characteristics. The information terminal 120 also receives from the image processing system 140 an "adjusted image" that has undergone various processes for adjusting the image characteristics, and displays it.

[0014] Note that the functions of the imaging device 110 and the functions of the information terminal 120 described above are examples, and some of the functions of the imaging device 110 may be possessed by the information terminal 120, and some of the functions of the information terminal 120 may be possessed by the imaging device 110.

[0015] 1, the imaging device 110 and the information terminal 120 are configured as separate entities, but the imaging device 110 and the information terminal 120 may be configured as an integrated entity. For example, the information terminal 120 may be a smartphone, tablet, or the like having an imaging function. Alternatively, the imaging device 110 may have a communication function for communicating with the image processing system 140 via the communication network 130.

[0016] The image processing system 140 is a system that executes various processes for adjusting image characteristics of an image received via the communication network 130. An image processing program is installed in the image processing system 140, and by executing the image processing program, the image processing system 140 functions as an image acquisition unit 150 and an image processing unit 160.

[0017] The image acquisition unit 150 receives an image from the information terminal 120 via the communication network 130, and stores the received image in the image storage unit 170. The image acquisition unit 150 also transmits, to the information terminal 120 via the communication network 130, an adjusted image in which various processes for adjusting image characteristics have been performed on the image stored in the image storage unit 170.

[0018] The image processing unit 160 reads out the image stored in the image storage unit 170 , executes various processes for adjusting the image characteristics, and stores the adjusted image in the image storage unit 170 .

[0019] According to the image generation system 100, for example, a user of the imaging device 110 can obtain an image with high visibility by transmitting the image to the image processing system 140 via the information terminal 120, even if the image was taken in a dark scene.

[0020] Furthermore, according to the image generation system 100, the user of the imaging device 110 can obtain an image with high visibility even if, for example, the information terminal 120 does not have a high-performance processor.

[0021] <Image processing system hardware configuration> Next, a description will be given of the hardware configuration of the image processing system 140. Fig. 2 is a diagram showing an example of the hardware configuration of the image processing system. The image processing system 140 is constructed by a computer, and as shown in Fig. 2, includes a processor 201, a ROM 202, a RAM 203, an HD 204, an HDD (Hard Disk Drive) controller 205, a display 206, an external device connection I / F (Interface) 208, a network I / F 209, a data bus 210, a keyboard 211, a pointing device 212, a DVD-RW (Digital Versatile Disk Rewritable) drive 214, and a media I / F 216.

[0022] Of these, the processor 201 includes a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) and controls the overall operation of the image processing system 140. The ROM 202 stores programs such as an IPL (Initial Program Loader) used to drive the processor 201. The RAM 203 is used as a work area for the processor 201. The HD 204 stores various data such as programs. The HDD controller 205 controls the reading and writing of various data from and to the HD 204 under the control of the processor 201. The display 206 displays various information such as a cursor, menus, windows, characters, or images. The external device connection I / F 208 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 209 is an interface for data communication using the communication network 130. The data bus 210 is an address bus, a data bus, or the like for electrically connecting the components such as the processor 201 shown in FIG. 2 .

[0023] The keyboard 211 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 212 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 214 controls reading and writing of various data from and to a DVD-RW 213, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 216 controls reading and writing (storing) of data from and to a recording medium 215, such as a flash memory.

[0024] Although the hardware configuration of the image processing system 140 has been described above, the hardware configuration of the imaging device 110 and the hardware configuration of the information terminal 120 are also similar.

[0025] <Functional configuration of the image processing unit> Next, a description will be given of the functional configuration of the image processing unit 160. Fig. 3 is a diagram showing an example of the functional configuration of the image processing unit.

[0026] As shown in FIG. 3, the image processing unit 160 includes an image reading unit 310, a divided image generating unit 320, a first modification unit 330, a reduced image generating unit 340, a second modification unit 350, a corresponding area identifying unit 360, a correction unit 370, and a combination unit 380.

[0027] The image reading unit 310 reads out from the image storage unit 170 the image that has been stored in the image storage unit 170 by the image acquisition unit 150. The image reading unit 310 notifies the divided image generation unit 320 and the reduced image generation unit 340 of the read image.

[0028] In the first embodiment, the image reading unit 310 notifies the divided image generating unit 320 and the reduced image generating unit 340 of the read image as an image in RGB format.

[0029] Therefore, if the image stored in the image storage unit 170 is a JPEG format image, the image reading unit 310 has a JPEG decoder that decodes the JPEG format image. Also, if the image stored in the image storage unit 170 is a YCbCr format image, the image reading unit 310 has a color space conversion unit that converts the YCbCr format image into an RGB format image.

[0030] The divided image generation unit 320 divides the image notified by the image reading unit 310 to generate a plurality of divided images. The divided image generation unit 320 divides the image into sizes that allow pixel values ​​to be changed by the first changing unit 330, which will be described later. The size that allows pixel values ​​to be changed is determined by, for example, the capacity of the GPU memory included in the processor 201.

[0031] The first modification unit 330 adjusts the image characteristics (brightness, color difference, etc.) of each of the multiple segmented images by modifying the pixel values ​​of at least some of the pixels constituting the multiple segmented images notified by the segmented image generation unit 320. The first modification unit 330 is assumed to have an arbitrary LLIE (Low Light Image Enhancement). The arbitrary LLE here includes, for example, a trained model (AI model) that has been trained in advance to adjust brightness or color difference. The arbitrary LLE may also include a trained model (AI model) that has been trained in advance to improve noise or sharpness in addition to brightness or color difference. In this case, the segmented images adjusted by the arbitrary LLE are segmented images with improved noise or sharpness in addition to brightness or color difference. The arbitrary LLE may also include, for example, a trained model (so-called generative AI) that has been trained on a large number of images.

[0032] The reduced image generation unit 340 reduces the image notified by the image reading unit 310 to generate a reduced image. Reducing an image here refers to image processing to inscribe the image within a frame smaller than the image and configured with a predetermined aspect ratio. Reducing an image includes, for example, reducing the length of each side of a rectangular image without changing the aspect ratio. Reducing an image also includes, for example, reducing a rectangular image so that the length of one side is shorter than any side of the original rectangular image. Therefore, a "reduced image" refers to an image obtained by reducing either or both the vertical and horizontal lengths compared to the image before reduction.

[0033] The reduced image generation unit 340 generates a reduced image by changing the image to a size that allows the pixel values ​​to be changed by the second change unit 350, which will be described later. The size that allows the pixel values ​​to be changed is determined by, for example, the capacity of the GPU memory included in the processor 201.

[0034] The second modification unit 350 adjusts the image characteristics (brightness, color difference, etc.) of the reduced image by modifying the pixel values ​​of at least some of the pixels constituting the reduced image notified by the reduced image generation unit 340. The second modification unit 350 includes an arbitrary LLE. The arbitrary LLE here includes, for example, a trained model (AI model) trained in advance to adjust brightness or color difference. The arbitrary LLE may also include a trained model (AI model) trained in advance to improve noise or sharpness in addition to brightness or color difference. In this case, the segmented image adjusted by the arbitrary LLE becomes a segmented image with improved noise or sharpness in addition to brightness or color difference. The arbitrary LLE may also include, for example, a trained model (so-called generative AI) trained on a large number of images. The LLE included in the second modification unit 350 and the LLE included in the first modification unit 330 may be the same LLE or different LLEs (different LLEs having similar functions).

[0035] The corresponding area identifying section 360 identifies areas in the reduced image in which the pixel values ​​have been changed by the second changing section 350 that correspond to each of the plurality of divided images in which the pixel values ​​have been changed by the first changing section 330.

[0036] The correction unit 370 acquires a plurality of divided images whose pixel values ​​have been changed by the first change unit 330. The correction unit 370 also acquires a reduced image whose pixel values ​​have been changed by the second change unit 350. The correction unit 370 corrects the pixel values ​​of each of the plurality of divided images whose pixel values ​​have been changed, using a correction value calculated based on the pixel values ​​of each corresponding region of the reduced image whose pixel values ​​have been changed. In this way, the correction unit 370 can reduce differences in image characteristics between the divided images whose image characteristics have been individually optimized by the first change unit 330.

[0037] The combining unit 380 generates an "adjusted image" by combining the plurality of divided images corrected by the correction unit 370, in which the differences in image characteristics have been reduced. The combining unit 380 also stores the generated adjusted image in the image storage unit 170.

[0038] Note that the combining unit 380 may have a JPEG encoder when storing the generated adjusted image in JPEG format.

[0039] <Specific examples of processing by each part of the image processing unit> Next, specific examples of the processing of each unit of the image processing unit 160 (here, the divided image generation unit 320, the first change unit 330, the reduced image generation unit 340, the second change unit 350, the corresponding area identification unit 360, and the correction unit 370) will be described.

[0040] (1) Specific example of processing by the divided image generating unit 320 First, a description will be given of a specific example of processing by the divided image generation unit 320. Fig. 4 is a diagram showing a specific example of processing by the divided image generation unit. As shown in Fig. 4, the divided image generation unit 320 has a division mode determination unit 410 and a division unit 420.

[0041] The division mode determination unit 410 determines the division mode when dividing the image 400 notified by the image reading unit 310. The division modes include: Split direction (split horizontally, split vertically, or split both horizontally and vertically), The number of divisions (for example, the number of divisions that will allow the number of pixels in each divided image to be within the number of pixels in a size that allows pixel values ​​to be changed), The number of pixels in the division direction of each divided image (whether to make it equal depending on the number of divisions, or to make it a fixed number of pixels (for example, a round number of pixels close to the maximum size of the pixel value that can be changed)), The division mode determination unit 410 may determine the division mode based on, for example, an instruction from a user of the image capture device 110, or may determine the division mode based on a predetermined division mode. Alternatively, the division mode determination unit 410 may determine the division mode based on the number of vertical and horizontal pixels of the image 400 notified by the image reading unit 310, the type of image capture device 110, the capacity of the GPU memory included in the processor 201, etc.

[0042] However, if the number of divisions is too large, subsequent processing becomes complicated, so it is desirable to keep the number of divisions as small as possible. Furthermore, when determining the division pattern, it is desirable to select a division pattern that is less likely to generate noise in subsequent processing.

[0043] The dividing unit 420 divides the image 400 according to the division mode determined by the division mode determination unit 410. In the example of Fig. 4, the image 400 is an image captured by the imaging device 110, which is an omnidirectional camera, and is an image in which a field of view of 360 degrees horizontally and 180 degrees vertically is uniformly recorded when the captured area is regarded as the surface of a sphere. The image 400 is an image of 2752 vertical pixels x 5504 horizontal pixels.

[0044] In the example of FIG. 4, the division mode determination unit 410 Deciding to divide the area vertically by a dividing line extending horizontally; Based on the number of pixels calculated based on the GPU memory capacity (here, 2.3 million pixels), a fixed number of pixels (the maximum number of pixels whose pixel value can be changed = a round number close to 2.3 million pixels) was determined for the vertical direction of each divided image. It shows the situation.

[0045] in particular, The division mode determination unit 410 calculates the number of pixels (417 pixels) by dividing the capacity of the GPU memory (2.3 million pixels) by the number of pixels in the horizontal direction of the image 400 (5504 pixels), Based on the calculated number of pixels (417 pixels), the vertical number of pixels for each divided image was determined to be 400 pixels. It shows the situation.

[0046] In the example of FIG. 4, divided images 401 to 407 indicate divided images obtained by dividing the image 400 by the dividing section 420 based on the division pattern determined by the division pattern determining section 410.

[0047] In the example of Figure 4, The divided image 401 indicates a divided image ranging from the 1st pixel to the 400th pixel in the vertical direction. The divided image 402 indicates a divided image ranging from the 401st pixel to the 800th pixel in the vertical direction. The divided image 403 indicates a divided image ranging from the 801st pixel to the 1200th pixel in the vertical direction. The divided image 404 indicates a divided image ranging from the 1201st pixel to the 1600th pixel in the vertical direction. The divided image 405 indicates a divided image ranging from the 1601st pixel to the 2000th pixel in the vertical direction. The divided image 406 indicates a divided image ranging from the 2001st pixel to the 2400th pixel in the vertical direction. A divided image 407 indicates a divided image ranging from the 2401st pixel to the 2752nd pixel in the vertical direction.

[0048] 4, the number of vertical pixels of each divided image is fixed at 400 pixels, but the number of vertical pixels of each divided image may be equalized depending on the number of divisions. For example, the division mode determination unit 410 may determine the number of divisions to be 7 so that the number of pixels of each divided image is within the adjustable number of pixels of the size (417 pixels), and may equally divide the number of vertical pixels of each divided image as 2752 pixels / 7=393 pixels.

[0049] (2) Specific example of processing by the first change unit 330 Next, a description will be given of a specific example of the processing by the first changing section 330. Fig. 5 is a diagram showing a specific example of the processing by the first changing section.

[0050] The first changing unit 330 changes the pixel values ​​of the divided images 401 to 407 notified by the divided image generating unit 320 in order, thereby adjusting the image characteristics of each of the divided images 401 to 407.

[0051] The example of FIG. 5 illustrates a state in which the first change unit 330 performs a process to increase the brightness of a low-brightness divided image 401, thereby outputting a high-brightness divided image 501. Similarly, the example of FIG. 5 illustrates a state in which the first change unit 330 performs a process to increase the brightness of a low-brightness divided image 402, thereby outputting a high-brightness divided image 502. Similarly, the example of FIG. 5 illustrates a state in which the first change unit 330 performs a process to increase the brightness of a low-brightness divided image 403, thereby outputting a high-brightness divided image 503. Similarly, the example of FIG. 5 illustrates a state in which the first change unit 330 performs a process to increase the brightness of a low-brightness divided image 404, thereby outputting a high-brightness divided image 504. Similarly, the example of FIG. 5 illustrates a state in which the first change unit 330 performs a process to increase the brightness of a low-brightness divided image 405, thereby outputting a high-brightness divided image 505. 5 shows a state in which the first changing unit 330 performs a process to increase the brightness of the divided image 406, which has low brightness, and outputs a divided image 506, which has high brightness. Similarly, the example of Fig. 5 shows a state in which the first changing unit 330 performs a process to increase the brightness of the divided image 407, which has low brightness, and outputs a divided image 507, which has high brightness.

[0052] (3) Specific example of processing by the reduced image generating unit 340 Next, a specific example of the processing performed by the reduced image generating unit 340 will be described. Fig. 6 is a diagram showing a specific example of the processing performed by the reduced image generating unit.

[0053] The example in Fig. 6 shows how the reduced image generation unit 340 changes the size of the image 400 (2752 pixels vertically × 5504 pixels horizontally) notified by the image reading unit 310 so that it is smaller than the capacity (2.3 million pixels) of the GPU memory, and generates a reduced image 600. In Fig. 6, the reduced image 600 is an image of 512 pixels vertically × 512 pixels horizontally.

[0054] In this way, the reduced image generated by the reduced image generating unit 340 does not need to maintain the aspect ratio of the image 400 notified by the image reading unit 310, and may have an aspect ratio different from that of the image 400.

[0055] (4) Specific Example of Processing by the Second Change Unit 350 Next, a description will be given of a specific example of the processing by the second changing section 350. Fig. 7 is a diagram showing a specific example of the processing by the second changing section.

[0056] The second modification unit 350 modifies the pixel values ​​of the reduced image 600 notified by the reduced image generation unit 340, thereby adjusting the image characteristics of the reduced image 600.

[0057] The example in FIG. 7 shows a state in which the second changing section 350 performs processing to increase the brightness of a reduced image 600 with low brightness, and outputs a reduced image 700 with high brightness.

[0058] (5) Specific examples of processing by the corresponding area identification unit 360 and the correction unit 370 Next, a description will be given of a specific example of processing by the corresponding area identifying unit 360 and the correction unit 370. Figures 8 to 11 are first to fourth diagrams showing a specific example of processing by the corresponding area identifying unit and the correction unit. As shown in Figures 8 to 11, the correction unit 370 has a divided area generating unit 820, an area-unit correction value calculating unit 830, a pixel-unit correction value calculating unit 840, and a pixel-unit correction unit 850.

[0059] 8 shows a specific example of processing by the corresponding region identifying section 360. The corresponding region identifying section 360 sequentially acquires high-luminance divided images 501 to 507 from the first changing section 330, and acquires a high-luminance reduced image 700 from the second changing section 350.

[0060] The corresponding area specifying unit 360 specifies corresponding areas in the high-brightness reduced image 700 that correspond to the high-brightness divided images 501 to 507. Specifically, The vertical starting pixel of each of the divided images 501 to 507 with high brightness is ps, The vertical end pixel of each of the divided images 501 to 507 with high brightness is pe, · The number of vertical pixels of image 400 is Po, The number of vertical pixels of the bright reduced image 700 is Pr, In this case, The starting pixel ps' of the corresponding area is ps'=Pr×(ps / Po), The end pixel pe' of the corresponding region is pe'=Pr×(pe / Po), It can be expressed as:

[0061] 8 shows, as an example, a state in which the corresponding area specifying unit 360 specifies a corresponding area in the high-brightness reduced image 700 that corresponds to the high-brightness divided image 504. As described above, in the case of the high-brightness divided image 504, ·ps=1201th pixel, pe=1600th pixel, Po=2752 pixels, ·Pr=512 pixels, Therefore, The starting pixel ps' of the corresponding area 800 = 512 × (1201 / 2752) = 223rd pixel, End pixel pe' of the corresponding area 800 = 512 × (1600 / 2752) = 298th pixel, This becomes:

[0062] 9 shows a specific example of processing by the divided region generating unit 820. The divided region generating unit 820 generates a region for which a correspondence relationship has been identified by the corresponding region identifying unit 360. - High brightness divided images 501 to 507, Each corresponding area in the reduced image 700 with high brightness, is divided into m parts vertically and n parts horizontally, generating m×n divided regions.

[0063] The example in FIG. 9 shows a state in which the divided image 503 with high brightness is divided into 4 parts vertically and 24 parts horizontally, and the corresponding region 800 is divided into 4 parts vertically and 24 parts horizontally.

[0064] 10 shows a specific example of processing by the area unit correction value calculation unit 830. The area unit correction value calculation unit 830 calculates the area unit correction value generated by the divided area generation unit 820. m × n divided regions for each of the divided images 501 to 507 with high brightness; m×n divided regions for each corresponding region in the reduced image 700 with high brightness; Specifically, the area unit correction value calculation unit 830 calculates a correction value for each divided area between Correction value of divided area (m, n)=(average value of pixel values ​​of each pixel in divided area (m, n) of the corresponding area in the reduced image (an example of the second calculated value)) / (average value of pixel values ​​of each pixel in divided area (m, n) of the divided image (an example of the first calculated value)) It is calculated as follows.

[0065] For example, the correction value of the R value of the divided area (4,1) of the divided image 503 with high brightness is The average value of the R value of each pixel included in the divided area (4,1) of the corresponding area 800 in the reduced image 700 with high brightness is calculated as follows: The average value of the R values ​​of the pixels included in the division area (4,1) of the division image 503 with high brightness, It is calculated by dividing by

[0066] Similarly, for example, the correction value of the G value of the divided area (4,1) of the divided image 503 with high brightness is The average value of the G value of each pixel included in the divided area (4,1) of the corresponding area 800 in the reduced image 700 with high brightness is calculated as follows: The average value of the G values ​​of the pixels included in the division area (4,1) of the division image 503 with high brightness, It is calculated by dividing by

[0067] Similarly, for example, the correction value of the B value of the divided area (4,1) of the divided image 503 with high brightness is The average value of the B value of each pixel included in the divided area (4,1) of the corresponding area 800 in the reduced image 700 with high brightness is calculated as follows: The average value of the B values ​​of the pixels included in the division area (4,1) of the division image 503 with high brightness, It is calculated by dividing by

[0068] In this way, the area unit correction value calculation section 830 calculates the R value correction value, the G value correction value, and the B value correction value for each divided area.

[0069] 11 shows a specific example of processing by the pixel-unit correction value calculation unit 840. The pixel-unit correction value calculation unit 840 calculates the R-value correction value, the G-value correction value, and the B-value correction value for each pixel based on the R-value correction value, the G-value correction value, and the B-value correction value calculated for each divided region by the region-unit correction value calculation unit 830.

[0070] Specifically, the pixel-unit correction value calculation unit 840 sets the correction value of the R value calculated by the area-unit correction value calculation unit 830 for each divided area as a representative value for each divided area, and interpolates the correction value of the R value of each pixel between the representative values. Similarly, the pixel-unit correction value calculation unit 840 sets the correction value of the G value calculated by the area-unit correction value calculation unit 830 for each divided area as a representative value for each divided area, and interpolates the correction value of the G value of each pixel between the representative values. Similarly, the pixel-unit correction value calculation unit 840 sets the correction value of the B value calculated by the area-unit correction value calculation unit 830 for each divided area as a representative value for each divided area, and interpolates the correction value of the B value of each pixel between the representative values.

[0071] Note that the pixel-by-pixel correction value calculation unit 840 may use any method for interpolating the correction values. For example, linear interpolation or bicubic interpolation may be used. The bicubic interpolation allows for smoother interpolation of the correction values. In the example of FIG. 10 , the first calculated value is described as the average R value, average G value, and average B value of each pixel in the divided region (m, n) of the divided image. However, the first calculated value may be the average value of either the luminance signal (Y value) or the color difference signal (Cb value, Cr value) obtained by converting the R value, G value, and B value of each pixel into the YCbCr format. Similarly, in the example of FIG. 10 , the second calculated value is described as the average R value, average G value, and average B value of each pixel in the divided region (m, n) of the corresponding region in the reduced image. However, the second calculated value may be the average value of either the luminance signal (Y value) or the color difference signal (Cb value, Cr value) obtained by converting the R value, G value, and B value of each pixel into the YCbCr format.

[0072] 11, graphs 1100 and 1110 show the R value correction values ​​interpolated by the pixel-by-pixel correction value calculation unit 840. In graphs 1100 and 1110, the black circles indicate the R value correction values ​​calculated for each divided area, and the mesh intersections indicate the R value correction values ​​for each pixel. However, due to space limitations, the mesh is coarse in graphs 1100 and 1110, and only the correction values ​​of some pixels are shown.

[0073] In this way, correction unit 370 first calculates correction values ​​on a region-by-region basis, and then calculates correction values ​​on a pixel-by-pixel basis by interpolation. This is because, when calculating correction values ​​based on pixel values ​​of a reduced image, if correction values ​​are calculated pixel-by-pixel from the beginning, the reduced image has a lower resolution than the divided images, and if the reduced image is corrected using the calculated correction values, the corrected divided images will be blurred. In contrast, with the above-mentioned method, the low resolution of the reduced image can be compensated for by interpolation, making it possible to prevent each divided image from becoming blurred.

[0074] The pixel-by-pixel R value correction value, G value correction value, and B value correction value for each of the multiple divided images calculated by the pixel-by-pixel correction value calculation unit 840 are notified to the pixel-by-pixel correction unit 850. This allows the pixel-by-pixel correction unit 850 to perform correction on each of the multiple divided images with high brightness, and generate "multiple corrected divided images."

[0075] <Image processing flow by image processing system> Next, we will explain the flow of image processing by the image processing system 140. Fig. 12 is a flowchart showing the flow of image processing by the image processing system.

[0076] In step S1201, the image reading unit 310 reads the target image from the image storage unit 170.

[0077] In step S1211, the divided image generating unit 320 divides the read image to generate a plurality of divided images.

[0078] In step S1212, the first change unit 330 adjusts the image characteristics by changing the pixel values ​​of each of the generated divided images.

[0079] In step S1221, the reduced image generating unit 340 changes the size of the read image and generates a reduced image.

[0080] In step S1222, the second modifying unit 350 adjusts the image characteristics of the generated reduced image by modifying the pixel values.

[0081] In step S1231, the corresponding area identifying section 360 identifies areas in the reduced image in which pixel values ​​have been changed that correspond to each of the multiple divided images in which pixel values ​​have been changed.

[0082] In step S1232, the correction unit 370 corrects the pixel values ​​of each of the divided images whose pixel values ​​have been changed, using a correction value calculated based on the pixel values ​​of the corresponding areas of the reduced image whose pixel values ​​have been changed. Details of the correction process (step S1231) will be described later.

[0083] In step S1233, the combining unit 380 combines the "plurality of divided images after correction" to generate the "adjusted image."

[0084] <Correction process flow by image processing system> Next, a description will be given of the details of the correction process (step S1231 in FIG. 12) performed by the image processing system 140. Fig. 13 is a flowchart showing the flow of the correction process performed by the image processing system.

[0085] In step S1301, the correction unit 370 acquires one divided image from among the plurality of divided images whose pixel values ​​have been changed.

[0086] In step S1302, the correction unit 370 acquires a corresponding area of ​​the reduced image whose pixel values ​​have been changed, which corresponds to one divided image acquired in step S1301.

[0087] In step S1303, the correction unit 370 divides one divided image acquired in step S1301 to generate a plurality of divided regions.

[0088] In step S1304, the correction unit 370 divides the corresponding area in the reduced image acquired in step S1302 to generate a plurality of divided areas.

[0089] In step S1305, correction unit 370 calculates a correction value for each area based on the multiple divided areas generated in step S1303 and the multiple divided areas generated in step S1304.

[0090] In step S1306, the correction unit 370 calculates a correction value for each pixel by interpolating the calculated correction value for each region.

[0091] In step S1307, the correction unit 370 corrects the pixel value of each pixel included in one divided image acquired in step S1301 using the calculated correction value for each pixel.

[0092] In step S1308, correction unit 370 determines whether the processes of steps S1301 to S1307 have been performed for all of the multiple image segments whose image characteristics have been adjusted by changing pixel values. If it is determined in step S1308 that there are any image segments for which the processes of steps S1301 to S1307 have not been performed (NO in step S1308), the process returns to step S1301.

[0093] On the other hand, if it is determined in step S1308 that the processes in steps S1301 to S1307 have been executed for all divided images (YES in step S1308), the correction process ends and the process returns to step S1233 in FIG.

[0094] <Example of an image after image processing> Next, a specific example of an image after image processing performed by the image processing system 140 will be described. Fig. 14 is a diagram showing an example of an image after image processing performed by the image processing system.

[0095] 14(a) shows, as a comparative example, an image after image processing when image processing is performed without performing correction processing (step S1232) in the image processing system 140. As shown in FIG. 14(a), when image processing is performed without performing correction processing (step S1232), among the divided images 501 to 507 adjusted to high brightness, Divided image 501 and divided image 502, Divided images 503 to 505, Therefore, a horizontal pattern is visible between divided images 501 and 502 and divided images 503 to 505. Similarly, if image processing is performed without performing correction processing (step S1232), among divided images 501 to 507 whose pixel values ​​have been changed to high brightness, Divided images 503 to 505, Divided image 506 and divided image 507, Therefore, a horizontal pattern is visible between divided images 503 to 505 and divided images 506 and 507.

[0096] On the other hand, Fig. 14(b) shows an image obtained when correction processing (step S1232) is performed in image processing by the image processing system 140. As shown in Fig. 14(b), when correction processing (step S1232) is performed, no difference in brightness occurs between the divided images after image processing, and therefore horizontal patterns are not visible.

[0097] Fig. 15 is a diagram showing an example of the brightness of pixels included in an image after image processing performed by an image processing system. Fig. 15(a) and Fig. 15(b) respectively show changes in brightness of pixels in the vertical direction at a predetermined position in the horizontal direction (the 3700th pixel in the example of Fig. 15) of the images shown in Fig. 14(a) and Fig. 14(b). That is, the horizontal axis of Fig. 15(a) and Fig. 15(b) represents the number of pixels in the vertical direction of the image after image processing, and the vertical axis represents the brightness of each pixel.

[0098] Comparing Fig. 15(a) and Fig. 15(b), for example, in Fig. 15(a), a brightness step occurs around the 800th pixel in the vertical direction, whereas in Fig. 15(b), the brightness step disappears. This corresponds to the fact that the horizontal pattern visible between divided images 501 and 502 and divided images 503 to 505 in Fig. 14(a) is not visible in Fig. 14(b).

[0099] 15(a) and 15(b), for example, in Fig. 15(a), the brightness of pixels from the 1400th pixel onwards in the vertical direction is low, whereas in Fig. 15(b), the brightness of pixels from the 1400th pixel onwards in the vertical direction is high. This corresponds to the fact that the horizontal pattern visible between divided images 503 to 505 and divided images 506 and 507 in Fig. 14(a) is not visible in Fig. 14(b).

[0100] In this way, the image processing system 140 can reduce differences in image characteristics between a plurality of divided images that occur when various processes for adjusting image characteristics are performed.

[0101] <Summary> As is clear from the above description, the image processing system 140 according to the first embodiment: Split an image and generate multiple split images. -Change the pixel values ​​of each of multiple divided images. Reduce the size of the image and generate a reduced version of the image. - Change pixel values ​​of the reduced image. In the reduced image in which pixel values ​​have been changed, areas corresponding to each of the plurality of divided images in which pixel values ​​have been changed are identified. Based on the pixel values ​​of the pixels included in the identified area, the pixel values ​​of the segmented image whose pixel values ​​have been changed are corrected. Combine multiple corrected split images.

[0102] As a result, the image processing system 140 according to the first embodiment can reduce differences in image characteristics between a plurality of divided images in which pixel values ​​have been changed.

[0103] [Second embodiment] In the first embodiment, the divided image generation unit 320 divides an image to generate a plurality of divided images so that the divided images do not overlap each other. However, the method of dividing an image by the divided image generation unit 320 is not limited to this. For example, when dividing an image, the divided image generation unit 320 may divide the image so that the divided images overlap each other.

[0104] In this case, when pixel-by-pixel correction unit 850 corrects the pixel values ​​of each of the multiple divided images using the pixel-by-pixel correction value calculated for each of the multiple divided images, it uses a correction value for the overlapping portion that is a weighted sum of the correction values ​​calculated for each divided image. Note that the weighting method for weighting the correction values ​​is arbitrary, and a fixed value (e.g., 0.5) may be used for weighting regardless of the overlapping position, or different values ​​may be used for weighting depending on the overlapping position.

[0105] [Third embodiment] In the above first embodiment, the image processing system 140 is described as executing image processing in response to an instruction from the information terminal 120. However, the image processing system 140 may be configured to execute image processing after charging the user of the imaging device 110 when receiving an instruction from the information terminal 120. In other words, the image processing system 140 may be a system that provides an image processing service on the cloud on the condition that the user is charged.

[0106] When the image processing system 140 provides an image processing service, the image processing system 140 may also provide a dedicated application used by a user receiving the image processing service. The application may include, for example, a user interface for specifying an image to be processed, a processing purpose, a representation format of the processed image, a compression format of the processed image, etc.

[0107] [Fourth embodiment] In the above first embodiment, it has been described that the image reading unit 310 notifies the read image as an RGB format image to the divided image generating unit 320 and the reduced image generating unit 340. Therefore, according to the first embodiment, when the luminance is adjusted as an image characteristic, the chrominance is also adjusted.

[0108] On the other hand, if the read image is notified to the divided image generation unit 320 and the reduced image generation unit 340 as an image in YCbCr format, it becomes possible to adjust only the luminance or only the color difference as an image characteristic.

[0109] That is, the image processing system 140 may be configured to allow the user to select an image format depending on the processing purpose.

[0110] [Other embodiments] In the above first embodiment, the image processing unit 160 has been described as being realized in the image processing system 140. However, some or all of the functional units of the image processing unit 160 may be realized in an image processing system, image processing device, or image processing terminal other than the image processing system 140.

[0111] In the first embodiment, the image processing system 140 executes the image processing program by itself. However, the image processing system 140 may be configured with, for example, multiple computers, and the image processing program may be installed in each of the computers, so that the image processing program is executed in a distributed computing format.

[0112] The disclosed technology may take the following forms as described below. (Appendix 1) a divided image generation unit that divides an image and generates a plurality of divided images; a first change unit that changes pixel values ​​of each of the plurality of divided images; a reduced image generating unit that reduces the image and generates a reduced image of the image; a second change unit that changes pixel values ​​of the reduced image; a corresponding area specifying unit that specifies areas in the reduced image that have been changed by the second changing unit that correspond to each of the plurality of divided images that have been changed by the first changing unit; a correction unit that corrects pixel values ​​of the divided image changed by the first change unit based on pixel values ​​of pixels included in the specified region; a combining unit that combines the plurality of divided images corrected by the correction unit; An image processing system having: (Appendix 2) the pixel values ​​include luminance and chrominance information; the first change unit changes pixel values ​​of the divided image so as to increase either or both of luminance and color difference of at least some of pixels constituting the divided image; the second modification unit modifies pixel values ​​of the reduced image so as to increase either or both of luminance and color difference of at least some of the pixels constituting the reduced image. 2. The image processing system of claim 1. (Appendix 3) The correction unit a first calculated value calculated from pixel values ​​of pixels included in a divided area obtained by further dividing the divided image whose pixel values ​​have been changed; a second calculated value calculated from pixel values ​​of pixels included in an area dividing the area corresponding to the divided image in the reduced image whose pixel values ​​have been changed; and calculating a correction value for each of the plurality of divided regions based on the 2. The image processing system of claim 1. (Appendix 4) the first calculated value is an average value of pixel values ​​of pixels included in the divided region, the second calculated value is an average value of pixel values ​​of pixels included in an area of ​​the reduced image that divides the area corresponding to the divided image; 4. The image processing system of claim 3. (Appendix 5) The correction unit interpolating correction values ​​for correcting pixel values ​​of each pixel included in each of the plurality of divided regions using the correction values ​​for each of the plurality of divided regions; correcting the pixel values ​​of the pixels included in each of the plurality of divided areas changed by the first change unit using the interpolated correction values ​​of the pixels; 4. The image processing system of claim 3. (Appendix 6) The plurality of divided images are Adjacent divided images have overlapping portions, The correction unit The overlapping portions of the adjacent divided images are corrected using a correction value obtained by weighting and adding the correction values ​​of each pixel calculated for each divided image. An image processing system according to any one of appendices 1 to 5. (Appendix 7) a divided image generation unit that divides an image and generates a plurality of divided images; a first change unit that changes pixel values ​​of each of the plurality of divided images; a reduced image generating unit that reduces the image and generates a reduced image of the image; a second change unit that changes pixel values ​​of the reduced image; a corresponding area specifying unit that specifies areas in the reduced image that have been changed by the second changing unit that correspond to each of the plurality of divided images that have been changed by the first changing unit; a correction unit that corrects pixel values ​​of the divided image changed by the first change unit based on pixel values ​​of pixels included in the specified region; a combining unit that combines the plurality of divided images corrected by the correction unit; An image processing device having: (Appendix 8) a divided image generating step of dividing the image and generating a plurality of divided images; a first modification step of modifying pixel values ​​of each of the plurality of divided images; a reduced image generating step of reducing the image and generating a reduced image of the image; a second modification step of modifying pixel values ​​of the reduced image; a corresponding area specifying step of specifying areas in the reduced image changed in the second changing step that correspond to each of the plurality of divided images changed in the first changing step; a correction step of correcting the pixel values ​​of the divided image changed in the first change step based on pixel values ​​of pixels included in the specified region; a combining step of combining the plurality of divided images corrected in the correcting step; An image processing method that performs (Appendix 9) On the computer, a divided image generating step of dividing the image and generating a plurality of divided images; a first modification step of modifying pixel values ​​of each of the plurality of divided images; a reduced image generating step of reducing the image and generating a reduced image of the image; a second modification step of modifying pixel values ​​of the reduced image; a corresponding area specifying step of specifying areas in the reduced image changed in the second changing step that correspond to each of the plurality of divided images changed in the first changing step; a correction step of correcting the pixel values ​​of the divided image changed in the first change step based on pixel values ​​of pixels included in the specified region; a combining step of combining the plurality of divided images corrected in the correcting step; An image processing program for executing the above.

[0113] The present invention is not limited to the configurations described in the above embodiments, but may be combined with other elements, etc. These aspects can be changed without departing from the spirit of the present invention, and can be appropriately determined depending on the application form. [Explanation of symbols]

[0114] 100: Image generation system 110: Imaging device 120: Information terminal 140: Image processing system 150: Image acquisition unit 160: Image processing unit 310: Image reading unit 320: Divided image generation unit 330: First change section 340: Reduced image generation unit 350: Second change section 360: Corresponding area identification unit 370: Correction unit 380:Joining part 410: Division mode determination unit 420 :Divided part 820 :Divided area generation unit 830: Area unit correction value calculation unit 840: Pixel-by-pixel correction value calculation unit 850: Pixel-by-pixel correction unit [Prior art documents] [Patent documents]

[0115] [Patent Document 1] Japanese Patent Application Publication No. 2018-56743

Claims

1. a divided image generation unit that divides an image and generates a plurality of divided images; a first change unit that changes pixel values ​​of each of the plurality of divided images; a reduced image generating unit that reduces the image and generates a reduced image of the image; a second modification unit that modifies pixel values ​​of the reduced image; a corresponding area specifying unit that specifies areas in the reduced image that have been changed by the second changing unit that correspond to each of the plurality of divided images that have been changed by the first changing unit; a correction unit that corrects pixel values ​​of the divided image changed by the first change unit based on pixel values ​​of pixels included in the specified region; a combining unit that combines the plurality of divided images corrected by the correction unit; An image processing system having:

2. the pixel values ​​include luminance and chrominance information; the first change unit changes pixel values ​​of the divided image so as to increase either or both of luminance and color difference of at least some of pixels constituting the divided image; the second modification unit modifies pixel values ​​of the reduced image so as to increase either or both of luminance and color difference of at least some of the pixels constituting the reduced image. The image processing system according to claim 1 .

3. The correction unit a first calculated value calculated from pixel values ​​of pixels included in a divided area obtained by further dividing the divided image whose pixel values ​​have been changed; a second calculated value calculated from pixel values ​​of pixels included in an area dividing the area corresponding to the divided image in the reduced image whose pixel values ​​have been changed; and calculating a correction value for each of the plurality of divided regions based on the The image processing system according to claim 1 .

4. the first calculated value is an average value of pixel values ​​of pixels included in the divided region, the second calculated value is an average value of pixel values ​​of pixels included in an area of ​​the reduced image that divides the area corresponding to the divided image; The image processing system according to claim 3 .

5. The correction unit interpolating correction values ​​for correcting pixel values ​​of each pixel included in each of the plurality of divided regions using the correction values ​​for each of the plurality of divided regions; correcting the pixel values ​​of the pixels included in each of the plurality of divided regions changed by the first change unit using the interpolated correction values ​​of the pixels; The image processing system according to claim 3 .

6. The plurality of divided images are Adjacent divided images have overlapping portions, The correction unit The overlapping portions of the adjacent divided images are corrected using a correction value obtained by weighting and adding the correction values ​​of each pixel calculated for each divided image. The image processing system according to claim 1 .

7. a divided image generation unit that divides an image and generates a plurality of divided images; a first change unit that changes pixel values ​​of each of the plurality of divided images; a reduced image generating unit that reduces the image and generates a reduced image of the image; a second modification unit that modifies pixel values ​​of the reduced image; a corresponding area specifying unit that specifies areas in the reduced image that have been changed by the second changing unit that correspond to each of the plurality of divided images that have been changed by the first changing unit; a correction unit that corrects pixel values ​​of the divided image changed by the first change unit based on pixel values ​​of pixels included in the specified region; a combining unit that combines the plurality of divided images corrected by the correction unit; An image processing device having:

8. a divided image generating step of dividing the image and generating a plurality of divided images; a first modification step of modifying pixel values ​​of each of the plurality of divided images; a reduced image generating step of reducing the image and generating a reduced image of the image; a second modification step of modifying pixel values ​​of the reduced image; a corresponding area specifying step of specifying areas in the reduced image changed in the second changing step that correspond to each of the plurality of divided images changed in the first changing step; a correction step of correcting the pixel values ​​of the divided image changed in the first change step based on pixel values ​​of pixels included in the specified region; a combining step of combining the plurality of divided images corrected in the correcting step; An image processing method that performs

9. On the computer, a divided image generating step of dividing the image and generating a plurality of divided images; a first modification step of modifying pixel values ​​of each of the plurality of divided images; a reduced image generating step of reducing the image and generating a reduced image of the image; a second modification step of modifying pixel values ​​of the reduced image; a corresponding area specifying step of specifying areas in the reduced image changed in the second changing step that correspond to each of the plurality of divided images changed in the first changing step; a correction step of correcting the pixel values ​​of the divided image changed in the first change step based on pixel values ​​of pixels included in the specified region; a combining step of combining the plurality of divided images corrected in the correcting step; An image processing program for executing the above.

Images