Image processing apparatus and method, image processing system, program and storage medium

Abstract

The present invention provides an image processing apparatus and method, an image processing system, a program, and a storage medium for obtaining color image data with a smaller data volume and high reproducibility. [Solution] In the image processing system, the imaging device that acquires an image has an image information acquisition unit 210 that acquires a color image 300 from the imaging unit and generates brightness value information (image brightness information) and color information (image color information) from the color image, a brightness image generation unit 211 that generates a brightness image 301 based on the image brightness information input from the image information acquisition unit 210 and outputs it to the recording unit, and a prompt generation unit 212 that generates a color information prompt 302 which is text indicating the characteristics of the color by analyzing the image color information, and the image restoration device generates an image using the brightness image and text.

Description

Technical Field

[0006] , , , ,

[0005] , , , ,

[0001] The present invention relates to an image processing apparatus and method, an image processing system, a program, and a storage medium, and particularly relates to a technique for reducing the data amount without degrading the quality of a captured image.

Background Art

[0002] Conventionally, an image generation technique is known that generates an image using a VLM (Visual Language Model) and a diffusion model with an input of a sentence called a prompt, such as DALL·E2 developed by OpenAI.

[0003] On the other hand, Patent Document 1 discloses a technique for colorizing a monochrome image based on the monochrome image and the result of a prediction model that has a prediction model for predicting the color information of pixels from specific classifications and corresponding color information.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In recent years, with the evolution of smartphones and digital cameras, imaging sensors have become higher resolution. Due to the increase in resolution, the data amount of captured color images also tends to increase, so there is concern about the pressure on the communication time and the capacity of the storage medium at the storage destination, and a reduction in the data amount is required.

[0006] Here, if we attempt to reduce the data size of a color image using the image generation technique described above with the prompt as input, the following problem arises. That is, when reconstructing an image using the reduced data size image data and prompt, because the prompt mainly consists of sentences and words, it lacks detailed information such as detail and color representation compared to the captured image.

[0007] On the other hand, the prior art disclosed in Patent Document 1 is a technique for colorizing monochrome images, and does not include any description regarding the reduction of the data size of color images.

[0008] This invention was made in view of the above-mentioned problems, and aims to obtain color image data with less data volume and high reproducibility. [Means for solving the problem]

[0009] To achieve the above objective, the image processing apparatus of the present invention comprises an acquisition means for acquiring an image, a first generation means for generating a brightness image showing the brightness of the acquired image, and a second generation means for generating text showing the characteristics of the acquired image. [Effects of the Invention]

[0010] According to the present invention, it is possible to obtain color image data with a smaller amount of data and with high reproducibility. [Brief explanation of the drawing]

[0011] [Figure 1] A block diagram showing an example configuration of an image processing system using an imaging device according to an embodiment of the present invention. [Figure 2] A block diagram showing an example of the hardware configuration of the imaging device in the first embodiment. [Figure 3] A conceptual diagram illustrating the mechanism for generating luminance images and color information prompts in the first embodiment. [Figure 4] A conceptual diagram illustrating the mechanism for restoring a color image in the first embodiment. [Figure 5] Flowchart of the process in the first embodiment. [Figure 6] Block diagram showing an example of the hardware configuration of the imaging device in the second embodiment. [Figure 7] Block diagram showing an example of the functional configuration of the auxiliary color information generation unit in the second embodiment. [Figure 8] Flowchart of the process in the second embodiment. [Figure 9] Conceptual diagram explaining the mechanism for generating the luminance image and the color information prompt in the third embodiment. [Figure 10] Flowchart of the process in the third embodiment. [Figure 11] Block diagram showing an example of the functional configuration of the luminance image generation unit in the fourth embodiment. [Figure 12] Conceptual diagram showing an example of the determination process by the edge detection determination unit in the fourth embodiment. [Figure 13] Flowchart of the process in the fourth embodiment. [Figure 14] Flowchart of the edge detection determination process in the fourth embodiment. [Figure 15] Conceptual diagram explaining the mechanism for generating the object image and the object information prompt in the fifth embodiment. [Figure 16] Flowchart of the process in the fifth embodiment. [Figure 17] Diagram showing details of the object image in the fifth embodiment.

Modes for Carrying Out the Invention

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and redundant explanations are omitted.

[0013] In the following embodiments, the case where the present invention is implemented as an image processing system using an imaging device will be described. However, as the imaging device, it is possible to implement using any electronic device having an imaging function. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game machines, robots, drones, and drive recorders. These are examples, and the present invention can also be implemented with other electronic devices. Further, the imaging device may be configured not as a single unit but as any electronic device having an imaging function and an image processing device that processes an image obtained from the electronic device.

[0014] <First Embodiment> ●Configuration First, the image processing system in the first embodiment of the present invention will be described. FIG. 1 is a diagram showing an example of the configuration of an image processing system 100 in the first embodiment of the present invention. As shown in FIG. 1 , the image processing system 100 includes an imaging device 101, an image restoration device 102, and a display device 103. Note that the imaging device 101, the image restoration device 102, and the display device 103 may each be independently configured, or at least a part thereof may be integrally configured. Further, it may have a recording device or the like for storing the restored image.

[0015] The imaging device 101 acquires image data corresponding to the optical image of the subject using lenses, image sensors, etc. The imaging device 101 also applies predetermined image processing to the image data to generate image data using luminance value information (luminance image) and text information using feature information of the image data (prompt information). Since the luminance image is composed of luminance value information, it is a grayscale image.

[0016] The image restoration device 102 communicates with the imaging device 101 via a communication network to acquire the luminance image and prompt information. Alternatively, the image restoration device 102 may acquire the luminance image and prompt information by connecting to a recording medium stored by the imaging device 101. The image restoration device 102 uses the luminance image and prompt information acquired from the imaging device 101 to restore the image. For example, a convolutional neural network (CNN) is used to restore the grayscale luminance image to a color image. At this time, the image restoration device 102 can determine what kind of color image to generate by referring to the text information of the color information prompt included in the prompt information. The display device 103 acquires and displays the color image restored from the image restoration device 102.

[0017] Next, with reference to Figure 2, an example of the hardware configuration of the imaging device 101 in the present invention will be described.

[0018] The imaging device 101 includes a CPU (Central Processing Unit) 202, a ROM (Read Only Memory) 203, a memory 204, an input unit 205, a display unit 206, an imaging unit 207, a recording unit 208, a communication unit 209, an image information acquisition unit 210, a brightness image generation unit 211, and a prompt generation unit 212. These components of the imaging device 101 are connected to each other via a system bus 201 so that data can be sent and received from one another.

[0019] The CPU 202 is one or more processors capable of executing programs. The CPU 202 implements each functional block by, for example, reading a program stored in the ROM 203 into the memory 204 and executing it. Note that the various programs necessary for the CPU 202 to operate are not limited to the ROM 203, but may also be stored in other storage areas such as a hard disk. The CPU 202 controls the operation of the display unit 206, imaging unit 207, recording unit 208, communication unit 209, image information acquisition unit 210, brightness image generation unit 211, and prompt generation unit 212 according to the program, based on the control signals supplied from the input unit 205 in response to user operations received by the input unit 205. This makes it possible to perform operations in response to user operations.

[0020] ROM203 is, for example, an electrically rewritable non-volatile memory that stores various programs necessary for the CPU202 to operate. Memory 204 is a recording area that temporarily holds data, implemented, for example, by RAM (Random Access Memory). The CPU 202 uses memory 204 as work memory according to a program stored, for example, in ROM 203.

[0021] The input unit 205 receives user input and generates control signals corresponding to those user inputs, which it then supplies to the CPU 202. These control signals may include signals to instruct the camera to take a picture or signals related to settings for taking pictures. The input unit 205 may also have physical input devices that accept user input, such as physical buttons or a touch panel. A touch panel is an input device configured to output coordinate information corresponding to the position of contact with a planar input unit. The input unit 205 may also accept user input through voice recognition or gaze detection.

[0022] The display unit 206 is implemented by a display such as an LCD and includes a mechanism for outputting display signals to display an image on the screen. If a touch panel is used as the input unit 205, the input unit 205 and the display may be configured as an integrated unit. In this case, for example, the touch panel is configured such that its light transmittance does not interfere with the display of the display, and is mounted on the upper layer of the display surface of the display, and the input coordinates on the touch panel are associated with the display coordinates on the display.

[0023] The imaging unit 207 is a mechanism that performs a series of imaging devices, including an imaging unit composed of a lens, a shutter with an aperture function, and a sensor such as a CCD or CMOS element that converts an optical image into an electrical signal, and an image processing unit that performs various image processing such as exposure control based on the signal from the imaging unit. The operation of the imaging unit 207 is controlled by the CPU 202, which enables imaging of a subject in response to user operation via the input unit 205. In this embodiment, the color image generated by the imaging unit 207 is described as being in RGB format, but it may be a RAW image, or the image processing unit may perform image processing and convert it to YUV format, HSV format, etc.

[0024] The communication unit 209 transmits and receives data with other terminals based on the control of the CPU 202. The communication unit 209 may be implemented, for example, by a NIC (Network Interface Card) for a wired LAN, and may be connected to the NIC of the image restoration device 102 to transmit luminance images, color information prompts, etc., stored in the recording unit 208 to the image restoration device 102 (external device).

[0025] The image information acquisition unit 210 includes a luminance conversion unit that converts RGB data into luminance values ​​using a predetermined algorithm such as the average method, weighted average method, or luminance method to generate image luminance information, and a color information conversion unit that generates image color information indicating the color information of each pixel. Alternatively, the image information acquisition unit 210 may be configured to receive and output image luminance information and image color information from the imaging unit 207, respectively.

[0026] The luminance image generation unit 211 generates a luminance image based on the image luminance information input from the image information acquisition unit 210, based on the control of the CPU 202, and outputs it to the recording unit 208.

[0027] The prompt generation unit 212 includes an image color information analysis unit that analyzes image color information input from the image information acquisition unit 210 based on the control of the CPU 202, and a color information prompt generation unit that generates a color information prompt indicating the color characteristics of the image from the analysis results.

[0028] The recording unit 208 is, for example, a semiconductor memory card, an SSD (Solid State Drive), and is a storage area for storing data such as images obtained by imaging. The recording unit 208 is also used as a storage location for luminance images generated by the luminance image generation unit 211 and color information prompts generated by the prompt generation unit 212. When both luminance images and color information prompts are stored in the recording unit 208, they may be associated with each other. For example, the color information prompt can be recorded as metadata (additional information) recorded in the data file that stores the image data of the luminance image. Alternatively, the color information prompt may be stored as a separate text file from the image data file of the luminance image.

[0029] ●Generation of luminance image and color information prompt Figure 3 is a conceptual diagram illustrating the mechanism for generating a luminance image and a color information prompt in this embodiment. The image information acquisition unit 210 acquires a color image 300 from the imaging unit 207 and generates brightness value information (image brightness information) and color information (image color information) from the color image. For example, the image information acquisition unit 210 generates image brightness information by converting the RGB-formatted color image 300 acquired from the imaging unit 207 into brightness values ​​using a predetermined algorithm.

[0030] The luminance image generation unit 211 then generates a luminance image 301 (grayscale image) from the image luminance information.

[0031] Meanwhile, the prompt generation unit 212 generates a color information prompt 302, which is text indicating color characteristics, by analyzing the image color information. For example, the prompt generation unit 212 generates a color information prompt from information obtained by analyzing the color components contained in the image through color clustering. Alternatively, it may generate a color information prompt from information such as the RGB color code value of a predetermined pixel in the image and the coordinate value indicating the position of the corresponding pixel. Note that the information contained in the color information prompt is not limited to these, and any information related to color obtained from the analysis results is acceptable.

[0032] ● Restoring color images Next, the mechanism for reconstructing a color image from a luminance image and a color information prompt in this embodiment will be described. Figure 4 is a conceptual diagram illustrating the mechanism by which the image restoration device 102 restores a color image from a luminance image and a color information prompt in this embodiment. The image restoration device 102 acquires the luminance image 301 and the color information prompt 302 shown in Figure 3 and restores a color image 401. In this embodiment, the image restoration device 102 restores a grayscale image to a color image by inference processing of a neural network using a learning model 400, as shown in Figure 4. However, the method of restoring a color image is not limited to this, and can also be achieved using, for example, an image generation AI or known techniques such as those described in the following document: Richard Zhang, Jun-Yan Zhu, “Real-Time User-Guided Image Colorization with Learned Deep Priors”, ACM Transactions on Graphics, May 8, 2017.

[0033] ● Processing Next, referring to the flowchart in Figure 5, an example of a series of processes in this embodiment when the imaging device 101 takes an image and saves the luminance image and color information prompt to the recording medium will be described. Note that the operation of each step is realized by the CPU 202 executing a program recorded in the ROM 203 and controlling other hardware as necessary.

[0034] In S501, the imaging unit 207 receives user input from the input unit 205, performs still image capture, and acquires a color image. The imaging unit 207 may also perform video recording and use the video frame images as color images. The imaging unit 207 outputs the acquired color image to the image information acquisition unit 210. The imaging unit 207 may also perform image processing on the color image to make it suitable for processing by the image information acquisition unit 210 before outputting it. The process then proceeds to S502.

[0035] In S502, the image information acquisition unit 210 separates the color image into image luminance information and image color information, outputs the image luminance information to the luminance image generation unit 211, outputs the image color information to the prompt generation unit 212, and proceeds to S503.

[0036] In S503, the luminance image generation unit 211 generates a luminance image from the image luminance information, outputs it to the recording unit 208, and proceeds to S504.

[0037] In S504, the prompt generation unit 212 generates a color information prompt from the image color information and outputs it to the recording unit 208. Note that the color information prompt may include not only descriptions of color elements included in the image (positive prompts) but also prompts that include descriptions of color elements not included in the image (negative prompts). The process then proceeds to S505.

[0038] In S505, the recording unit 208 saves the luminance image and the color information prompt. At this time, both the luminance image and the color information prompt may be saved in association. For example, the color information prompt can be recorded as metadata recorded in the data file that stores the image data of the luminance image. Alternatively, the color information prompt may be saved as a separate text file from the image data file of the luminance image. After that, the process ends.

[0039] As described above, according to the first embodiment, the amount of image data can be reduced and colors can be restored with high fidelity by separating and saving the luminance image and color information prompt. In other words, since the image data can be saved as a grayscale image using only the image luminance information from the image data, the amount of data can be reduced. Furthermore, since the grayscale image can be restored to a color image by generating a color information prompt from the image color information of the image data, it becomes possible to restore a color image with high color reproducibility. As a result, even when shooting with a camera with a high-resolution sensor, for example, the image data size is reduced, thus preventing data storage pressure.

[0040] <Second Embodiment> Next, a second embodiment of the present invention will be described. In the second embodiment, an example of a mechanism for further improving color reproducibility during image restoration is described, which involves adding auxiliary color information to enhance the accuracy of the color information prompt.

[0041] In this embodiment, we will describe the parts that are particularly different from the first embodiment described above, and we will omit explanations of parts that are substantially the same as those in the first embodiment as appropriate. The configuration of the image processing system 100 in the second embodiment is the same as that described in the first embodiment with reference to Figure 1. However, the configuration of the imaging device 101 is different from the configuration in the first embodiment, so it will be described below.

[0042] Figure 6 is a block diagram showing the hardware configuration of the imaging device 101 in the second embodiment. The hardware configuration of the imaging device 101 shown in Figure 6 is the same as that of the imaging device 101 shown in Figure 2, with the addition of an auxiliary color information generation unit 603, and the operation of the imaging unit 602 and the prompt generation unit 604 differs from that of the first embodiment. The other components are the same as those shown in Figure 2, so the same reference numerals are used and their explanation is omitted.

[0043] The imaging unit 602 outputs imaging information such as shooting settings and subject detection results to the auxiliary color information generation unit 603. The auxiliary color information generation unit 603 acquires imaging information such as shooting settings and subject detection results from the imaging unit 602, analyzes color-related information from the imaging information to generate auxiliary color information, and outputs it to the prompt generation unit 604. The prompt generation unit 604 generates a color information prompt using the image color information from the image information acquisition unit 210 and the auxiliary color information from the auxiliary color information generation unit 603.

[0044] Figure 7 is a block diagram showing an example of the functions of the auxiliary color information generation unit 603 in the imaging device of the second embodiment.

[0045] The imaging unit 602, based on the control of the CPU 202, performs imaging and image processing using various information such as user setting information for image settings desired by the user, shooting setting information indicating information about the settings at the time of imaging, and subject detection information from the AF (autofocus) function at the time of imaging.

[0046] The user setting acquisition unit 701 acquires user setting information of image settings set by the user according to their preferences from the imaging unit 602 and outputs it to the prompt generation unit 604 as auxiliary color information. User setting information is information related to the color of the image set by the user, such as a user proset that allows the user to set the image to their preferred color tone. User setting information is not limited to this and may also include information obtained by numerically inputting the color temperature or learning the user's setting preferences.

[0047] The imaging setting acquisition unit 702 acquires shooting setting information from the imaging unit 602, which is information related to the settings used when taking images, and outputs it to the prompt generation unit 604 as auxiliary color information. The shooting setting information is, for example, white balance setting information and information related to settings at the time of shooting, such as color temperature information (Kelvin). However, the shooting setting information is not limited to this and may also include color-related information such as location information of the shooting location, time, season, and manufacturer information. For example, time and season information determines the color temperature and leaf color of the restored image depending on the time of day and season, such as morning, evening, spring, and autumn. Also, since the characteristics of color creation by each manufacturer differ, it is possible to restore the color tone of a specified manufacturer by, for example, an AI model that has learned images associated with each manufacturer.

[0048] The subject color detection unit 703 acquires subject detection information and image data from the imaging unit 602. Subject detection can be performed using conventional methods, such as those used in AF (autofocus) functions. The subject color detection unit 703 detects information about the subject's color from the subject information and image data, and outputs it as auxiliary color information to the prompt generation unit 604. For example, if information indicating a person has been recognized is acquired, the unit detects the color of the shirt and pants. The detected subject is not limited to people; it may also include animals, buildings, the sky, etc.

[0049] The prompt generation unit 604 generates a color information prompt that indicates the characteristics of a color using the auxiliary color information input from the imaging setting acquisition unit 702 and the image color information acquired from the image information acquisition unit 210.

[0050] Figure 8 is a flowchart of the process in the second embodiment. Here, we describe an example of a series of processes when the imaging device 101 takes an image and saves the luminance image and color information prompt to the recording medium. The operation of each step is realized by the CPU 202 executing a program recorded in the ROM 203 and controlling other hardware as necessary. In the process shown in Figure 8, processes S501 to S503 and S505 are the same as the processes described with reference to Figure 5 in the first embodiment, so their explanation is omitted.

[0051] In S801, the auxiliary color information generation unit 603 acquires information such as shooting settings and subject detection results from the imaging unit 602, generates auxiliary color information, and outputs it to the prompt generation unit 604. Then, the process proceeds to S802.

[0052] In S802, the prompt generation unit 604 generates a color information prompt using the image color information from the image information acquisition unit 210 and the auxiliary color information from the auxiliary color information generation unit 603, outputs it to the recording unit 208, and proceeds to S505.

[0053] As described above, according to the second embodiment, by obtaining auxiliary color information from shooting information, user setting information, subject recognition information, etc., a more suitable color information prompt can be generated, thereby further improving color reproduction during image restoration.

[0054] <Third Embodiment> Next, a third embodiment of the present invention will be described. In the third embodiment, an example of a mechanism for further reducing the amount of data in a luminance image by converting the luminance image into a binary edge image will be described.

[0055] In this embodiment, we will describe the parts that are particularly different from the first embodiment described above, and we will omit explanations of parts that are substantially the same as those in the first embodiment as appropriate. Furthermore, the image processing system in the third embodiment will be the image processing system 100 that was described in the first embodiment using Figures 1, 2, and 4.

[0056] Figure 9 is a conceptual diagram illustrating the mechanism for generating a luminance image and a color information prompt in this embodiment. The luminance image generation unit 211 generates a binary edge image 901 from the image luminance information received from the image information acquisition unit 210. The edge image 901 is a luminance image in which the contours (edges) of an object are detected with high accuracy, for example, using the Canny method (Canny Edge Detection), and the edges are emphasized.

[0057] Furthermore, the image restoration device 102 can restore the color image 300 at the time of acquisition without losing detail by using the edge image 901 and the color information prompt 302 to restore it as a color image 401.

[0058] Next, referring to the flowchart in Figure 10, an example of a series of processes when the imaging device 101 takes an image and saves the edge image 901 and color information prompt to the recording medium will be described. Note that the operation of each step is realized by the CPU 202 executing a program recorded in the ROM 203 and controlling other hardware as necessary. In addition, in the process shown in Figure 10, the processes S501, S502, and S504 are the same as the processes described with reference to Figure 5 in the first embodiment, so their explanation will be omitted.

[0059] In S1001, the luminance image generation unit 211 detects the edges of the luminance image and generates an edge image 901. The process then proceeds to S504.

[0060] Furthermore, in S1002, the recording unit 208 saves the edge image 901 and the color information prompt. The recording unit 208 may also save the edge image 901 and the color information prompt in association, similar to the first embodiment described above. After that, the process is terminated.

[0061] As described above, according to the third embodiment, since a binary edge image using image brightness information is saved from the image data, it is possible to further reduce the amount of data. In addition, by restoring a color image from the color information prompt and the edge image, it is possible to restore details such as composition and color with high fidelity. As a result, even when shooting with a camera with a high-resolution sensor, for example, the image data size is reduced, thus preventing data capacity issues.

[0062] However, the first and third embodiments are in a trade-off relationship in terms of image data volume and detail reproduction. Specifically, firstly, while the first embodiment uses grayscale images, the third embodiment uses binary edge images, thus reducing the amount of data per pixel. Therefore, the third embodiment is superior in terms of image data volume. On the other hand, grayscale images express brightness gradation more accurately than binary edge images, so the first embodiment is superior in terms of detail reproduction. Therefore, it is advisable to select the appropriate embodiment depending on the function of the system to which the present invention is applied.

[0063] <Fourth Embodiment> Next, a fourth embodiment of the present invention will be described. In the fourth embodiment, an example of a mechanism that reduces the amount of data in a luminance image and improves color reproduction by adaptively using a grayscale image and a binary edge image in the generation of a luminance image will be described.

[0064] In this embodiment, we will describe the parts that are particularly different from the first and third embodiments described above, and we will omit explanations of parts that are substantially the same as those in the first and third embodiments as appropriate. Furthermore, as the image processing system in the fourth embodiment, we will use the image processing system 100 described with reference to Figures 1, 2, and 4, as in the first and third embodiments.

[0065] Figure 11 is a block diagram showing the functional configuration of the luminance image generation unit 211 in the fourth embodiment. In the fourth embodiment, the luminance image generation unit 211 has an image information acquisition unit 210, and the edge detection determination unit 1101 analyzes the image luminance information received from the image information acquisition unit 210 to determine whether or not edge detection of an object is possible. Furthermore, based on the analysis result, it outputs either grayscale image information 1102 or edge image information 1103.

[0066] Figure 12 is a conceptual diagram showing an example of an analysis method for determining whether edge detection is possible and a method for selecting an image to output based on the analysis results. Figure 12(a) is a conceptual diagram showing the image brightness information received from the image information acquisition unit 210. Figure 12(b) shows the divided regions, each of which is obtained by dividing the image luminance information into three vertical and three horizontal sections (a total of nine divisions), as divided images 1201 to 1209. Note that the division method is not limited to this, and various modifications and changes are possible.

[0067] Figure 12(c) shows an example of the results of the edge detection determination unit 1101 determining whether or not an image from divided image 1201 to divided image 1209 is an image in which edge detection is possible. The black circles and black stars in Figure 12(c) indicate areas where it is difficult to extract details by edge detection, while the unmarked areas indicate areas in which edge detection is possible. The black circles indicate areas where edge detection is difficult due to low contrast, such as clouds. The black stars indicate areas where, due to the complex depiction of grass and leaves, parts other than the main edges are also detected as edges, resulting in an inability to discern the details of the object.

[0068] In Figure 12(b), segmented image 1203 is low contrast due to clouds, and segmented images 1204, 1207, and 1208 are difficult to detect edges in due to their complex depiction of grass.

[0069] Next, referring to the flowchart in Figure 13, an example of a series of processes when the imaging device 101 takes an image and saves grayscale image information 1102, edge image information 1103, and color information prompt to the recording medium will be described. Note that the operation of each step is realized by the CPU 202 executing a program recorded in the ROM 203 and controlling other hardware as necessary. Furthermore, in the process shown in Figure 13, the processes S501, S502, and S505 are the same as the processes described with reference to Figure 5 in the first embodiment, so their explanation will be omitted.

[0070] In S1300, the edge detection and determination unit 1101 performs edge detection and analysis on the image brightness information, and outputs either grayscale image information 1102, edge image information 1103, or both, depending on the analysis result. The process then proceeds to S505.

[0071] Figure 14 is a flowchart detailing the edge detection determination process performed in S1300. In this embodiment, a method using contrast ratio analysis is described as a way to determine whether or not an edge can be detected.

[0072] In S1401, the edge detection and determination unit 1101 divides the image brightness information and proceeds to S1402. Here, as an example, the divided image 1201 is divided into divided image 1209 as shown in Figure 12(b). In S1402, the edge detection determination unit 1101 initializes a determination map (not shown) that holds the analysis results of the segmented images, and proceeds to S1403. In S1403, the edge detection determination unit 1101 obtains a histogram of the image brightness information of one segmented image (for example, segmented image 1201), calculates the contrast ratio, and proceeds to S1404.

[0073] In S1404, the edge detection determination unit 1101 determines whether the contrast ratio of the divided image is within a predetermined range. Here, the contrast ratio is compared with a reference value, and based on the comparison, it is determined whether the edge image is appropriate for reproducing the color image. For example, in the case of a luminance image with 8 bits per pixel, low contrast is generally considered to be between 3:1 and 1:1, while high contrast, which hinders edge detection, is considered to be between 100:1 and 255:1. However, the reference value is not limited to these.

[0074] In low-contrast situations, edge detection may fail, and objects are likely to disappear when an edge image is generated. Therefore, edge images are unsuitable for reproducing color images. On the other hand, in high-contrast situations that hinder edge detection, there are many fine objects and noise, and it is highly likely that an edge image is generated that captures unnecessary details as edges. Therefore, in this case too, edge images are unsuitable for reproducing color images.

[0075] Note that the determination method in S1404 may be other than those described above. For example, the determination may be made by checking whether or not there are multiple peaks in the histogram. If there are multiple peaks, it is highly likely that the image contains fine objects or a lot of noise, and therefore the edge image can be judged as unsuitable for reproducing a color image.

[0076] If the comparison result is low contrast or high contrast that interferes with edge detection, the edge image is deemed unsuitable for detail reproduction during playback, as described above, and the process proceeds to S1406. Otherwise, the edge image is deemed suitable for detail reproduction during playback, and the process proceeds to S1405.

[0077] In S1405, the edge detection determination unit 1101 determines that the edge image is appropriate, generates edge image information 1103, and proceeds to S1407. On the other hand, in S1406, the edge detection determination unit 1101 determines that the edge image is inappropriate, generates grayscale image information 1102, and proceeds to S1407.

[0078] In S1407, the edge detection decision unit 1101 stores the decision result from S1404 for the segmented image 1201 in the decision map. In the case of segmented image 1201, it stores that edge image information 1103 was selected, as shown in Figure 12(c). Then, the process proceeds to S1408.

[0079] In S1408, the edge detection determination unit 1101 checks whether the determination and generation of output images have been completed for all divided images. If it is not yet finished, it returns to S1403; if it is finished, it terminates the process and proceeds to the process in S505 shown in Figure 13.

[0080] Furthermore, when saving grayscale image information 1102 or edge image information 1103 to the recording unit 208 in S505, the judgment map is also saved, and by using the judgment map when restoring the color image by the image restoration device 102, it becomes possible to generate a highly reproducible color image.

[0081] As described above, according to the fourth embodiment, based on the result of determining whether or not edge detection is possible with respect to the image brightness information, edge image information 1103 is saved for the area where edge detection is possible and used for restoring the color image. This makes it possible to reduce the amount of image data for the area in which edge image information 1103 is selected, while maintaining the same level of detail reproduction as in the first embodiment.

[0082] <Fifth Embodiment> Next, a fifth embodiment of the present invention will be described. In the fifth embodiment, an example of a mechanism that maintains image detail while reducing the amount of data for images where edge detection is difficult is described, by converting the luminance image into a binary object image and the color information prompt into an object information prompt.

[0083] In this embodiment, we will describe the parts that are particularly different from the first embodiment described above, and we will omit details of parts that are substantially the same as those in the first embodiment as appropriate. Also, as the image processing system in the fifth embodiment, we will use the image processing system 100 described with reference to Figures 1, 2, and 4, as in the first embodiment.

[0084] Figure 15 is a conceptual diagram illustrating the mechanism for generating object images and object information prompts in the fifth embodiment. The luminance image generation unit 211 generates a luminance image from the image luminance information received from the image information acquisition unit 210, detects objects contained in the luminance image, and generates a binary object image 1501 containing only the outlines (frames) of the objects. It also adds object information related to each detected object to the object image 1501. The luminance image generation unit 211, for example, detects the region and type of each object contained in the luminance image using panoptic segmentation, and generates an object frame that emphasizes the outline of each object region, as well as object information such as the type and position of the object. Note that a captured image may be used as the image for panoptic segmentation instead of a luminance image.

[0085] Object information can be information representing the relationships between objects, or any information related to objects. Object information can be added by directly embedding it as a string in the object image. Alternatively, information linking the object frame to the object information can be generated, and the object information and placement coordinates can be saved together as metadata in the object image data. In other words, any method of saving that associates the object frame with the object information is acceptable.

[0086] The prompt generation unit 212 analyzes the object frame, object information, and image color information together to generate object information prompts 1502, which are text indicating the color characteristics corresponding to each object. For example, the prompt generation unit 212 analyzes the color code values ​​within the target object from the pixel positions and object frame included in the image color information. It then generates the base color information of the object and the type of object as object information prompts. The color information may consist of multiple colors, and multiple colors may be set for each region.

[0087] When the image restoration device 102 restores the color image 401, it uses the aforementioned object image 1501 and object information prompt 1502 to restore the image, thereby enabling the restoration of all objects visible in the color image 300 at the time of acquisition without losing any detail.

[0088] Figure 16 is a flowchart of the process in the fifth embodiment. Here, an example of a series of processes when the imaging device 101 takes an image and saves the object image 1501 and the object information prompt 1502 to the recording medium is described. Note that the operation of each step is realized by the CPU 202 executing a program recorded in the ROM 203 and controlling other hardware as necessary. In the process shown in Figure 16, the processes S501 to S503 are the same as the processes described with reference to Figure 5 in the first embodiment, so their explanation is omitted.

[0089] In S1601, the luminance image generation unit 211 detects each object from the luminance image, generates an object frame from the contour of the detected object area, and generates an object image 1501. It also generates object information from the object detection results, adds it to the object image 1501, and proceeds to S1602.

[0090] In S1602, the prompt generation unit 212 generates an object information prompt 1502, which is a prompt corresponding to each object, from the object frame, object information, and image color information, and proceeds to S1603.

[0091] In S1603, the recording unit 208 saves the object image 1501 and the object information prompt 1502. The recording unit 208 may also save the object image 1501 and the object information prompt 1502 in association, similar to the first embodiment described above. After that, the process terminates.

[0092] Figure 17 shows the details of the object image 1501 in the fifth embodiment. Each object included in the object image 1501 has an object name embedded in the center of the frame, which is a combination of the object type, which is object information, and an alphabet for individual recognition. The object names are: 1701: Lawn A, 1702: Road, 1703: Lawn B, 1704: Tree A, 1705: Tree B, 1706: Building A, 1707: Building B, 1708: Sky, 1709: Cloud. Note that objects other than those described are separate individuals of the same type, so their description is omitted in this embodiment. The location where the object name is embedded does not have to be in the center of the frame; it can be anywhere where the object frame and the object name can correspond. The prompt generation unit 212 generates an object information prompt 1502 corresponding to each of the aforementioned object names. Furthermore, information indicating the relationships between each object may be added. For example, if the relationship is between 1704: Tree A and 1705: Tree B, information representing the relationship between the objects, such as "1704: Tree A is positioned in front of 1705: Tree B," may be added to the object information prompt 1502. By adding information indicating the relationships between objects, it is possible to maintain the detail when reproducing an image, even if the object image 1501 has less information than the edge image 901 described in the third embodiment.

[0093] As described above, according to the fifth embodiment, by saving an object image 1501 in which only the frame of an object included in the luminance image is extracted, and an object information prompt 1502 which is a prompt corresponding to each object, and using these to restore the color image, it becomes possible to reduce the amount of data for the areas where edge image information was not selected, while maintaining equivalent detail reproduction for each object, compared to the fourth embodiment.

[0094] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0095] <Summary> This embodiment includes the following configuration.

[0096] (Item 1) Means for acquiring images, A first generation means for generating a brightness image showing the brightness of the acquired image, A second generation means for generating text that indicates the characteristics of the acquired image, An image processing apparatus characterized by having (Item 2) The image processing apparatus according to item 1, characterized in that the features of the image include the color features of the image. (Item 3) The system further includes an image information acquisition means that converts the aforementioned image into brightness values ​​to generate brightness information, and also generates color information indicating the color information of each pixel of the aforementioned image. The first generation means generates the brightness image based on the brightness information, The second generation means generates the text based on the color information. The image processing apparatus according to item 2, characterized in that (Item 4) The image processing apparatus according to item 3, characterized in that the second generation means generates text indicating the color information and position information of each pixel of the image. (Item 5) The image processing apparatus according to any one of items 1 to 4, characterized in that the features of the image include user setting information set by the user, imaging information when the image was taken, and color information of each subject included in the image. (Item 6) The image processing apparatus according to item 5, characterized in that the user setting information is information based on the color tone of an image set by the user. (Item 7) The image processing apparatus according to item 5 or 6, characterized in that the imaging information includes at least one of the following: color temperature information, location information, time, season, and manufacturer information of the imaging device that captured the image. (Item 8) The image processing apparatus according to item 3, characterized in that the first generation means generates a brightness image with enhanced contours extracted from the image based on the brightness value. (Item 9) The image processing apparatus according to item 3, characterized in that the first generation means analyzes the contrast of the luminance values ​​for each of the plurality of divided images obtained by dividing the acquired image, and generates either a luminance image with enhanced contours extracted from the image or a luminance image without enhanced contours according to the analyzed contrast. (Item 10) The image processing apparatus according to item 9, characterized in that the first generation means generates a luminance image with the contours emphasized when the contrast is within a predetermined range, and generates a luminance image without contour emphasis when the contrast is not within the range. (Item 11) The image processing apparatus according to item 9 or 10, characterized in that the first generation means further generates information indicating whether a brightness image with enhanced contours or a brightness image without enhanced contours was generated for each of the plurality of divided images. (Item 12) The image processing apparatus according to any one of items 8 to 11, characterized in that the luminance image with the enhanced contour is an image in which the luminance values ​​of the image are represented by binary values. (Item 13) The image processing apparatus according to any one of items 1 to 12, further comprising a storage means for storing the luminance image and the text. (Item 14) The image processing apparatus according to item 13, characterized in that the storage means stores the text as additional information for the brightness image. (Item 15) The image processing apparatus according to any one of items 1 to 14, further comprising communication means for transmitting the luminance image and the text to an external device. (Item 16) The first generation means is, The object included in the aforementioned brightness image is detected, An object image consisting of the frame of the aforementioned object, Object information relating to the aforementioned object, Generate, The second generation means further generates the text based on the frame and the object information. The image processing apparatus according to item 3, characterized in that (Item 17) The image processing apparatus according to item 16, characterized in that the first generation means assigns the object information to each object constituting the object image. (Item 18) The image processing apparatus according to item 17, characterized in that the object information includes at least one of information that allows each object to be individually recognized and information that represents the relationships between each object. (Item 19) The image processing apparatus according to any one of items 1 to 15, characterized in that the acquisition means is an imaging means. (Item 20) The image processing apparatus according to any one of items 1 to 15, characterized in that the acquisition means acquires the image from an external source. (Item 21) An image processing device described in any one of items 1 to 20, A reproduction device that generates an image using the luminance image and the text. An image processing system characterized by having the following features. (Item 22) The image processing system according to item 18, further comprising a display device for displaying an image generated by the reproduction device. (Item 23) The process of acquiring images, A first generation step of generating a brightness image showing the brightness of the acquired image, A second generation step involves generating text that describes the features of the acquired image, An image processing method characterized by having the following features. (Item 24) A program for causing a computer to function as one of the means of an image processing apparatus described in any one of items 1 through 18. (Item 25) A computer-readable storage medium containing the program described in item 24.

[0097] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0098] 100…Image processing system, 101…Imaging device, 102…Image restoration device, 103…Display device, 201…System bus, 202…CPU, 203…ROM, 204…Memory, 205…Input unit, 206…Display unit, 207…Imaging unit, 208…Recording unit, 209…Communication unit, 210…Image information acquisition unit, 211…Brightness image generation unit, 212…Prompt generation unit, 300…Color image, 301…Brightness image, 302…Color information prompt, 400… Image restoration learning model, 401... Restored color image, 601... Imaging device, 602... Imaging unit, 603... Auxiliary color information generation unit, 604... Prompt generation unit, 701... User setting acquisition unit, 702... Imaging setting acquisition unit, 703... Subject color detection unit, 901... Edge image, 1101... Edge detection judgment unit, 1102... Grayscale image information, 1103... Edge image information, 1501... Object image, 1502... Object information prompt

Claims

1. Means for acquiring images, A first generation means for generating a brightness image showing the brightness of the acquired image, A second generation means for generating text that indicates the characteristics of the acquired image, An image processing apparatus characterized by having

2. The image processing apparatus according to claim 1, characterized in that the features of the image include the color features of the image.

3. The system further includes an image information acquisition means that converts the aforementioned image into brightness values ​​to generate brightness information, and also generates color information indicating the color information of each pixel of the aforementioned image. The first generation means generates the brightness image based on the brightness information, The second generation means generates the text based on the color information. The image processing apparatus according to claim 2.

4. The image processing apparatus according to claim 3, characterized in that the second generation means generates text indicating the color information and position information of each pixel of the image.

5. The image processing apparatus according to claim 1, characterized in that the features of the image include user setting information set by the user, imaging information when the image was taken, and color information of each subject included in the image.

6. The image processing apparatus according to claim 5, characterized in that the user setting information is information based on the color tone of an image set by the user.

7. The image processing apparatus according to claim 5, characterized in that the imaging information includes at least one of the following: color temperature information, location information, time, season, and manufacturer information of the imaging device that captured the image.

8. The image processing apparatus according to claim 3, characterized in that the first generation means generates a brightness image with enhanced contours extracted from the image based on the brightness value.

9. The image processing apparatus according to claim 3, wherein the first generation means analyzes the contrast of the luminance values ​​for each of the plurality of divided images obtained by dividing the acquired image, and generates either a luminance image with enhanced contours extracted from the image or a luminance image without enhanced contours according to the analyzed contrast.

10. The image processing apparatus according to claim 9, characterized in that the first generation means generates a luminance image with the contours emphasized when the contrast is within a predetermined range, and generates a luminance image without contour emphasis when the contrast is not within the range.

11. The image processing apparatus according to claim 9, characterized in that the first generation means further generates information indicating whether a brightness image with the contours emphasized or a brightness image without the contours emphasized was generated for each of the plurality of divided images.

12. The image processing apparatus according to claim 8, characterized in that the luminance image with the contours enhanced is an image in which the luminance values ​​of the image are represented by binary values.

13. The image processing apparatus according to claim 1, further comprising a storage means for storing the luminance image and the text.

14. The image processing apparatus according to claim 13, characterized in that the storage means stores the text as additional information for the brightness image.

15. The image processing apparatus according to claim 1, further comprising communication means for transmitting the luminance image and the text to an external device.

16. The first generating means is The object included in the aforementioned brightness image is detected, An object image consisting of the frame of the aforementioned object, Object information relating to the aforementioned object, Generate, The second generation means further generates the text based on the frame and the object information. The image processing apparatus according to claim 3.

17. The image processing apparatus according to claim 16, characterized in that the first generation means assigns the object information to each object constituting the object image.

18. The image processing apparatus according to claim 17, characterized in that the object information includes at least one of information that allows each object to be recognized individually and information that represents the relationships between each object.

19. The image processing apparatus according to claim 1, characterized in that the acquisition means is an imaging means.

20. The image processing apparatus according to claim 1, characterized in that the acquisition means acquires the image from an external source.

21. An image processing apparatus according to any one of claims 1 to 20, A reproduction device that generates an image using the luminance image and the text. An image processing system characterized by having the following features.

22. The image processing system according to claim 21, further comprising a display device for displaying an image generated by the reproduction device.

23. The process of acquiring images, A first generation step of generating a brightness image showing the brightness of the acquired image, A second generation step involves generating text that describes the characteristics of the acquired image, An image processing method characterized by having the following features.

24. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 18.

25. A computer-readable storage medium storing the program described in claim 24.

Images