Image processing apparatus, image processing method, and image processing program

Abstract

Provided is an improved image processing device etc. which uses the visual perception of vertebrates such as humans. An image processing device (100) comprises: an image acquisition unit (101) which acquires an image; a first image processing unit (110) which performs first image processing with respect to the acquired image, and which includes a first sampling unit (112) that performs first sampling for extracting from the acquired image one or more samples to be processed, and a color detection unit (113) that detects the color of the one or more extracted samples; and a second image processing unit (120) which performs second image processing that differs from the first image processing with respect to the acquired image, and which includes a second sampling unit (122) that performs second sampling for extracting from the acquired image one or more samples to be processed, and a color reduction unit (123) that reduces the color of the one or more extracted samples.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

Background Art

[0002] Due to the spread of video distribution and web conferencing under high-speed networks, there is concern about the increase in transfer data. There is a high need to provide and share accurate visual information with external devices via a network. However, depending on specific situations such as not only entertainment but also remote collaborative work and telemedicine (for example, satellite communication, mountainous areas, etc.), a high-speed network may not be available.

[0003] Patent Document 1 discloses an image compression system including an image operation device that operates under program control, an image compression device that operates under program control, and an image compression operation device that allows a user to specify an input source of a target image file and an output destination of a compressed image file to operate an image compression process. The image compression device performs character recognition on a compressed image compressed using reference compression rate data individually for each compression target image input from the image operation device, and a decision tree in which a plurality of nodes, which are data including a compression rate, are respectively associated with a node including a compression rate higher than the compression rate and a node including a compression rate lower than the compression rate and stored, based on reference difference rate data, difference rate data obtained by comparing reference image character recognition result data, compression image character recognition result data, specifies a compression rate, compresses the compression target image at the specified compression rate, repeats the character recognition, the specification of the compression rate, and the compression at the specified compression rate the number of times indicated by evaluation count data, and outputs a compressed result image obtained by the repetition.

[0004] Patent Document 2 discloses a video camera imaging apparatus having a pair of video cameras for left and right eyes, an image recognition device that inputs and processes the video signals of the video cameras, and a monitor device that inputs and displays the video signals from the image recognition device, which displays an imitation image of an image that can be visually obtained by the naked eye when a human looks at an object or actually, and imitates the fixation movement of a human by moving the pair of video cameras to a desired position.

[0005] Patent Document 3 describes a method including receiving raw image data corresponding to a series of raw images and processing the raw image data by an encoder of a processing device to generate encoded data. The encoder is characterized by an input / output conversion that substantially mimics the input / output conversion of at least one retinal cell of a vertebrate retina. The method also includes processing the encoded data by applying a dimensionality reduction algorithm to the encoded data to generate encoded data with reduced dimensions. The dimensionality reduction algorithm is configured to compress the amount of information contained in the encoded data. Devices and systems that can be used with such a method are also described.

[0006] Patent Document 4 discloses a method including the steps of receiving raw image data corresponding to a series of raw images, processing the raw image data to generate data encoded using an encoder characterized by an input / output conversion that substantially mimics the input / output conversion of a vertebrate retina, the processing including applying a spatio-temporal conversion to the raw image data to generate retinal output cell response values, the application of the spatio-temporal conversion including the application of a single-stage spatio-temporal conversion including a series of weights directly determined from experimental data generated using a stimulus including a natural scene, generating data encoded based on the retinal output cell response values, and applying a first machine vision algorithm to data generated at least partially based on the encoded data.

Prior Art Documents

Patent Document

[0007]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0008] In order to cope with limited network bandwidth, there is a need to more appropriately reduce transfer data within a range where there is no problem in image recognition at the transfer destination. The above Patent Documents 3 and 4 use an encoder characterized by an input / output conversion that substantially mimics the input / output conversion of the vertebrate retina, but there is still room for improvement.

[0009] This invention has been made to solve such problems, and an object thereof is to provide an improved image processing apparatus, an image processing method, and an image processing program that utilize the visual recognition of vertebrates such as humans.

Means for Solving the Problems

[0010] An image processing apparatus according to a first aspect of the present invention includes an image acquisition unit that acquires an image, a first image processing unit that performs first image processing on the acquired image, the first image processing unit including a first sampling unit that performs first sampling for extracting one or more samples to be processed from the acquired image, and a color detection unit that detects the color of the one or more extracted samples. A second image processing unit that performs a second image processing different from the first image processing on the acquired image, the second image processing unit including a second sampling unit that performs a second sampling for extracting one or more samples to be processed from the acquired image, and a color reduction unit that reduces the colors of the one or more extracted samples. It is provided with.

[0011] The image processing method according to the second aspect of the present invention includes a step of acquiring an image, A step of performing a first image processing on the acquired image, the step including performing a first sampling for extracting one or more samples to be processed from the acquired image, and detecting the colors of the one or more extracted samples. A step of performing a second image processing different from the first image processing on the acquired image, the step including performing a second sampling for extracting one or more samples to be processed from the acquired image, and reducing the colors of the one or more extracted samples. It includes.

[0012] The image processing program according to the third aspect of the present invention includes a process of acquiring an image, A process of performing a first image processing on the acquired image, the process including performing a first sampling for extracting one or more samples to be processed from the acquired image, and detecting the colors of the one or more extracted samples. A process of performing a second image processing different from the first image processing on the acquired image, the process including performing a second sampling for extracting one or more samples to be processed from the acquired image, and reducing the colors of the one or more extracted samples. Causes a computer to execute operations including.

Advantages of the Invention

[0013] According to the present invention, it is possible to provide a new image processing apparatus, an image processing method, and an image processing program that utilize the visual recognition of vertebrates such as humans.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0015] The present disclosure relates to a technique for executing image processing using image recognition of vertebrates such as humans. For example, a glaucoma patient may have no visual field defect but no subjective symptoms. That is, such a patient may not notice that the object is not visible. The present disclosure proposes an image processing method that uses such human vision and recognition to reduce image data within a range where there is no problem with recognition.

[0016] The image processing apparatus according to some embodiments can be used to appropriately convert the image data captured by a camera into a low-resolution image. Also, an image (or video) transfer system including the image processing apparatus according to some embodiments can be used to capture an image, reduce the image data, and then convert it into a high-definition image after transferring it via a network with limited bandwidth. The image processing apparatus according to some embodiments can be used to convert the image data captured by a low-resolution camera into a high-definition image.

[0017] Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. However, the present invention is not limited to the following embodiments. Also, for clarity of explanation, the following description and drawings are appropriately simplified.

[0018] FIG. 1 is a cross-sectional view of a human right eye seen from above the head. The lens 303 in the human eye 300 is located behind the pupil 302, has the ability to change its focal length, focuses objects at varying distances from the observer onto the observer's retina 320, and is sent to the observer's brain via the optic nerve 340, where it is visually interpreted by the brain. The retina 320 refers to the main part of the inner surface of the eye (such as that of a human, observer, etc.) that is equipped with a group of visual sensors on the side opposite the pupil 302 of the eye. The fovea 310 refers to a relatively small central part of the retina that is equipped with a large number of visual sensors capable of providing the sharpest vision and the most sensitive color detection in the eye. The macula lutea 312 is the area in the eye or on the retina that receives the most amount of light and is therefore also called the area of "the sharpest vision".

[0019] Figure 2 is a front view explaining an exemplary distribution of different retinal cells in the human eye. Cones 11 (the first retinal cells) are densely packed in the macula lutea 312. Only cones 11 are densely packed in the fovea 310. Rods 12 (the second retinal cells) are densely packed around the macula lutea 312. Since there are no photoreceptor cells in the optic disc 345, light cannot be sensed. The visual field corresponding to the optic disc 345 is a blind spot called the Mariotte blind spot.

[0020] Figure 3 is a front view explaining an exemplary distribution of the first retinal cells (cones) in the human eye. These cones 11 recognize colors (e.g., RGB). A large number of cones 11 (e.g., about 6 million in one eye) are densely packed in the macula lutea 312 at the center of the retina 320.

[0021] Figure 4 is a front view explaining an exemplary distribution of the second retinal cells (rods) in the human eye. Rods 12 do not recognize colors, but are more sensitive to light than cones 11 and respond to even a small amount of light. Therefore, rods 12 can recognize the shape of an object quite well even in the dark.

[0022] It is a conceptual diagram explaining an image processing method that mimics different retinal cells in the human eye. An image of a subject (e.g., a pigeon in FIG. 5) is acquired using a camera (e.g., an image sensor) (step 1). Next, a first image process (compression process) that mimics the first retinal cells (e.g., cone cells) of the human eye is performed on the acquired image (step 2). Sampling is performed based on the distribution of cone cells as shown in FIG. 3 (e.g., the number of samplings is 6 million), and recognition processing of the color information of the image (e.g., RGB color information, YCbCr information, HSV information, etc.) in each cone cell is performed. The image data after sampling and the color information corresponding to each cone cell are transmitted to an external device or the like. In this way, the image data reduced by the sampling of the first image process can be transmitted to an external device or the like.

[0023] Similarly, a second image process (compression process) that mimics the second retinal cells (e.g., rod cells) of the human eye is performed on the acquired image (step 3). Sampling is performed based on the distribution of rod cells as shown in FIG. 4 (e.g., the number of samplings is 120 million), and reduction processing (conversion to monochrome) of the color information of the image (e.g., RGB color information, YCbCr information, HSV information, etc.) in each rod cell is performed. The number of samplings in the second image process is significantly larger than the number of samplings in the first image process. The image data after sampling and the monochrome information corresponding to each rod cell are transmitted to an external device or the like. In this way, the image data reduced by the sampling of the second image process can be transmitted to an external device or the like. Note that either step 2 or step 3 may be performed first.

[0024] Finally, based on the image data and color information after the first image processing and the image data and monochrome information after the second image processing, a synthesis process (for example, a restoration process) is performed (step 4). Note that there are 6 million cone cells and 120 million rod cells, whereas the axons of ganglion cells that transmit visual information to the brain are about 1 million in one eye. The brain restores images from such limited information. By mimicking such human visual recognition processing, it can be applied to an image transfer system that transfers data via a network with limited bandwidth. Hereinafter, several specific embodiments will be described.

[0025] Embodiment 1 FIG. 6 is a block diagram showing the configuration of an image processing apparatus according to Embodiment 1. The image processing apparatus 100 includes an image acquisition unit 101, a first image processing unit 110, a second image processing unit 120, and a synthesis unit 150. The image processing apparatus 100 is realized by one or more computers. The image processing apparatus 100 in FIG. 6 incorporates all the components, but some components (for example, the synthesis unit 150) may be configured by another computer connected via a network.

[0026] The image acquisition unit 101 acquires image data obtained by imaging a subject using an image sensor (for example, a CCD (Charge-Coupled Device) sensor or a CMOS (Complementary MOS) sensor). The image may be a still image or a moving image. The image acquisition unit 101 may be, for example, a camera or may simply acquire image data from a camera.

[0027] The first image processing unit 110 performs a predetermined image processing (first image processing) mimicking the first retinal cells (for example, cone cells) on the image data from the image acquisition unit 101. The first image processing unit 110 includes a sampling unit 112 and a color detection unit 113.

[0028] The sampling unit 112 extracts samples from the image data from the image acquisition unit 101 based on, for example, a predetermined sampling matrix (template). Samples that are not extracted are discarded. The predetermined sampling matrix indicates the samples to be extracted from n×m processing blocks (details will be described later with reference to FIGS. 7 to 9). The sampling matrix is determined based on the distribution of the first retinal cells (for example, cone cells) as shown in FIG. 3. The number of samples to be extracted (the first number) can be arbitrarily set in consideration of the compression ratio of the image. In this way, the image data can be reduced by the compressive sampling process of the sampling unit 112.

[0029] The color detection unit 113 detects color information (for example, RGB data) for each sample extracted by the sampling unit 112 from the image from the image acquisition unit 101.

[0030] Also, the first image processing unit 110 can perform encoding processing and various compression processes. For example, the dynamic range or the luminance range may be compressed to a range that does not cause problems in recognition.

[0031] As described above, the sampled image data and the identified color information are sent to the synthesis unit 150 by the first image processing that mimics the first retinal cells (for example, cone cells).

[0032] On the other hand, the second image processing unit 120 also performs a predetermined image processing (second image processing) different from that of the first image processing unit 110, which mimics the second retinal cells (for example, rod cells), on the image data from the image acquisition unit 101. The second image processing unit 120 includes a sampling unit 122 and a color reduction unit 123.

[0033] The sampling unit 122 extracts samples from the image data from the image acquisition unit 101, for example, based on a predetermined sampling matrix. The predetermined sampling matrix is determined based on the distribution of the second retinal cells (for example, rod cells) as shown in FIG. 4. The samples not extracted are discarded. The number to be extracted (the second number) can be set to any number greater than the first number. Thus, the image data can be reduced by the compressive sampling process of the sampling unit 122.

[0034] The color reduction unit 123 reduces the color (RGB) of the image from the image acquisition unit 101 and converts it into a monochrome image or a grayscale image. Thereby, the image data can be reduced.

[0035] Also, the second image processing unit 120 can perform encoding processing and various compression processes. For example, the dynamic range or the luminance range may be compressed to a range that does not cause problems in recognition.

[0036] As described above, the image data sampled and color-reduced by the second image processing imitating the second retinal cells (for example, rod cells) is sent to the synthesis unit 150.

[0037] The synthesis unit 150 synthesizes the image data from the first image processing unit 110 and the image data from the second image processing unit 120. At this time, high-resolution conversion of the image may be performed using deep learning.

[0038] Here, with reference to FIGS. 7 to 9, an example of the distributed arrangement of a plurality of different sensor units will be described. FIG. 7 is a front view for explaining an exemplary distribution of a plurality of different sensor units according to Embodiment 1. This is a group of sensors imitating retinal cells. In FIG. 7, 11×11 processing blocks are arranged. Among these, the first sensor unit 21 (hatched processing blocks in FIG. 7) corresponds to the first retinal cells (for example, cone cells 11). On the other hand, the second sensor unit 22 (processing blocks filled with gray in FIG. 8) corresponds to the second retinal cells (for example, rod cells 12).

[0039] As described above, only one or more first sensor units 21 corresponding to the first retinal cells (for example, cone cells 11) are arranged at the center of the sampling matrix. Also, one or more second sensor units 22 corresponding to the second retinal cells (for example, rod cells 12) are arranged relatively densely around the central part where one or more first sensor units 21 are concentrated.

[0040] FIG. 8 is a diagram for explaining an exemplary distribution of the first sensor unit (corresponding to cone cells) according to Embodiment 1. Among 11×11 (total 121) processing blocks in the sampling matrix, 31 first sensor units are dispersedly arranged. Only the first sensor unit 21 is arranged in the central 3×3 processing blocks.

[0041] FIG. 9 is a diagram for explaining an exemplary distribution of the second sensor unit (corresponding to rod cells) according to Embodiment 1. Among 11×11 (total 121) processing blocks in the sampling matrix, 90 second sensor units are dispersedly arranged.

[0042] The distributions shown in FIGS. 8 and 9 are merely examples, and various modifications and variations can be made. However, the number of first sensor units configured to recognize colors is larger than the number of second sensors configured to reduce colors. Also, in the central part, the first sensor units (corresponding to cone cells) are distributed such that the number of first sensor units is larger than the number of second sensor units. Further, around this central part, the second sensor units are distributed such that the number of second sensor units is larger than the number of first sensor units. Note that the central part can refer to a part of the two central regions among the regions quartered in the X direction and the Y direction as shown in FIGS. 3 and 8.

[0043] According to the present embodiment described above, by executing two different image processes that mimic human visual recognition, image data can be appropriately reduced. Also, thereafter, by executing a synthesis process, it can be appropriately restored.

[0044] Embodiment 2 FIG. 10 is a block diagram showing the configuration of the image processing apparatus according to Embodiment 2. In Embodiment 2, random sampling for extracting samples at a specific probability is performed. Samples that are not extracted are discarded. That is, instead of the predetermined sampling matrix as described above, based on a specific probability, an area for performing image processing is randomly determined from a large number of divided processing blocks within the image. This specific probability is determined based on the distribution of the first retinal cells (e.g., cone cells) or the second retinal cells (e.g., rod cells) among the retinal cells of many people (subjects).

[0045] Also, in this embodiment, it is effective to change the distribution according to the object and purpose. For example, in the case of a low-light camera, since high sensitivity is emphasized, it can be achieved by increasing the ratio corresponding to rod cells. Also, in this embodiment, the spatial distribution of cone cells suitable for high-precision image processing such as machine learning can be set. By setting these according to the purpose, it becomes possible to design a camera having characteristics that cannot be achieved by the actual human eyeball.

[0046] The image processing apparatus 200 includes an image acquisition unit 201, a block division unit 205, a first image processing unit 210, a second image processing unit 220, and a synthesis unit 250. The image processing apparatus 100 is realized by one or more computers. The image processing apparatus 200 in FIG. 10 incorporates all the components, but some components (e.g., the synthesis unit 150) may be configured by another computer connected via a network.

[0047] The image acquisition unit 201 acquires image data obtained by imaging a subject using an image sensor (for example, a CCD (Charge-Coupled Device) sensor or a CMOS (Complementary MOS) sensor). The image may be a still image or a moving image. The image acquisition unit 201 may be, for example, a camera or may simply acquire image data from a camera.

[0048] The block division unit 205 divides the image from the image acquisition unit 101 into processing blocks and supplies it to the first image processing unit 210 and the second image processing unit 220. The unit of the processing block can be arbitrarily set by the designer. Here, the image is divided into n×m processing blocks. Note that each processing block may be arranged at equal intervals (for example, see FIGS. 7 to 9) or may be arranged at unequal intervals like retinal cells (see FIGS. 2 to 4).

[0049] As shown in FIG. 7, by adding (binning) the signals of pixels at equal intervals, it is possible to increase the apparent sensitivity. However, when treating 2×2 pixels as one pixel, since four signals are added in signal processing, the readout noise from the imaging device also increases by four times. On the other hand, if large elements can be mixed at the semiconductor design stage, it is possible to reduce the readout noise. In conventional cameras, an imaging device with large elements arranged at equal intervals was manufactured and sold as a digital camera. However, even in that case, the size of the elements was about twice as large, and it was difficult to obtain a dramatic effect. In order to solve this problem, it is necessary to increase the size of high-sensitivity elements. For that purpose, by randomly arranging the processing blocks, it is possible to create a location for element placement. On the other hand, a location where there is a defect (no element) occurs, but it is possible to supplement the missing information by storing, reproducing, and inferring that location through image processing.

[0050] The first image processing unit 210 performs predetermined image processing (first image processing) that mimics the first retinal cells (for example, cone cells) on the image data divided into a plurality of processing blocks from the block division unit 205. The first image processing unit 210 includes a random sampling unit 212 and a color detection unit 213.

[0051] The random sampling unit 212 randomly extracts samples from the processing blocks divided by the block division unit 205 based on a specific probability. FIG. 11 is a graph showing an exemplary probability distribution for a plurality of first sensor units in a specific region. For example, based on the probability distribution shown in FIG. 11, the first sensor units can be randomly sampled. The number to be extracted (the first number) can be arbitrarily set in consideration of the compression ratio of the image. As a result, the first sensor units can be extracted so as to be dispersed in the same manner as the distribution shown in FIG. 3 or FIG. 8 (that is, the first sensor units are concentrated in the central part). In this way, the image data can be reduced by the random sampling process of the random sampling unit 212.

[0052] The color detection unit 213 recognizes color information (for example, RGB data) for each sample of the image extracted by the random sampling unit 212.

[0053] Also, the first image processing unit 210 can execute encoding processing and various compression processes. For example, the dynamic range or the luminance range may be compressed to a range that does not cause problems in recognition.

[0054] As described above, the sampled image data and the identified color information are sent to the synthesis unit 250 by the first image processing that mimics the first retinal cells (for example, cone cells).

[0055] On the other hand, the second image processing unit 220 also performs a predetermined compression process (second image processing), which mimics the second retinal cells (e.g., rod cells), on the image data divided into a plurality of processing blocks from the block division unit 205. The second image processing unit 220 includes a random sampling unit 222 and a color reduction unit 223.

[0056] The random sampling unit 222 randomly extracts samples from the processing blocks divided by the block division unit 221 based on a specific probability. FIG. 12 is a graph showing an exemplary probability distribution for a plurality of second sensor units in a specific region. For example, based on the probability distribution shown in FIG. 12, the second sensor units can be randomly sampled. As a result, second sensor units that are dispersed in the same manner as the distribution shown in FIG. 4 or FIG. 9 (i.e., the second sensor units are concentrated around the central part) can be extracted. In this way, the image data can be reduced by the random sampling process of the random sampling unit 222.

[0057] The number to be extracted (the second number) can be set to any number greater than the first number. In this way, the image data can be reduced by the compression sampling process of the random sampling unit 222.

[0058] The color reduction unit 223 reduces the color of the image from the image acquisition unit 201 and converts it into a monochrome image or a grayscale image. Thereby, the image data can be reduced.

[0059] Also, the second image processing unit 220 can execute encoding processing and various compression processes. For example, the dynamic range or the luminance range may be compressed to a range that does not cause problems in recognition.

[0060] As described above, the image data that has been sampled and color-reduced by the second image processing mimicking the second retinal cells (e.g., rod cells) is sent to the synthesis unit 250.

[0061] The synthesizing unit 250 synthesizes the image data from the first image processing unit 210 and the image data from the second image processing unit 220. At this time, high-resolution conversion of the image may be performed using deep learning.

[0062] According to the embodiment described above, by executing two different image processes that mimic human visual recognition, the image data can be appropriately reduced, and then, by executing the synthesis process, it can be appropriately restored. Also, by performing random sampling, the distribution can be changed according to the object and purpose.

[0063] Embodiment 3 Embodiment 3 is a modification of Embodiment 2. In FIG. 13, the same components as those in Embodiment 2 are denoted by the same reference numerals as in FIG. 10, and the description thereof will be omitted as appropriate. In Embodiment 3, the first image processing unit 210 has a block dividing unit 211, and the second image processing unit 220 has a block dividing unit 221.

[0064] In this embodiment, the block dividing unit 211 and the block dividing unit 221 may perform different dividing processes. The block dividing unit 211 divides the image into n×m processing blocks. Each processing block may be arranged at equal intervals (for example, see FIGS. 7 to 9), or may be arranged at unequal intervals like retinal cells (see FIGS. 2 to 4). The random sampling unit 212 performs random sampling on the image divided into n×m processing blocks. As described above, based on the probability distribution shown in FIG. 11, the first sensor unit can be randomly sampled.

[0065] The block division unit 221 divides the image into n×m processing blocks. The block division unit 221 may divide the image into a different number of processing blocks than the block division unit 211. Each processing block may be arranged at equal intervals (for example, see FIGS. 7 to 9), or may be arranged at unequal intervals like retinal cells (see FIGS. 2 to 4). The random sampling unit 222 performs random sampling on the image divided into n×m processing blocks. As described above, the second sensor unit can be randomly sampled based on the probability distribution shown in FIG. 12.

[0066] According to the embodiment described above, by executing two different image processes that mimic human visual recognition, the image data can be appropriately reduced, and then, by executing the synthesis process, it can be appropriately restored.

[0067] FIG. 14 is a block diagram showing a hardware configuration example of the image processing apparatuses 100 and 200 (hereinafter referred to as the image processing apparatus 100 etc.). Referring to FIG. 14, the image processing apparatus 100 etc. includes a network interface 1201, a processor 1202, and a memory 1203. The network interface 1201 is used to communicate with other network node apparatuses constituting the communication system. The network interface 1201 may be used to perform wireless communication. For example, the network interface 1201 may be used to perform wireless LAN communication defined in the IEEE 802.11 series, or mobile communication defined in 3GPP (3rd Generation Partnership Project). Alternatively, the network interface 1201 may include, for example, a network interface card (NIC) compliant with the IEEE 802.3 series.

[0068] The processor 1202 reads and executes software (computer program) from the memory 1203, and thus, in the above-described embodiment, those described using a flowchart or sequence Image processingDevice 10 0,200 Perform processes such as. The processor 1202 may be, for example, a microprocessor, an MPU (Micro Processing Unit), or a CPU (Central Processing Unit). The processor 1202 may include a plurality of processors.

[0069] The memory 1203 is composed of a combination of volatile memory and non-volatile memory. The memory 1203 may include storage located away from the processor 1202. In this case, the processor 1202 may access the memory 1203 via an I / O interface (not shown).

[0070] In the example of FIG. 14, the memory 1203 is used to store a group of software modules. The processor 1202 reads and executes these groups of software modules from the memory 1203, thereby performing the Image processing Device 10 0,200 processes such as.

[0071] As described with reference to FIG. 14, each of the processors included in an image processing device 100 or the like executes one or more programs including a group of instructions for causing a computer to perform the algorithms described with reference to the drawings.

[0072] As described above, the composition unit 150 can be realized by a separate computer from the image processing device. Therefore, in this case, the hardware configuration of the composition unit 150 is also as shown in FIG. 14.

[0073] Furthermore, in the various embodiments described above, as described in the procedure of the processing in the image processing apparatus, the present disclosure may also take the form of an image processing method. This image processing method includes a step of acquiring an image, and a step of performing a first image processing on the acquired image, the first image processing including performing a first sampling to extract one or more samples to be processed from the acquired image, and detecting the colors of the one or more extracted samples; and a step of performing a second image processing different from the first image processing on the acquired image, the second image processing including performing a second sampling to extract one or more samples to be processed from the acquired image, and reducing the colors of the one or more extracted samples. For other examples, they are as described in the various embodiments above. Also, the image processing program is a program for causing a computer to execute such an image processing method.

[0074] In the above example, the program can be stored in various types of non-transitory computer readable media and supplied to a computer. The non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media are magnetic recording media (such as flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (such as magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, DVD (Digital Versatile Disc), BD (Blu-ray (registered trademark) Disc), semiconductor memories (such as mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). Also, the program may be supplied to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer readable media can supply the program to the computer via wired communication channels such as electric wires and optical fibers, or wireless communication channels.

[0075] Note that the present invention is not limited to the above embodiments and can be appropriately modified without departing from the gist. For example, in the above-described embodiments, mainly the retinal cells of the human eye were described, but it is also possible to apply them to the retinal cells of other vertebrates. Also, a plurality of the examples described above can be implemented in appropriate combinations.

[0076] Some or all of the above embodiments can be described as follows in the appended claims, but are not limited thereto. (Appended Claim 1) An image acquisition unit that acquires an image, A first image processing unit that performs first image processing on the acquired image, the first image processing unit including: a first sampling unit that performs first sampling to extract one or more samples to be processed from the acquired image; and a color detection unit that detects the colors of the one or more extracted samples. A second image processing unit that performs second image processing different from the first image processing on the acquired image, the second image processing unit including: a second sampling unit that performs second sampling to extract one or more samples to be processed from the acquired image; and a color reduction unit that reduces the colors of the one or more extracted samples. An image processing apparatus comprising the above. (Appendix 2) The image processing apparatus according to Appendix 1, wherein the number of samplings of the first sampling unit is less than the number of samplings of the second sampling unit. (Appendix 3) The first image processing unit performs first image processing that mimics a first retinal cell among the retinal cells of vertebrates. The image processing apparatus according to Appendix 1, wherein the second image processing unit performs second image processing that mimics a second retinal cell among the retinal cells of vertebrates. (Appendix 4) The image processing apparatus according to Appendix 3, wherein the first retinal cell is a cone cell and the second retinal cell is a rod cell. (Appendix 5) The first sampling unit performs first sampling based on a sampling matrix determined based on the distribution of the first retinal cells. The image processing apparatus according to Appendix 3, wherein the second sampling unit performs second sampling based on a sampling matrix determined based on the distribution of the second retinal cells. (Appendix 6) The first sampling unit performs first random sampling according to a probability distribution determined based on the distribution of the first retinal cells. The second sampling unit performs second random sampling according to a probability distribution determined based on the distribution of the second retinal cells, and the image processing apparatus according to Supplementary Note 3. (Supplementary Note 7) In the distribution of the first retinal cells, in the central part, the number of the first retinal cells is more concentrated than the number of the second retinal cells. In the distribution of the second retinal cells, around the central part, the number of the second retinal cells is more concentrated than the number of the first retinal cells, and the image processing apparatus according to Supplementary Note 5 or 6. (Supplementary Note 8) The image processing apparatus according to any one of Supplementary Notes 1 to 7, further comprising a combining unit that combines the image data processed by the first image processing unit and the image data processed by the second image processing unit. (Supplementary Note 9) A step of acquiring an image, A step of performing first image processing on the acquired image, the step including performing first sampling to extract one or more samples to be processed from the acquired image, and detecting the colors of the one or more extracted samples. A step of performing second image processing different from the first image processing on the acquired image, the step including performing second sampling to extract one or more samples to be processed from the acquired image, and reducing the colors of the one or more extracted samples. An image processing method including the above steps. (Supplementary Note 10) The number of samplings in the first sampling is less than the number of samplings in the second sampling, and the image processing method according to Supplementary Note 9. (Supplementary Note 11) The step of performing the first image processing performs first image processing imitating the first retinal cells among the retinal cells of vertebrates. The step of performing the second image processing performs second image processing imitating the second retinal cells among the retinal cells of vertebrates, and the image processing method according to Supplementary Note 9. (Supplementary Note 12) The image processing method according to Supplementary Note 11, wherein the first retinal cell is a cone cell and the second retinal cell is a rod cell. (Supplementary Note 13) The first sampling is performed based on a sampling matrix determined based on the distribution of the first retinal cells, The second sampling is performed based on a sampling matrix determined based on the distribution of the second retinal cells, the image processing method according to Supplementary Note 11. (Supplementary Note 14) The first sampling is performed by first random sampling according to a probability distribution determined based on the distribution of the first retinal cells, The second sampling is performed by second random sampling according to a probability distribution determined based on the distribution of the second retinal cells, the image processing method according to Supplementary Note 11. (Supplementary Note 15) In the distribution of the first retinal cells, in the central part, the number of the first retinal cells is more densely concentrated than the number of the second retinal cells, In the distribution of the second retinal cells, around the central part, the number of the second retinal cells is more densely concentrated than the number of the first retinal cells, the image processing method according to Supplementary Note 13 or 14. (Supplementary Note 16) The image processing method according to any one of Supplementary Notes 9 to 15, further including a step of synthesizing the image data processed by the first image processing and the image data processed by the second image processing. (Supplementary Note 17) A process of acquiring an image, A process of performing first image processing on the acquired image, which includes performing first sampling to extract one or more samples to be processed from the acquired image, and a process of detecting the colors of the extracted one or more samples. A process of performing a second image process different from the first image process on the acquired image, the process including performing a second sampling to extract one or more samples to be processed from the acquired image, and a process of reducing the colors of the extracted one or more samples. An image processing program that causes a computer to execute operations including the above.

[0077] This application claims priority based on Japanese Patent Application No. 2020-170261 filed on October 8, 2020, and incorporates all of its disclosures herein.

Explanation of Reference Numerals

[0078] 11 Cone cell 12 Rod cell 21 First sensor unit 22 Second sensor unit 100 Image processing apparatus 101 Image acquisition unit 110 First image processing unit 112 Sampling unit 113 Color detection unit 120 Second image processing unit 122 Sampling unit 123 Color reduction unit 150 Synthesis unit 200 Image processing apparatus 201 Image acquisition unit 205 Block division unit 210 First image processing unit 211 Block division unit 212 Random sampling unit 213 Color detection unit 220 Second image processing unit 221 Block division unit 222 Random sampling unit 223 Color reduction unit 250 Synthesis unit 300 Eye 302 Pupil 303 Lens 310 Fovea 312 Macula 320 retina 340 optic nerve 345 optic disc

Claims

1. An image acquisition unit that acquires an image; A first image processing unit that performs first image processing on the acquired image, the first image processing unit including a first sampling unit that performs first sampling to extract one or more samples to be processed from the acquired image, and a color detection unit that detects the colors of the one or more extracted samples; A second image processing unit that performs second image processing different from the first image processing on the acquired image, the second image processing unit including a second sampling unit that performs second sampling to extract one or more samples to be processed from the acquired image, and a color reduction unit that reduces the colors of the one or more extracted samples; An image processing apparatus comprising the above.

2. The image processing apparatus according to claim 1, wherein the number of samplings of the first sampling unit is less than the number of samplings of the second sampling unit.

3. The first image processing unit performs first image processing imitating a first retinal cell among the retinal cells of vertebrates; The second image processing unit performs second image processing imitating a second retinal cell among the retinal cells of vertebrates. The image processing apparatus according to claim 1.

4. The image processing apparatus according to claim 3, wherein the first retinal cell is a cone cell and the second retinal cell is a rod cell.

5. The first sampling unit performs first sampling based on a sampling matrix determined based on the distribution of the first retinal cells; The second sampling unit performs second sampling based on a sampling matrix determined based on the distribution of the second retinal cells. The image processing apparatus according to claim 3.

6. The first sampling unit performs first random sampling according to a probability distribution determined based on the distribution of the first retinal cells; The second sampling unit performs second random sampling according to a probability distribution determined based on the distribution of the second retinal cells. The image processing apparatus according to claim 3.

7. In the distribution of the first retinal cells, in the central portion, the number of the first retinal cells is more concentrated than the number of the second retinal cells. In the distribution of the second retinal cells, in the vicinity of the central portion, the number of the second retinal cells is denser than the number of the first retinal cells. The image processing apparatus according to claim 5 or 6.

8. The image processing apparatus according to any one of claims 1 to 7, further comprising a combining unit that combines the image data processed by the first image processing unit and the image data processed by the second image processing unit.

9. A step of acquiring an image, A step of performing first image processing on the acquired image, the step including performing first sampling to extract one or more samples to be processed from the acquired image, and detecting the colors of the one or more extracted samples. A step of performing second image processing different from the first image processing on the acquired image, the step including performing second sampling to extract one or more samples to be processed from the acquired image, and reducing the colors of the one or more extracted samples. An image processing method including the above steps.

10. In the image processing method according to claim 9, the number of samplings in the first sampling is less than the number of samplings in the second sampling.

11. The step of performing the first image processing performs first image processing imitating the first retinal cells among the retinal cells of vertebrates. The step of performing the second image processing performs second image processing imitating the second retinal cells among the retinal cells of vertebrates. The image processing method according to claim 9.

12. In the image processing method according to claim 11, the first retinal cells are cone cells, and the second retinal cells are rod cells.

13. The first sampling is performed based on a sampling matrix determined based on the distribution of the first retinal cells. The second sampling is performed based on a sampling matrix determined based on the distribution of the second retinal cells. The image processing method according to claim 11.

14. The first sampling performs first random sampling according to a probability distribution determined based on the distribution of the first retinal cells. The second sampling performs second random sampling according to a probability distribution determined based on the distribution of the second retinal cells. The image processing method according to claim 11.

15. In the distribution of the first retinal cells, in the central part, the number of the first retinal cells is more densely concentrated than the number of the second retinal cells. The image processing method according to claim 13 or 14, wherein in the distribution of the second retinal cells, around the central part, the number of the second retinal cells is more densely concentrated than the number of the first retinal cells.

16. The image processing method according to any one of claims 9 to 15, further comprising a step of synthesizing the image data processed by the first image processing and the image data processed by the second image processing.

17. A process of acquiring an image, A process of performing first image processing on the acquired image, the process comprising performing first sampling to extract one or more samples to be processed from the acquired image, and detecting the colors of the one or more extracted samples. A process of performing second image processing different from the first image processing on the acquired image, the process comprising performing second sampling to extract one or more samples to be processed from the acquired image, and reducing the colors of the one or more extracted samples. An image processing program for causing a computer to execute operations including these.

Images