Image processing device, image processing method and program

Abstract

An imaging processing device (100) has a determination unit that determines, on the basis of detected color information of an object to be written on, color information of an output image generated from a written content image, which is an image of written content written on the object to be written on.

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In recent years, a technology has been developed that extracts information written on a blackboard or whiteboard and outputs information on the extracted information. For example, Patent Document 1 discloses a technology that captures an image of a blackboard or whiteboard, extracts an image corresponding to the information written on the board from the captured image, and outputs the extracted image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-30369 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the technology described in Patent Document 1, the extracted image is output as monochrome image data as is, so it is expected that the visibility of the image may be poor. Even if the extracted image is output as a color image as is, the visibility of the image may be poor because the appearance may differ when the blackboard writing is viewed directly in physical space and when viewed on a display.

[0005] Generally, blackboard writing is done in a form suitable for the object being written on. Therefore, when an image of the blackboard writing is extracted and combined with a background image different from the object being written on and output as an output image, it is expected that the visibility of the blackboard writing image in the output image may be poor.

[0006] Therefore, an object of the present disclosure is to provide an image processing device, an image processing method, and a program that make it possible to improve the visibility of a handwriting content image, which is an image of the handwriting content written on a writing target. [Means for solving the problem]

[0007] In order to solve the above problems, an image processing device of one embodiment according to the present disclosure has a determination unit that determines color information of an output image generated from a writing content image, which is an image of the writing content written on the writing object, based on color information of the detected writing object. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram for explaining an overview of an image processing device 100 and an input device 200 according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram for explaining an example of a functional configuration of the system 1 according to the present embodiment. [Diagram 3] 11 is a diagram for explaining an example of an image acquired by an acquisition unit 11 according to the embodiment. FIG. [Figure 4] 11 is a diagram for explaining an example of an extraction process performed by an extraction unit 12 according to the embodiment. FIG. [Diagram 5] 11 is a diagram for explaining an example of a process for extracting a writing target area by the extraction unit 12 according to the embodiment. FIG. [Figure 6] 11 is a diagram for explaining an example of a result of an extraction process of a writing target area by the extraction unit 12 according to the embodiment. FIG. [Figure 7] 11 is a diagram for explaining an example of a correction method by a correction unit 13 according to the embodiment. FIG. [Figure 8] 11 is a diagram for explaining an example of a correction method by a correction unit 13 according to the embodiment. FIG. [Figure 9] 11 is a diagram for explaining an example of a correction method by a correction unit 13 according to the embodiment. FIG. [Figure 10] 11 is a diagram for explaining an example of correction by the correction unit 13 according to the embodiment using a two-character curve for hues corresponding to hue angles of 50 degrees to 70 degrees. FIG. [Figure 11]11A to 11D are diagrams for explaining a specific example of execution of correction processing using a filter by the correction unit 13 according to the embodiment. [Figure 12] 11 is a diagram for explaining an example of a correction process for the contour of the handwritten content image 22 by the correction unit 13 according to the embodiment. FIG. [Figure 13] FIG. 13 is a diagram showing an example of blurring of handwriting 4. [Figure 14] FIG. 13 is a diagram illustrating an example of a method for suppressing color emphasis. [Figure 15] FIG. 13 is a diagram illustrating an example of a function. [Figure 16] FIG. 13 is a diagram illustrating an example of a function. [Figure 17] FIG. 13 is a diagram illustrating an example of a function. [Figure 18] 1 is a diagram for explaining an example of an output by the output device 300 according to the present embodiment. FIG. [Figure 19] 1 is a diagram for explaining an example of the flow of operations of the system 1 according to the present embodiment. [Figure 20] FIG. 13 is a diagram showing an example of area division in application example 2. [Figure 21] FIG. 13 is a diagram showing an example of a local region in Application Example 3. [Figure 22] FIG. 11 is a diagram for explaining an example of the flow of operations of the system 1 according to the application example 3. [Figure 23] FIG. 11 is a diagram for explaining an example of a functional configuration of a system 1 according to an application example 4. [Figure 24] FIG. 13 is a diagram showing an example of degeneration of handwriting pixels according to Application Example 5. [Diagram 25] 11 is a diagram for explaining an example of a correction process performed by the correction unit 13 according to the present embodiment based on behavior detection information indicating whether or not the writer is writing on the writing target image 21. FIG. [Figure 26] 11 is a diagram for explaining an example of a correction process performed by the correction unit 13 according to the present embodiment based on behavior detection information indicating whether or not the writer is writing on the writing target image 21. FIG. [Figure 27]13 is a diagram for explaining a correction process of a writing content image based on a positional relationship between a writer 3 and a writing target 2 by a correction unit 15 according to the embodiment. FIG. [Figure 28] 1 is a block diagram showing an example of a hardware configuration of an information processing device having a system 1 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0010] The present disclosure will be described in the following order. 1. Embodiment 1-1.Background 1-2. Overall configuration example 1-3. Example of functional configuration 2. Example of operation 3. Application Examples 3-1. Application example 1 3-2. Application example 2 3-3. Application example 3 3-4. Application example 4 3-5. Application example 5 3-6. Application example 6 3-7. Application Example 7 4. Hardware configuration example 5. Summary

[0011] <1. Embodiment> <<1-1.Background>> First, the background of the embodiments of the present disclosure will be described.

[0012] In recent years, a technology has been developed that extracts writing (hereinafter referred to as written content 4) on a writing object 2 such as a blackboard or whiteboard (hereinafter referred to as writing object 2) from an image captured of the writing object 2, and outputs the extracted writing content 4 as an image. In a lecture, meeting, or other such situation, an image of the writing content 4 (hereinafter referred to as a writing content image 22) is output to a display or the like, so that participants who are, for example, at a remote location, can easily check the writing content.

[0013] However, when the captured handwritten content 4 is output as the handwritten content image 22 as it is, it is assumed that the visibility of the handwritten content image 22 is poor. The reason for this is that the appearance may differ when the handwritten content is viewed directly in the physical space and when the handwritten content image 22 is viewed on a display, for example.

[0014] Furthermore, the colors frequently used when writing may generally differ for each type of writing object 2. Therefore, it is expected that the visibility of the writing content image 22 included in the output image may be poor depending on the combination of the color used for writing and the background color of the output image.

[0015] The technical concept according to an embodiment of the present disclosure has an aspect conceived by focusing on the above points, and enables correction of the form of the handwritten content image so as to improve the visibility of the handwritten content image. Hereinafter, configuration examples and operation examples according to an embodiment of the present disclosure will be described in detail.

[0016] <<1-2. Overall configuration example>> Next, an overview of an image processing device 100 and an input device 200 according to an embodiment of the present disclosure will be described with reference to FIG.

[0017] Fig. 1 is a diagram for explaining an overview of an image processing device 100 and an input device 200 according to an embodiment of the present disclosure. Fig. 1 illustrates the image processing device 100 and the input device 200 connected to the image processing device 100.

[0018] The writing object 2 is an object on which visual information (writing content 4) such as dots, lines, characters, sentences, mathematical expressions, symbols, pictures, figures, or images is written. The writing object 2 is a blackboard, a whiteboard, electronic paper, a touch panel, or the like.

[0019] The writer 3 performs an action on the writing target 2. For example, the writer 3 writes writing content 4 on the writing target 2.

[0020] The writing content 4 is visual information written on the writing object 2. As described above, the writing content 4 is written on the writing object 2 with chalk, a marker, a stylus, a finger, or the like. The writing content 4 may be in various colors. For example, if the writing object 2 is a blackboard, the writing content 4 may be white, red, yellow, or the like.

[0021] The input device 200 is a device that inputs information about the physical space in which the input device 200 is installed. The input device 200 includes, for example, an imaging device and a voice input device. The imaging device includes a lens system composed of an imaging lens, an aperture, a zoom lens, a focus lens, etc., a drive system that performs focusing and zooming operations on the lens system, and a solid-state image sensor array that photoelectrically converts imaging light obtained by the lens system to generate an imaging signal. The voice input device includes a microphone that collects surrounding sounds, a microphone amplifier circuit that amplifies and processes the audio signal obtained by the microphone, an A / D converter, and a signal processing circuit such as a noise canceller. The input device 200 outputs image data converted into digital signals and audio data at the time of imaging.

[0022] The input device 200 can capture an image of an object in a physical space as the capture target. The input device 200 may capture an image of a writing target 2 on which handwritten content 4 is written in a physical space as the capture target, and output the image obtained by capturing the image (hereinafter referred to as a captured image 20) in association with the capture time to the image processing device 100. The captured image 20 may include other than the writing target 2 and the handwritten content 4. In this case, the input device 200 outputs the captured image in which other than the writing target 2 and the handwritten content 4 are also captured to the image processing device 100.

[0023] The function of the input device 200 may be included in the writing target 2. For example, the input device 200 and the writing target 2 may be realized as an electronic whiteboard. The input device 200 as an electronic whiteboard may acquire an image corresponding to the captured image 20 described above by scanning the state of the writing target 2. In this case, the input device 200 acquires an image of the writing target 2 on which the writing content 4 is written, and provides it to the image processing device 100. After being provided to the image processing device 100, the image may be treated in the same way as the captured image 20. Note that the image acquired by the input device 200 and the writing target 2 realized as an electronic whiteboard may include only the writing target 2 and the writing content 4.

[0024] The image processing device 100 is a device that extracts a handwritten content image 22 from a captured image 20 input by an input device 200, and corrects the form of the extracted handwritten content image 22. The image processing device 100 outputs an image including the corrected handwritten content image 22 (hereinafter referred to as an output image 25) to an output device 300 (not shown in FIG. 1) described later.

[0025] Here, the form of the handwritten content image 22 refers to, for example, the color, width, contour, etc. of the handwritten content image 22. Details of correction of the color, width, and contour of the handwritten content image 22 by the image processing device 100 will be described later. Note that the image processing device 100 may be connected to the input device 200 by wire or wirelessly.

[0026] <<1-3. Example of functional configuration>> An example of the functional configuration of the system 1 according to this embodiment will be described below. Fig. 2 is a diagram for explaining an example of the functional configuration of the system 1 according to this embodiment. As shown in Fig. 2, the system 1 includes an image processing device 100, an input device 200, and an output device 300.

[0027] [1-3-1. Input device 200] The input device 200 inputs a captured image 20 and outputs the captured image 20 to the image processing device 100 .

[0028] [1-3-2. Image processing device 100] The image processing device 100 is a device that controls the overall operation of the system 1. The image processing device 100 is realized by any device such as a PC (Personal Computer), a smartphone, or a tablet terminal.

[0029] The image processing device 100 extracts a handwritten content image 22 from a captured image input by the input device 200, and the image processing device 100 corrects the shape of the extracted handwritten content image 22 and generates an output image 25 that includes the corrected handwritten content image 22 and has a background of a predetermined color.

[0030] As shown in FIG. 2, the image processing device 100 includes an acquisition unit 11, an extraction unit 12, a correction unit 13, an output unit 14, a detection unit 15, and a determination unit 16.

[0031] (1-3-2-1. Acquisition part 11) The acquisition unit 11 has a function of acquiring a captured image 20 from an input device 200. The captured image 20 may include other objects besides the writing target 2 on which the writing content 4 has been written.

[0032] Here, an example of the captured image 20 acquired by the acquisition unit 11 according to the present embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining an example of the captured image 20 acquired by the acquisition unit 11 according to the present embodiment. Fig. 3 shows a captured image 20a. The captured image 20a includes a writing target image 21a, a writing content image 22a, and a writer image 23a.

[0033] In the example of FIG. 3, the captured image 20a includes not only a writing target image 21a and a writing content image 22a, but also a writer image 23a and other areas.

[0034] As described above, the writing target image 21 refers to an area (image) in the captured image 20 in which the writing target 2 is captured. Also, the writing content image 22 refers to an area (image) in the captured image 20 in which the writing content 4 is captured.

[0035] In addition, the captured image 20 may have a size of the handwriting content image 22 equal to or larger than a predetermined ratio. An image on which white balance correction has been performed may be used as the captured image 20. When the input device 200 is an electronic whiteboard, it may be easy for the acquisition unit 11 to acquire an image including only the writing target image 21 and the handwriting content image 22. On the other hand, even when the input device 200 is an electronic whiteboard, the acquisition unit 11 may acquire the captured image 20 from the input device 200, in which the imaging device has captured the electronic whiteboard.

[0036] (1-3-2-2. Extraction part 12) The extraction unit 12 extracts the handwriting image 22 from the captured image 20. Specifically, the extraction unit 12 extracts the handwriting image 22 independent of the handwriting target image 21 and the like. That is, the extraction unit 12 generates image data including only the handwriting image 22. Examples of methods for extracting the handwriting image 22 include binarization processing and edge detection.

[0037] FIG. 4 is a diagram for explaining an example of the extraction process by the extraction unit 12 according to the present embodiment. FIG. 4 shows a captured image 20b after the extraction process. The captured image 20b after the extraction process includes a written content image 22b. The writing target image 21 has been removed to leave a blank space 21b. As shown in FIG. 4, the written content image 22b can be extracted. Here, in the example shown in FIG. 4, the written content image 22b is white.

[0038] However, there may be a situation where it is difficult to extract only the handwriting content image 22 by the binarization process by the extraction unit 12. For example, as shown in Fig. 3, the captured image 20 may contain a writer image 23 and other areas in addition to the writing target image 21 and the handwriting content image 22, and if a binarization process is performed, images other than the handwriting content image 22 will also be extracted. In order to extract only the handwriting content image 22, it is first necessary to remove the writer image 23 and other areas from the captured image 20.

[0039] The extraction unit 12 performs an extraction process of the writing target area in order to remove other areas. Also, the extraction unit 12 performs a separation process of the writer image 23 in order to remove the writer image 23. Details of the extraction process of the writing target area and the separation process of the writer image 23 will be described below.

[0040] (1-3-2-2-1. Extraction process of writing area) First, the process of extracting the target area to be written on will be described. Specifically, the process of extracting the target area to be written on refers to a process of extracting an area specified by specifying a plurality of points, for example, four points, in the captured image 20 as the target area to be written on. The target area to be written on here refers to an image obtained by removing other areas from the captured image 20.

[0041] Hereinafter, a description will be given with reference to Fig. 5. Fig. 5 is a diagram for explaining an example of the extraction process of the writing target area by the extraction unit 12 according to the present embodiment. Fig. 5 shows a captured image 20c and a captured image 20d after the writing target area has been extracted.

[0042] Here, the captured image 20c includes the writing target image 21c and the writing content image 22c, as well as the writer image 23c and other regions. The extraction unit 12 generates a captured image 20d after the writing target region is extracted based on the writing target image 21c and a plurality of points designated to surround the region of the writing target 2. The captured image 20d after the writing target region is extracted includes the writing target image 21c, the writing content image 22c, and a part of the writer 23d.

[0043] (1-3-2-2-2. Separation process of the scribe image 23 and the handwriting content image 22) Next, a process of separating the writer image 23 and the handwriting content image 22 will be described.

[0044] The captured image 20d after extraction of the writing target area, which is extracted by the extraction process of the captured image 20d after extraction of the writing target area described above, includes a part 23d of the writer image. In order to extract the writing content image 22 from the captured image 20d after extraction of the writing target area, the extraction unit 12 needs to remove the part 23d of the writer image from the captured image 20d after extraction of the writing target area.

[0045] Specifically, the process of separating the writer image 23 from the writing content image 22 refers to, for example, pattern-recognizing the shape of a portion 23 of the writer image from the captured image 20 after the writing target area has been extracted, and excluding the recognized writer image 23.

[0046] The extraction unit 12 performs a separation process between the writer image 23 and the writing content image 22, and performs a binarization process on the executed image, thereby generating a captured image 20e after the extraction process, which includes the writing target image 21e and the writing content image 22c, as shown in Figure 6.

[0047] In this way, by executing the process of extracting the region to be written and the process of separating the writer image 23, it becomes possible to correctly execute the process of extracting the handwritten content image 22. Note that, although the above describes an example in which the process of extracting the region to be written is executed first, it may also be possible to execute the process of separating the writer image 23 first.

[0048] (1-3-2-3. Correction unit 13) The correction unit 13 corrects the form of the handwritten content image 22 extracted by the extraction unit 12. Specifically, the correction unit 13 corrects the form of the handwritten content image 22 extracted by the extraction unit 12 so that the visibility of the handwritten content image 22 in the output image 25 is improved.

[0049] Note that the correction of the color of the handwritten content image 22 by the correction unit 13 refers to correcting three color attributes of the handwritten content image 22. For example, examples of the three color attributes include hue, saturation, and brightness. The correction unit 13 corrects at least one of the hue, saturation, and brightness of the color of the handwritten content image 22. Note that the correction unit 13 may perform correction of either or both of the saturation and brightness when the purpose is to improve visibility.

[0050] Of course, even if a scale other than the three attributes of color is used, the color correction process by the correction unit 13 may be executed. For example, when correcting the color of YUV data, the correction unit 13 may convert the YUV data into HSV data and execute correction of either or both of saturation and lightness.

[0051] Hereinafter, as an example only, a case where the correction unit 13 corrects either or both of the saturation and the brightness will be described.

[0052] The correction unit 13 determines a color correction processing method based on a combination of the type of the writing target image 21 and the background color of the output image 25, and performs color correction processing of the writing content image 22 using the determined correction processing method.

[0053] Here, the background of the output image 25 refers to the background of the image output by the output device 300. As an example, the writing target image 21 can be used as the background. In this way, the color of the background can be set to a color of the same family as the color corresponding to the writing target 2, or a color of a different family can be set. For example, when the writing target 2 is a blackboard, the color of the background can be set to any color other than the dark green or black corresponding to the writing target 2, such as white. In addition, the color of the background can be fixed for each frame of the output image 25, or can be set to a different color for each output image 25.

[0054] When it is assumed that the background color of the output image 25 is a fixed color, the correction unit 13 may determine the color correction processing method based only on the type of the writing target image 21.

[0055] Next, the determination of the color correction processing method will be described. The determination of the color correction processing method here means, for example, the determination of a filter that corrects the saturation or brightness of the handwriting content image 22. Here, the filter means a relationship in which, when saturation or brightness is input, saturation or brightness corresponding to the input saturation or brightness is output.

[0056] The filter for correcting the saturation and the filter for correcting the brightness may be determined independently. The correction unit 13 uses the filter to correct the saturation and brightness corresponding to each color of the handwriting content image 22 extracted by the extraction unit 12.

[0057] The correction unit 13 may determine filters for saturation and brightness corresponding to a combination of the type of the writing target image 21 and the background color of the output image 25. Specifically, the correction unit 13 may correct the brightness of the color of the writing content image 22 based on the difference between the brightness of the color of the writing target 2 and the brightness of the background color. More specifically, the correction unit 13 may correct the brightness of the color of the writing content image 22 so as to invert the brightness when the difference between the brightness of the color of the writing target 2 and the brightness of the background color is equal to or greater than a predetermined value. That is, the correction unit 13 may correct the brightness of the color of the writing content image 22 so that the relationship of brightness between the multiple colors of the writing content image 22 is inverted from the relationship of brightness between the multiple colors before correction.

[0058] For example, when the writing target image 21 is a blackboard and the background color of the output image 25 is white, the correction unit 13 may determine a filter that corrects the white writing content image 22 to black. This is because when the white writing content image 22 is composited onto a white background, the visibility of the writing content image 22 decreases. Another reason is that when the writing content image 22 is composited onto a white background, the visibility increases if the color of the writing content image 22 is black.

[0059] In this case, the background color can be identified as follows. As an example only, when the color of the image to be written on is set as the background color, the color of the image to be written on can be identified as the background color by image recognition from the image to be written on 21 for each frame of the captured image 20. The color of the image to be written on may be known by user input. When the color of the image to be written on or another color is specified by user settings as the background color, the specified color can be identified as the background color.

[0060] Furthermore, when the handwritten content image 22 has a plurality of colors, the correction unit 13 may determine a filter that increases the difference in saturation and brightness between the plurality of colors in the handwritten content image 22 compared to the difference in brightness between the plurality of colors in the pre-correction handwritten content image 22. Specifically, the correction unit 13 may determine a filter that increases the saturation and brightness of the colors in the handwritten content image 22 that are higher than the saturation and brightness of other colors and decreases the saturation and brightness of the colors that are lower than the saturation and brightness of other colors.

[0061] Specific examples of the above-mentioned filters will be described below with reference to Fig. 7 to Fig. 9. Fig. 7 to Fig. 9 are diagrams for explaining an example of a correction method by the correction unit 13 according to the present embodiment. The specific examples of the filters described below may be usable for correcting both saturation and brightness.

[0062] FIG. 7 shows a graph G1 indicating the relationship between input and output. The "input" shown here is the saturation or brightness of each color of the handwriting content image 22 extracted by the extraction unit 12. The "output" shown here is the corrected saturation or brightness corresponding to each "input". In the graph G1 shown in FIG. 7, when the saturation or brightness is equal to or higher than a predetermined value, the saturation or brightness of the "output" is higher than that of the "input". On the other hand, as shown in FIG. 7, when the saturation or brightness is equal to or lower than a predetermined value, the saturation or brightness of the "output" is lower than that of the "input".

[0063] The curve of the graph G1 is expressed by, for example, the formula (1).

[0064] OUTPUT norm =1 / {1+e (s-INPUT)*γ}···(1)

[0065] In equation (1), s is the amount of shift along the horizontal axis, γ is a coefficient, and INPUT and OUTPUT are the input and output, respectively.

[0066] 8 also shows a graph G2 showing a relationship between input and output, which is different from that shown in FIG. 7. Unlike graph G1, graph G2 has a wide range of saturation or brightness where "output" is higher than "input." For example, when saturation is corrected by a filter as shown in graph G2, the saturation of colors other than achromatic colors or colors close to achromatic colors becomes higher in the handwriting content image 22 extracted by the extraction unit 12.

[0067] The filters may also include filters that invert saturation and brightness. In this case, inversion of saturation and brightness refers to correcting a color to have a lower saturation or brightness that is higher than other colors, and correcting a color to have a higher saturation or brightness that is lower than other colors.

[0068] 9 shows a graph G3 showing the relationship between input and output. The graph G3 has an output result opposite to that of the graph G1. Specifically, when the saturation or brightness is equal to or higher than a certain level, the saturation or brightness of the "output" is lower than that of the "input", while when the saturation or brightness is equal to or lower than a certain level, the saturation or brightness of the "output" is higher than that of the "input".

[0069] The curve of the graph G3 is expressed by, for example, the formula (2).

[0070] OUTPUT inv =1 / {1+e (INPUT-s)*γ}···(2)

[0071] In equation (2), like equation (1), s is the amount of shift along the horizontal axis, γ is a coefficient, and INPUT and OUTPUT are the input and output, respectively.

[0072] It should be noted that formulas (1) and (2) are merely examples, and filters using other formulas may be used.

[0073] The filter shown in the graph G3 is used, for example, to correct the handwriting content image 22, which is an image of the handwriting content 4 written on a blackboard, and generate an output image 25 with a white background. Usually, when writing on a blackboard, the writer 3 often writes the white handwriting content 4 using white chalk. When the output image 25 is generated from the handwriting content image 22 corresponding to the white handwriting content 4 and the white background, it may be difficult to distinguish between the handwriting content image 22 and the white background. Therefore, for example, when the type of the writing target 2 is a blackboard and the background color is white, the correction unit 13 may use the filter shown in the graph G3 to correct the color of the handwriting content image 22 so as to invert the brightness of the white handwriting content image 22.

[0074] The correction unit 13 may correct the saturation and brightness of the color of the handwritten content image 22 using the filters shown in FIGS.

[0075] The correction unit 13 may use a filter other than the filters that increase the saturation difference and the brightness difference shown in Fig. 7 to Fig. 9. In other words, the correction unit 13 may use a filter that does not increase the saturation difference and the brightness difference. For example, the correction unit 13 may use a filter that outputs the input saturation and brightness without changing them, or a filter that inverts the input saturation and brightness and outputs them.

[0076] Here, consider the case where the writing target image 21 is a blackboard, the writing content image 22 is yellow, and the output image 25 is generated with a white background. The brightness of the yellow writing content image 22 written on the blackboard is higher than the other colors of the writing content image 22 written on the blackboard, and may be used to emphasize the yellow writing content image 22 more than the writing content images 22 of other colors. However, when the writing target image 21 is a blackboard and the output image 25 is generated with a white background, a filter that inverts the brightness is used, and the yellow writing content image 22 with high brightness becomes lower in brightness after correction. Therefore, a situation may occur in which it is difficult to distinguish between the corrected yellow writing content image 22 and the corrected white writing content image 22.

[0077] If it is anticipated that such a situation may occur, the correction unit 13 may perform further correction, for example, after the correction using a filter, so as to increase the brightness of the yellow handwritten content image 22.

[0078] A situation similar to the above situation may occur in the handwritten content image 22 of a hue other than the yellow hue. Therefore, the correction unit 13 may correct a color of a predetermined hue according to a combination of the type of the handwritten content image 21 and the background information. Specifically, the correction unit 13 may correct the lightness of a hue of the handwritten content image 22 whose lightness difference with the background color of the output image 25 is equal to or less than a predetermined value, among the colors of the handwritten content image 22, so that the lightness of the hue changes more than the other colors of the hue. Specifically, the correction unit 13 may correct the lightness of the handwritten content image 22 so that the lightness difference becomes 20% or more when a lightness difference of 10% or less of the minimum lightness and maximum lightness occurs in the relationship between the lightness of the color of the handwritten content image 22 after correction and the lightness of the background color. For example, when the minimum lightness is 0 and the maximum lightness is 255, the correction unit 13 may correct the lightness of the handwritten content image 22 so that the lightness difference becomes 51 or more when the lightness difference is 25 or less.

[0079] In the correction according to the brightness difference, the correction unit 13 may change the output of the brightness corresponding to a predetermined hue when performing the correction using a filter according to a combination of the type of the writing target image 21 and the background information. Hereinafter, an example will be described with reference to FIG. 10. FIG. 10 is a diagram for explaining an example of correction by the correction unit 13 according to this embodiment using a two-character curve in a hue corresponding to a hue angle of 50 degrees to 70 degrees. FIG. 10 shows a graph G4. In the graph G4, the value of the coefficient corresponding to each hue is shown. Here, the coefficient is the coefficient γ in Equation (1) and Equation (2). In FIG. 10, an example of correction in a hue corresponding to a hue angle of 50 degrees to 70 degrees has been described, but of course, a similar correction may be performed in a hue in another range.

[0080] In the above, the correction of the brightness according to the brightness difference with the background color of the output image 25 has been described. However, a similar correction of the brightness may be performed in consideration of the influence of lighting on the handwritten content 4. For example, when the handwritten content 4 is illuminated by a lighting device, the handwritten content image 22 corresponding to the handwritten content 4 may appear to have a color of a predetermined hue different from the original color of the handwritten content 4 due to the influence of the lighting. When the correction unit 13 corrects the color of the handwritten content image 22 having a color different from the original color, a situation may occur in which the differentiation due to the difference in color is not achieved as originally intended by the writer 3. Therefore, when the occurrence of such a situation is expected, the correction unit 13 may correct the brightness of the color of the predetermined hue and emphasize the difference between the handwritten content image 22 of the corrected hue color and the handwritten content image 22 of another color.

[0081] In this way, the correction unit 13 can output the handwriting content image 22 appropriately according to the type of the handwriting target image 21. This function makes it possible to improve the visibility of the handwriting content image 22 in the output image 25 and to cancel the influence of lighting on the handwriting target image 21.

[0082] Next, a specific example of the execution of the correction process using a filter by the correction unit 13 described above will be described with reference to Fig. 11. Fig. 11 is a diagram for explaining a specific example of the execution of the correction process using a filter by the correction unit 13.

[0083] The upper part of Fig. 11 shows a captured image 20f after extraction processing by the extraction unit 12. The captured image 20f includes a writing target image 21f, and a white character 22W, a line 22R of color R, a line 22B of color B, and a line 22Y of color Y as a writing content image 22f. Here, the type of the writing target image 21 is a blackboard. The correction unit 13 corrects the captured image 20f to generate an output image 25g whose background color is white. For example, the color R is red, the color B is blue, and the color Y is yellow.

[0084] 11 shows an output image 25g corrected by the correction unit 13. The output image 25g includes a white background image 24g, corrected black characters 22Wg, a corrected line 22Rg of color R, a corrected line 22Bg of color B, and a corrected line 22Yg of color Y. Here, the line 22Rg of color R, the line 22Bg of color B, and the line 22Yg of color Y are obtained by correcting the lightness of the line 22R of color R, the line 22B of color B, and the line 22Y of color Y, respectively.

[0085] In this way, the handwritten content image 22 can be corrected to enhance visibility according to the background color of the output image 25.

[0086] The filter determination and correction target are not limited to the above example. For example, in the above, a filter for converting the color of the writing content image 22 is determined based on the type of the writing target 2 and the background color of the output image 25, but a filter may also be determined for each partial portion of the captured image 20.

[0087] Further, the correction unit 13 may correct the contour of the handwritten image 22 in addition to correcting the color of the handwritten image 22. Correcting the contour of the handwritten image 22 refers to, for example, a process of emphasizing the contour of the handwritten image 22 and erasing the portion of the handwritten image 22 other than the contour. Hereinafter, an example of the correction process of the contour of the handwritten image 22 by the correction unit 13 will be described with reference to Fig. 12. Fig. 12 is a diagram for explaining an example of the correction process of the contour of the handwritten image 22 by the correction unit 13 according to this embodiment.

[0088] The upper part of Fig. 12 shows a captured image 20h after extraction processing by the extraction unit 13. The captured image 20h includes a writing target image 21h and a writing image 22h. Here, the correction unit 13 may correct the contour of the writing image 22h. The lower part of Fig. 12 shows an output image 25i including the writing image 22i corrected by the correction unit 13. The output image 25i includes a base image 24i. The writing image 22i is the writing image 22h with the color and contour corrected.

[0089] The correction unit 13 may correct the contours of a part of the handwritten content image 22 and may not correct the contours of the other parts of the handwritten content image 22. For example, after performing a saturation or brightness correction process using a filter, the correction unit 13 may correct the contours of the handwritten content image 22 of a predetermined hue among the handwritten content images 22 after the correction process, and may not correct the contours of the handwritten content images 22 of a color other than the predetermined hue. In addition, for example, the correction unit 13 may correct the contours of the handwritten content images 22 that have been recognized as characters by character recognition or the like among the handwritten content images 22. The target for which the correction unit 13 performs contour correction is not limited to the above example.

[0090] In this way, the correction unit 13 can perform correction so as to emphasize the contour of the handwritten content image 22. This function makes it possible to express that the handwritten content image 22 with the contour corrected has a different meaning from other handwritten content images 22.

[0091] (1-3-2-4. Output section 14) The output unit 14 controls to output an output image 25 including the handwritten content image 22 corrected by the correction unit 13. Specifically, the output unit 14 generates the output image 25 by superimposing the handwritten content image 22, the form of which has been corrected by the correction unit 13, on a background image. As an example of such a background image, the output unit 14 can generate the output image 25 by superimposing the handwritten content image 22, the form of which has been corrected by the correction unit 13, on the writing target image 21 of the captured image 20. Note that the background image may be an image of any color designated by user settings or the like. The output image 25 thus generated is output to the output device 300, which will be described later. The output unit 14 may output the output image 25 to the output device 300, which will be described later, in real time. The output unit 14 may also accumulate the output image 25 in a buffer or storage (not shown) and cause the accumulated output image 25 to be output to the output device 300 at any timing.

[0092] (1-3-2-5. One aspect of the color correction challenge) As one aspect, when the color of the handwritten content image 22 is corrected by the correction unit 13, the visibility of the handwritten content image 22 in the output image 25 is improved.

[0093] While there is such an aspect, the color correction by the correction unit 13 emphasizes the saturation and brightness of the handwriting image 22. Therefore, the color correction has an aspect of emphasizing not only the color of the handwriting 4 included in the handwriting image 22 but also the color of the writing object 2. For example, when the writing object 2 is a blackboard, chalk smudges and the like occur in the handwriting written with chalk or the like. Therefore, in addition to the color of the chalk, the color of the blackboard surface is reflected in the handwriting image 22 in the places corresponding to the chalk smudges.

[0094] FIG. 13 is a diagram showing an example of blurred writing content 4. FIG. 13 shows, as an example, an example of writing with white chalk on a blackboard, which is an example of writing target 2. Furthermore, FIG. 13 shows an excerpt of the letter "A" from the character string "ABCD" as an example of writing content 4 written with white chalk. Furthermore, FIG. 13 shows an excerpt of the lower left end of the letter "A" and its surroundings 22A, which are further excerpted and enlarged, and the enlarged view is shown as reference numeral 22A1. Note that in FIG. 13, the color of the blackboard is shown by hatching, and the color of the chalk is shown in white.

[0095] As shown by reference numeral 22A1 in FIG. 13, a part of the handwritten content 4 may include chalk smudges c1 to c6 in addition to the chalk written at the lower left end of the letter "A". In the handwritten content image 22 including the chalk smudges c1 to c6, not only the saturation and brightness of the chalk but also the chalk smudges c1 to c6 are color-enhanced. As a result, as shown by reference numeral 22A2 in FIG. 13, the saturation and brightness of the green color of the chalk smudges c1 to c6, that is, the color of the green color corresponding to the board surface of the blackboard, are emphasized as the chalk smudges C1 to C6. The chalk smudges C1 to C6, the saturation and brightness of which are emphasized in this way, are one of the factors that impede the visibility of the handwritten content 4. For example, the chalk smudges C1 to C6 located inside the letter "A" become clear, and as a result, the outline of the letter "A" appears unclear, and the letter "A" appears to be colored in the blackboard, for example, greenish, blurred, or dull.

[0096] 13 shows an example in which the writing object 2 is a blackboard, but even when the writing object 2 is a whiteboard, smudges and unevenness of the marker ink may occur, which may similarly impair the visibility of the written content 4. Furthermore, when the writing object 2 is a whiteboard, the sense of incongruity may be further aggravated by an improper setting of the white balance of the imaging device used for shooting. For example, when a whiteboard is shot in a yellowed state, the written content 4 may also appear yellowed.

[0097] (1-3-2-6. One aspect of the problem-solving approach) Therefore, in an embodiment of the present disclosure, color information of the output image 25 generated from the handwriting image 22 is determined based on color information of the handwriting object 2 detected from the captured image 20. As one aspect, the embodiment of the present disclosure has a color emphasis suppression function that suppresses the emphasis of saturation or brightness of handwriting pixels having a color corresponding to the vicinity of the color of the handwriting object 2 among the handwriting pixels that are pixels included in the handwriting image 22.

[0098] As just one example, the coefficient used to enhance the saturation or brightness of writing content pixels that are close to the statistical value of the hue of the writing object 2, for example the average value, is determined to be smaller than the coefficient used to enhance the saturation or brightness of writing content pixels that are not close to the statistical value of the hue of the writing object 2.

[0099] FIG. 14 is a diagram showing an example of a method for suppressing color emphasis. In FIG. 14, an enlarged view 22A1 shown in FIG. 13 is shown, and a color emphasis result when the suppression function of color emphasis according to the embodiment of the present disclosure is applied is shown as reference numeral 22A3. When the suppression function of color emphasis according to the embodiment of the present disclosure is applied, as shown in FIG. 14, the saturation or brightness of the chalk is emphasized, while the emphasis of the saturation or brightness of the chalk smudges c1 to c6 corresponding to the vicinity of the color of the writing target 2 is suppressed. Therefore, as shown by reference numeral 22A3 in FIG. 14, the chalk smudges C11 to C16 in which the emphasis of the saturation or brightness is suppressed also suppress the decrease in the visibility of the writing content 4. For example, as a result of suppressing the outline of the letter "A" from becoming unclear, it is possible to suppress the letter "A" from appearing to be the color of the blackboard, for example, greenish, blurred, or dull.

[0100] Therefore, according to the color emphasis suppression function according to the embodiment of the present disclosure, it is possible to suppress a decrease in visibility of the handwritten content 4 due to color emphasis.

[0101] Hereinafter, functional units that realize the function of suppressing color emphasis according to an embodiment of the present disclosure, such as the detection unit 15 and the determination unit 16, will be described.

[0102] (1-3-2-7. Detection unit 15) The detection unit 15 detects color information of the writing target 2 from the writing target image 21. As an embodiment, the processing when the writing target 2 is a blackboard will be taken as an example. In this case, the writing content 4 is likely to be a brighter color than the writing target 2. For this reason, the detection unit 15 applies a filter to the writing target image 21, which replaces the pixel values ​​of the pixel to be processed, such as hue, saturation, and brightness, with the pixel value of a surrounding pixel having the minimum brightness value among the surrounding pixels located within a predetermined range from the pixel to be processed. As an example only, each pixel included in the writing target image 21 is processed by the filter in the same order as raster scanning. In addition, the range of surrounding pixels to be replaced with pixel values ​​may be arbitrary. For example, the surrounding pixels may include four or eight neighboring pixels to the left, right, top, and bottom of the pixel to be processed, or the surrounding pixels may include pixels located up to a predetermined number of pixels from the pixel to be processed. By applying such a filter to the writing target image 21, in one aspect, the pixel values ​​of pixels having a color corresponding to chalk or the like, for example a color darker than white or yellow, for example the color of the board surface of a blackboard, are extracted by the filter and replaced with the pixel value of the pixel to be processed. As a result, a writing target image 21 from which the writing content 4 has been removed is obtained. Then, the detection unit 15 calculates statistics of the pixels included in the writing target image 21 from which the writing content 4 has been removed, for example an average value, a mode value, a median value, etc. As a result, color information of the writing target 2 is detected.

[0103] Note that, although the filter replacing the pixel value of the pixel to be processed with the pixel value of the surrounding pixel having the minimum brightness among the surrounding pixels located within a predetermined range from the pixel to be processed has been illustrated here as merely an example, other filters can also be applied to the captured image 20. For example, a filter replacing the pixel value of the pixel to be processed with a statistical value of the pixel values ​​of the surrounding pixels having the most frequent brightness among the surrounding pixels located within a predetermined range from the pixel to be processed can also be applied to the writing target image 21.

[0104] In addition, although the case where the writing object 2 is a blackboard has been given here merely as an example, the color information of the writing object 2 can be detected in a similar manner even when the writing object 2 is something other than a blackboard. As another example, when the writing object 2 is a whiteboard, a filter that replaces the pixel value of a pixel to be processed with the pixel value of a surrounding pixel having the maximum brightness among surrounding pixels located within a predetermined range from the pixel to be processed may be applied to the writing object image 21.

[0105] (1-3-2-8. Decision unit 16) The determination unit 16 determines a color emphasis suppression coefficient to be multiplied by the coefficient used by the correction unit 13 to emphasize the color of the handwriting image 22, based on the color information of the writing target 2 detected by the detection unit 15. Hereinafter, the coefficient used by the correction unit 13 to emphasize the saturation or brightness of the handwriting pixel will be referred to as a "color emphasis coefficient" in order to distinguish the label from the color emphasis suppression coefficient.

[0106] For example, the determination unit 16 sets the color emphasis suppression coefficient to be multiplied by the color emphasis coefficient of the writing content pixel having a hue near the statistical value of the hue of the writing object 2 to a value smaller than the color emphasis suppression coefficient of the writing content pixel having a hue that is not near the statistical value of the hue of the writing object 2. In this way, by multiplying the color emphasis coefficient of the writing content pixel having a hue near the statistical value of the hue of the writing object 2 by a color emphasis suppression coefficient with a smaller value, color emphasis of the writing content pixel having a hue near the statistical value of the hue of the writing object 2 is suppressed.

[0107] More specifically, the determination unit 16 can set a function that defines the correspondence between the hue and the color emphasis / suppression coefficient based on the color information of the writing object 2 detected by the detection unit 15.

[0108] 15 to 17 are diagrams showing examples of functions. Fig. 15 shows an example in which a function f1 is set in which the value of a color emphasis suppression coefficient corresponding to a hue within a predetermined range, for example, ±α, from the statistical value of the hue of the writing object 2 is set to 0, and the value of a color emphasis suppression coefficient corresponding to a hue outside the range is set to 1.

[0109] 16 also shows an example in which a function f2 is set in which the value of the color emphasis suppression coefficient increases monotonically and nonlinearly as it departs from the statistical value of the hue of the writing object 2. For example, in the function f2, the minimum value of the color emphasis suppression coefficient can be set as small as 0, and the maximum value of the color emphasis suppression coefficient can be set as large as 1. Note that, although FIG. 16 shows an example of the function f2 in which the value of the color emphasis suppression coefficient increases monotonically and nonlinearly, it is also possible to set a function in which the value of the color emphasis suppression coefficient increases linearly as the function f2.

[0110] 17 shows an example in which a function f3 is set in which the increment of the color emphasis suppression coefficient corresponding to an increase in hue and the increment of the color emphasis suppression coefficient corresponding to a decrease in hue are different between the two directions, toward the hue increase direction and the hue decrease direction, away from the hue statistical value of the writing object 2. Furthermore, in function f3, the increment of the monotonous increase of the color emphasis suppression coefficient corresponding to a decrease in hue in the hue decrease direction is set to be larger than the increment of the monotonous increase of the color emphasis suppression coefficient corresponding to an increase in hue in the hue increase direction.

[0111] The magnitude relationship of the increment of the monotonous increase of the color emphasis suppression coefficient in the two directions of the hue increasing direction and the hue decreasing direction can be set according to the visual characteristics. As an example, the visual characteristics have an aspect of low sensitivity to blue compared to other colors, while high sensitivity to skin color. For this reason, the increment of the monotonous increase of the color emphasis suppression coefficient corresponding to the direction including a hue corresponding to skin color among the two directions of the hue increasing direction and the hue decreasing direction can be set smaller than the increment of the monotonous increase of the color emphasis suppression coefficient corresponding to the other direction. Alternatively, the increment of the monotonous increase of the color emphasis suppression coefficient corresponding to the direction including a hue corresponding to blue among the two directions of the hue increasing direction and the hue decreasing direction can be set larger than the increment of the monotonous increase of the color emphasis suppression coefficient corresponding to the other direction.

[0112] 17 shows an example in which the color emphasis suppression coefficient increases linearly, but the color emphasis suppression coefficient may be increased nonlinearly. Also, the monotonically increasing increment of the color emphasis suppression coefficient corresponding to the direction including a hue corresponding to skin color among the two directions of the hue increasing direction and the hue decreasing direction may be set to be larger than the monotonically increasing increment of the color emphasis suppression coefficient corresponding to the other direction. Alternatively, the monotonically increasing increment of the color emphasis suppression coefficient corresponding to the direction including a hue corresponding to blue among the two directions of the hue increasing direction and the hue decreasing direction may be set to be smaller than the monotonically increasing increment of the color emphasis suppression coefficient corresponding to the other direction.

[0113] 15 to 17 show functions that define the correspondence between the hue and the color emphasis suppression coefficient, but the color emphasis suppression coefficient does not necessarily have to be determined according to a function. For example, it goes without saying that a lookup table or the like that defines the correspondence between the hue and the color emphasis suppression coefficient can be used instead of a function.

[0114] A color emphasis suppression coefficient corresponding to the hue of the handwriting content pixel is determined for each handwriting content pixel according to these functions f1 to f3. By multiplying the color emphasis suppression coefficient determined for each handwriting content pixel in this way by the color correction coefficient, i.e., the above-mentioned "γ", it is possible to incorporate the above-mentioned color emphasis suppression function into the color emphasis by the correction unit 13.

[0115] According to these functions f1 to f3, the color emphasis suppression coefficient of the handwriting content pixel having a hue close to the statistical value of the hue of the writing object 2 can be set to a value smaller than the color emphasis suppression coefficient of the handwriting content pixel having a hue not close to the statistical value of the hue of the writing object 2. Furthermore, according to the functions f2 and f3, the monotonically increasing increment of the color emphasis suppression coefficient corresponding to the increasing direction and the decreasing direction of the hue starting from the statistical value of the hue of the writing object 2 is smoother than that of the function f1, so that the flicker phenomenon that appears with the sharp increase and decrease of the color emphasis can be suppressed. Furthermore, according to the function f3, the monotonically increasing increment of the color emphasis suppression coefficient in the increasing direction or the decreasing direction of the hue to which the eye has low sensitivity is set larger than that of the functions f1 and f2, or the monotonically increasing increment of the color emphasis suppression coefficient in the increasing direction or the decreasing direction of the hue to which the eye has high sensitivity is set smaller, so that the flicker phenomenon that appears with the sharp increase and decrease of the color emphasis can be more effectively suppressed.

[0116] Although the example in which the color emphasis suppression coefficient is determined based on the hue has been given here merely as an example, the color emphasis suppression coefficient may be determined based on at least one of the saturation and the brightness.

[0117] [1-3-3. Output device 300] The output device 300 is a device that outputs information based on the control of the image processing device 100. The output device 300 is realized by a display device such as a CRT display device, a liquid crystal display device, a plasma display device, an EL display device, a laser projector, an LED projector, or a lamp.

[0118] The output device 300 receives the output image 25 from the output unit 14 and outputs the output image 25. The output device 300 may output the output image 25 in a stream format as a moving image. In other words, the output device 300 may output the output image 25 in real time.

[0119] The output device 300 may output the output image 25 at the timing of receiving the output image 25 from the output unit 14. On the other hand, the output device 300 may store the output image 25 received from the output unit 14 and output it at a later timing. Note that the output device 300 may receive the output image 25 stored in a buffer or storage inside or outside the image processing device 100 and output the output image 25 as a still image or a moving image.

[0120] As described above, the output device 300 is realized by various display devices. The output device 300 may be configured by a plurality of display devices. Here, a specific example of the output device 300 will be described with reference to FIG. 18. FIG. 18 is a diagram for explaining an example of output by the output device 300 according to this embodiment. FIG. 18 shows the input device 200 and the output devices 300a, 300b, and 300c.

[0121] As shown in Fig. 18, the output device 300 may be a display device such as output devices 300a and 300b. The output device 300 may be a tablet terminal such as output device 300c. The output devices 300a, 300b, and 300c each output an output image 25p. Note that, like the output device 300c, another terminal may be connected to the image processing device 100 and access the output image 25p. Of course, the output of the output image 25 by the output device 300 is not limited to the above example.

[0122] In this way, by outputting the output image 25 on various display devices, it becomes possible for an individual to check the output image 25 according to his or her own situation.

[0123] <2. Example of operation> Next, an example of the flow of operations of the system 1 according to the present embodiment will be described. Fig. 19 is a diagram for explaining an example of the flow of operations of the system 1 according to the present embodiment. The process shown in Fig. 19 is merely an example and can be started when the captured image 20 is acquired.

[0124] 19, the acquisition unit 11 acquires a captured image 20 in which a writing target image 21 is captured by the input device 200 (S1101). Next, the extraction unit 12 extracts a writing content image 22 from the captured image 20 (step S1102). In addition, the detection unit 15 detects color information of the writing target 2 from the captured image 20 (step S1103).

[0125] After that, the determination unit 16 and the correction unit 13 repeat the processes of step S1104 and step S1105 described below a number of times corresponding to the number of handwriting pixels, which are pixels included in the handwriting image 22 extracted in step S1102.

[0126] That is, the determination unit 16 determines a color emphasis suppression coefficient to be multiplied by the coefficient used by the correction unit 13 to enhance the color of the handwritten content image 22 based on the color information of the writing object 2 detected in step S1103 (step S1104). Then, the correction unit 13 executes according to the color emphasis coefficient obtained by multiplying the color emphasis coefficient used to correct the color of the form of the handwritten content image 22 by the color emphasis suppression coefficient determined in step S1104 (step S1105).

[0127] Thereafter, the output unit 14 generates an output image 25 in which the writing content image 22, the form of which has been corrected by the correction unit 13, is superimposed on the writing target image 21 of the captured image 20, and outputs the output image 25 to the output device 300 (step S1106), thereby completing the processing.

[0128] The order of execution of steps S1102 and S1103 shown in Fig. 19 is not limited to the example shown in the figure. For example, the process of step S1103 may be executed before the process of step S1102, or the processes of steps S1102 and S1103 may be executed in parallel.

[0129] <3. Application Examples> The above-described embodiment is merely an example, and various applications are possible.

[0130] <<3-1. Application example 1>> In the above embodiment, the example of calculating the statistical values ​​of pixels included in the writing target image 21 from which the writing content 4 has been removed by applying a filter to the captured image 20 has been given as merely one example, but the method of detecting color information of the writing target 2 is not limited to this. As another example, the detection unit 15 can detect color information of the writing target 2 by calculating the statistical values ​​of the remaining pixels by excluding pixels corresponding to the writing content image extracted by the extraction unit 12 from the pixels included in the writing target image 21.

[0131] <<3-2. Application Example 2>> In the above embodiment, the example in which the color emphasis suppression coefficient of the writing content pixel is determined based on the color information of the writing target 2 detected from the entire writing target image 21 has been given as merely one example, but the present invention is not limited to this.

[0132] FIG. 20 is a diagram showing an example of area division in application example 2. FIG. 20 shows an example in which the writing target image 21j is divided into three areas E1 to E3. In this case, the detection unit 15 detects color information of the writing target 2 for each of the areas E1 to E3 by calculating a statistical value of the hue of pixels other than the writing content pixels among the pixels included in each of the areas E1 to E3 of the writing target image 21j. Then, the determination unit 16 can set a function that determines a color emphasis suppression coefficient of the writing content pixels belonging to each of the areas E1 to E3 of the writing target image 21j. This can reduce the influence of blurring of the writing content 4 and the influence of sunlight or lighting.

[0133] Here, in FIG. 20, an example is given in which the writing target image 21j is divided into three areas E1 to E3, but the method of dividing the areas is not limited to this. For example, as the size of the writing target 2 becomes larger, the influence of blurring of the writing content 4 becomes more noticeable on the writing content image, and the influence of sunlight and lighting also becomes greater. For this reason, the pixel density per unit area of ​​the writing target 2 can be calculated from the writing target image 21, and the number of divisions can be set according to the pixel density. For example, as the pixel density per unit area of ​​the writing target 2 becomes larger, a smaller number of divisions can be set, or as the pixel density per unit area of ​​the writing target 2 becomes smaller, a larger number of divisions can be set. Note that, in FIG. 20, an example is given in which the writing target image 21j is divided left and right, but it can also be divided up and down, or up and down, left and right.

[0134] <<3-3. Application example 3>> Following the above-mentioned Application Example 2, the detection unit 15 can set a different function for each handwriting pixel based on the statistical value of the hue of the local area including the handwriting pixel and its surrounding pixels. FIG. 21 is a diagram showing an example of a local area in Application Example 3. FIG. 21 shows, as an example, an example of writing with white chalk on a blackboard, which is an example of a writing object 2. Furthermore, FIG. 21 shows a character string "ABCD" as an example of handwriting 4 written with white chalk. In the example shown in FIG. 21, for each handwriting pixel Pi corresponding to the character string "ABCD", a local area L of a predetermined size including the handwriting pixel Pi and its surrounding pixels is set. Pi Calculate the hue statistics of the local area L Pi Based on the statistical values ​​of the hues, the determination unit 16 sets a function for determining a color emphasis suppression coefficient for the writing content pixel Pi. This makes it possible to suppress color emphasis of the background of the writing object 2, which corresponds to smudges of chalk or marker, even if the sunlight or lighting conditions are different at each point of the writing object 2.

[0135] Fig. 22 is a diagram for explaining an example of the flow of operations of the system 1 according to application example 3. The process shown in Fig. 22 can also be started when the captured image 20 is acquired, similar to the process shown in Fig. 19.

[0136] 22, the acquisition unit 11 acquires a captured image 20 in which a writing target image 21 is captured by the input device 200 (S1101). Next, the extraction unit 12 extracts a writing content image 22 from the captured image 20 (step S1102).

[0137] After that, the detection unit 15, the determination unit 16, and the correction unit 13 repeat the processes of steps S1103 to S1105 described below a number of times corresponding to the number of handwriting pixels, which are pixels included in the handwriting image 22 extracted in step S1102.

[0138] That is, the detection unit 15 detects color information of a local region including the handwriting pixel and its surrounding pixels (step S2101). Then, the determination unit 16 determines a color emphasis suppression coefficient to be multiplied by the coefficient used by the correction unit 13 to enhance the color of the handwriting image 22 based on the color information of the local region detected in step S2101 (step S1104). Then, the correction unit 13 executes according to the color emphasis coefficient obtained by multiplying the color emphasis coefficient used to correct the color of the form of the handwriting image 22 by the color emphasis suppression coefficient determined in step S1104 (step S1105).

[0139] Thereafter, the output unit 14 generates an output image 25 in which the writing content image 22, the form of which has been corrected by the correction unit 13, is superimposed on the writing target image 21 of the captured image 20, and outputs the output image 25 to the output device 300 (step S1106), thereby completing the processing.

[0140] <<3-4. Application Example 4>> In the above embodiment, the color emphasis suppression function that suppresses color emphasis of the handwriting content image 22 included in the output image 25 based on the color information of the handwriting object 2 has been given as an example only, but it is also possible to provide a base determination function that determines color information of a base image on which the handwriting content image 22 is superimposed based on the color information of the handwriting object 2. It goes without saying that the color emphasis suppression function and the base determination function can be executed individually or in combination.

[0141] Fig. 23 is a diagram for explaining an example of a functional configuration of the system 1 according to Application Example 4. Fig. 23 shows an example of a functional configuration in which a color emphasis suppression function and a background determination function are implemented in combination. The image processing device 110 shown in Fig. 23 differs from the image processing device 100 shown in Fig. 2 in that it has a first determination unit 16 and a second determination unit 17.

[0142] Among these, the first determination unit 16 has the same function as the determination unit 16 shown in FIG. 2. The second determination unit 17 determines color information of the base image on which the writing content image 22 is superimposed based on color information of the writing object 2. As an example only, the second determination unit 17 determines a color corresponding to the statistical value of the hue of the writing object 2 as the color of the base image. In addition, when the statistical value of the hue of the writing object 2 corresponds to the color of a blackboard, the second determination unit 17 can also determine the color of the blackboard, for example, "black" or "dark green", as the color of the base image. In addition, when the statistical value of the hue of the writing object 2 corresponds to the color of a whiteboard, the second determination unit 17 can also determine the color of the whiteboard, for example, "white", as the color of the base image.

[0143] <<3-5. Application Example 5>> In the above embodiment, the example of color enhancement for all handwriting pixels is given as an example, but color enhancement does not necessarily have to be performed for all handwriting pixels. FIG. 24 is a diagram showing an example of degeneration of handwriting pixels according to application example 5. FIG. 24 shows the character "a" as an example of handwriting 4 as an example. For example, as shown in FIG. 24, the character "a" can be degenerated to a slightly thinner character "a" by applying a morphology filter to the character "a". The handwriting pixels after degeneration obtained in this way can correspond to the core of the handwriting. Then, the handwriting pixels are narrowed down to those that do not correspond to the degenerated handwriting pixels, and the color enhancement based on the color enhancement suppression coefficient can be determined.

[0144] <<3-6. Application Example 6>> As described above, the correction unit 13 corrects the handwriting image 22 based on the information of the writing target image 21 and the background color of the output image 25. Alternatively, the correction unit 13 may correct the handwriting image 22 based on the state of the writer detected from the captured image 20. The correction process by the correction unit 13 based on the state of the writer will be described below.

[0145] First, the state of the writer may be motion information of the writer. The correction unit 13 may correct the form of the handwriting content image 22 further based on the motion information of the writer. Here, the motion information refers to, for example, behavior detection information indicating whether the writer is writing on the writing target image 21. At this time, it may be detected from the captured image 20 that the writer is writing on the writing target image 21.

[0146] The motion of the writer is detected by performing behavior recognition of the writer. Specifically, by performing behavior recognition on each frame of a still image or video, the timing at which the writer performs a motion can be captured.

[0147] 25 and 26, a description will be given of a correction process performed by the correction unit 13 based on behavior detection information indicating whether or not the writer is writing on the writing target image 21. Figs. 25 and 26 are diagrams for describing an example of a correction process performed by the correction unit 13 according to this embodiment based on behavior detection information indicating whether or not the writer is writing on the writing target image 21.

[0148] When a writer who is not writing on the writing target image 21 is detected, the correction unit 13 corrects the color of the writing content image 22 based on the combination of the type of the writing target 2 and the background color of the output image 25, as described above. On the other hand, when a writer who is writing on the writing target 2 is detected, a correction different from that when a writer who is not writing on the writing target 2 is detected is performed on the writing target image 21.

[0149] FIG. 25 shows a captured image 20j acquired by the acquisition unit 11. The captured image 20j includes a writer image 23j that does not write on the writing target image 21j. FIG. 25 also shows an output image 25k. The output image 25k shows a base image 24k and a corrected writing content image 22k. In the example of FIG. 25, the writing content image 22k is the same as the writing content image 22j.

[0150] 26 shows a captured image 20l acquired by the acquisition unit 11. The captured image 20l includes a writing target image 21l, a writing content image 22j, and a writer image 23l. Here, the writer image 23l is an image of a writer who is in the middle of writing.

[0151] Here, the correction unit 13 detects that the writer 3 is writing, and corrects the color and width of the handwriting content image 22j. FIG. 26 shows an output image 25m. The output image 25m includes a base image 24m and a corrected handwriting content image 22m. Here, the handwriting content image 22m has a changed color and is wider than the handwriting content image 22j.

[0152] As shown in Figures 25 and 26, for example, the correction unit 13 may correct the writing content image 22 while the writer 3 is writing so that a person viewing the output image 25 can understand that the writer 3 is writing.

[0153] 25 and 26, an example has been described in which the correction unit 13 performs correction to widen (expand) the width of the handwriting image 22 when the detection unit 12 detects that the writer 3 is writing. However, on the other hand, when the detection unit 12 detects that the writer 3 is writing, the correction unit 13 may perform processing to narrow (reduce) the width of the handwriting image 22 or perform correction of the outline of the handwriting image 22, for example, by hollowing out the handwriting image 22 to leave only the outline portion.

[0154] In this way, it is enough to simply check the still image or video of the completed writing state to understand the written content image 22. This function can save the viewer time and effort when, for example, the viewer wants to check the written content image 22 later.

[0155] <<3-7. Application Example 7>> Furthermore, the information relating to the state of the writer 3 may be positional relationship information indicating a positional relationship between the writer 3 and the writing target image 21. The correction unit 15 may correct the writing content image 22 further based on the positional relationship information. Here, the positional relationship between the writer 3 and the writing target image 21 refers to the position of the writer 3 relative to the writing target image 21, etc. The positional relationship information may include a time corresponding to the positional relationship between the writer 3 and the writing target image 21, and a writing content image 22 among the writing content images 22 that corresponds to the positional relationship.

[0156] The positional relationship information can be obtained by providing a distance measuring device in the system 1. Here, the distance measuring device is, for example, a device that includes a distance measuring sensor and is capable of obtaining the distance between the distance measuring sensor and an object.

[0157] Hereinafter, the correction process of the handwriting content image 22 based on the positional relationship between the writer 3 and the writing target image 21 by the correction unit 15 will be described with reference to Fig. 27. Fig. 27 is a diagram for explaining the correction process of the handwriting content image 22 based on the positional relationship between the writer 3 and the writing target image 21 by the correction unit 15 according to this embodiment.

[0158] FIG. 27 shows a lecture. FIG. 27 shows a note-taking target 2a, a scribe 3a, a part of the note-taking content 4a, a student 5, an output device 300, and a plurality of distance measuring devices 400. The output image 25 output by the output device 300 may be installed so as to be visible only to the scribe 3. Here, the scribe 3 is standing in a position that hides the note-taking content 4. Therefore, only a part of the note-taking content 4a is visible to the student 5. Here, the correction unit 15 may correct the color of the note-taking content image 22 based on the positional relationship between the scribe 3 and the note-taking target image 21.

[0159] An example will be described with reference to Fig. 27. The multiple distance measuring devices 400 acquire the distance between each of the distance measuring devices 400 and the writer 3. At this time, the position of the writer with respect to the image 21 to be written on is detected.

[0160] Here, the correction unit 15 may correct the writing content image 22 so as to notify the writer 3 that the writing content image 22 is being hidden when the position of the writer detected from the writing target image 21 is in a position that hides the writing content image 22 for a predetermined period of time.

[0161] Specifically, when the positional relationship information indicates that the change in the positional relationship between the writer 3 and the writing content image 22 is equal to or less than a predetermined amount for a predetermined time, the correction unit 15 may correct the writing content image 22 corresponding to the positional relationship. For example, when the change in the position of the writer 3 is equal to or less than a predetermined amount for a predetermined time, the correction unit 15 may correct the color of the writing content image 22, for example, the color of the hidden writing content image 22 or the color near the hidden writing content image 22, to a predetermined color so as to notify the writer 3 that the writing content image 22 present at the position of the writing target 2 close to the position of the writer 3 is hidden.

[0162] 27, when the change in the position of the writer 3 is equal to or smaller than a predetermined amount during a predetermined time, the correction unit 15 corrects the color of the writing content 4a and the writing content image 22 hidden by the writer 3 to a predetermined color. In the example of FIG. 27, the corrected writing content image 22 is output by the output device 300 and notified to the writer 3.

[0163] In this way, it is possible to notify the transcriber that the student cannot view the written content image 22 because the written content image 22 is hidden. This function enables the transcriber to act in a way that allows the student to receive a more comfortable lecture.

[0164] The information indicating the positional relationship between the writer 3 and the writing target 2 may be acquired using an imaging device other than the distance measuring device 400. As the imaging device, for example, the input device 200 may be used. Furthermore, when the writing target 2 is an electronic whiteboard, the writing target 2 may directly output the output image 25.

[0165] <4. Hardware configuration example> Next, a hardware configuration example of the image processing device 100, the input device 200, and the output device 300 according to an embodiment of the present disclosure will be described. FIG. 28 is a block diagram showing a hardware configuration example of the image processing device 100, the input device 200, and the output device 300 according to an embodiment of the present disclosure. Referring to FIG. 28, the image processing device 100, the input device 200, and the output device 300 have, for example, a processor 871, a ROM 872, a RAM 873, a host bus 874, a bridge 875, an external bus 876, an interface 877, an input device 878, an output device 879, a storage 880, a drive 881, a connection port 882, and a communication device 883. Note that the hardware configuration shown here is an example, and some of the components may be omitted. In addition, components other than the components shown here may be further included.

[0166] (Processor 871) The processor 871 functions, for example, as an arithmetic processing device or control device, and controls all or part of the operation of each component based on various programs recorded in the ROM 872, the RAM 873, the storage 880, or the removable recording medium 901.

[0167] (ROM 872, RAM 873) The ROM 872 is a means for storing the programs read into the processor 871, data used in calculations, etc. The RAM 873 temporarily or permanently stores, for example, the programs read into the processor 871 and various parameters that change appropriately when the programs are executed.

[0168] In addition, the processor 871, ROM 872, RAM 873, and software cooperate to realize the functions of the acquisition unit 11, extraction unit 12, correction unit 13, output unit 14, detection unit 15, determination unit 16, input device 200, and output device 300 described above.

[0169] (host bus 874, bridge 875, external bus 876, interface 877) The processor 871, ROM 872, and RAM 873 are connected to one another via, for example, a host bus 874 capable of high-speed data transmission. On the other hand, the host bus 874 is connected to an external bus 876 having a relatively low data transmission speed via, for example, a bridge 875. In addition, the external bus 876 is connected to various components via an interface 877.

[0170] (Input Device 878) The input device 878 may be, for example, a mouse, a keyboard, a touch panel, a button, a switch, a lever, etc. Furthermore, a remote controller capable of transmitting a control signal using infrared rays or other radio waves may be used as the input device 878. The input device 878 may also include an audio input device such as a microphone.

[0171] (Output Device 879) The output device 879 is a device capable of visually or audibly notifying the user of acquired information, such as a display device such as a CRT (Cathode Ray Tube), LCD, or organic EL, an audio output device such as a speaker or a headphone, a printer, a mobile phone, or a facsimile. The output device 879 according to the present disclosure also includes various vibration devices capable of outputting tactile stimulation. The output device 879 realizes the function of the output device 300.

[0172] (Storage 880) The storage 880 is a device for storing various types of data. For example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device may be used as the storage 880.

[0173] (Drive 881) The drive 881 is a device that reads information recorded on a removable recording medium 901 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information to the removable recording medium 901.

[0174] (Removable recording media 901) The removable recording medium 901 is, for example, a DVD medium, a Blu-ray (registered trademark) medium, an HD DVD medium, various semiconductor storage media, etc. Of course, the removable recording medium 901 may be, for example, an IC card equipped with a non-contact type IC chip, an electronic device, etc.

[0175] (Connection port 882) The connection port 882 is a port for connecting an external device 902, such as a Universal Serial Bus (USB) port, an IEEE 1394 port, a Small Computer System Interface (SCSI), an RS-232C port, or an optical audio terminal.

[0176] (External connection device 902) The external connection device 902 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.

[0177] (Communication Device 883) The communication device 883 is a communication device for connecting to a network, such as a communication card for wired or wireless LAN, Bluetooth (registered trademark), or WUSB (Wireless USB), a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. By using the communication device 883, wireless communication between the image processing device 100 and a terminal device as the output device 300 is realized.

[0178] <5. Summary> An embodiment of the present disclosure has been described above with reference to FIGS. 1 to 28. As described above, the image processing device 100 according to the embodiment of the present disclosure determines color information of the output image 25 generated from the handwriting image 22 based on color information of the writing target 2 detected from the captured image 20. As one aspect, the image processing device 100 according to the embodiment of the present disclosure has a color enhancement suppression function that suppresses enhancement of saturation or brightness of writing content pixels having a color corresponding to the vicinity of the color of the writing target 2 among the writing content pixels that are pixels included in the writing content image 22. Therefore, according to the image processing device 100 according to the embodiment of the present disclosure, it is possible to suppress a decrease in visibility of the writing content 4 due to color enhancement.

[0179] Although the preferred embodiment of the present disclosure has been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

[0180] In addition, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects.

[0181] In addition, the processes described in this specification using flowcharts and sequence diagrams do not necessarily have to be performed in the order shown in the drawings. Some processing steps may be performed in parallel. In addition, additional processing steps may be employed, and some processing steps may be omitted.

[0182] The present technology can also be configured as follows. (1) A determination unit that determines color information of an output image generated from a writing content image, which is an image of writing content written on the writing target, based on color information of the detected writing target; An image processing device comprising: (2) The color information of the writing target is a statistical value of pixels included in the writing target image obtained by removing the writing content from the writing target image by applying a predetermined filter to the writing target image, which is an image of the writing target. The image processing device according to (1) above. (3) the filter replaces the pixel value of a pixel to be processed with the pixel value of a surrounding pixel having a minimum brightness value among surrounding pixels located within a predetermined range from the pixel to be processed; The image processing device according to (2) above. (4) the filter replaces the pixel value of the pixel to be processed with a statistical value of pixel values ​​of surrounding pixels located within a predetermined range from the pixel to be processed, the surrounding pixels having a brightness that is a most frequent value; The image processing device according to (2) above. (5) the filter replaces the pixel value of a pixel to be processed with the pixel value of a surrounding pixel having a maximum brightness among surrounding pixels located within a predetermined range from the pixel to be processed; The image processing device according to (2) above. (6) The color information of the writing object is a statistical value of pixels included in a writing object image, which is an image of the writing object, other than pixels corresponding to the writing content image. The image processing device according to (1) above. (7) The color information of the writing object is detected for each area of ​​the writing object. The image processing device according to any one of (1) to (6). (8) the color information of the writing object is detected for each writing content pixel based on a statistical value of a hue of a local region including a writing content pixel, which is a pixel included in the writing content image, and peripheral pixels of the writing content pixel; The image processing device according to (1) above. (9) The determination unit determines a color emphasis suppression coefficient to be multiplied by a color emphasis coefficient of a writing content pixel, which is a pixel included in the writing content image, based on color information of the writing object. The image processing device according to any one of (1) to (8). (10) the determination unit determines a color emphasis suppression coefficient to be multiplied by a color emphasis coefficient of a writing content pixel having a hue close to a statistical value of a hue of the writing object to be a value smaller than a color emphasis suppression coefficient of a writing content pixel having a hue not close to a statistical value of a hue of the writing object; The image processing device according to (9) above. (11) the determination unit determines, for each of the writing content pixels, a color emphasis suppression coefficient corresponding to a hue possessed by the writing content pixel according to a function in which a value of a color emphasis suppression coefficient corresponding to a hue within a predetermined range from a statistical value of a hue of the writing object is set to 0, and a value of a color emphasis suppression coefficient corresponding to a hue outside the range is set to 1; The image processing device according to (10) above. (12) the determination unit determines, for each of the writing content pixels, a color emphasis suppression coefficient corresponding to a hue possessed by the writing content pixel according to a function in which a value of the color emphasis suppression coefficient monotonically increases nonlinearly as the value deviates from a statistical value of the hue of the writing object; The image processing device according to (10) above. (13) the determination unit determines, for each of the writing content pixels, a color emphasis suppression coefficient corresponding to a hue possessed by the writing content pixel according to a function in which an increment of the color emphasis suppression coefficient corresponding to an increase in hue and an increment of the color emphasis suppression coefficient corresponding to a decrease in hue are different between two directions away from the statistical value of the hue of the writing object, in a hue increase direction and a hue decrease direction; The image processing device according to (10) above. (14) a magnitude relationship between increments of monotonically increasing the color emphasis suppression coefficient in two directions, that is, a hue increasing direction and a hue decreasing direction, is set according to visual characteristics; The image processing device according to (13) above. (15) an increment of monotonous increase of the color emphasis suppression coefficient corresponding to a direction including a hue corresponding to a skin color out of the two directions, the hue increasing direction and the hue decreasing direction, is set to be smaller than an increment of monotonous increase of the color emphasis suppression coefficient corresponding to the other direction; The image processing device according to (14) above. (16) an increment of monotonous increase of the color emphasis suppression coefficient corresponding to a direction including a hue corresponding to blue out of two directions, the hue increasing direction and the hue decreasing direction, is set to be larger than an increment of monotonous increase of the color emphasis suppression coefficient corresponding to the other direction; The image processing device according to (14) above. (17) The determination unit determines color information of a base image on which the writing content image is superimposed based on color information of the writing object. The image processing device according to any one of (1) to (16). (18) The determination unit determines a color corresponding to a statistical value of a hue of the writing object as a color of the base image. The image processing device according to (17) above. (19) determining color information of an output image generated from a writing content image, which is an image of the writing content written on the writing target, based on color information of the detected writing target; An image processing method in which the processing is performed by a computer. (20) determining color information of an output image generated from a writing content image, which is an image of the writing content written on the writing target, based on color information of the detected writing target; A program that causes a computer to carry out processing. [Explanation of symbols]

[0183] 100 Image processing device 11 Acquisition Department 12 Extraction part 13 Correction section 14 Output section 15 Detection section 16 Decision Section 200 Input Device 300 Output Device 400 Rangefinder

Claims

1. A determination unit that determines color information of an output image generated from a writing content image, which is an image of writing content written on the writing target, based on color information of the detected writing target; having The determination unit determines a color emphasis suppression coefficient to be multiplied by a color emphasis coefficient of a writing content pixel, which is a pixel included in the writing content image, based on color information of the writing object; the determination unit determines a color emphasis suppression coefficient to be multiplied by a color emphasis coefficient of a writing content pixel having a hue close to a statistical value of a hue of the writing object to be a value smaller than a color emphasis suppression coefficient of a writing content pixel having a hue that is not close to a statistical value of a hue of the writing object; the determination unit determines, for each of the writing content pixels, a color emphasis suppression coefficient corresponding to a hue possessed by the writing content pixel according to a function in which an increment of the color emphasis suppression coefficient corresponding to an increase in hue and an increment of the color emphasis suppression coefficient corresponding to a decrease in hue are different between two directions, a hue increase direction and a hue decrease direction, away from the statistical value of the hue of the writing object; Image processing device.

2. The color information of the writing target is a statistical value of pixels included in the writing target image obtained by removing the writing content from the writing target image by applying a predetermined filter to the writing target image, which is an image of the writing target. The image processing device according to claim 1 .

3. the filter replaces the pixel value of a pixel to be processed with the pixel value of a surrounding pixel having a minimum brightness value among surrounding pixels located within a predetermined range from the pixel to be processed; The image processing device according to claim 2 .

4. the filter replaces the pixel value of the pixel to be processed with a statistical value of pixel values ​​of surrounding pixels located within a predetermined range from the pixel to be processed, the surrounding pixels having a brightness that is a most frequent value; The image processing device according to claim 2 .

5. the filter replaces the pixel value of a pixel to be processed with the pixel value of a surrounding pixel having a maximum brightness among surrounding pixels located within a predetermined range from the pixel to be processed; The image processing device according to claim 2 .

6. The color information of the writing object is a statistical value of pixels included in a writing object image, which is an image of the writing object, other than pixels corresponding to the writing content image. The image processing device according to claim 1 .

7. The color information of the writing object is detected for each area of ​​the writing object. The image processing device according to claim 1 .

8. the color information of the writing object is detected for each writing content pixel based on a statistical value of a hue of a local region including a writing content pixel, which is a pixel included in the writing content image, and peripheral pixels of the writing content pixel; The image processing device according to claim 1 .

9. a magnitude relationship between increments of monotonically increasing the color emphasis suppression coefficient in two directions, that is, a hue increasing direction and a hue decreasing direction, is set in accordance with visual characteristics; The image processing device according to claim 1 .

10. an increment of monotonous increase of the color emphasis suppression coefficient corresponding to a direction including a hue corresponding to a skin color out of the two directions, the hue increasing direction and the hue decreasing direction, is set to be smaller than an increment of monotonous increase of the color emphasis suppression coefficient corresponding to the other direction; The image processing device according to claim 9 .

11. a monotonically increasing increment of the color emphasis suppression coefficient corresponding to a direction including a hue corresponding to blue out of two directions, the hue increasing direction and the hue decreasing direction, is set to be larger than a monotonically increasing increment of the color emphasis suppression coefficient corresponding to the other direction; The image processing device according to claim 9 .

12. The determination unit determines color information of a base image on which the writing content image is superimposed based on color information of the writing object. The image processing device according to claim 1 .

13. The determination unit determines a color corresponding to a statistical value of a hue of the writing object as a color of the base image. The image processing device according to claim 12.

14. determining color information of an output image generated from a writing content image, which is an image of the writing content written on the writing target, based on color information of the detected writing target; The computer executes the process, The process of determining includes a process of determining a color emphasis suppression coefficient to be multiplied by a color emphasis coefficient of a writing content pixel, which is a pixel included in the writing content image, based on color information of the writing object; the determining process includes a process of determining a color emphasis suppression coefficient to be multiplied by a color emphasis coefficient of a writing content pixel having a hue close to a statistical value of a hue of the writing object to be a value smaller than a color emphasis suppression coefficient of a writing content pixel having a hue that is not close to a statistical value of a hue of the writing object; the determining process includes a process of determining, for each of the writing content pixels, a color emphasis suppression coefficient corresponding to a hue possessed by the writing content pixel according to a function in which an increment of the color emphasis suppression coefficient corresponding to an increase in hue and an increment of the color emphasis suppression coefficient corresponding to a decrease in hue are different between two directions, that is, a hue increasing direction and a hue decreasing direction, from the statistical value of the hue of the writing object; Image processing methods.

15. determining color information of an output image generated from a writing content image, which is an image of the writing content written on the writing target, based on color information of the detected writing target; The process is executed by a computer, The process of determining includes a process of determining a color emphasis suppression coefficient to be multiplied by a color emphasis coefficient of a writing content pixel, which is a pixel included in the writing content image, based on color information of the writing object; the determining process includes a process of determining a color emphasis suppression coefficient to be multiplied by a color emphasis coefficient of a writing content pixel having a hue close to a statistical value of a hue of the writing object to be a value smaller than a color emphasis suppression coefficient of a writing content pixel having a hue that is not close to a statistical value of a hue of the writing object; the determining process includes a process of determining, for each of the writing content pixels, a color emphasis suppression coefficient corresponding to a hue possessed by the writing content pixel according to a function in which an increment of the color emphasis suppression coefficient corresponding to an increase in hue and an increment of the color emphasis suppression coefficient corresponding to a decrease in hue are different between two directions, that is, a hue increasing direction and a hue decreasing direction, from the statistical value of the hue of the writing object; program.

Images