IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, VIDEO TRANSMISSION / RECEIVED SYSTEM, AND PROGRAM

Abstract

To make it possible to perform more suitable image quality adjustment for image signals in accordance with the details of a video represented by the image signals.SOLUTION: An image processing apparatus comprises: an area determination unit that determines a plurality of areas from frame images in a video; a luminance value distribution generation unit that generates a luminance value distribution for every area; and a contrast conversion unit that compresses the dynamic range of a specific area of the plurality of areas by using a compression method according to the luminance value distribution generated for every area to perform contrast conversion for the specific area.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, a video transmission / reception system, and a program. [Background technology]

[0002] The following Patent Document 1 discloses a technique for compressing the dynamic range of an image signal for each pixel and then encoding the image signal. This technique can further reduce the number of bits required for encoding an image signal. Summary of the Invention [Problem to be solved by the invention]

[0003] However, the technology disclosed in the above-mentioned Patent Document 1 compresses the dynamic range for all pixels in an image, which may result in inappropriate degradation of image quality in areas of the image that are considered important (i.e., areas where high image quality is preferable).

[0004] SUMMARY OF THE PRESENT DISCLOSURE In order to solve the above-mentioned problems of the conventional technology, an object of the present invention is to make it possible to perform more suitable image quality adjustment on an image signal in accordance with the content of the image represented by the image signal. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the image processing device of the present invention includes an area determination unit that determines a plurality of areas from a frame image in a video, a brightness value distribution generation unit that generates a brightness value distribution on an area-by-area basis, and a contrast conversion unit that performs contrast conversion on a specific area by compressing the dynamic range of the specific area among the plurality of areas using a compression method according to the brightness value distribution generated on an area-by-area basis. Effect of the Invention

[0006] According to the present invention, it is possible to perform more suitable image quality adjustment on an image signal in accordance with the content of a video image represented by the image signal. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing a system configuration of a video conference system according to an embodiment; [Diagram 2] FIG. 2 is a diagram showing a functional configuration of a video processing unit according to an embodiment; [Diagram 3] A flowchart showing a procedure of processing by a terminal device according to an embodiment. [Figure 4] FIG. 13 is a diagram showing an example of a region determined by a region determination unit according to an embodiment; [Diagram 5] FIG. 1 is a diagram for explaining a first conversion parameter generation method by a conversion parameter generation unit according to an embodiment; [Figure 6] FIG. 11 is a diagram for explaining a second conversion parameter generation method by a conversion parameter generation unit according to an embodiment; [Figure 7] FIG. 11 is a diagram for explaining a third conversion parameter generation method by a conversion parameter generation unit according to an embodiment; [Figure 8] FIG. 13 is a diagram showing another example of a region determined by a region determination unit according to an embodiment; [Figure 9] FIG. 11 is a diagram for explaining a fourth conversion parameter generation method by an overlap area processing unit according to an embodiment; [Figure 10] FIG. 13 is a diagram showing an example of generation of conversion parameters for an overlapping area by a conversion parameter generation unit according to an embodiment; [Figure 11] FIG. 13 is a diagram for explaining a fifth conversion parameter generation method by a conversion parameter generation unit according to an embodiment; [Figure 12] FIG. 13 is a diagram for explaining a sixth conversion parameter generation method by a conversion parameter generation unit according to an embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] [One embodiment] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0009] (System configuration of video conference system 1) FIG. 1 is a diagram showing a system configuration of a video conference system 1 according to an embodiment. As shown in FIG. 1, the video conference system 1 is an example of a "video transmission / reception system" and includes a plurality of terminal devices 200 (terminal device 200A and terminal device 200B) and a server 300. The terminal device 200A is provided at a first base. The terminal device 200B is provided at a second base. The terminal device 200A, the terminal device 200B, and the server 300 are connected to a network 400 such as the Internet or an intranet. FIG. 1 shows the configuration of the terminal device 200A in detail, but the terminal device 200B also has a similar configuration to the terminal device 200A. As a result, the terminal device 200A and the terminal device 200B can hold a video conference by transmitting and receiving video to and from each other via the network 400. Hereinafter, when there is no need to distinguish between the terminal device 200A and the terminal device 200B (i.e., when a description is given that applies to both the terminal device 200A and the terminal device 200B), the terminal device 200A will be referred to as the "terminal device 200." The terminal device 200A and the terminal device 200B are examples of an "image processing device."

[0010] As shown in FIG. 1, the terminal device 200 includes a camera module 10, a video processing unit 20, a video CODEC unit 30, a video output processing unit 40, a microphone array 50, an audio output unit 60, an audio processing unit 70, a network processing unit 80, an overall processing unit 90, an operation unit 100, a RAM 110, a recording device 120, a video characteristic analysis unit 130, and a RAM 140.

[0011] The camera module 10 is an example of an "imaging device." The camera module 10 captures an image of a conference scene. The camera module 10 has a lens 11, an imaging unit 12 (image sensor), and a DSP 13. The imaging unit 12 generates image data (RAW data) by converting an image collected through the lens 11 into an electrical signal. The DSP 13 generates image data (YUV data) by performing known camera image processing such as Bayer conversion and 3A control on the image data (RAW data) output from the imaging unit 12.

[0012] The image processor 20 performs various image processing including dynamic range compression processing on the image data (YUV data) output from the camera module 10. Details of the functions of the image processor 20 will be described later with reference to Fig. 2. The image processor 20 uses the RAM 110 as a buffer when performing various image processing.

[0013] The video CODEC unit 30 is an example of an "encoding unit." The video CODEC unit 30 encodes and decodes video data (video stream data) transmitted and received between the other terminal device 200. For example, the video CODEC unit 30 encodes video data output from the video processing unit 20 by a video endocer, and transmits the encoded video data to the other terminal device 200 via the network processing unit 80. Also, for example, the video CODEC unit 30 acquires video data transmitted from the other terminal device 200 (video data encoded by the other terminal device 200) via the network processing unit 80, and decodes the video data by a video dedocer. Then, the video CODEC unit 30 outputs the decoded video data to the video output processing unit 40. The video CODEC unit 30 is configured by a CODEC circuit or software using a compression standard such as H.264 / 265.

[0014] The video output processing unit 40 displays a video based on the video data on the display of the touch panel unit 41A. For example, the video output processing unit 40 displays a video based on the video data decoded by the video CODEC unit 30 (i.e., a video of another location) on the display of the touch panel unit 41A. Also, for example, the video output processing unit 40 displays a video based on the video data output from the camera module 10 (i.e., a video of the own location) on the display of the touch panel unit 41A. Also, the video output processing unit 40 performs contrast conversion processing on the video data decoded by the video CODEC unit 30 (processing for restoring the contrast of the video data that has been contrast converted by the other terminal device 200).

[0015] The microphone array 50 has a microphone array 51 and an A / D converter 52. The microphone array 51 collects the voices of the participants of the video conference and outputs an audio signal (analog signal). The A / D converter 52 converts the audio signal (analog signal) of the voice output from the microphone array 51 into a digital signal, and outputs the converted audio signal (digital signal) to the audio processing unit 70.

[0016] The audio output unit 60 has a D / A converter 62 and a speaker 61. The D / A converter 62 converts an audio signal (digital signal) transmitted from another terminal device 200 into an analog signal. The speaker 61 receives the audio signal (analog signal) converted by the D / A converter 62 and outputs the audio of the participants of the video conference collected at another base.

[0017] The audio processing unit 70 performs predetermined audio processing (e.g., codec processing, noise cancellation (NC), etc.) on audio data constituting the video data received from the other terminal device 200. Then, the audio processing unit 70 outputs the audio data after the audio processing to the audio output unit 60. At the same time, the audio processing unit 70 performs echo cancellation (EC) processing on the audio data that is input by looping around to the microphone array 50 while grasping the audio data to be output to the audio output unit 60. Also, the audio processing unit 70 performs predetermined audio processing (e.g., codec processing, noise cancellation (NC), etc.) on audio data constituting the video data to be transmitted to the other terminal device 200. Then, the audio processing unit 70 outputs the audio data after the audio processing to the network processing unit 80.

[0018] The network processing unit 80 transmits the encoded video data output from the video CODEC unit (encoder) 30 to another terminal device 200 as a destination via the network 400. The network processing unit 80 also receives encoded video data transmitted from another terminal device 200 via the network 400. Then, the network processing unit 80 outputs the video data to the video CODEC unit (encoder) 30. The network processing unit 80 also has a function (network state detection unit) of monitoring the network bandwidth in order to determine encoding parameters (QP value, etc.). The network processing unit 80 also has a function (other station function discrimination unit) of acquiring information on the functions and performance of the other station in order to optimize the settings of the encoding parameters (QP value, etc.) and the transmission mode.

[0019] The overall processing unit 90 performs overall control of the terminal device 200. The overall processing unit 90 is configured with a CPU, ROM, RAM, SSD, etc. For example, the overall processing unit 90 performs mode setting and status management for each module and block according to instructions from an operator. The overall processing unit 90 also has a function of arbitrating the right to use the system memory (RAM) and the access right to the system bus.

[0020] Moreover, the overall processing unit 90 sets the shooting mode of the camera module 10. The shooting mode settings of the camera module 10 may include automatic setting items (e.g., photometry conditions, etc.) that are automatically set according to the environment, and manual setting items that are manually set by an operator's operation input. The manual setting items include a reduction rate L (%) of the contrast width of a non-important area in the frame image. Furthermore, the manual setting items include a reduction rate M (%) of the contrast width of an intermediate area in the frame image. These settings are made by the operator operating the operation unit 100, and are stored in a memory (RAM) provided in the terminal device 200. These settings are then used by the image processing unit 20.

[0021] The operation unit 100 includes various input devices (for example, a touch panel, operation buttons, a remote control, etc.) The operation unit 100 accepts various inputs (for example, various settings, calling conference participants, etc.) by an operator operating the various input devices.

[0022] Recording device 120 records, in a storage device (memory, HDD, etc.), recording data that is a combination of audio data output from audio processing unit 70 and video data output from camera module 10. Recording device 120 can play back the recording data stored in the storage device.

[0023] The video characteristic analysis unit 130 has a detection unit 131 and a motion determination unit 132. The detection unit 131 detects an area where a human face exists from a frame image constituting the video data output from the camera module 10. The motion determination unit 132 detects an area where a person is moving from a frame image constituting the video data output from the camera module 10. The video characteristic analysis unit 130 outputs the detection result of each area to the video processing unit 20. Note that the video characteristic analysis unit 130 uses the RAM 140 as a buffer when detecting each area.

[0024] (Functional configuration of the video processing unit 20) FIG. 2 is a diagram showing a functional configuration of the image processing unit 20 according to an embodiment. As shown in FIG. 2, the image processing unit 20 includes an area determination unit 21, a brightness value distribution generation unit 22, a conversion parameter generation unit 23, an overlapping area processing unit 24, and a contrast conversion unit 25. Each function shown in FIG. 2 is a function for performing contrast conversion on the image data output from the camera module 10. The image data output from the camera module 10 includes a plurality of continuous frame images. For example, the image data output from the camera module 10 includes a predetermined number of frame images (for example, 24, 30, etc.) per second. Each frame image is displayed as one frame of the image by the image signal that constitutes the frame image. The image processing unit 20 can perform contrast conversion on the entire image data by performing contrast conversion on each of the plurality of frame images.

[0025] The area determination unit 21 determines an important area, a non-important area, and an intermediate area in a frame image constituting the video data output from the camera module 10. For example, the area determination unit 21 determines an area in the frame image where a human face is detected by the detection unit 131 of the video characteristics analysis unit 130 as an important area. Also, for example, the area determination unit 21 determines an area in the frame image where a human movement is detected by the movement determination unit 132 of the video characteristics analysis unit 130 as an important area. Note that the area determination unit 21 may determine an area in the frame image designated by an operator's operation of the touch panel unit 41A as an important area. Also, the area determination unit 21 determines an area other than the important area in the frame image as a non-important area. Furthermore, the area determination unit 21 determines an area of ​​the boundary between the important area and the non-important area in the frame image as an intermediate area.

[0026] The luminance value distribution generating unit 22 generates a histogram (an example of a "luminance value distribution") representing the distribution of luminance values ​​throughout the entire frame image, based on the luminance value of each pixel throughout the entire frame image. The luminance value distribution generating unit 22 also calculates a weighted average value Pc of luminance values ​​throughout the entire frame image, based on the generated histogram representing the distribution of luminance values ​​throughout the entire frame image. The luminance value distribution generating unit 22 also generates a histogram representing the distribution of luminance values ​​in each important area determined by the area determining unit 21 in the frame image, based on the luminance value of each pixel in the important area. The luminance value distribution generating unit 22 also calculates a weighted average value Pa of luminance values ​​in each important area determined by the area determining unit 21 in the frame image, based on the generated histogram representing the distribution of luminance values ​​in the important area.

[0027] The conversion parameter generating unit 23 generates contrast conversion parameters for the non-important area and the intermediate area determined by the area determining unit 21. For example, the conversion parameter generating unit 23 generates conversion parameters for the non-important area determined by the area determining unit 21 based on the weighted average value Pc calculated by the brightness value distribution generating unit 22 and the reduction rate L (%) of the contrast width of the non-important area previously set by the overall processing unit 90. Also, for example, the conversion parameter generating unit 23 generates contrast conversion parameters for the intermediate area determined by the area determining unit 21 based on the reduction rate M (%) of the contrast width of the intermediate area previously set by the overall processing unit 90. At this time, the conversion parameter generating unit 23 generates the contrast conversion parameters for the intermediate area using one of the first to third conversion parameter generating methods according to the magnitude relationship between the weighted average value Pc and the weighted average value Pa, which will be described later with reference to Figs. 5 to 7. The conversion parameter generating unit 23 generates the contrast conversion parameters in a LUT (Look Up Table) format. This is because the LUT format is suitable for real-time processing of video data.

[0028] When there is an area where the intermediate areas determined by the area determination unit 21 overlap each other (hereinafter referred to as the "overlapping area"), the overlapping area processing unit 24 generates conversion parameters for the overlapping area having a contrast width of N(%) (where 100(%)>N(%)>L(%)) using a fourth conversion parameter generation method described later with reference to Figures 8 to 10.

[0029] The contrast conversion unit 25 performs contrast conversion on the image captured by the camera module 10 by compressing the dynamic range for each area determined by the area determination unit 21 based on the conversion parameters generated by the conversion parameter generation unit 23. Specifically, the contrast conversion unit 25 compresses the luminance value of each of the pixels included in each area determined by the area determination unit 21 by converting the luminance value based on the conversion parameters generated by the conversion parameter generation unit 23. For example, the contrast conversion unit 25 does not perform contrast conversion on the important area in order to maintain image quality. For example, the contrast conversion unit 25 performs contrast conversion on the non-important area using the contrast conversion parameters with a contrast width of L (%) generated by the conversion parameter generation unit 23. For example, the contrast conversion unit 25 performs contrast conversion on the intermediate area using the contrast conversion parameters with a contrast width of M (%) generated by the conversion parameter generation unit 23. For example, the contrast conversion unit 25 performs contrast conversion on the overlapping area using the contrast conversion parameters with a contrast width of N (%) generated by the overlapping area processing unit 24. In this way, the contrast conversion unit 25 performs contrast conversion using appropriate contrast conversion parameters for each area determined by the area determination unit 21, thereby efficiently reducing the amount of code generated in the encoding process while maintaining subjective image quality.

[0030] (Processing procedure by terminal device 200) FIG. 3 is a flowchart showing a procedure of processing by the terminal device 200 according to an embodiment.

[0031] First, the overall processing unit 90 performs initial settings for the system (each module) of the terminal device 200, and sets the camera module 10 in a state in which imaging is possible (step S1).

[0032] Next, the overall processing unit 90 sets the shooting mode of the camera module 10 (step S2). The shooting mode settings of the camera module 10 may include automatic setting items (e.g., photometry conditions, etc.) that are automatically set according to the environment, and manual setting items that are manually set by an operator's operation input. The manual setting items include a reduction rate L (%) of the contrast width of the non-important area. The manual setting items also include a reduction rate M (%) of the contrast width of the intermediate area. The manual setting items also include the number of stages of the intermediate area. These setting values ​​are set to appropriate values ​​based on the shooting environment, the bandwidth of the transmission line, various conditions, etc.

[0033] Next, the terminal device 200 starts the video conference (step S3). For example, the terminal device 200 starts the video conference by transmitting a communication request to the other terminal device 200 that is the connection destination. Alternatively, the terminal device 200 starts the video conference by receiving a communication request transmitted from the other terminal device 200 that is the connection destination. Note that the terminal device 200 may start recording by the recording device 120 at this timing.

[0034] Next, the video characteristic analysis unit 130 and the video processing unit 20 acquire one frame image constituting the video data output from the camera module 10 (step S4).

[0035] Next, the area determining section 21 of the image processing section 20 determines important areas and non-important areas in the frame image acquired in step S4 (step S5).

[0036] Next, the brightness value distribution generation unit 22 of the image processing unit 20 generates a histogram representing the distribution of brightness values ​​in the entire frame image by aggregating the brightness values ​​of each pixel in the entire frame image acquired in step S4 (step S6).

[0037] Next, the brightness value distribution generating unit 22 of the image processing unit 20 aggregates the brightness values ​​of each pixel of the frame image acquired in step S4 for each important area determined in step S5, thereby generating a histogram representing the distribution of brightness values ​​for each important area determined in step S5 (step S7).

[0038] Next, the brightness value distribution generating section 22 of the image processing section 20 calculates a weighted average value Pc of the brightness values ​​of the entire frame image based on the histogram of the entire frame image generated in step S6 (step S8).

[0039] Next, the conversion parameter generation unit 23 of the image processing unit 20 generates conversion parameters for the non-important area determined in step S5 based on the weighted average value Pc calculated in step S8 and the reduction rate L (%) of the contrast width of the non-important area set in step S2 (step S9).

[0040] Next, the brightness value distribution generating unit 22 of the image processing unit 20 calculates a weighted average value Pa of the brightness values ​​for each important area based on the histogram for each important area generated in step S7 (step S10). For example, when two important areas A1 and A2 are determined in step S5, the brightness value distribution generating unit 22 calculates a weighted average value Pa1 of the important area A1 based on the histogram of the important area A1, and a weighted average value Pa2 of the important area A2 based on the histogram of the important area A2.

[0041] Next, the conversion parameter generating unit 23 of the image processing unit 20 judges whether or not the condition {Pa=Pc} is satisfied (step S11). Here, the weighted average value Pa and the weighted average value Pc do not have to be completely equal, and may have a predetermined tolerance within a range in which the luminance is perceived to be almost the same visually.

[0042] In step S11, when it is determined that the condition {Pa = Pc} is satisfied (step S11: Yes), the conversion parameter generation unit 23 generates a contrast conversion parameter for the intermediate area based on the reduction rate M (%) of the contrast width of the intermediate area set in step S2 by the third conversion parameter generation method described later with reference to FIG. 7 (step S15). Then, the terminal device 200 proceeds to step S16.

[0043] On the other hand, in step S11, when it is determined that the condition {Pa = Pc} is not satisfied (step S11: Yes), the conversion parameter generation unit 23 determines whether the condition {Pa < Pc} is satisfied (step S12).

[0044] In step S12, when it is determined that the condition {Pa < Pc} is satisfied (step S12: Yes), the conversion parameter generation unit 23 generates a contrast conversion parameter for the intermediate area based on the reduction rate M (%) of the contrast width of the intermediate area set in step S2 by the first conversion parameter generation method described later with reference to FIG. 5 (step S13). Then, the terminal device 200 proceeds to step S16.

[0045] On the other hand, in step S12, when it is determined that the condition {Pa < Pc} is not satisfied (step S12: No), the conversion parameter generation unit 23 generates a contrast conversion parameter for the intermediate area based on the reduction rate M (%) of the contrast width of the intermediate area set in step S2 by the second conversion parameter generation method described later with reference to FIG. 6 (step S14). Then, the terminal device 200 proceeds to step S16.

[0046] In step S16, the conversion parameter generation unit 23 determines whether contrast conversion parameters for all intermediate areas have been generated.

[0047] If it is determined in step S16 that contrast conversion parameters have not been generated for all intermediate areas (step S12: No), the terminal device 200 returns the process to step S11.

[0048] On the other hand, if it is determined in step S16 that contrast conversion parameters have been generated for all intermediate areas (step S12: Yes), the conversion parameter generation unit 23 determines whether there are any areas where intermediate areas overlap (hereinafter referred to as "overlapping areas") (step S17).

[0049] If it is determined in step S17 that there is an overlapping area (step S17: Yes), the conversion parameter generating unit 23 generates conversion parameters for the overlapping area by a fourth conversion parameter generating method described later with reference to Figures 8 to 10 (step S18). Then, the terminal device 200 advances the process to step S19.

[0050] On the other hand, if it is determined in step S17 that there is no overlapping area (step S17: No), the terminal device 200 advances the process to step S19.

[0051] In step S19, the contrast conversion section 25 of the image processing section 20 performs contrast conversion processing on each of the intermediate area and non-important area in the frame image in accordance with the corresponding conversion parameters generated by the conversion parameter generation section 23.

[0052] Then, the video processor 20 outputs the frame image after the contrast conversion process in step S19 to the video CODEC unit 30 (step S20).

[0053] After that, the terminal device 200 judges whether the video conference has ended (step S21). If it is judged in step S21 that the video conference has not ended (step S21: No), the terminal device 200 returns the process to step S4. On the other hand, if it is judged in step S21 that the video conference has ended (step S21: Yes), the terminal device 200 ends the series of processes shown in FIG. 3.

[0054] (An example of an area determined by the area determination unit 21) FIG. 4 is a diagram showing an example of a region determined by the region determining unit 21 according to an embodiment.

[0055] 4, important areas A1, A2, and A3 are determined in a frame image by the area determination unit 21 of the image processing unit 20. In addition, the area determination unit 21 determines an area other than the important areas A1, A2, and A3 in the frame image as a non-important area C.

[0056] 4, an intermediate area B1 is provided on the boundary between the important area A1 and the non-important area C in the frame image. An intermediate area B2 is provided on the boundary between the important area A2 and the non-important area C. An intermediate area B3 is provided on the boundary between the important area A3 and the non-important area C.

[0057] The contrast conversion unit 25 of the image processing unit 20 does not perform contrast conversion processing on the important areas A1, A2, and A3 in which important information exists. This allows the contrast conversion unit 25 to suppress deterioration of image quality in the important areas A1, A2, and A3.

[0058] On the other hand, the contrast conversion unit 25 performs contrast conversion on the non-important area C to set the contrast width to L (%).

[0059] However, when there is a significant difference between the image quality levels of the important areas A1, A2, and A3 and the image quality level of the non-important area C, there is a risk that the unnatural difference in image quality levels at the boundaries between the important areas A1, A2, and A3 and the non-important area C will become prominent.

[0060] Therefore, the terminal device 200 of the present embodiment provides intermediate areas B1, B2, and B3 at the boundaries between the important areas A1, A2, and A3 and the non-important area C, and performs contrast conversion on the intermediate areas B1, B2, and B3 such that the contrast width is M (%) (where 100 (%) > M (%) > L (%)).

[0061] As a result, the contrast conversion unit 25 can gently change the image quality level between the important areas A1, A2, and A3 and the non-important area C, and thus can make the boundaries between the important areas A1, A2, and A3 and the non-important area C less prominent.

[0062] However, the contrast range of the intermediate areas B1, B2, and B3 is determined by the conversion parameters generated by the conversion parameter generation unit 23, as will be described later with reference to FIGS. 5 to 7.

[0063] (First conversion parameter generation method) FIG. 5 is a diagram for explaining a first conversion parameter generation method by the conversion parameter generation unit 23 according to an embodiment. When it is determined that the condition {Pa (weighted average value of important areas) < Pc (weighted average value of the entire frame image)} is satisfied, the conversion parameter generation unit 23 uses the first conversion parameter generation method described below to generate contrast conversion parameters for the intermediate areas.

[0064] FIG. 5(a) is a histogram showing the distribution of luminance values in the entire frame image. This histogram is generated by the luminance value distribution generation unit 22 based on the luminance values of each pixel in the entire frame image. Also, FIG. 5(a) shows the weighted average value Pc of the luminance values in the entire frame image.

[0065] Fig. 5(b) is a histogram showing the distribution of luminance values ​​in the important area A1 (see Fig. 4) of the frame image. This histogram is generated based on the luminance values ​​of each pixel in the important area A1 by the luminance value distribution generating unit 22. Fig. 5(b) also shows the weighted average value Pa1 of the luminance values ​​in the important area A1.

[0066] FIG. 5(c) is a graph showing the relationship between the input and output of luminance values ​​in contrast conversion processing.

[0067] In the graph of FIG. 5(c), a dashed straight line 501 is applied to the important area A1, and represents the relationship between the input and output of the luminance value when the contrast width is not reduced.

[0068] In addition, in the graph of Figure 5 (c), the solid line 502 is applied to the non-important area C (see Figure 4) and represents the relationship between the input and output of the brightness value when the contrast width is reduced to L (%).

[0069] In addition, in the graph of Figure 5 (c), the straight line 503 shown by the dashed line is applied to the intermediate area B1 (the boundary between the important area A1 and the non-important area C), and represents the relationship between the input and output of the brightness value when the contrast width is reduced to M (%).

[0070] In the first conversion parameter generation method, first, the conversion parameter generation unit 23 obtains a straight line 502 so that the input value of the straight line 501 and the input value of the straight line 502 intersect at the weighted average value Pc of the luminance values ​​in the entire frame image, as shown in FIG. 5(c).

[0071] Next, the conversion parameter generating unit 23 calculates a straight line 503 so that the contrast width (M(%)) of the straight line 503 is expanded toward the lower value side than the contrast width (L(%)) of the straight line 502 (i.e., so as not to cut the contrast on the low value side), as shown in FIG. 5(c).

[0072] Then, the conversion parameter generation unit 23 determines the obtained straight line 503 as a conversion parameter to be applied to the intermediate area B1.

[0073] According to the first conversion parameter generation method described above, when the luminance of the important area is relatively low compared to the luminance of the entire frame, the luminance difference from the important area can be reduced in the low-luminance area in the intermediate area, thereby making the boundary between the important area and the non-important area less conspicuous.

[0074] (Second conversion parameter generation method) FIG. 6 is a diagram for explaining a second conversion parameter generation method by the conversion parameter generation unit 23 according to an embodiment. When it is determined that neither the condition {Pa = Pc} nor the condition {Pa < Pc} is satisfied, the conversion parameter generation unit 23 generates a contrast conversion parameter for the intermediate area using the second conversion parameter generation method described below.

[0075] FIG. 6(a) is a histogram showing the distribution of luminance values in the entire frame image. This histogram is generated by the luminance value distribution generation unit 22 based on the luminance values of each pixel in the entire frame image. Also, FIG. 6(a) shows the weighted average value Pc of the luminance values in the entire frame image.

[0076] FIG. 6(b) is a histogram showing the distribution of luminance values in the important area A2 (see FIG. 4) of the frame image. This histogram is generated by the luminance value distribution generation unit 22 based on the luminance values of each pixel in the important area A2. Also, FIG. 6(b) shows the weighted average value Pa2 of the luminance values in the important area A2.

[0077] FIG. 6(c) is a graph showing the relationship between the input and output of luminance values in the contrast conversion process.

[0078] In the graph of FIG. 6(c), a dashed straight line 601 is applied to the important area A2, and represents the relationship between the input and output of the luminance value when the contrast width is not reduced.

[0079] In addition, in the graph of FIG. 6(c), the solid line 602 is applied to the non-important area C, and represents the relationship between the input and output of the luminance value when the contrast width is reduced to L (%).

[0080] In addition, in the graph of Figure 6 (c), the straight line 603 shown by the dashed line is applied to the intermediate area B2 (the boundary between the important area A2 and the non-important area C), and represents the relationship between the input and output of the brightness value when the contrast width is reduced to M (%).

[0081] In the second conversion parameter generation method, first, the conversion parameter generation unit 23 obtains a straight line 602 so that the input value of the straight line 601 and the input value of the straight line 602 intersect at the weighted average value Pc of the luminance values ​​in the entire frame image, as shown in FIG. 6(c).

[0082] Next, the conversion parameter generating unit 23 obtains a straight line 603 such that the contrast width (M(%)) of the straight line 603 is expanded toward the higher value side than the contrast width (L(%)) of the straight line 602 (i.e., the contrast on the higher value side is not cut), as shown in FIG. 6(c).

[0083] Then, the transformation parameter generating unit 23 determines the obtained straight line 603 as the transformation parameters to be applied to the intermediate area B2.

[0084] According to the second conversion parameter generation method described above, when the brightness of an important area is relatively high compared to the brightness of the entire frame, the brightness difference between the important area and the high brightness area in the intermediate area can be reduced, thereby making the boundary between the important area and the non-important area less noticeable.

[0085] (Third conversion parameter generation method) 7 is a diagram for explaining a third conversion parameter generation method by the conversion parameter generation unit 23 according to an embodiment. When it is determined that the condition {Pa=Pc} is satisfied, the conversion parameter generation unit 23 generates a contrast conversion parameter for the intermediate area using the third conversion parameter generation method described below.

[0086] Fig. 7(a) is a histogram showing the distribution of luminance values ​​in the entire frame image. This histogram is generated based on the luminance values ​​of each pixel in the entire frame image by the luminance value distribution generating unit 22. Fig. 7(a) also shows the weighted average value Pc of the luminance values ​​in the entire frame image.

[0087] Fig. 7(b) is a histogram showing the distribution of luminance values ​​in important area A3 (see Fig. 4) of the frame image. This histogram is generated based on the luminance values ​​of each pixel in important area A3 by luminance value distribution generating unit 22. Fig. 7(b) also shows a weighted average value Pa3 of the luminance values ​​in important area A3.

[0088] FIG. 7(c) is a graph showing the relationship between the input and output of luminance values ​​in contrast conversion processing.

[0089] In the graph of FIG. 7C, a dashed straight line 701 is applied to the important area A3, and represents the relationship between the input and output of the luminance value when the contrast width is not reduced.

[0090] In addition, in the graph of FIG. 7(c), the solid line 702 is applied to the non-important area C, and represents the relationship between the input and output of the luminance value when the contrast width is reduced to L (%).

[0091] In addition, in the graph of Figure 7 (c), the straight line 703 shown by the dashed line is applied to the intermediate area B3 (the boundary between the important area A3 and the non-important area C), and represents the relationship between the input and output of the brightness value when the contrast width is reduced to M (%).

[0092] In the third conversion parameter generation method, first, the conversion parameter generation unit 23 calculates a straight line 702 so that the input value of the straight line 701 and the input value of the straight line 702 intersect at the weighted average value Pc of the luminance values ​​in the entire frame image, as shown in FIG. 7(c).

[0093] Next, the conversion parameter generating unit 23 obtains a straight line 703 such that the contrast width (M(%)) of the straight line 703 is expanded on both the high-value side and the low-value side more than the contrast width (L(%)) of the straight line 702, as shown in FIG. 7(c) (i.e., such that the contrast on the high-value side and the contrast on the low-value side are not completely cut).

[0094] Then, the transformation parameter generating unit 23 determines the obtained straight line 703 as the transformation parameters to be applied to the intermediate area B3.

[0095] According to the third conversion parameter generation method described above, when the brightness of an important area is similar to the brightness of the entire frame, the brightness difference between the important area and the intermediate area in the region near the weighted average brightness value Pc can be reduced, thereby making the boundary between the important area and the non-important area less noticeable.

[0096] (Another example of the area determined by the area determination unit 21) FIG. 8 is a diagram showing another example of the region determined by the region determining section 21 according to an embodiment.

[0097] 8, an important area A1 and an important area A2 are determined in a frame image by the area determination section 21 of the image processing section 20. In addition, an area other than the important areas A1 and A2 is determined as a non-important area C in the frame image by the area determination section 21.

[0098] 8, an intermediate area B1 is provided on the boundary between the important area A1 and the non-important area C in the frame image. Also, an intermediate area B2 is provided on the boundary between the important area A2 and the non-important area C.

[0099] Here, in the example shown in FIG. 8, there is an overlapping area D in the frame image where part of the intermediate area B1 and part of the intermediate area B2 overlap.

[0100] For the intermediate area B1, the conversion parameter generation unit 23 generates conversion parameters for reducing the contrast width to M (%) so that the contrast width becomes wider in the direction toward lower brightness values ​​than the non-important area C, as illustrated in Figure 5.

[0101] In addition, for the intermediate area B2, the conversion parameter generation unit 23 generates conversion parameters for reducing the contrast width to M (%) so that the contrast width becomes wider in the direction toward higher brightness values ​​than the non-important area C, as illustrated in Figure 6.

[0102] However, for overlap area D, the overlap area processing unit 24 generates conversion parameters for reducing the contrast width to N(%) (where 100(%)>N(%)>L(%)) using the fourth conversion parameter generation method described with reference to Figures 9 and 10.

[0103] (Fourth conversion parameter generation method) Fig. 9 is a diagram for explaining a fourth conversion parameter generation method by the overlap area processor 24 according to an embodiment. Fig. 10 is a diagram showing an example of generation of conversion parameters for the overlap area D by the overlap area processor 24 according to an embodiment.

[0104] For the overlap area D shown in FIG. 8, the overlap area processing unit 24 generates a contrast conversion parameter for the overlap area D by using a fourth conversion parameter generating method described below.

[0105] Fig. 9 is a graph showing the relationship between the input and output of luminance values ​​in contrast conversion processing. In the graph of Fig. 9, a dashed straight line 901 is applied to important areas A1 and A2, and shows the relationship between the input and output of luminance values ​​when the contrast width is not reduced. Also, in the graph of Fig. 9, a solid straight line 902 is applied to non-important area C, and shows the relationship between the input and output of luminance values ​​when the contrast width is reduced to L (%).

[0106] In the graph of FIG. 9, a straight line 903 indicated by a dashed dotted line is applied to the overlap area D, and represents the relationship between the input and output of the luminance value when the contrast width is reduced to N (%).

[0107] As shown in Fig. 9, the overlapping area processing unit 24 obtains the straight line 903 so that the contrast width (N(%)) of the straight line 903 is expanded on both the high-value side and the low-value side more than the contrast width (L(%)) of the straight line 902 (i.e., so that the contrast on the high-value side and the contrast on the low-value side are not completely cut off). At this time, the overlapping area processing unit 24 determines the OR range of the contrast width (M(%)) of the intermediate area B1 and the contrast width (M(%)) of the intermediate area B2 as the contrast width (N(%)) of the straight line 903 (i.e., the overlapping area D) as shown in Figs. 10(a) to (c). Then, the overlapping area processing unit 24 determines the obtained straight line 903 as the conversion parameter to be applied to the overlapping area D.

[0108] (Fifth conversion parameter generation method) 11 is a diagram for explaining a fifth conversion parameter generation method by the conversion parameter generation unit 23 according to an embodiment. When the distribution of luminance values ​​in the important area A has a relatively narrow range, the conversion parameter generation unit 23 may generate a contrast conversion parameter common to all areas by using a fifth conversion parameter generation method described below, instead of the first to third conversion parameter generation methods described using FIGS. 5 to 7.

[0109] Fig. 11(a) is a histogram showing the distribution of luminance values ​​in the entire frame image. This histogram is generated based on the luminance values ​​of each pixel in the entire frame image by the luminance value distribution generating unit 22. Fig. 11(a) also shows the weighted average value Pc of the luminance values ​​in the entire frame image.

[0110] FIG. 11(b) is a histogram showing the distribution of luminance values ​​in important area A of a frame image. This histogram is generated by luminance value distribution generating unit 22 based on the luminance values ​​of each pixel in important area A. As shown in FIG. 11(b), the distribution of luminance values ​​in important area A has a relatively narrow range. FIG. 11(b) also shows the weighted average value Pa of the luminance values ​​in important area A. Note that the fifth conversion parameter is easy to apply when the luminance distribution of the important area is narrow, and by generating a contrast conversion parameter common to all areas, it is possible to make the boundary between the important area and the non-important area inconspicuous without having an intermediate area even if the contrast is reduced.

[0111] FIG. 11C is a graph showing the relationship between the input and output of luminance values ​​in contrast conversion processing.

[0112] In the graph of FIG. 11(c), a dashed straight line 1101 represents the relationship between the input and output of the luminance value when the contrast width is not reduced.

[0113] In addition, in the graph of Figure 11 (c), the solid line 1102 is applied to all areas of the frame image and represents the relationship between the input and output of brightness values ​​when the contrast width is reduced to L (%).

[0114] 11(c), the conversion parameter generating unit 23 obtains a straight line 1102 that reduces the contrast width to L (%) so as to maintain the contrast when the input luminance value is near the weighted average value Pa, and to reduce the contrast when the input luminance value is not near the weighted average value Pa. Then, the conversion parameter generating unit 23 determines the obtained straight line 1102 as the conversion parameter to be applied to all areas of the frame image.

[0115] According to the fifth conversion parameter generation method, contrast in some areas is maintained, while contrast conversion is performed on other areas using common conversion parameters regardless of area type, thereby making it possible to reduce the amount of encoded data while maintaining the relative luminance relationship between multiple areas.

[0116] (Sixth conversion parameter generation method) 12 is a diagram for explaining a sixth conversion parameter generation method by the conversion parameter generation unit 23 according to an embodiment. When the distribution of luminance values ​​in the important area A has a relatively wide range, the conversion parameter generation unit 23 may generate a contrast conversion parameter common to all areas by using a sixth conversion parameter generation method described below, instead of the first to third conversion parameter generation methods described using FIGS. 5 to 7.

[0117] Fig. 12(a) is a histogram showing the distribution of luminance values ​​in the entire frame image. This histogram is generated based on the luminance values ​​of each pixel in the entire frame image by the luminance value distribution generating unit 22. Fig. 12(a) also shows the weighted average value Pc of the luminance values ​​in the entire frame image.

[0118] FIG. 12(b) is a histogram showing the distribution of luminance values ​​in important area A of a frame image. This histogram is generated by luminance value distribution generating unit 22 based on the luminance values ​​of each pixel in important area A. As shown in FIG. 12(b), the distribution of luminance values ​​in important area A has a relatively wide range. FIG. 12(b) also shows the weighted average value Pa of the luminance values ​​in important area A. Note that the sixth conversion parameter is easy to apply when the luminance distribution of the important area is narrow, and by generating a contrast conversion parameter common to all areas, it is possible to make the boundary between the important area and the non-important area inconspicuous without having an intermediate area even if the contrast is reduced.

[0119] FIG. 12(c) is a graph showing the relationship between the input and output of luminance values ​​in contrast conversion processing.

[0120] In the graph of FIG. 12(c), a dashed straight line 1201 represents the relationship between the input and output of the luminance value when the contrast width is not reduced.

[0121] Also, in the graph of Figure 12 (c), the solid line 1202 is applied to all areas of the frame image and represents the relationship between the input and output of brightness values ​​when the contrast width is reduced to L (%).

[0122] 12(c), the conversion parameter generation unit 23 obtains a straight line 1102 that reduces the contrast width to L (%) so that a straight line 1201 and a straight line 1202 intersect at a weighted average value Pa. Then, the conversion parameter generation unit 23 determines the obtained straight line 1202 as the conversion parameter to be applied to all areas of the frame image.

[0123] According to the sixth conversion parameter generation method, contrast conversion is performed on all areas using common conversion parameters regardless of area type, without maintaining the contrast of some areas, so that the amount of data to be encoded can be reduced while maintaining the relative luminance relationship between multiple areas.

[0124] Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0125] The applications, devices, and systems to which the present invention is applicable are not limited to those described in the above embodiment. In other words, the present invention is applicable to any applications, devices, and systems that aim to reduce the amount of data in video data by narrowing the contrast width of frame images in the video data.

[0126] In the above embodiment, the area where a human face is detected and the area where a human movement is detected are defined as the "important area", but the present invention is not limited to this. In other words, the "important area" may be any area that shows a subject (e.g., a document showing characters or images, a whiteboard, a person on a security camera, etc.) that is preferably displayed with a relatively high image quality depending on the purpose of use. [Explanation of symbols]

[0127] 1. Video conferencing system (video transmission and reception system) 10 Camera module (imaging device) 20 Video Processing Section 21 Area determination section 22 Brightness value distribution generation unit 23 Conversion parameter generation section 24 Overlapping area processing unit 25 Contrast conversion section 30 Video CODEC section (encoding section) 130 Video characteristics analysis section 131 Detection unit 132 Motion Judgment Unit 200, 200A, 200B Terminal device (image processing device) 300 Servers 400 Network A1, A2, A3 Important Areas B1, B2, B3 Intermediate area C Non-critical areas D. Overlap Area [Prior art documents] [Patent documents]

[0128] [Patent Document 1] International Publication No. 95 / 022228

Claims

1. A region determination unit that determines a plurality of regions from a frame image in a video; a luminance value distribution generating unit that generates a luminance value distribution in units of the region; a contrast conversion unit that performs contrast conversion on a specific region among the plurality of regions by compressing a dynamic range of the specific region using a compression method according to the luminance value distribution generated on a region-by-region basis; Equipped with The area determination unit is determining an important area and a non-important area from the frame image, and determining an intermediate area that is a boundary between the important area and the non-important area as the specific area; The contrast conversion unit is A compression method according to the luminance value distribution generated for each region is used to compress the dynamic range of the intermediate area, thereby performing contrast conversion of the intermediate area.

13. An image processing device comprising:

2. The contrast conversion unit is A process of compressing the dynamic ranges of the non-important area and the intermediate area is performed so that the dynamic range of the intermediate area is narrower than the dynamic range of the important area, and the dynamic range of the non-important area is narrower than the dynamic range of the intermediate area.

2. The image processing device according to claim 1,

3. The contrast conversion unit is No contrast conversion is performed on the important areas 3. The image processing device according to claim 2.

4. The luminance value distribution generating unit A weighted average value Pc of the luminance values ​​in the entire area of ​​the frame image; A weighted average value Pa of the luminance values ​​in the important area; Further calculate The contrast conversion unit is A process of compressing the dynamic range of the intermediate area is performed using a conversion parameter according to the magnitude relationship between the weighted average value Pc and the weighted average value Pa.

4. The image processing device according to claim 2, wherein the first and second inputs are input to the image processing apparatus.

5. The contrast conversion unit is When the weighted average value Pa is smaller than the weighted average value Pc, the dynamic range of the intermediate area is compressed so as to be expanded to a lower value side than the dynamic range of the non-important area.

5. The image processing device according to claim 4.

6. The contrast conversion unit is When the weighted average value Pa is greater than the weighted average value Pc, the dynamic range of the intermediate area is compressed so as to be expanded toward the higher value side than the dynamic range of the non-important area.

6. The image processing device according to claim 4, wherein the first and second inputs are input to the image processing device.

7. The contrast conversion unit is When the weighted average value Pa and the weighted average value Pc are substantially the same, the dynamic range of the intermediate area is compressed so as to be expanded toward the lower value side and the higher value side, respectively, than the dynamic range of the non-important area.

7. The image processing device according to claim 4, wherein the image processing device performs the image processing according to claim 5.

8. The contrast conversion unit is When there is an overlapping area where a plurality of the intermediate areas overlap, the dynamic range of the overlapping area is compressed so as to include the dynamic range of each of the plurality of intermediate areas.

8. The image processing device according to claim 4, wherein the image processing device performs the image processing according to claim 5.

9. A region determination step of determining a plurality of regions from a frame image in a video; a luminance value distribution generating step of generating a luminance value distribution on a region-by-region basis; a contrast conversion step of compressing a dynamic range of a specific region among the plurality of regions using a compression method according to the luminance value distribution generated for each region, thereby converting the contrast of the specific region; Including, The region determination step includes: determining an important area and a non-important area from the frame image, and determining an intermediate area that is a boundary between the important area and the non-important area as the specific area; The contrast conversion step includes: A compression method according to the luminance value distribution generated for each region is used to compress the dynamic range of the intermediate area, thereby performing contrast conversion of the intermediate area.

13. An image processing method comprising:

10. A video transmission / reception system having a plurality of terminal devices, At least one of the plurality of terminal devices, A region determination unit that determines a plurality of regions from a frame image in a video; a luminance value distribution generating unit that generates a luminance value distribution in units of the region; a contrast conversion unit that performs contrast conversion on a specific region among the plurality of regions by compressing a dynamic range of the specific region using a compression method according to the luminance value distribution generated on a region-by-region basis; Equipped with The area determination unit is determining an important area and a non-important area from the frame image, and determining an intermediate area that is a boundary between the important area and the non-important area as the specific area; The contrast conversion unit is A compression method according to the luminance value distribution generated for each region is used to compress the dynamic range of the intermediate area, thereby performing contrast conversion of the intermediate area. A video transmission and reception system comprising:

11. Computer, a region determination unit that determines a plurality of regions from frame images in a video; a luminance value distribution generating unit that generates a luminance value distribution in units of the region; and a contrast conversion unit that performs contrast conversion on a specific region by compressing a dynamic range of the specific region among the plurality of regions using a compression method according to the luminance value distribution generated on a region-by-region basis; A program for causing the device to function as a The area determination unit is determining an important area and a non-important area from the frame image, and determining an intermediate area that is a boundary between the important area and the non-important area as the specific area; The contrast conversion unit is A compression method according to the luminance value distribution generated for each region is used to compress the dynamic range of the intermediate area, thereby performing contrast conversion of the intermediate area. program.

Images