Image processing device, method and program

Abstract

An image processing device according to one embodiment of the present invention comprises an extraction unit that extracts one or more processing target regions on the basis of characteristic elements of pixels of a moving image, a feature measurement unit that measures features of pixels in a region obtained by tracking the extracted processing target regions in continuous frames of the moving image, and a mask generation unit that generates image mask information that is information indicating the shapes and positions of the processing target regions extracted by the extraction unit when an average value of the features measured by the feature measurement unit falls within a predetermined range.

Description

[Technical field]

[0001] The present invention relates to an image processing device, a method, and a program. [Background technology]

[0002] There is a technology that uses a program running on a computer and a calculator such as an image processing chip to automatically reconstruct (image process) three-dimensional contents from contents captured as single perspective images or stereo images (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2011-18269 Summary of the Invention [Problem to be solved by the invention]

[0004] As mentioned above, when reconstructing 3D content from content captured as monocular or stereo images, the presence of transparent objects in the content poses a problem. For example, many contents include objects (hereinafter referred to as transparent objects) through which another subject located farther away than the actual subject can be seen, such as glass, smoke used in stage productions, or fog that appears depending on the weather.

[0005] It is easy for humans to recognize the presence or absence of this transmissive object and its depth position, but it is difficult for a computer to correctly grasp the presence or absence of this transmissive object and to determine the depth position at which this transmissive object exists.

[0006] Because it is difficult to determine the presence or absence of a transparent object and its depth position, when 3D content is automatically reconstructed by a computer, for example, if an object that is faintly visible further away than the fog in a foggy landscape, such as object A in this case, is processed by the computer, whether or not it can be extracted will vary depending on the darkness or lightness of the fog.

[0007] Here, if the thickness of the fog is near the boundary point where object A can or cannot be extracted, object A will alternate between being able to be extracted and not being able to be extracted. As a result, when 3D content is created as a video, the results of the depth information processing will change relatively frequently from frame to frame, resulting in a problem of flicker and a reduction in image quality.

[0008] The present invention has been made in light of the above-mentioned circumstances, and an object of the present invention is to provide an image processing device, method, and program that can improve the quality when an image contains semi-transparent objects. [Means for solving the problem]

[0009] An image processing device according to one aspect of the present invention includes an extraction unit that extracts one or more processing target regions based on feature elements of pixels of a moving image, a feature amount measurement unit that measures feature amounts of pixels in a region obtained by tracking the extracted processing target region in consecutive frames of the moving image, and , of all pixels in the area obtained by tracing the extracted processing target area and a mask generating section that generates, when the average of the feature amount is within a predetermined range, information indicating the shape and position of the processing target region extracted by the extracting section as mask information for the image.

[0010] An image processing method according to one aspect of the present invention is performed by an image processing device. circle A method for The extraction unit of the image processing device Extracting one or more target regions based on pixel features of the video image First step and, A feature amount measuring unit of the image processing deviceMeasure the feature amount of pixels in the area obtained by tracking the extracted processing target area in successive frames of the video image. The second step and, A mask generation unit of the image processing device The feature amount measuring unit In the second step Measured , of all pixels in the area obtained by tracing the extracted processing target area When the average of the feature amount is within a predetermined range, the extraction unit In the first step Generate information indicating the shape and position of the extracted processing target region as image mask information. The third step And, Yes do. Effect of the Invention

[0011] According to the present invention, it is possible to improve the quality when an image contains semi-transparent objects. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing an application example of an image processing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a flowchart showing an example of a processing operation performed by the semi-transparent region extraction unit. [Diagram 3] FIG. 3 is a diagram showing an example of a semi-transparent region extracted from an image by the semi-transparent region extraction unit. [Figure 4] FIG. 4 is a flowchart illustrating an example of a processing operation of the feature amount measuring unit. [Diagram 5] FIG. 5 is a flowchart illustrating an example of a processing operation of the original image acquisition unit. [Figure 6] FIG. 6 is a flowchart illustrating an example of a processing operation of the feature amount comparison unit. [Figure 7] FIG. 7 is a flowchart illustrating an example of the processing operation of the mask generation unit. [Figure 8] FIG. 8 is a block diagram showing an example of the hardware configuration of a semi-transparent area processing device of an image processing system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an application example of an image processing system according to an embodiment of the present invention. As shown in FIG. 1, an image processing system according to an embodiment of the present invention includes a semi-transparent region processing device (image processing device) 100 and a 3D image processing device 120. The semi-transparent region processing device 100 includes a semi-transparent region extraction unit 11, a feature amount measurement unit 12, an original image acquisition unit 13, a feature amount comparison unit 14, and a mask generation unit 15. The 3D image processing device 120 also includes a depth map generating unit 21 and a flick management unit 22.

[0014] The semi-transparent region processing device 100 extracts semi-transparent regions, such as fog or smoke in a stage production, from an image, and optimizes depth information for these semi-transparent regions. The 3D image processing device 120 may be integrated with the semi-transparent region processing device 100 .

[0015] The semi-transparent region extraction unit 11 of the semi-transparent region processing device 100 extracts a semi-transparent region in an image and processes information about this region. In the following explanation, it is assumed that the extracted semi-transparent region is a smoke region.

[0016] FIG. 2 is a flowchart showing an example of a processing operation performed by the semi-transparent region extraction unit. The color of the smoke varies depending on the stage lighting, the environment, and other conditions, so here we will explain the case where the smoke is whitish in color as an example.

[0017] The semi-transparent region extraction unit 11 checks the pixel values ​​of the whitish region in an arbitrary frame of a moving image, and holds n regions in which a group of pixels exists whose pixel values ​​are within a specified value range U as region A1, region A2, region A3, ..., region An (S11). Examples of the value range U here can be shown in the following (1) and (2). (1) (For RGB) 240 <R≦255、240<G≦255、240<B≦255 (2) (YUV(YCbCr)) 240 <Y≦255、-10<Cb≦0、-10<Cr≦0

[0018] Here, the range U may be hue information or luminance information as long as it is pixel information obtained from an image, and is not particularly limited as long as the characteristics of each pixel can be accurately extracted. In addition, the range U may be freely changed by an image editor based on experience by capturing the characteristics of each image, or an optimal solution may be set by a computer (including machine learning, etc.).

[0019] FIG. 3 is a diagram showing an example of a semi-transparent region extracted from an image by the semi-transparent region extraction unit. In the example shown in FIG. 3, it is shown that areas A1, A2, and A3 in image G1 are extracted as areas corresponding to whitish smoke.

[0020] Next, the semi-transparent region extraction unit 11 holds pixel values ​​In of representative points X1, X2, X3, ..., Xn in regions A1, A2, A3, ..., and An (S12). At this time, the representative point may be the center of gravity of region An, or the point in region An whose (x, y) coordinates are the minimum.

[0021] Next, the semi-transparent region extraction unit 11 tracks region A1, region A2, region A3, . . . , region An in successive frames of the moving image (S13). The semi-transparent region extraction unit 11 passes region information of the following region to the feature amount measurement unit 12 (S14). This region information includes shape information and position information of the region. The procedure of the processing by the feature amount measurement unit 12 will be described later.

[0022] Next, the semi-transparent region extraction unit 11 determines whether or not an average pixel value Kn, which will be described later and which has been returned from the feature amount measurement unit 12, is within the range U described above (S15). If the average pixel value Kn is not within the range U (No in S15), the semi-transparent region extraction unit 11 deletes (discards) the region An (S17), and the process related to the region An ends.

[0023] On the other hand, if the average pixel value Kn is within the range U (Yes in S15), the semi-transparent region extraction unit 11 passes the region information related to the region An to the mask generation unit 15 (S16), and returns to the process of S11. The region information of the region An includes the shape information MAn of the region An and the position information NAn of the region An. The processing operation of the mask generation unit 15 will be described later.

[0024] Next, a description will be given of the processing operation of the feature amount measuring unit 12. Fig. 4 is a flowchart showing an example of the processing operation of the feature amount measuring unit. First, the feature amount measuring unit 12 continues to measure all pixel values ​​within the area An in the information passed from the semi-transparent area extracting unit 11 for each frame (S21).

[0025] The feature amount measurement unit 12 calculates the average value of all the pixel values ​​measured in S21 and stores this average value as the average pixel value Kn (S22). The feature amount measurement unit 12 returns the average pixel value Kn of the region An stored in S22 to the semi-transparent region extraction unit 11 (S23).

[0026] Next, the processing operation of the original image acquisition unit 13 and the processing operation of the feature amount comparison unit 14 in FIG. 1 will be described in order. By providing the original image acquisition unit 13 and the feature comparison unit 14, if a problem occurs in the processing performed by the aforementioned semi-transparent region extraction unit 11 and feature measurement unit 12, the problem can be resolved by performing compensation, thereby making it possible to maintain high image quality.

[0027] For this reason, the group consisting of the semi-transparent region extraction unit 11 and the feature measurement unit 12, and the group consisting of the original image acquisition unit 13 and the feature comparison unit 14 may operate independently of each other, or the two groups may exchange information with each other and operate in a mutually complementary manner.

[0028] Specific issues that can be expected during processing operations by the aforementioned semi-transparent region extraction unit 11 and feature measurement unit 12 include, for example, (1) "Since there are a significantly large number of regions having features similar to smoke, the amount of calculations becomes significantly large, and therefore the processing time increases" and (2) "It is difficult to obtain good results due to the characteristics of the image (for example, in cases where the lights keep flashing frequently and rapidly, such as in a live venue)."

[0029] FIG. 5 is a flowchart illustrating an example of a processing operation of the original image acquisition unit. First, the original image acquisition unit 13 uses an imaging device (not shown) to capture images in advance that include a range that is planned to be photographed, based on a photography plan (S31). Next, the original image acquisition unit 13 analyzes each image captured in S31, and stores a set W of pixel features of each analyzed image in an internal memory or the like (S32). The set of feature quantities W may include various numerical values ​​indicating the characteristics of the pixels of the image, such as the RGB values ​​and luminance values ​​of all the pixels. Then, in response to a request from the feature amount comparison unit 14, the original image acquisition unit 13 passes the set W of feature amounts of the pixels of the image to the feature amount comparison unit 14 (S33).

[0030] FIG. 6 is a flowchart illustrating an example of a processing operation of the feature amount comparison unit. First, the feature comparison unit 14 receives the values ​​of the feature set W of the pixels of the original image from the original image acquisition unit 13 . Then, the feature comparison unit 14 compares this feature set W with the feature V of pixels of consecutive frames of the moving image to be processed (S41).

[0031] The feature quantity comparison unit 14 calculates Z according to the following formula (1). Z = |V - W|... Formula (1)

[0032] The feature quantity comparison unit 14 determines whether the calculated value of Z is within the value range T specified by the following formula (2) (S42). 0 < T ≤ a (a is a positive number)... Formula (2)

[0033] If it is Yes in S42, the feature quantity comparison unit 14 sets the n regions where the pixel groups within the above value range U exist as region B1, region B2, region B3, ···, region Bn, and stores them in an internal memory or the like (S43).

[0034] Here, if the set of feature quantities W and the feature quantity V are of the same type, they may be numerical values representing hue or numerical values representing luminance. Also, a in the above formula (2) may be a value set as an appropriate value based on empirical rules by the image editor, or a value calculated by calculation processing.

[0035] Next, the pixel values Jn of the representative points Y1, Y2, Y3, ···, Yn within the regions B1, B2, B3, ···, Bn are stored in an internal memory or the like (S44). This representative point may be the centroid of the region Bn or a point where the coordinate values (x, y) of the region Bn are the minimum values. Finally, the feature quantity comparison unit 14 passes the shape information MBn and the position information NBn, which are the region information within the region Bn, to the mask generation unit 15 (S45), and returns to the process of S41.

[0036] Figure 7 is a flowchart showing an example of the processing operation of the mask generation unit. First, the mask generation unit 15 receives at least one of the region information (shape and position information) of the region An from the aforementioned semi-transparent region extraction unit 11 or the region information (shape and position information) of the region Bn from the feature quantity comparison unit 14 (S51). Hereinafter, it will be described assuming that both the region information of the region An and the region information of the region Bn are received.

[0037] The mask generation unit 15 calculates the moving averages FMAn, FMBn, FNAn, and FNBn for the F (variable) frames going back from the current time for the shape information MAn, MBn and the position information NAn, NBn in the region information received in S51 (S52).

[0038] Here, the value of F may be set as any positive number. As an example of a guideline for F, in a 60fps video, a value in the range of about 20 to 60 is suitable. The value of F may be changed according to the processing load.

[0039] The mask generation unit 15 determines whether the values of the moving averages FMAn and FMBn calculated in S52 are included in the value range GM represented by the following formula (3), and whether the values of the moving averages FNAn and FNBn calculated in S52 are included in the range GN represented by the following formula (4) (S53). 0 < GM ≤ b (b is a positive number) … formula (3) 0 < GN ≤ c (c is a positive number) … formula (4)

[0040] If the result in S53 is Yes, the mask generation unit 15 passes the shape information MAn, MBn and the position information NAn, NBn, which are the region information, as the mask information at this time to the depth map generation unit 21 of the 3D image processing device 120 (S54). Also, if the result in S53 is No, the mask generation unit 15 does not pass the region information to the depth map generation unit 21 (S55) and ends the processing. By calculating the moving average in S52 above and determining this value in S53, the generation of mask information is performed gently, so that image flicker can be prevented. Note that the above depth map generation unit 21 may be held within the image processing system, or may be used in an external processing device or module, etc.

[0041] Furthermore, the depth map generating unit 21, which has received the mask information, generates depth information for each image based on this mask information. Furthermore, when flicking is recognized in the depth information generated by the depth map generating unit 21, the flick managing unit 22 notifies this to the semi-transparent region extracting unit 11 and the original image acquiring unit 13 of the semi-transparent region processing device 100. Upon receiving this notification, the semi-transparent region extracting unit 11 and the original image acquiring unit 13 start the above-mentioned processing.

[0042] FIG. 8 is a block diagram showing an example of the hardware configuration of a semi-transparent area processing device in an image processing system according to an embodiment of the present invention. 8, the semi-transparent region processing device 100 according to the above embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115. The same is true for the 3D image processing device 120.

[0043] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network NW. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0044] An input device 200 and an output device 300 that are attached to the semi-transparent region processing device 100 and are used by a user or the like are connected to the input / output interface 113 . The input / output interface 113 takes in operation data input by a user or the like through an input device 200 such as a keyboard, a touch panel, a touchpad, a mouse, etc., and outputs output data to an output device 300 including a display device using liquid crystal or organic EL (Electro Luminescence), etc., for display. The input device 200 and the output device 300 may be devices built into the translucent region processing device 100, or may be input devices and output devices of other information terminals that can communicate with the translucent region processing device 100 via the network NW.

[0045] The program memory 111B is a non-transient tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and stores programs necessary for executing various control processes, etc. according to one embodiment.

[0046] The data memory 112 is a tangible storage medium that is a combination of, for example, the above-mentioned nonvolatile memory and a volatile memory such as a RAM (Random Access Memory), and is used to store various data acquired and created during various processing steps.

[0047] A semi-transparent area processing device 100 according to one embodiment of the present invention can be configured as a data processing device having a semi-transparent area extraction unit 11, a feature measurement unit 12, an original image acquisition unit 13, a feature comparison unit 14, and a mask generation unit 15 shown in FIG. 1 as software-based processing function units.

[0048] Each information storage unit used as a working memory by each unit of the translucent area processing device 100 can be configured by using the data memory 112 shown in Fig. 8. However, these configured storage areas are not essential components within the translucent area processing device 100, and may be areas provided in a storage device such as an external storage medium such as a Universal Serial Bus (USB) memory, or a database server located in the cloud.

[0049] The processing function units in each of the above-mentioned semi-transparent region extraction unit 11, feature amount measurement unit 12, original image acquisition unit 13, feature amount comparison unit 14, and mask generation unit 15 can all be realized by reading and executing a program stored in the program memory 111B by the above-mentioned hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0050] The image processing device according to this embodiment extracts one or more processing target regions based on the feature elements of pixels in a video, measures the feature amounts of pixels in the region obtained by tracking the extracted processing target regions in successive frames of the video, and generates information indicating the shape and position of the extracted processing target region as image mask information when the average of the measured feature amounts is within a predetermined range. This makes it possible to prevent flickering in video caused by the movement of regions where objects in the background are not visible.

[0051] The methods described in each embodiment may be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), an optical disc (CD-ROM, DVD, MO, etc.), or a semiconductor memory (ROM, RAM, flash memory, etc.), and may be distributed by transmitting the program via a communication medium. The programs stored on the medium include a setting program that configures the software means (including not only execution programs but also tables and data structures) that the computer executes. The computer that realizes the present device reads the program recorded on the recording medium, and in some cases, configures the software means by the setting program, and executes the above-mentioned processing by controlling the operation of the software means. The recording medium in this specification is not limited to a recording medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory provided inside the computer or in a device connected via a network.

[0052] The present invention is not limited to the above-mentioned embodiment, and can be modified in various ways without departing from the gist of the present invention. The embodiments may be combined as appropriate, and in that case, the combined effect can be obtained. Furthermore, the above-mentioned embodiment includes various inventions, and various inventions can be extracted by combinations selected from the multiple components disclosed. For example, if the problem can be solved and the effect can be obtained even if some components are deleted from all the components shown in the embodiment, the configuration from which the components are deleted can be extracted as an invention. [Explanation of symbols]

[0053] 100...Translucent area processing device 120...3D image processing device 11...Semi-transparent area extraction part 12…Feature measurement section 13…Original image acquisition section 14…Feature comparison section 15…Mask generation section

Claims

1. an extraction unit that extracts one or more processing target regions based on feature elements of pixels of a moving image; a feature amount measuring unit that measures feature amounts of pixels within an area obtained by tracking the extracted processing target area in successive frames of the video; a mask generation unit that generates information indicating a shape and a position of the processing target area extracted by the extraction unit as image mask information when an average of the feature amounts of all pixels in the area obtained by tracking the extracted processing target area, measured by the feature amount measurement unit, is within a predetermined range; An image processing device comprising:

2. The extraction unit is extracting an area in the image where pixel values ​​are within a predetermined range as the processing target area; The image processing device according to claim 1 .

3. The mask generation unit calculating a moving average of information indicating a shape and a position of the processing target area extracted by the extraction unit in a predetermined number of frames going back from the current frame among the consecutive frames, and generating, when the calculated moving average is within a predetermined value range, information indicating the shape and the position of the processing target area extracted by the extraction unit in the predetermined number of frames going back as the mask information; The image processing device according to claim 1 .

4. a specifying unit that, when a difference between a feature amount of each pixel of the captured image and a feature amount of a pixel in each successive frame of the moving image falls within a predetermined value range, specifies one or more regions in which a pixel falling within the value range exists as a processing target region; The mask generation unit generating information indicating the shape and position of the identified processing target region as image mask information; The image processing device according to claim 1 .

5. 1. A method performed by an image processing device, comprising: A first step of extracting one or more processing target regions based on feature elements of pixels of a moving image by an extraction unit of the image processing device; a second step of measuring, by a feature amount measuring unit of the image processing device, feature amounts of pixels in an area obtained by tracking the extracted processing target area in successive frames of the moving image; a third step of generating, by a mask generation unit of the image processing device, information indicating a shape and a position of the processing target region extracted by the extraction unit in the first step as image mask information when an average of the feature amounts of all pixels in the region obtained by tracking the extracted processing target region, which is measured in the second step by the feature amount measurement unit, is within a predetermined range; An image processing method comprising the steps of:

6. The first step by the extraction unit includes: extracting an area in the image where pixel values ​​are within a predetermined range as the processing target area, The image processing method according to claim 5.

7. The third step by the mask generation unit is calculating a moving average of information indicating the shape and position of the processing target area extracted in the first step by the extraction unit in a predetermined number of frames going back from the current frame among the consecutive frames, and generating, when the calculated moving average is within a predetermined value range, information indicating the shape and position of the processing target area extracted in the first step by the extraction unit in the predetermined number of frames going back as the mask information; The image processing method according to claim 5.

8. An image processing program that causes a processor to function as each unit of the image processing device according to any one of claims 1 to 4.

Images