Image processing apparatus, method for processing image, and program

Abstract

To contribute to a technique for enabling more appropriate detection or estimation of a region of a specific object in a captured image.SOLUTION: The image processing apparatus includes: detection means for detecting positions of a plurality of feature points of a specific object on the basis of a first captured image in which the specific object is captured; determination means for determining the reference color of the specific object based on at least one of the positions of the plurality of feature points and on one of the first captured image and a second captured image in which the specific object is captured; estimation means for obtaining a first region, which is estimated as a region of the specific object, on the basis of the positions of the plurality of feature points; determination means for determining whether colors at positions of the first region are similar to the reference color; and adjustment means for excluding a region having a color determined by the determination means not to be similar to the reference color from the first region.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In recent years, research into mixed reality (MR) has focused on technology that overlays virtual space information onto real space in real time and presents it to users. In mixed reality, for example, a composite image is displayed in which an image of virtual space corresponding to the position and orientation of an imaging device is overlaid on an image of real space captured by an imaging device.

[0003] In this case, if the positional relationship between the real object and the virtual object is specific, the sense of distance between the objects may be expressed by not displaying the virtual object in the area of ​​the specific real object (real object) in the captured image. For example, when a user wearing an HMD holds a real object (such as their own hand or a tool) in front of a virtual object, if the virtual object is not rendered in the area of ​​the real object in the captured image, it is possible to display the real object as if it were in front of the virtual object. This allows the user to easily grasp the positional relationship between the virtual object and the real object, making it easy to verify work using a real hand or tool in the virtual space.

[0004] Therefore, in order to accurately represent the positional relationship between real objects and virtual objects, it is necessary to detect the area of ​​the real object and measure the distance to the real object. Furthermore, in order to prevent the user from experiencing motion sickness, it is preferable that the detection of the area of ​​the real object and the distance measurement be performed in real time (for example, at a frequency of about 60 fps).

[0005] Patent Documents 1 and 2 describe techniques for extracting a face or human body area in an image on a pixel-by-pixel basis using color information. Patent Document 1 describes a technique for detecting a rectangular area representing a face based on facial features and extracting a certain range of human skin color from the peak value of a color histogram. Patent Document 2 describes a technique for recognizing human body parts and extracting human skin color from the rectangular area of ​​the human body part using a trained classifier. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-166263 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-103242 Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Documents 1 and 2, it is assumed that the entire face or body of a third party shown in an image is visible, and that a rectangular area including the entire face or body can be detected. On the other hand, in MR, it is ideal to be able to detect all areas, such as the hands and arms, of a person using an HMD. However, since it is rare that all of these areas are included in an image, it may be impossible to detect the appropriate area.

[0008] Therefore, an object of the present invention is to contribute to a technology that enables a region of a specific object in a captured image to be more appropriately detected or estimated. [Means for solving the problem]

[0009] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. a detection means for detecting positions of a plurality of feature points on a specific object based on a first captured image of the specific object; a determination means for determining a reference color of the specific object based on at least one position of the plurality of feature points and the first captured image or a second captured image of the specific object; an estimation means for acquiring a first region that is an area estimated as an area of ​​the specific object based on the positions of the plurality of feature points; a determining means for determining whether the color at each position in the first region is similar to the reference color; an adjustment means for excluding from the first region a region of a color determined by the determination means to be not similar to the reference color; The image processing device is characterized by having:

[0010] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. a detecting step of detecting positions of a plurality of feature points on a specific object based on a first captured image of the specific object; a determining step of determining a reference color of the specific object based on at least one position of the plurality of feature points and the first captured image or a second captured image of the specific object; an estimation step of acquiring a first region that is an area estimated as an area of ​​the specific object based on the positions of the plurality of feature points; a determining step of determining whether or not the color at each position in the first region is similar to the reference color; an adjustment step of excluding from the first region a region of a color determined to be not similar to the reference color in the determination step; The image processing method is characterized by comprising the steps of: [Effects of the Invention]

[0011] The present invention can contribute to a technique that enables a region of a specific object in a captured image to be more appropriately detected or estimated. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a hardware configuration diagram of an image processing device according to a first embodiment. [Figure 2] 1 is a functional configuration diagram of an image processing device according to a first embodiment. [Figure 3] 10 is a flowchart of a color registration process according to the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating color registration according to the first embodiment. [Figure 5] 1A and 1B are diagrams illustrating estimation of color tendency of the entire hand and arm according to the first embodiment. [Figure 6] 4 is a flowchart of an image generation process according to the first embodiment. [Figure 7] FIG. 10 is a diagram illustrating estimation of color tendency of the entire hand and arm according to the second embodiment. [Figure 8] 10 is a flowchart of a color registration process according to the third embodiment. [Figure 9] FIG. 10 is a diagram illustrating color registration according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. While the following describes a technique for detecting "hand and arm regions" from a captured image, a region of a specific object may be detected instead of the "hand and arm regions." The specific object may be, for example, a foot, a human face, an animal, a vehicle, or a piece of equipment worn by a person (such as a stick or glove of a specific color).

[0014] <Embodiment 1> The hardware configuration of an image processing device 100 according to the first embodiment will be described with reference to FIG. The image processing device 100 includes a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, an input interface 104, and an output interface 105.

[0015] The CPU 101 is a control unit that controls each component of the image processing device 100 .

[0016] The RAM 102 is used as a working area when the CPU 101 controls each component.

[0017] The ROM 103 stores a control program, various application programs, data, etc. The CPU 101 loads the control program stored in the ROM 103 into the RAM 102 and executes it, thereby realizing the processing of each functional configuration within the image processing device 100 shown in FIG.

[0018] An input signal in a format that can be processed by the image processing device 100 is input to the input interface 104 from an external device (such as an imaging device).

[0019] The output interface 105 outputs a display image in a format that can be processed by an external device (such as a display device).

[0020] 2 is a software block diagram of the image processing device 100 according to embodiment 1. An imaging device 220 and a display device 230 are connected to the image processing device 100.

[0021] The image capturing device 220 is, for example, a video see-through type HMD (Head Mounted Display). The video see-through HMD is a camera incorporated in a display. A video see-through HMD is an HMD in which an image of a subject is displayed on the display unit in real time. The image capture device 220 captures an image of a subject such as a hand in each frame, thereby obtaining a captured image. Here, the image capture device 220 is configured in stereo, and the captured image is a stereo image.

[0022] The display device 230 is, for example, an HMD or a display (such as a PC monitor).

[0023] The image processing device 100 includes an image acquisition unit 201 , a registration unit 202 , a data storage unit 203 , a color region extraction unit 204 , a depth acquisition unit 205 , a model generation unit 206 , a display image generation unit 207 , a control unit 210 , and a detection unit 218 .

[0024] The image acquisition unit 201 acquires a captured image from the imaging device 220. The image acquisition unit 201 also stores the captured image in the data storage unit 203.

[0025] The registration unit 202 acquires color information of the hand and arm based on the "captured image acquired from the image acquisition unit 201" and the "position and posture of the hand acquired from the detection unit 218." Then, the registration unit 202 registers the color information of the hand and arm in a color registration table in the data storage unit 203. Hereinafter, the hand and arm may be collectively referred to as the "hand and arm."

[0026] In this way, the registration unit 202 registers the color of the hand or arm as a registered color in the color registration table. Furthermore, the registration unit 202 may treat the color of an area other than the hand or arm as a non-registered color based on the position and posture of the hand, and delete the non-registered color from the registered colors in the color registration table. A registered color is a color of an area that is extracted from a captured image, and a non-registered color is a color of an area that is not extracted from a captured image. The color registration table is expressed in a color space such as RGB, YUV, HSV, or Lab. The color registration table records whether each color is a registered color or a non-registered color. In order to reduce the number of color registrations and ensure stable detection, the color registration table may store registered colors and non-registered colors by expanding them in the color space.

[0027] The color region extraction unit 204 extracts (detects) the region of the arm to be modeled from the captured image based on the "captured image and color registration table" stored in the data storage unit 203. As a result, the color region extraction unit 204 generates an "image of only the region extracted from the captured image" as an extracted image.

[0028] The depth acquisition unit 205 calculates (acquires) depth (depth information) based on the captured image recorded in the data storage unit 203. As a result, the depth acquisition unit 205 generates a depth image that represents the depth in the captured image. Depth typically represents the distance from the imaging device 220 that acquired the captured image to the object depicted in the pixel. For example, if a method such as Semi-Global Matching (SGM) is used, the depth of the entire image can be calculated based on the stereo image. At this time, the accuracy of the depth image can be improved by limiting the area from which the depth is calculated using the extracted image acquired from the color region extraction unit 204 (i.e., by masking the captured image).

[0029] The model generation unit 206 generates a three-dimensional model of the arm based on the "extracted image acquired from the color region extraction unit 204" and the "depth image acquired from the depth acquisition unit 205." For example, the model generation unit 206 generates a two-dimensional polygon based on the extracted image. Then, the model generation unit 206 generates a three-dimensional polygon based on the two-dimensional polygon and the depth information of the depth image.

[0030] The two-dimensional polygons may be generated by Marching Squares or the like using the extracted image. The two-dimensional polygons may be generated by determining the contour of an object in the extracted image and using Delaunay triangulation based on the contour. If the object to be modeled is limited to an area such as a hand whose shape is known in advance, a three-dimensional model may be generated using model fitting or the like.

[0031] Display image generation unit 207 generates a composite image by combining a CG model (a model of a virtual object) stored in data storage unit 203 with the captured image, and controls display device 230 to display the composite image. At this time, display image generation unit 207 first draws the captured image as the background. Next, display image generation unit 207 enables a depth test that determines the depth relationship between multiple images to be combined, and renders only the depth of the 3D model of the arm. Finally, display image generation unit 207 renders the CG model. At this time, the "area of ​​the CG model that overlaps with the 3D model of the arm and is located further back than the 3D model of the arm" is made transparent. This allows display image generation unit 207 to display an image of the arm, rather than a CG model, at the position of the 3D model of the arm.

[0032] The control unit 210 controls each component of the image processing device 100 .

[0033] The detection unit 218 detects the position of an object, such as the hand of a user using the HMD, based on the captured image stored in the data storage unit 203, and stores the detected position in the data storage unit 203. To detect the position of an object, for example, a method can be used in which a model for estimating the position of an object is generated in advance using Deep Learning (DL), and the position of the object is estimated based on the model. Using DL, a detector is prepared that has previously been trained based on an "image containing a hand" and "correct coordinates of the hand." The detector then uses the processed target image as input to detect the coordinates of the hand in the target image as the position and posture of the hand. The coordinates of the hand include the coordinates (positions) of multiple feature points on the hand. The coordinates of the hand include, for example, the coordinates of a rectangular frame containing the hand, the coordinates of the wrist, and the coordinates of the joint points of the hand. In each embodiment, the method for detecting (estimating) the position and posture of the hand is not limited. The captured image acquired by the detection unit 218 from the data storage unit 203 may be the same as or different from the captured image used by the registration unit 202.

[0034] A flowchart of color registration processing according to the first embodiment will be shown with reference to the flowchart in FIG. 3. The processing of this flowchart is executed by the control unit 210 controlling each block each time the captured image in the data storage unit 203 is updated by the image acquisition unit 201. In other words, the processing of this flowchart is executed for each captured image (for each frame). Hereinafter, a captured image that is the target of the processing of this flowchart will be referred to as a "target image." Note that this processing is executed asynchronously with the processing performed by the display image generation unit 207.

[0035] In step S301, the control unit 210 determines whether the processing of this flowchart is already being executed. If it is determined that the processing of this flowchart is already being executed, the processing from step S303 onwards is not performed. If it is determined that the processing of this flowchart is not being executed, the processing proceeds to step S303. In step S301, the processing of the flowchart in FIG. 3 may take longer than the update rate of the captured image, so this determination is made to prevent the processing of this flowchart from being executed redundantly.

[0036] In step S303, the registration unit 202 acquires information about the position and orientation of the hand from the detection unit 218. Then, the registration unit 202 extracts a reference area that is more likely to be a hand area than a threshold value, based on the position and orientation of the hand (at least one of the coordinates of a plurality of feature points). In the example of FIG. 4, the reference area 441 is a filled-in area surrounded by "joint points that constitute the back of the hand, among the plurality of feature points." Note that the area estimated to be the inside of the back of the hand is likely to fall within the actual inside of the hand, even if there is some error. For this reason, in FIG. 4, the area inside the hand is extracted as the reference area 441.

[0037] In step S304, the registration unit 202 determines reference colors for the hand region and the arm region (hereinafter, the two regions are collectively referred to as the "hand / arm region") based on the color of the reference region of the hand. The reference color is a color that serves as a reference for determining the hand / arm region. FIG. 5 shows an example of the process for determining the reference color in the target image. In FIG. 5, it is assumed that the colors of the entire hand / arm region tend to be distributed around a straight line in color space 500. A straight line 502 (reference axis) that represents the tendency of the entire hand / arm region 503 is obtained by fitting to the reference region 501. The straight line 502 represents the characteristics of the set of multiple colors in the reference region of the hand (color set) in the color space. The color space may be a three-dimensional space such as RGB, YUV, HSV, or Lab. The color space may be a two-dimensional space in which only the UV component of YUV is extracted, for example. The registration unit 202 determines the color on the straight line 502 in the color space as the reference color.

[0038] In step S305, the registration unit 202 acquires information about the position and orientation of the hand from the detection unit 218. The registration unit 202 roughly estimates a hand / arm region in the target image based on the position and orientation of the hand (coordinates of multiple feature points). For example, in the example of FIG. 4, among the joint points of the hand position and orientation 411, the registration unit 202 newly estimates, as the position of a feature point in the arm region, "a position obtained by extending a line connecting the wrist joint point and the joint points of the bases of the fingers (the base of the thumb, the base of the index finger, and the base of the middle finger) by a certain magnification." Then, the registration unit 202 estimates, as the arm region, a region enclosed by lines connecting three feature points of the estimated arm region and the wrist joint point. The registration unit 202 also estimates, as the hand region, "a circular region centered on the joint point," "a region in which the joint points constituting the back of the hand are filled in," and "a region in which the joint points of each finger are connected by a line."

[0039] In step S306, the registration unit 202 generates a registration candidate area based on the estimated arm area. In the example of FIG. 4, the area where the estimated arm area is filled in is the registration candidate area. It is generated as 421.

[0040] In step S307, the registration unit 202 generates a registration area by adjusting the registration candidate area so that it is included in the actual hand / arm area. The registration unit 202 excludes areas of the registration candidate area whose color is determined to be unsimilar to the reference color. Specifically, the registration unit 202 determines, for example, among the pixels (positions) of the registration candidate area, "pixels having a color that is a certain distance or more in color space from the line 502" as pixels that are unsimilar to the reference color. The registration unit 202 then excludes areas of pixels that are unsimilar to the reference color from the registration candidate area. In the example of FIG. 4, areas of the registration candidate area 421 that are not included in the actual arm area are excluded to generate the registration area 431. According to the processing of step S307, the registration unit 202 generates the registration area based not only on the positions of the feature points but also on the reference color, thereby enabling the registration area to be detected as a more appropriate hand / arm area.

[0041] In step S308, the registration unit 202 generates a non-registration candidate area based on the position and posture of the hand (at least one of the coordinates of a plurality of feature points). In the example of Fig. 4, the registration unit 202 generates a window-frame-shaped area that does not include a rectangular area that includes the range corresponding to the hand position and posture 411 as the non-registration candidate area 451.

[0042] In step S309, the registration unit 202 generates a non-registration area by adjusting the non-registration candidate area so that it does not include the actual arm area. The registration unit 202 excludes areas of a color similar to the reference color from the non-registration candidate area. Specifically, for example, the registration unit 202 determines that, among the pixels (positions) of the non-registration candidate area, "pixels having a color that is less than a certain distance in color space from the line 502" are pixels similar to the reference color. Then, the registration unit 202 excludes areas of pixels similar to the reference color from the non-registration candidate area. In this way, the registration unit 202 generates a non-registration area. In the example of FIG. 4, areas included in the actual arm area are excluded from the non-registration candidate area 451, and a non-registration area 461 is generated.

[0043] In step S310, the registration unit 202 registers the colors included in the registration area as registered colors in the color registration table. In the example of FIG. 4, the registration unit 202 registers the colors included in the registration area 431 as registered colors in the color registration table 470. The registration unit 202 saves the colors included in the non-registration area 461 as non-registered colors. At this time, if the same color is included in both the registered colors and the non-registered colors, it may be possible to set the color to be used preferentially as either the registered colors or the non-registered colors. For this reason, the registration unit 202 may correct the colors included in the registered colors in the color registration table based on the non-registered colors.

[0044] The processes of steps S306 and S307 may be performed after the processes of steps S308 and S309 are completed. In this case, in step S307, the registration unit 202 may adjust the registration candidate region based on the reference color and the non-registration region. For example, the registration unit 202 may adjust the registration candidate region so that the registration region includes "a region that is not included in the non-registration region and has a color similar to the reference color."

[0045] In the above description, the registration unit 202 executes processes such as "determining a reference color," "estimating a hand / arm region," "adjusting a registered region," and "registering a registered color." The image processing device 100 may include a determination unit that determines a reference color, an estimation unit that estimates a hand / arm region, and an adjustment unit that adjusts a registered region, which share part of the processes performed by the registration unit 202.

[0046] The image generation process according to the first embodiment will be described with reference to the flowchart in Fig. 6. The process in the flowchart in Fig. 6 is executed by the control unit 210 when the captured image in the data storage unit 203 is updated by the image acquisition unit 201. Here, the process in the flowchart in Fig. 5 The captured image of the target and the captured image to be processed in the flowchart of FIG. 6 may be different captured images.

[0047] In step S601, the color region extraction unit 204 reads the color registration table from the data storage unit 203. The color region extraction unit 204 extracts (detects) a region of a color (registered color) registered in the color registration table from the captured image. As a result, the color region extraction unit 204 operates as an object detection unit that detects a hand / arm region from the captured image.

[0048] In step S602, the depth acquisition unit 205 generates (acquires) a depth image (depth information). At this time, the depth acquisition unit 205 may improve the accuracy of the depth image by generating the depth image limited to the area (extraction image) of the registered color extracted in step S601.

[0049] In step S603, the model generation unit 206 generates a three-dimensional model of the object based on the "extracted image generated in step S602" and the "depth image generated in step S601."

[0050] In step S604, display image generation unit 207 displays on display device 230 a display image obtained by combining the captured image with a CG model (a model of a virtual object) stored in data storage unit 203. At this time, display image generation unit 207 first draws the captured image as a background. Next, display image generation unit 207 enables a depth test that determines the depth relationship between multiple images to be combined, and renders and draws a thickness model that makes the three-dimensional model of the object transparent. Finally, display image generation unit 207 renders and draws the CG model. Here, in the area where the thickness model and CG model overlap, the CG model that is deeper than the thickness model is made transparent, and pixels of the captured image are displayed in that area. In this way, display image generation unit 207 can display the arm at the position of the arm in real space.

[0051] According to the first embodiment, a registration area can be generated based on not only the position and orientation of the detected hand area but also the reference color. Therefore, a registration area that more closely resembles the actual hand / arm area can be detected (generated). Then, by extracting the color of the entire skin including the arm from the registration area and using this color information, the hand / arm area can also be extracted (detected) in a real-time captured image.

[0052] <Embodiment 2> In the first embodiment, the image processing device 100 determines the colors to be registered in the registration table based on colors that are less than a certain distance from a straight line in the color space. However, depending on the color space and shooting environment, it may be difficult to approximate the color area of ​​the arm with a straight line. Therefore, in the second embodiment, curved fitting is performed instead of a straight line.

[0053] FIG. 7 shows an example of estimating the color tendency of the entire hand / arm in the second embodiment. A curve 702 (reference curve) that represents the tendency of the entire hand / arm region 703 is obtained by performing fitting or the like on a reference region 701 in a color space 700. The curve can be expressed, for example, by the arc of a sphere or ellipsoid. Curve 702 can be obtained by optimizing to minimize the distance between this curve and the color of each pixel in the reference region. Curve 702 represents the characteristics of the set in color space of the "set of multiple colors in the reference region of the hand (color set)." The registration unit 202 determines the colors on curve 702 in the color space as the reference colors.

[0054] The registration unit 202 then generates a registration area by excluding from the registration candidate area any color area that is a certain distance or more from the curve 702 in the color space as an area of ​​a color that is not similar to the reference color. The registration unit 202 generates a non-registration area by excluding from the non-registration candidate area any color area that is less than a certain distance from the curve 702 as an area of ​​a color that is similar to the reference color.

[0055] According to the second embodiment, even when the color region of the arm is distributed around a curve, it is possible to register a color that allows extraction of the arm region.

[0056] In the above embodiments, an example has been described in which a composite image (display image) obtained by combining a captured image and a CG model (a model of a virtual object) is displayed on a video see-through HMD. However, the HMD may be an optical see-through HMD that allows the user to view the outside through the display surface of the HMD. In this case, the image processing device 100 displays on the display device 230 a display image in which the CG model is superimposed on a thickness model in which a three-dimensional model of the object is made transparent. This allows the user to view the real space through the display surface of the HMD and also view the CG model. In other words, the user can view a space in which the CG model is appropriately arranged in the real space.

[0057] <Embodiment 3> In the third embodiment, in addition to hand position information and color information, depth information (depth information) is used to register colors.

[0058] A flowchart of color registration processing according to the third embodiment will be shown with reference to the flowchart of FIG. 8. The processing of this flowchart is executed by the control unit 210 controlling each block each time the captured image in the data storage unit 203 is updated by the image acquisition unit 201. In other words, the processing of this flowchart is executed for each group of captured images of a certain frame. Hereinafter, a captured image for which color registration is performed by the processing of this flowchart will be referred to as a "target image." Note that this processing is executed asynchronously with the processing performed by the display image generation unit 207. In addition, in parallel with the description of the flowchart of FIG. 8, a specific example will be described using FIG. 9, which shows an example of color registration.

[0059] In step S801, similar to step S301, the control unit 210 determines whether the processing of this flowchart is already being executed. If it is determined that the processing of this flowchart is already being executed, the processing from step S801 onwards is not performed. If it is determined that the processing of this flowchart is not already being executed, the process proceeds to step S802. This determination is made in step S801 because the processing of the flowchart in FIG. 8 may take longer than the update rate of the captured image.

[0060] In step S802, the depth acquisition unit 205 generates (acquires) a depth image (depth information) of the target image. In the example of Fig. 9, a depth image 921 corresponding to the input image 900 is generated.

[0061] In step S803, the registration unit 202 acquires information about the position and orientation of the hand from the detection unit 218. Then, the registration unit 202 extracts a reference area that is more likely to be a hand area than a threshold value, based on the position and orientation of the hand (at least one of the coordinates of a plurality of feature points). In the example of FIG. 9, the detection unit 218 acquires a position and orientation 911 of the hand in the input image 900. The reference area 931 is a filled-in area surrounded by "joint points that constitute the back of the hand, among the plurality of feature points." Note that the area estimated to be the inside of the back of the hand is likely to fall within the actual inside of the hand, even if there is some error. For this reason, in FIG. 9, the area inside the hand is extracted as the reference area 931.

[0062] In step S804, the registration unit 202 acquires information on the position and orientation of the hand from the detection unit 218, and acquires a depth image from the depth acquisition unit 205. The registration unit 202 also generates a registration candidate area based on the information on the position and orientation of the hand and the depth image. For example, since the position and orientation of the hand is determined as a three-dimensional position from the image capture device 220, the distance of the hand from the image capture device 220 to each joint point can be acquired from the information on the position and orientation of the hand. Then, from the depth image, the distance of the hand in the target image can be acquired. The distance between the object captured in each pixel and the camera can be obtained based on the hand position and orientation information and the depth image. Therefore, the registration unit 202 selects only pixels of the target image that capture an object at a distance close to the distance from the image capture device 220 to the hand, and defines the remaining pixels as a registration candidate area. The "closest distance to the image capture device 220 to the hand" may be defined as a range obtained by adding an offset to the range from the minimum to the maximum distance between each joint point of the hand and the image capture device 220. The range of distances used to determine the registration candidate area may also be limited using a pre-set or estimated user arm length. Furthermore, the range of distances used to determine the registration candidate area may also be limited using information such as the distance to a UI displayed in an application that the user directly operates with their hand (pre-set distance information). In the example of FIG. 9, a region including the estimated hand / arm region is generated as the registration candidate area 941. The registration candidate area 941 includes fine noise on the hand and a sleeve region located close to the distance of the hand.

[0063] In step S805, the registration unit 202 generates a registration area by adjusting the registration candidate area so that it is included in the actual arm area. The registration unit 202 first removes noise areas from the registration candidate area. For example, the registration unit 202 removes areas such as small dots by performing image contraction or expansion processing. The registration unit 202 also removes areas of a color determined to be dissimilar to the reference color from the registration candidate area. Specifically, the registration unit 202 determines, for example, among the pixels (positions) of the registration candidate area, "pixels having a color that is a certain distance or more from the line 502 in color space" as pixels dissimilar to the reference color. The registration unit 202 then removes areas of pixels dissimilar to the reference color from the registration candidate area. In the example of FIG. 9, dot-like noise areas and sleeve areas that are not included in the actual arm area are removed from the registration candidate area 941, and a registration area 951 is generated. According to the process of step S805, the registration unit 202 generates the registration area based not only on the positions and depth information of the feature points but also on the reference color, so that the registration area can be detected as a more appropriate hand / arm area.

[0064] In step S806, the registration unit 202 acquires information about the position and orientation of the hand from the detection unit 218 and acquires a depth image from the depth acquisition unit 205. The registration unit 202 generates a non-registration candidate area based on the information about the position and orientation of the hand and the depth information. For example, the registration unit 202 retains only pixels of the target image that depict an object that is a distance from the image capture device 220 that is significantly different from the distance from the image capture device 220 to the hand, and sets the remaining pixels as a non-registration candidate area. In this case, the "distance significantly different from the distance from the image capture device 220 to the hand" may be defined as a range that is sufficiently farther from the hand by adding a large offset to the maximum distance from the image capture device 220 to each joint point of the hand. In the example of FIG. 9, the registration unit 202 generates, as the non-registration candidate area 961, a floor area or an area of ​​an object that appears to be farther away from the image capture device 220 that is sufficiently farther from the image capture device 220 than the distance from the image capture device 220 to the hand.

[0065] In step S807, the registration unit 202 generates a non-registration region by adjusting the non-registration candidate region so that it does not include the actual arm region. The registration unit 202 first removes noise regions from the registration candidate region. The registration unit 202 removes areas such as small dots by, for example, performing contraction or expansion processing in image processing. Then, the registration unit 202 removes areas of a color similar to the reference color from the non-registration candidate region. Specifically, for example, the registration unit 202 determines that, among the pixels (positions) of the non-registration candidate region, "pixels having a color that is less than a certain distance in color space from the line 502" are pixels similar to the reference color. Then, the registration unit 202 removes the area of ​​pixels similar to the reference color from the non-registration candidate region. In this way, the registration unit 202 generates a non-registration region. In the example of FIG. 9, the non-registration candidate region 961 is removed from the area included in the actual arm region and noise on the ceiling, thereby generating a non-registration region 971.

[0066] In step S808, the registration unit 202 registers the color included in the registration area as a registered color. 9, the registration unit 202 registers the colors included in the registration area 951 as registered colors in the color registration table 981. The registration unit 202 saves the colors included in the non-registration area 971 as non-registered colors. At this time, if the same color is included in both the registered colors and the non-registered colors, it may be possible to set the color to be used preferentially as either the registered colors or the non-registered colors. For this reason, the registration unit 202 may correct the colors included in the registered colors in the color registration table 981 based on the non-registered colors.

[0067] The processes of steps S804 and S805 may be performed after the processes of steps S806 and S807 are completed. In this case, in step S805, the registration unit 202 may adjust the registration candidate region based on the reference color and the non-registration region. For example, the registration unit 202 may adjust the registration candidate region so that the registration region includes "a region of a color similar to the reference color that is not included in the non-registration region."

[0068] In the above description, the registration unit 202 executes processes such as "determining a reference color," "estimating a hand / arm region," "adjusting a registered region," and "registering a registered color." The image processing device 100 may include a determination unit that determines a reference color, an estimation unit that estimates a hand / arm region, and an adjustment unit that adjusts a registered region, which share part of the processes performed by the registration unit 202.

[0069] Only the color registration process of the flowchart in Fig. 8 of embodiment 3 may be executed, or it may be executed while dynamically switching between this and the color registration process of the flowchart in Fig. 3 of embodiment 2. For example, the two processes may be switched depending on the brightness of the scene being photographed, the presence or absence of sleeves, how the arms are photographed, etc. For example, when an arm or wrist area is detected from the target image within the hand / arm area, the process of the flowchart in Fig. 8 may be executed, and in other cases the process of the flowchart in Fig. 3 may be executed.

[0070] According to embodiment 3, color registration is performed using depth information (depth information) in addition to hand position information and color information, making it possible to register hand color more stably than in embodiments 1 and 2.

[0071] Furthermore, in the above, "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be read as "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2." Conversely, "If A is greater than (higher than) B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be read as "If A is greater than (higher than) B, proceed to step S1; if A is less than (lower than) B, proceed to step S2." Therefore, unless a contradiction arises, "greater than or equal to A" may be read as "greater than (higher; longer; more) than A," and "less than or equal to A" may be read as "less than (lower; shorter; fewer) than A." Furthermore, "greater than (higher; longer; more) than A" may be read as "greater than or equal to A," and "less than (lower; shorter; fewer) than A" may be read as "less than or equal to A."

[0072] The various controls described above may or may not be performed by a single piece of hardware (e.g., a general-purpose processor or a dedicated processor). The entire device may be controlled by sharing the processing among multiple pieces of hardware (e.g., multiple general-purpose processors, multiple dedicated processors, or a combination of one or more general-purpose processors and one or more dedicated processors).

[0073] In this specification, the term "processor" is used in a broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0074] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0075] In the above-described embodiment, the present invention has been described as being applied to an image processing device, but the present invention is not limited to this example and can be applied to any electronic device that can perform image processing on captured images. The image processing device may be a computer, a smartphone, a tablet terminal, a digital camera, or a home appliance.

[0076] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more general-purpose processors in the computer of the system or device read and execute the program. It can also be realized by a dedicated processor that realizes one or more of the functions.

[0077] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) a detection means for detecting positions of a plurality of feature points on a specific object based on a first captured image of the specific object; a determination means for determining a reference color of the specific object based on at least one position of the plurality of feature points and the first captured image or a second captured image of the specific object; an estimation means for acquiring a first region that is an area estimated as an area of ​​the specific object based on the positions of the plurality of feature points; a determining means for determining whether the color at each position in the first region is similar to the reference color; an adjustment means for excluding from the first region a region of a color determined by the determination means to be not similar to the reference color; 1. An image processing device comprising: (Configuration 2) The image processing device further includes a registration unit that registers color information included in the first region adjusted by the adjustment unit as color information of the specific object in a specific table. 2. The image processing device according to configuration 1, (Configuration 3) further comprising an object detection means for detecting the specific object in a third captured image in which the specific object is captured, based on color information registered in the specific table; 3. The image processing device according to configuration 2. (Configuration 4) a depth acquisition means for acquiring depth information of a region of the specific object detected by the object detection means; and an image generating means for generating an image by combining the third captured image and the model of the virtual object based on the depth information acquired by the depth acquisition means, the third captured image, and the model of the virtual object. 4. The image processing device according to configuration 3. (Configuration 5) the registration means corrects the color information registered in the specific table as color information of the specific object based on a color included in a second area that is an area estimated to be not an area of ​​the specific object; 5. The image processing device according to any one of configurations 2 to 4. (Configuration 6) a depth acquisition means for acquiring depth information of the first captured image, the estimation means acquires the second region based on the positions of the plurality of feature points and the depth information acquired by the depth acquisition means. 6. The image processing device according to configuration 5. (Configuration 7) the estimation means acquires a second region that is estimated as a region that is not a region of the specific object based on at least one position of the plurality of feature points; the determining means determines whether or not the color at each position in the second region is similar to the reference color; the adjustment means excludes from the second region a region of a color determined by the determination means to be similar to the reference color. 7. The image processing device according to any one of configurations 1 to 6. (Configuration 8) The estimation means estimating a position of a new feature point on the specific object based on the position of at least one of the plurality of feature points; acquiring the first region based on the positions of the plurality of feature points and the positions of the estimated new feature points; 8. The image processing device according to any one of configurations 1 to 7. (Configuration 9) the determining means determines, as the reference color, a color according to a reference axis that represents a characteristic of a set in a color space of a set of a plurality of colors in an area surrounded by the plurality of feature points; the determining means determines a color that is located at a distance from the reference axis that is less than a certain distance as a color that is similar to the reference color; 9. The image processing device according to any one of configurations 1 to 8. (Configuration 10) the determining means determines, as the reference color, a color according to a reference curve that represents a set characteristic in a color space of a set of a plurality of colors in an area surrounded by the plurality of feature points; The determining means determines a color whose distance from the reference curve is less than a certain distance as a color similar to the reference color. 9. The image processing device according to any one of configurations 1 to 8. (Configuration 11) a depth acquisition means for acquiring depth information of the first captured image, the estimation means acquires the first region based on the positions of the plurality of feature points and the depth information acquired by the depth acquisition means. 11. The image processing device according to any one of configurations 1 to 10. (Configuration 12) The estimation means further acquires the first region based on preset distance information. 12. The image processing device according to configuration 11. (Configuration 13) When a specific region of the specific object is detected, the estimation means estimates the first region based on the positions of the plurality of feature points and the depth information acquired by the depth acquisition means. to get, 12. The image processing device according to configuration 11. (Configuration 14) The first captured image and the second captured image are the same image. 14. The image processing device according to any one of configurations 1 to 13. (method) a detecting step of detecting positions of a plurality of feature points on a specific object based on a first captured image of the specific object; a determining step of determining a reference color of the specific object based on at least one position of the plurality of feature points and the first captured image or a second captured image of the specific object; an estimation step of acquiring a first region that is an area estimated as an area of ​​the specific object based on the positions of the plurality of feature points; a determining step of determining whether or not the color at each position in the first region is similar to the reference color; an adjustment step of excluding from the first region a region of a color determined to be not similar to the reference color in the determination step; An image processing method comprising: (program) A program for causing a computer to function as each means of the image processing device according to any one of configurations 1 to 14. [Explanation of symbols]

[0078] 100: Image processing device, 218: Detection unit, 204: Color region extraction unit, 202: Registration unit (estimation unit, determination unit, adjustment unit)

Claims

1. a detection means for detecting positions of a plurality of feature points on a specific object based on a first captured image of the specific object; a determination means for determining a reference color of the specific object based on at least one position of the plurality of feature points and the first captured image or a second captured image of the specific object; an estimation means for acquiring a first region that is an area estimated as an area of ​​the specific object based on the positions of the plurality of feature points; a determining means for determining whether the color at each position in the first region is similar to the reference color; an adjustment unit that excludes from the first region a region of a color that is determined by the determination unit to be not similar to the reference color; 1. An image processing device comprising:

2. The image processing device further includes a registration unit that registers color information included in the first region adjusted by the adjustment unit as color information of the specific object in a specific table.

2. The image processing device according to claim 1, wherein:

3. further comprising an object detection means for detecting the specific object in a third captured image in which the specific object is captured, based on color information registered in the specific table; 3. The image processing device according to claim 2.

4. a depth acquisition means for acquiring depth information of a region of the specific object detected by the object detection means; an image generating means for generating an image by combining the third captured image and the model of the virtual object based on the depth information acquired by the depth acquisition means, the third captured image, and a model of the virtual object; further comprising 4. The image processing device according to claim 3.

5. the registration means corrects the color information registered in the specific table as the color information of the specific object based on a color included in a second region that is an area estimated to be not an area of ​​the specific object; 3. The image processing device according to claim 2.

6. a depth acquisition means for acquiring depth information of the first captured image, the estimation means acquires the second region based on the positions of the plurality of feature points and the depth information acquired by the depth acquisition means.

6. The image processing device according to claim 5,

7. the estimation means acquires a second region that is estimated as a region that is not a region of the specific object based on at least one position of the plurality of feature points; the determining means determines whether or not the color at each position in the second region is similar to the reference color; the adjustment means excludes from the second region a region of a color determined by the determination means to be similar to the reference color; 2. The image processing device according to claim 1, wherein:

8. The estimation means Based on at least one position of the plurality of feature points, The positions of new feature points are estimated, acquiring the first region based on the positions of the plurality of feature points and the positions of the estimated new feature points; 2. The image processing device according to claim 1, wherein:

9. the determining means determines, as the reference color, a color according to a reference axis that represents a characteristic of a set in a color space of a set of a plurality of colors in an area surrounded by the plurality of feature points; the determining means determines a color that is located at a distance from the reference axis that is less than a certain distance as a color that is similar to the reference color; 2. The image processing device according to claim 1, wherein:

10. the determining means determines, as the reference color, a color according to a reference curve that represents a set characteristic in a color space of a set of a plurality of colors in an area surrounded by the plurality of feature points; The determining means determines a color whose distance from the reference curve is less than a certain distance as a color similar to the reference color.

2. The image processing device according to claim 1, wherein:

11. a depth acquisition means for acquiring depth information of the first captured image, the estimation means acquires the first region based on positions of the plurality of feature points and the depth information acquired by the depth acquisition means.

2. The image processing device according to claim 1, wherein:

12. the estimation means further acquires the first region based on preset distance information.

12. The image processing device according to claim 11.

13. when a specific region of the specific object is detected, the estimation means acquires the first region based on positions of the plurality of feature points and the depth information acquired by the depth acquisition means; 12. The image processing device according to claim 11.

14. the first captured image and the second captured image are the same image; 2. The image processing device according to claim 1, wherein:

15. a detecting step of detecting positions of a plurality of feature points on a specific object based on a first captured image of the specific object; a determining step of determining a reference color of the specific object based on at least one position of the plurality of feature points and the first captured image or a second captured image of the specific object; an estimation step of acquiring a first region that is an area estimated as an area of ​​the specific object based on positions of the plurality of feature points; a determining step of determining whether or not the color at each position in the first region is similar to the reference color; an adjustment step of excluding from the first region a region of a color determined to be not similar to the reference color in the determination step; An image processing method comprising:

16. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 14.

Images