Image processing device, image processing method, and program

Abstract

To provide an image processing device, an image processing method, and a program which can appropriately discriminate a shadow of an object to be photographed.SOLUTION: An image processing device comprises: a detection unit which detects an object and a shadow region included in an image; an extraction unit which extracts feature points of the object and the shadow region; and a shadow identification unit which searches the feature point of the shadow region corresponding to the feature point of the object, on the basis of the direction of a light source, and identifies the shadow region in which the feature point corresponds to the object as the shadow region corresponding to the object.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] There is a known technology for identifying the shadow of a subject from an image of the subject. Also, attempts are being made to generate a three-dimensional stereoscopic image of the subject from the image of the subject. In such cases, it is believed that if information on the shadow of the subject is available, it is possible to improve the accuracy and detail of the three-dimensional stereoscopic image of the subject.

[0003] For example, Patent Document 1 listed below discloses a three-dimensional object detection device that has a real shadow area acquisition unit that acquires a real shadow area, which is a shadow area formed by the vehicle when it receives light, based on an image of the area around the vehicle, a virtual shadow area calculation unit that calculates a virtual shadow area, which is a shadow area that the vehicle is predicted to form on the road surface when it receives light, and a three-dimensional object detection unit that detects three-dimensional objects around the vehicle based on the results of comparing the shape of the real shadow area with the shape of the virtual shadow area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-91327 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technique described in Patent Document 1 detects surrounding three-dimensional objects from one's own shadow when photographing with a camera, but does not determine the shadow of the object being photographed. In this way, when the shadow of the subject being photographed cannot be easily identified, no means has been proposed for determining whether the shadow is that of the subject being photographed.

[0006] In view of the above-mentioned problems, the present disclosure has an object to provide an image processing device, an image processing method, and a program capable of appropriately determining the shadow of a subject being photographed. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the objective, the image processing device according to the present disclosure includes a detection unit that detects objects and shadow areas included in an image, an extraction unit that extracts feature points of the objects and the shadow areas, and a shadow identification unit that searches for feature points of the shadow areas that correspond to the feature points of the object based on a light source direction, and identifies the shadow areas that correspond to the feature points of the object as the shadow areas corresponding to the object.

[0008] In order to solve the above-mentioned problems and achieve the objective, the image processing method disclosed herein includes the steps of detecting an object and a shadow area included in an image, extracting feature points of the object and the shadow area, and searching for feature points of the shadow area that correspond to the feature points of the object based on a light source direction, and identifying the shadow area that corresponds to the feature point of the object as the shadow area corresponding to the object.

[0009] In order to solve the above-mentioned problems and achieve the objective, the program disclosed herein causes a computer to execute the steps of detecting objects and shadow areas included in an image, extracting feature points of the objects and shadow areas, searching for feature points of the shadow areas that correspond to the feature points of the object based on a light source direction, and identifying the shadow areas that correspond to the feature points of the object as the shadow areas corresponding to the object. Effect of the Invention

[0010] According to the present disclosure, it is possible to provide an image processing device, an image processing method, and a program capable of appropriately determining the shadow of a subject being photographed. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an overview of an image processing system according to the present disclosure. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of an image processing system according to the present disclosure. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of an image processing device according to the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating a configuration example of a server device according to the present disclosure. [Diagram 5] FIG. 5 is a flowchart showing the flow of the image processing method according to the present disclosure. [Figure 6] FIG. 6 is a diagram illustrating a first example of an imaging target of the image processing device according to the present disclosure. [Figure 7] FIG. 7 is a diagram showing a second example of an imaging target of the image processing device according to the present disclosure. [Figure 8] FIG. 8 is a diagram showing a result of executing the feature point extraction process on a second example of an imaging target by the image processing device according to the present disclosure. [Figure 9] FIG. 9 is a diagram showing a result of executing the light source direction identification process on a second example of an imaging target of the image processing device according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments described below.

[0013] (Image Processing System Overview) First, an overview of an image processing system according to the present disclosure will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an overview of an image processing system according to the present disclosure. As shown in Fig. 1, an image processing system 1 includes an image processing device 100, a server device 200, and a network N. A brief description of these components and a detailed description thereof will be given later.

[0014] As shown in Fig. 1, a person P2 holding an image processing device 100 captures an image of a person P1 as a subject. Although Fig. 1 depicts the person P1 as a subject of the image processing device 100, the subject of the image processing device 100 is not limited to the person P1 and may be any object, for example, an animal such as a dog or a cat, or a fruit such as an apple or a peach. As shown in Fig. 1, the image processing device 100 also captures the shadow of the subject at the same time. Then, the image processing device 100 detects the shadow of the subject by performing image processing on the captured image.

[0015] Note that the image processing device 100 may only execute a process of capturing an image of a subject, transmit the captured image to the server device 200 via the network N, and execute a process of detecting the shadow of the subject in the server device 200.

[0016] As described above, the image processing system 1 functions as a single system with the image processing device 100 at the center and multiple devices transmitting and receiving information to and from each other.

[0017] (Image Processing System Configuration) Next, the configuration of the image processing system 1 according to the present disclosure will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the image processing system according to the present disclosure. As shown in Fig. 2, the image processing system 1 according to the present disclosure includes an image processing device 100, a server device 200, and a network N. The configurations of these will be briefly described below.

[0018] The image processing system 1 includes an image processing device 100, a server device 200, and a network N.

[0019] The image processing device 100 is an information processing device having an imaging function. The image processing device 100 may be, for example, an information processing device such as a camera, a smartphone, a tablet terminal, a wearable terminal, a mobile phone, or a PDA (Personal Digital Assistant). In the example shown in FIG. 1, the image processing device 100 is a camera.

[0020] The server device 200 is an information processing device that executes various types of information processing. For example, the server device 200 executes image processing on an image captured by the image processing device 100. The server device 200 may be realized by an information processing device such as a personal computer (PC), a work station (WS), or a computer having a server function.

[0021] The network N connects the image processing device 100 and the server device 200 in a wired or wireless manner so that they can communicate with each other. If the network N is wired, it may be realized by ETHERNET (registered trademark) defined in IEEE802.3, or a control signal line of USB (Universal Serial Bus) and a video signal line of SDI (Serial Digital Interface). If the network N is wireless, it may be realized by Bluetooth (registered trademark) or a wireless LAN (Local Area Network) defined in IEEE802.11.

[0022] (Configuration of image processing device) Next, a configuration example of the image processing device 100 according to the present disclosure will be described with reference to Fig. 3. Fig. 3 is a diagram showing a configuration example of the image processing device according to the present disclosure. As shown in Fig. 3, the image processing device 100 according to the present disclosure includes a communication unit 110, a storage unit 120, a control unit 130, and an imaging unit 140. Below, these configurations will be described in order.

[0023] The communication unit 110 is a communication module that allows the image processing device 100 to communicate with an external device such as the server device 200. The communication unit 110 communicates data with the external device by wireless communication or wired communication. In the case of wireless communication, the communication unit 110 may include a communication antenna, an RF (Radio Frequency) circuit, and other circuits for communication processing. In the case of wired communication, the communication unit 110 may include, for example, a LAN (Local Area Network) terminal, a transmission circuit, and other circuits for communication processing.

[0024] The storage unit 120 is a storage device that stores various types of information. The storage unit 120 includes a main storage device and an auxiliary storage device. The main storage device may be realized by a semiconductor memory element such as a random access memory (RAM), a read only memory (ROM), or a flash memory. The auxiliary storage device may be realized by a hard disk, a solid state drive (SSD), an optical disk, or the like.

[0025] As shown in FIG. 3, the storage unit 120 includes an image data storage unit 121.

[0026] The image data storage unit 121 stores image data captured by the imaging unit 140. For example, the image data storage unit 121 may store information such as an identification number for identifying image data and the date and time when the image data was captured, in association with the image data.

[0027] The control unit 130 is a controller that manages and controls the image processing device 100. The control unit 130 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like executing various programs stored in the storage unit 120 using a RAM as a working area. The control unit 130 may also be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0028] As shown in FIG. 3, the control unit 130 includes an acquisition unit 131, a detection unit 132, an extraction unit 133, a light source direction identification unit 134, a shadow identification unit 135, and an obstruction identification unit 136. The control unit 130 realizes these functions and executes these processes by reading and executing a program (software) from the storage unit 120. The control unit 130 may be realized by an electronic circuit. The control unit 130 may execute these processes by one CPU, or may include multiple CPUs and execute these processes in parallel by the multiple CPUs. Below, these configurations will be described in order.

[0029] The acquisition unit 131 acquires image data captured by the imaging unit 140. When the acquisition unit 131 acquires the image data captured by the imaging unit 140, the acquisition unit 131 stores the acquired image data in the image data storage unit 121. Then, the acquisition unit 131 may perform data conversion on the image data captured by the imaging unit 140 using camera parameters including internal parameters and external parameters and depth information measured by the imaging unit 140 to obtain image data in a parallel projection state. That is, in this case, the acquisition unit 131 may convert the image data of the perspective projection captured by the imaging unit 140 into image data of the parallel projection. Note that, here, the perspective projection is imaging light input to a specific viewpoint. Also, the parallel projection is imaging light incident parallel to a viewpoint assumed to be infinitely far away.

[0030] The detection unit 132 detects an object and a shadow area included in the image data acquired by the acquisition unit 131 (or image data converted into parallel projection). Note that, here, the object refers to an object having a shape other than a shadow that appears in the image. For example, the detection unit 132 detects an object and a shadow area captured from the image data, and executes a process of classifying the detected object. For example, the detection unit 132 may be realized by using semantic segmentation, which is a method of classifying pixels of image data according to which object class they belong to. Note that the detection of a shadow area included in an image may be realized by having a model such as a CNN (Convolutional Neural Network) learn the luminance characteristics of the shadow area.

[0031] In the present embodiment, the detection unit 132 associates an object with a shadow region based on the object and the shadow region detected from the image data. The detection unit 132 associates the object with its shadow region when the positional relationship between the object and the shadow region in the coordinate system of the image data satisfies a predetermined condition. The predetermined condition here may be set arbitrarily, but for example, indicates that the end of the object and the end of the shadow region are within a predetermined distance range in the coordinate system of the image data. That is, the detection unit 132 associates the object with the shadow region when there is a shadow region whose end is within a predetermined distance range for the end of an object. For example, when the object is a person, when the person's feet (end of the object) and the end of the shadow region (the foot of the shadow) are within a predetermined distance range, the person and the shadow region are associated. Note that the association here refers to linking the shadow region associated with the object as the shadow of the object.

[0032] In this way, the object and the shadow area can be associated based on the object and the shadow area detected from the image data. However, the object and the shadow area may not be associated appropriately only by the method using the object and the shadow area. For example, if an obstacle exists in front of the object, the object may be hidden by the obstacle, and the end of the object in the image data may be separated from the end of the shadow area, and it may be determined that the predetermined condition is not satisfied. In contrast, in this embodiment, the object and the shadow area can be associated appropriately by the processing of the extraction unit 133, the light source direction identification unit 134, and the shadow identification unit 135 described below. A specific description will be given below.

[0033] The extraction unit 133 extracts feature points of the object and the shadow region. Here, the feature point refers to a position of the object or the shadow region that is characteristic of the object or the shadow region, and may refer to, for example, a corner of the shape of the object or the shadow region. The extraction unit 133 may extract the feature points by using a method such as Scale Invariant Feature Transform (SIFT). The scale invariant feature transform is a method for extracting feature amounts that are invariant to the enlargement / reduction and rotation of an image, and includes a process of detecting feature points and a process of describing rotation-invariant feature amounts for the detected feature points. This makes it possible to extract clear features that are invariant even when the image is enlarged / reduced or rotated.

[0034] The light source direction identification unit 134 acquires the light source direction. The light source direction refers to the direction of the light source that casts the shadow of the object in the image represented by the image data. In other words, the light source direction refers to the traveling direction of light in the image represented by the image data.

[0035] The light source direction identifying unit 134 may acquire the light source direction by any method. For example, the light source direction may be associated with and set in advance for the image data. However, it is preferable that the light source direction identifying unit 134 identifies the light source direction based on the image data. Specifically, in this embodiment, the light source direction identifying unit 134 calculates the light source direction based on the feature points of the object and the feature points of the shadow region that are associated with each other as satisfying a predetermined condition.

[0036] More specifically, the light source direction specifying unit 134 performs matching between the feature points of an object and the feature points of a shadow region associated with the object. For example, the light source direction specifying unit 134 extracts the feature points of the matching object and the feature points of the shadow region based on the positional relationship between the object and the shadow region associated with the object. For example, the light source direction specifying unit 134 may perform matching by detecting whether the shadow region associated with the object is on the left or right side of the object, and if the shadow region is on the left side, matching between the feature points of the object and the left side of the top of the shadow region. The light source direction specifying unit 134 calculates the light source direction based on the feature points of the matching object and the feature points of the shadow region.

[0037] In this embodiment, the light source direction specifying unit 134 may determine the light source direction as a direction parallel to a line connecting the feature points of the associated object and the feature points of the shadow region. More specifically, the light source direction specifying unit 134 may determine the light source direction as a direction parallel to a line connecting the feature points of the matching object and the feature points of the shadow region. When there are multiple sets of matched feature points, the light source direction specifying unit 134 may calculate the angle between the line connecting the feature points of the matched object and the feature points of the shadow region and the horizontal direction in the coordinate system of the image for all sets of matched feature points, and calculate the median of each angle. Then, the light source direction specifying unit 134 may determine the direction inclined by the median angle with respect to the horizontal direction as the light source direction.

[0038] The shadow identification unit 135 searches for feature points of a shadow region that correspond to the feature points of an object based on the light source direction. Then, the shadow identification unit 135 identifies a shadow region having feature points that correspond to the feature points of the object as a shadow region that corresponds to the object. In this embodiment, the shadow identification unit 135 associates an object and a shadow region that have not been associated with a shadow region by the detection unit 132 (an object and a shadow region that do not satisfy a predetermined condition) based on the light source direction.

[0039] Any method may be used to associate an object with a shadow region based on the light source direction, but the method in this embodiment will be described below. Here, a line that passes through a feature point of an object that does not satisfy a predetermined condition and is parallel to the light source direction is defined as a parallel line. In this case, the shadow identification unit 135 extracts feature points within a predetermined distance range from the parallel lines from among the feature points of the shadow region that does not satisfy the predetermined condition as feature points of the shadow region corresponding to the feature points of the object (feature points of the object that pass through the parallel lines). Then, the shadow identification unit 135 associates the shadow region having the extracted feature points with the object. In this way, by associating an object with a shadow region based on the light source direction, for example, an object that does not satisfy a predetermined condition and a shadow region can be appropriately associated.

[0040] The obstruction identifying unit 136 identifies an obstruction that obstructs the object and the shadow area from satisfying a predetermined condition based on the position of the object and the shadow area associated based on the light source direction. Specifically, the obstruction identifying unit 136 identifies an object that is located near the object and the shadow area and whose shadow covers the shadow of the target object as an obstruction. For example, the obstruction identifying unit 136 may identify an object that is located between the object associated based on the light source direction and the shadow area as an obstruction.

[0041] The imaging unit 140 captures an image. The imaging unit 140 includes an optical element and an imaging element. The optical element is an element that constitutes an optical system, such as a lens, a mirror, a prism, and a filter. The imaging element is an element that converts light incident through the optical element into an image signal, which is an electrical signal. The imaging element is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0042] The imaging unit 140 may also obtain depth information for each position of the image. For example, the depth information may be obtained for each position of the image information using a TOF (Time Of Flight) method. In this case, the imaging unit 140 includes a light projecting element and a light receiving element, and measures the distance to the object by measuring the time it takes for a photon emitted from the light projecting element to hit the object and for the reflected light to be received by the light receiving element. For example, an LED (Light Emitting Diode), an LD (Laser Diode), or a VCSEL (Vertical Cabity Surface Emitting LASER) can be used as the light projecting element. For example, the above-mentioned CCD sensor or CMOS sensor can be used as the light receiving element.

[0043] (Server device configuration) In the above description, the image processing device 100 including the imaging unit 140 executes the image processing method for associating an object with a shadow area from a captured image. However, an image processing device not including an imaging unit may acquire image data captured by the imaging unit and execute this image processing method. For example, a server device 200 may be used as an image processing device not including an imaging unit. A configuration example of the server device 200 will be described with reference to FIG. 4. FIG. 4 is a diagram showing a configuration example of a server device according to the present disclosure. As shown in FIG. 4, the server device 200 according to the present disclosure includes a communication unit 210, a storage unit 220, a control unit 230, a sound output unit 240, and a display unit 250. Note that, among the configurations of the server device 200, the communication unit 210, the storage unit 220, and the control unit 230 are the same as the communication unit 110, the storage unit 120, and the control unit 130 of the image processing device 100, respectively, and therefore description thereof will be omitted. Therefore, in the following, only the sound output unit 240 and the display unit 250, which have different configurations from the image processing device 100, will be described.

[0044] The sound output unit 240 outputs sound. For example, when the processing in the server device 200 is completed, the sound output unit 240 may output a sound notifying the completion of the processing. The sound output unit 240 may be a speaker, and the speaker converts an electric signal into sound by a diaphragm. That is, the speaker vibrates the diaphragm with a predetermined amplitude and frequency based on a control command given by the electric signal, thereby vibrating the air in contact with the diaphragm to output sound.

[0045] The display unit 250 displays various types of information. For example, the display unit 250 may display a GUI (Graphical User Interface) for executing the processing of the server device 200. The display unit 250 may be realized by a liquid crystal display, an organic EL (Electro Luminescence) display, a micro LED (Light Emitting Diode) display, or the like.

[0046] This allows the server device 200 to execute image processing at high speed without the image processing device 100 being equipped with a high-performance arithmetic processing device.

[0047] (Image processing method) Next, an image processing method according to the present disclosure will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the flow of the image processing method according to the present disclosure. Below, the image processing method according to the present disclosure will be described step by step along the flow shown in Fig. 5.

[0048] First, the image processing device 100 converts an image captured by the imaging unit 140 from a perspective projection state to a parallel projection state by using camera parameters and depth information (step S101). Next, the image processing device 100 extracts a shadow area of ​​an object from the converted input image (step S102). Next, the image processing device 100 extracts an object area from the converted input image (step S103). Note that steps S102 and S103 may be performed simultaneously in parallel.

[0049] Next, the image processing device 100 associates each of the extracted objects with a shadow region (step S104). That is, here, the image processing device 100 associates objects that satisfy the above-mentioned predetermined condition with a shadow region. Next, the image processing device 100 extracts feature points of each of the objects (step S105). Next, the image processing device 100 extracts feature points of each of the shadow regions (step S106). Note that steps S105 and S106 may be performed simultaneously in parallel.

[0050] Next, the image processing device 100 matches the feature points of each object region associated as satisfying a predetermined condition with the feature points of the shadow region (step S107). Next, for each set of matched feature points, the image processing device 100 extracts the angle between the line connecting the feature points of the matched object and the feature points of the shadow and the horizontal direction in the image, calculates the median of all angles, and calculates the angle between the feature points of the object and the feature points of the shadow region (step S108), and specifies the direction inclined by the calculated angle with respect to the horizontal direction as the light source direction. Next, the image processing device 100 draws a line parallel to the light source direction through the feature points of the target object not associated with a shadow (the object not associated as not satisfying the predetermined condition), and detects the feature points of the shadow region in the vicinity of the line (the feature points of the shadow region within a predetermined distance range of the line), thereby confirming the relationship between the feature points of the target object and the shadow region (step S109). Next, the image processing device 100 determines that the shadow region including the detected feature points of the shadow region is the shadow of the target object, and associates the object with the shadow region (step S110).

[0051] This makes it possible to link the shadow of an object even when the bottom (end) of the object is not visible. Therefore, by linking an object with its shadow, it is possible to improve the accuracy of generating a 3D image of the object by using shadow information.

[0052] (Specific examples of image processing) A specific example of the image processing described above will be described with reference to Fig. 6 to Fig. 9. Fig. 6 is a diagram showing a first example of an image capture target of the image processing device according to the present disclosure. Fig. 6 shows a person P1 and his shadow SD1 as a first example of an image capture target. In this case, the end of the person P1, which is an object, and the end of the shadow SD1, which is a shadow region, are close to each other, and therefore are associated with each other as satisfying the above-mentioned predetermined condition.

[0053] Next, a second example of the imaging target will be described with reference to FIG. 7. FIG. 7 is a diagram showing a second example of the imaging target of the image processing device according to the present disclosure. FIG. 2 shows a case where an obstruction G1 is in front of a person P1, and the state of the bottom of the person P1 is unclear. In this case, the end of the shadow SD1 projected on the ground and the end of the person P1 are located apart from each other, so that they do not satisfy a predetermined condition and cannot be associated with each other. Therefore, the shadow of the target object can be determined by the method described above. A specific example will be described below.

[0054] Next, a specific example of the result of the feature point extraction process after detecting the object and the shadow region will be described with reference to Fig. 8. Fig. 8 is a diagram showing the feature point extraction process executed on the second example of the imaging target of the image processing device according to the present disclosure. As described above, the feature point indicates the corner of the object, etc. For example, Fig. 8 shows that the person feature point PFT1 at the corner of the shoulder of person P1 and the shadow feature point SDFT1 at the corner of the shadow SD1 are extracted by the feature point extraction process.

[0055] Next, a specific example of the shadow identification process will be described with reference to Fig. 9. Fig. 9 is a diagram showing a result of executing the light source direction identification process on a second example of an image capture target of the image processing device according to the present disclosure. Fig. 9 shows that, by the shadow identification process, a straight line L1 is drawn from the person feature point PF1 of a person P1 to which no shadow is associated at an angle calculated by the light source direction identification process, and a shadow SD1 having a shadow feature point SDFT1 of a shadow SD1 located nearby on the straight line L1 is associated as the shadow of the person P1.

[0056] (Composition and Effects) The image processing device 100 according to the present disclosure includes a detection unit 132 that detects objects and shadow areas included in an image, an extraction unit 133 that extracts feature points of the objects and shadow areas, and a shadow identification unit 135 that searches for feature points of the shadow area corresponding to the feature points of the object based on the light source direction and identifies the shadow area corresponding to the feature points of the object as the shadow area corresponding to the object.

[0057] According to this configuration, the feature points of the object and the shadow are associated with each other using the light source direction, and the object and the shadow are associated with each other. Therefore, even if the bottom of the object is not visible, the position of the shadow relative to the object can be estimated using the light source direction, and the object and the shadow can be appropriately associated, and the shadow of the subject to be photographed can be appropriately determined. By using the association between the object and its shadow in this way, it is possible to improve the accuracy by using the shadow information when generating a 3D stereoscopic image of the object.

[0058] The detection unit 132 of the image processing device 100 according to the present disclosure associates an object with a shadow region when the positional relationship between the object and the shadow region satisfies a predetermined condition, and the image processing device 100 further includes a light source direction identification unit 134 that calculates a light source direction based on feature points of the associated object and feature points of the shadow region. The shadow identification unit 135 searches for feature points of the shadow region that correspond to feature points of the object that do not satisfy the predetermined condition, based on the light source direction.

[0059] According to this configuration, the light source direction can be calculated with high accuracy by identifying the light source direction based on the feature points of the associated object and the feature points of the shadow region. Therefore, by using the light source direction when generating a 3D stereoscopic image of the object, it is possible to contribute to detailed 3D stereoscopic images of the object.

[0060] The light source direction identifying unit 134 of the image processing device 100 according to the present disclosure determines, as the light source direction, a direction parallel to a line connecting the feature point of the associated object and the feature point of the shadow region.

[0061] According to this configuration, the direction parallel to the line connecting the feature point of the associated object and the feature point of the shadow region can be set as the light source direction, which contributes to the detailed 3D image of the object by using the light source direction when generating the 3D image of the object.

[0062] The image processing device 100 according to the present disclosure further includes an obstruction identification unit 136 that identifies an obstruction, which is an object that prevents the object and the shadow area from satisfying a predetermined condition, based on the position of the object and the shadow area that are associated based on the light source direction.

[0063] According to this configuration, an obstruction can be identified based on the positions of the object and the shadow area associated based on the light source direction, which can be useful in generating a 3D image of the obstruction at the same time as generating a 3D image of the object.

[0064] The image processing method according to the present disclosure includes the steps of detecting an object and a shadow area included in an image, extracting feature points of the object and the shadow area, searching for feature points of the shadow area corresponding to the feature points of the object based on a light source direction, and identifying the shadow area corresponding to the feature points of the object as the shadow area corresponding to the object.

[0065] With this configuration, it becomes possible to appropriately link objects and shadows, and by utilizing the link between an object and its shadow, it becomes possible to use the shadow information when generating a three-dimensional stereoscopic image of the object to improve accuracy, etc.

[0066] A program according to the present disclosure causes a computer to execute the steps of detecting objects and shadow areas contained in an image, extracting feature points of the objects and shadow areas, searching for feature points of the shadow areas corresponding to the feature points of the object based on the light source direction, and identifying the shadow areas corresponding to the feature points of the object as shadow areas corresponding to the object.

[0067] With this configuration, it becomes possible to appropriately link objects and shadows, and by utilizing the link between an object and its shadow, it becomes possible to use the shadow information when generating a three-dimensional stereoscopic image of the object to improve accuracy, etc.

[0068] Although the embodiment of the present disclosure has been described above, the embodiment is not limited by the contents of this embodiment. In addition, the above-mentioned components include those that a person skilled in the art can easily imagine, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-mentioned components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the above-mentioned embodiment. [Explanation of symbols]

[0069] 100 Image processing device 110 Communications Department 120 Storage section 121 Image data storage unit 130 Control section 131 Acquisition Department 132 Detection unit 133 Extraction part 134 Light source direction identification part 135 Shadow Identification Department 136 Inhibitor Identification Department 140 Imaging unit 200 Server device 210 Communications Department 220 Storage section 221 Image data storage unit 230 Control Unit 231 Acquisition Department 232 Detection unit 233 Extraction part 234 Light source direction identification part 235 Shadow Identification Department 236 Inhibitor Identification Department 240 Sound output section 250 Display section N Network

Claims

1. A detection unit that detects objects and shadow areas included in an image; an extraction unit that extracts feature points of the object and the shadow region; a shadow identification unit that searches for a feature point of the shadow region corresponding to a feature point of the object based on a light source direction, and identifies the shadow region corresponding to the feature point of the object as the shadow region corresponding to the object. Image processing device.

2. the detection unit associates the object with the shadow region when a positional relationship between the object and the shadow region satisfies a predetermined condition; a light source direction specifying unit that calculates the light source direction based on the associated feature points of the object and the associated feature points of the shadow region; the shadow identification unit searches for feature points of the shadow region corresponding to feature points of the object that do not satisfy the predetermined condition; The image processing device according to claim 1 .

3. the light source direction identification unit determines, as the light source direction, a direction parallel to a line connecting a feature point of the associated object and a feature point of the shadow region; The image processing device according to claim 2 .

4. an obstruction identifying unit that identifies an obstruction that obstructs the object and the shadow area from satisfying the predetermined condition based on the positions of the object and the shadow area associated based on the light source direction, 4. The image processing device according to claim 2 or 3.

5. Detecting objects and shadow regions included in an image; extracting feature points of the object and the shadow region; searching for a feature point of the shadow region corresponding to a feature point of the object based on a light source direction, and identifying the shadow region corresponding to the feature point of the object as the shadow region corresponding to the object. Image processing methods.

6. Detecting objects and shadow regions included in an image; extracting feature points of the object and the shadow region; searching for a feature point of the shadow region corresponding to a feature point of the object based on a light source direction, and identifying the shadow region corresponding to the feature point of the object as the shadow region corresponding to the object; A program that causes a computer to execute the following.

Images